Layered/incremental snapshots

CHANGELOG.md

@@ -7,6 +7,22 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 ### Added
 
 ### Changed
+* **Breaking:** Snapshots use ABI 7 and config schema v4. Existing snapshots
+  must be regenerated.
+* **Breaking:** `PtRootFinder` receives a `SnapshotMemoryReader` and returns a
+  `Result<Vec<u64>>`. Read snapshot bytes with `snapshot.read_exact(gpa, buffer)?`
+  and return roots with `Ok(roots)`.
+* Related snapshots share immutable memory layers. This reduces capture time
+  and memory use. A guest can read data it overwrote before a snapshot from the
+  older layers of that snapshot.
+* KVM: deleting a memory slot flushes only that slot's guest mappings on Linux
+  6.12 and later. This speeds up the first guest call after a snapshot,
+  restore, or unmap changes memory slots.
+* **Breaking:** `Sandbox::snapshot` installs the captured snapshot into the
+  sandbox and removes `map_region` and `map_file_cow` regions. Pages of these
+  regions that the guest has not mapped into its page tables are lost.
+* `vmem::walk_va_spaces` merges adjacent mappings of the same kind into one
+  mapping.
 
 ### Removed
 

docs/paging-development-notes.md

@@ -116,10 +116,10 @@ the metadata at the top of the scratch region and grows downward.
 
 When the host takes a snapshot of a guest, it will traverse the guest
 page tables, collecting every (non-page-table) physical page that is
-mapped (outside of the scratch map) in the guest. It will write out a
-new compacted snapshot with precisely those pages in order, and a new
-set of page tables which produce precisely the same virtual memory
-layout, except for the scratch map.
+mapped (outside of the scratch map) in the guest. It will reuse pages
+unchanged since earlier snapshots, write the rest in order to a new
+compacted blob, and build a new set of page tables which produce
+precisely the same virtual memory layout, except for the scratch map.
 
 ### Pre-sizing the scratch region
 

docs/snapshot-oci-format.md

@@ -22,24 +22,28 @@ path/
   blobs/sha256/
     <manifest-digest>                 OCI image manifest JSON
     <config-digest>                   Hyperlight config JSON
-    <snapshot-digest>                 raw memory bytes
-                                      (`memory_size` bytes)
+    <data-digest>                     one raw blob per snapshot blob:
+                                      guest data, then page tables
     <transport-digest>                canonical virtqueue rings
 ```
 
 Four blob kinds per tag:
 
 * **manifest** (`application/vnd.oci.image.manifest.v1+json`). Tiny JSON
-  pointer record selected via `index.json`. References one config and
-  two layers by digest.
-* **config** (`application/vnd.hyperlight.snapshot.config.v3+json`). The
+  pointer record selected via `index.json`. References one config,
+  the snapshot blobs, and one transport layer by digest.
+* **config** (`application/vnd.hyperlight.snapshot.config.v4+json`). The
   snapshot descriptor: arch, hypervisor, CPU vendor, ABI version,
   resume address and captured registers, memory and transport layout,
-  registered host functions, snapshot generation counter, and namespaced
-  application metadata. Loaded eagerly and fully parsed.
-* **layer / memory** (`application/vnd.hyperlight.snapshot.memory.v1`).
-  The raw guest memory image, exactly `memory_size` bytes. mmap'd on
-  restore.
+  registered host functions, the start address, data length, and page-table
+  length of each blob, the blob with the page tables for a restore, snapshot
+  generation counter, and namespaced application metadata.
+  Loaded eagerly and fully parsed.
+* **layers / data** (`application/vnd.hyperlight.snapshot.memory.v2`).
+  One raw, host-page-aligned blob per snapshot blob, in config order. Each
+  blob holds guest data followed by the page-table tree of the snapshot that
+  created it. Each blob is mmap'd on restore, and the guest maps only its
+  data.
 * **layer / transport**
   (`application/vnd.hyperlight.snapshot.transport.v1`). A bounded
   binary image of the canonical G2H and H2G rings. Loading checks ring
@@ -48,6 +52,8 @@ Four blob kinds per tag:
 The runtime queue protocol and canonical checkpoint are described in
 [Virtqueue host and guest communication](./virtio-host-guest-communication.md).
 
+Manifest layers are ordered data, then transport.
+
 Blob filenames are the sha256 of the blob bytes, so identical blobs
 across tags are stored once.
 
@@ -95,7 +101,7 @@ A single saved `Snapshot` consists of exactly:
   config blob for tooling visibility,
 * one **manifest** blob (referenced by that index entry),
 * one **config** blob (referenced by the manifest's `config` field),
-* one memory **layer** blob,
+* one or more data **layer** blobs,
 * one transport **layer** blob.
 
 Saving two snapshots under different tags into the same `path`
@@ -138,12 +144,12 @@ podman), `go-containerregistry` (crane), and `regclient`.
 ## Read semantics
 
 `Snapshot::load(path, reference)` reads a snapshot. It does not check
-the manifest, config, memory, or transport blobs against their sha256 digests.
+the manifest, config, data, or transport blobs against their sha256 digests.
 `reference` is an
 [`OciReference`], either a tag that matches the
 `org.opencontainers.image.ref.name` annotation or the manifest
 digest returned by `save`. `Snapshot::checked_load` adds the digest
-check on all four blobs, catching accidental corruption on disk.
+check on every blob, catching accidental corruption on disk.
 Both run every other check (OCI structure, descriptor sizes, schema
 versions, arch / hypervisor / CPU vendor / ABI tags, layout bounds,
 entrypoint bounds). The caller is responsible for trusting the source.

docs/snapshot-versioning.md

@@ -18,28 +18,29 @@ A snapshot carries four independently evolvable version markers:
   size), and the calling convention for guest function entry. A change
   to any of these breaks older snapshots unless the loader adds a
   compat path.
-* **Snapshot blob encoding**, `MT_SNAPSHOT_V1`
-  (`application/vnd.hyperlight.snapshot.memory.v1`), aliased as
+* **Snapshot blob encoding**, `MT_SNAPSHOT_V2`
+  (`application/vnd.hyperlight.snapshot.memory.v2`), aliased as
   `MT_SNAPSHOT_CURRENT`. This is the on-wire format of the snapshot
   blob: framing, section ordering, alignment, dirty/zero-page elision,
   anything about how the bytes are packed inside the OCI layer.
 * **Transport blob encoding**, `MT_TRANSPORT_V1`
   (`application/vnd.hyperlight.snapshot.transport.v1`), aliased as
   `MT_TRANSPORT_CURRENT`. This is the binary encoding of canonical
   virtqueue state stored outside the memory layer.
-* **Config schema**, `MT_CONFIG_V3`
-  (`application/vnd.hyperlight.snapshot.config.v3+json`), aliased as
+* **Config schema**, `MT_CONFIG_V4`
+  (`application/vnd.hyperlight.snapshot.config.v4+json`), aliased as
   `MT_CONFIG_CURRENT`. This is the JSON shape of the config blob:
   field names, types, required vs optional, the descriptors the loader
   needs in order to reconstruct the sandbox (memory sizes, buffer
   sizes, `abi_version`, `hyperlight_version`, etc.). Renaming a field,
   changing its type, or adding a required field is a schema change and
-  bumps this constant. Version 3 describes the virtqueue-only memory layout
-  and requires a transport layer. Config v1 and v2 are incompatible with
-  the current ABI.
+  bumps this constant. Version 4 describes snapshot blobs, the blob with
+  the page tables, and transport geometry, and requires a transport
+  layer. Config v1, v2, and v3 are incompatible with the current ABI.
 
 ABI 6 requires checkpoint readiness after guest initialization and permits
-H2G prefill to omit retained slots. Earlier snapshots must be regenerated.
+H2G prefill to omit retained slots. ABI 7 stores snapshot memory as
+layers. Earlier snapshots must be regenerated.
 
 The `OCI_LAYOUT_VERSION` constant is pinned by the OCI image-layout
 spec at `1.0.0`.