Layered/incremental snapshots

CHANGELOG.md

@@ -32,7 +32,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
   the configured level is above `OFF` rather than whether the tracing state was
   allocated.
 * **Breaking:** Virtqueue rings and pools occupy host-owned scratch before page
-  tables. Snapshots use ABI 5 and config schema v3. Existing snapshots must be
+  tables. Snapshots use ABI 6 and config schema v4. Existing snapshots must be
   regenerated.
 * Host virtqueue access uses checked copies and atomics across mapped scratch.
   Snapshot admission checks geometry, canonical rings, and distinct, aligned
@@ -49,13 +49,26 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 * Require guest logs and all host and guest function calls to use virtqueues.
 * Keep registered Rust guest return values typed until transport encoding so
   external byte results avoid intermediate FlatBuffer copies.
-* Store canonical virtqueue rings in versioned OCI transport layers. Config v3
+* Store canonical virtqueue rings in versioned OCI transport layers. Config v4
   rejects snapshots without transport state.
 * Running snapshots checkpoint dirty virtqueues before capture. Ordinary calls
   keep their deferred result path.
 * Reject snapshot capture while guest-owned transport buffers are retained.
 * Use the reclaimed stack pages to raise the default G2H and H2G pools to 12
   and 8 pages.
+* **Breaking:** `PtRootFinder` receives a `SnapshotMemoryReader` and returns a
+  `Result<Vec<u64>>`. Read snapshot bytes with `snapshot.read(gpa, buffer)?`
+  and return roots with `Ok(roots)`.
+* Related snapshots share immutable memory layers. This reduces capture time
+  and memory use.
+* KVM: deleting a memory slot flushes only that slot's guest mappings on Linux
+  6.12 and later. This speeds up the first guest call after a snapshot,
+  restore, or unmap changes memory slots.
+* **Breaking:** `Sandbox::snapshot` installs the captured snapshot into the
+  sandbox and removes `map_region` and `map_file_cow` regions. Pages of these
+  regions that the guest has not mapped into its page tables are lost.
+* `vmem::walk_va_spaces` merges adjacent mappings of the same kind into one
+  mapping.
 
 ### Removed
 * `RunPool` and the run-specific `AllocError::InvalidAlign` variant.

docs/paging-development-notes.md

@@ -116,10 +116,10 @@ the metadata at the top of the scratch region and grows downward.
 
 When the host takes a snapshot of a guest, it will traverse the guest
 page tables, collecting every (non-page-table) physical page that is
-mapped (outside of the scratch map) in the guest. It will write out a
-new compacted snapshot with precisely those pages in order, and a new
-set of page tables which produce precisely the same virtual memory
-layout, except for the scratch map.
+mapped (outside of the scratch map) in the guest. It will reuse pages
+unchanged since earlier snapshots, write the rest in order to a new
+compacted blob, and build a new set of page tables which produce
+precisely the same virtual memory layout, except for the scratch map.
 
 ### Pre-sizing the scratch region
 

docs/snapshot-oci-format.md

@@ -22,24 +22,27 @@ path/
   blobs/sha256/
     <manifest-digest>                 OCI image manifest JSON
     <config-digest>                   Hyperlight config JSON
-    <snapshot-digest>                 raw memory bytes
-                                      (`memory_size` bytes)
+    <data-digest>                     one raw blob per data layer
+    <page-tables-digest>              page-table storage bytes
     <transport-digest>                canonical virtqueue rings
 ```
 
-Four blob kinds per tag:
+Five blob kinds per tag:
 
 * **manifest** (`application/vnd.oci.image.manifest.v1+json`). Tiny JSON
   pointer record selected via `index.json`. References one config and
-  two layers by digest.
-* **config** (`application/vnd.hyperlight.snapshot.config.v3+json`). The
+  data layers, one page-table layer, and one transport layer by digest.
+* **config** (`application/vnd.hyperlight.snapshot.config.v4+json`). The
   snapshot descriptor: arch, hypervisor, CPU vendor, ABI version,
   resume address and captured registers, memory and transport layout,
-  registered host functions, snapshot generation counter, and namespaced
-  application metadata. Loaded eagerly and fully parsed.
-* **layer / memory** (`application/vnd.hyperlight.snapshot.memory.v1`).
-  The raw guest memory image, exactly `memory_size` bytes. mmap'd on
-  restore.
+  registered host functions, layer start addresses and live ranges, page-table
+  length, snapshot generation counter, and namespaced application metadata.
+  Loaded eagerly and fully parsed.
+* **layers / data** (`application/vnd.hyperlight.snapshot.memory.v2`).
+  One raw, host-page-aligned blob per snapshot data layer, in layer order.
+  Each blob is mmap'd on restore.
+* **layer / page tables** (`application/vnd.hyperlight.snapshot.page-tables.v1`).
+  The page-table tree in host-page-aligned storage, separate from guest data.
 * **layer / transport**
   (`application/vnd.hyperlight.snapshot.transport.v1`). A bounded
   binary image of the canonical G2H and H2G rings. Loading checks ring
@@ -48,6 +51,8 @@ Four blob kinds per tag:
 The runtime queue protocol and canonical checkpoint are described in
 [Virtqueue host and guest communication](./virtio-host-guest-communication.md).
 
+Manifest layers are ordered data, page tables, transport.
+
 Blob filenames are the sha256 of the blob bytes, so identical blobs
 across tags are stored once.
 
@@ -95,7 +100,8 @@ A single saved `Snapshot` consists of exactly:
   config blob for tooling visibility,
 * one **manifest** blob (referenced by that index entry),
 * one **config** blob (referenced by the manifest's `config` field),
-* one memory **layer** blob,
+* one or more data **layer** blobs,
+* one page-table **layer** blob,
 * one transport **layer** blob.
 
 Saving two snapshots under different tags into the same `path`
@@ -138,12 +144,12 @@ podman), `go-containerregistry` (crane), and `regclient`.
 ## Read semantics
 
 `Snapshot::load(path, reference)` reads a snapshot. It does not check
-the manifest, config, memory, or transport blobs against their sha256 digests.
+the manifest, config, data, page-table, or transport blobs against their sha256 digests.
 `reference` is an
 [`OciReference`], either a tag that matches the
 `org.opencontainers.image.ref.name` annotation or the manifest
 digest returned by `save`. `Snapshot::checked_load` adds the digest
-check on all four blobs, catching accidental corruption on disk.
+check on every blob, catching accidental corruption on disk.
 Both run every other check (OCI structure, descriptor sizes, schema
 versions, arch / hypervisor / CPU vendor / ABI tags, layout bounds,
 entrypoint bounds). The caller is responsible for trusting the source.

docs/snapshot-versioning.md

@@ -7,7 +7,7 @@ existing snapshots loadable, or while rejecting them with a clear error.
 
 ## What is versioned
 
-A snapshot carries four independently evolvable version markers:
+A snapshot carries five independently evolvable version markers:
 
 * **Memory blob ABI**, `SNAPSHOT_ABI_VERSION` (a `u32` inside the
   config blob, defined in
@@ -18,25 +18,29 @@ A snapshot carries four independently evolvable version markers:
   size), and the calling convention for guest function entry. A change
   to any of these breaks older snapshots unless the loader adds a
   compat path.
-* **Snapshot blob encoding**, `MT_SNAPSHOT_V1`
-  (`application/vnd.hyperlight.snapshot.memory.v1`), aliased as
+* **Snapshot blob encoding**, `MT_SNAPSHOT_V2`
+  (`application/vnd.hyperlight.snapshot.memory.v2`), aliased as
   `MT_SNAPSHOT_CURRENT`. This is the on-wire format of the snapshot
   blob: framing, section ordering, alignment, dirty/zero-page elision,
   anything about how the bytes are packed inside the OCI layer.
+* **Page-table blob encoding**, `MT_PAGE_TABLES_V1`
+  (`application/vnd.hyperlight.snapshot.page-tables.v1`), aliased as
+  `MT_PAGE_TABLES_CURRENT`. This is the encoding of the guest page-table
+  tree stored after the data layers.
 * **Transport blob encoding**, `MT_TRANSPORT_V1`
   (`application/vnd.hyperlight.snapshot.transport.v1`), aliased as
   `MT_TRANSPORT_CURRENT`. This is the binary encoding of canonical
   virtqueue state stored outside the memory layer.
-* **Config schema**, `MT_CONFIG_V3`
-  (`application/vnd.hyperlight.snapshot.config.v3+json`), aliased as
+* **Config schema**, `MT_CONFIG_V4`
+  (`application/vnd.hyperlight.snapshot.config.v4+json`), aliased as
   `MT_CONFIG_CURRENT`. This is the JSON shape of the config blob:
   field names, types, required vs optional, the descriptors the loader
   needs in order to reconstruct the sandbox (memory sizes, buffer
   sizes, `abi_version`, `hyperlight_version`, etc.). Renaming a field,
   changing its type, or adding a required field is a schema change and
-  bumps this constant. Version 3 describes the virtqueue-only memory layout
-  and requires a transport layer. Config v1 and v2 are incompatible with
-  the current ABI.
+  bumps this constant. Version 4 describes data layers, page tables, and
+  transport geometry, and requires a transport layer. Config v1, v2,
+  and v3 are incompatible with the current ABI.
 
 The `OCI_LAYOUT_VERSION` constant is pinned by the OCI image-layout
 spec at `1.0.0`.
@@ -68,7 +72,7 @@ out at build time rather than in production.
 Compile-time tripwires in
 [src/hyperlight_host/src/sandbox/snapshot/tripwires.rs](../src/hyperlight_host/src/sandbox/snapshot/tripwires.rs)
 hold a copy of every value that defines the format:
-`SNAPSHOT_ABI_VERSION`, the snapshot and config media-type strings, the
+`SNAPSHOT_ABI_VERSION`, every `_CURRENT` media-type string, the
 OCI layout version, the `HyperlightPEB` size, every `OutBAction` and
 `VmAction` discriminant, and `BASE_ADDRESS`. If the source value
 drifts from the copy in `tripwires.rs`, the crate fails to compile.

proposals/0003-incremental-snapshots/README.md

@@ -14,8 +14,8 @@ immutable layers. Each layer has one shared read-only blob, and the live guest
 address ranges that the layer gives.
 
 An incremental snapshot points at the blobs of the parent and adds one new
-layer. That layer holds the data that changed, and the page tables for a
-restore. Sharing a blob costs one cloned `Arc`, because a blob is read-only.
+layer. That layer holds the data that changed. The snapshot also holds its
+page tables. Sharing a blob costs one cloned `Arc`, because a blob is read-only.
 Thus a snapshot stores the changes only, which lowers memory use and makes a
 snapshot faster to take.
 
@@ -48,13 +48,15 @@ and snapshot B is taken.
 flowchart LR
   subgraph A["Snapshot A"]
     LA["layer<br>live p0 p1 p2 p3"]
+    PTA["page tables A"]
   end
   subgraph B["Snapshot B"]
     LB0["layer<br>live p0 p1 p3"]
     LB1["layer<br>live p2'"]
+    PTB["page tables B"]
   end
-  BLOB0["blob 0<br>data p0 p1 p2 p3<br>page tables"]
-  BLOB1["blob 1<br>data p2'<br>page tables"]
+  BLOB0["blob 0<br>data p0 p1 p2 p3"]
+  BLOB1["blob 1<br>data p2'"]
   LA --> BLOB0
   LB0 --> BLOB0
   LB1 --> BLOB1
@@ -64,18 +66,16 @@ Snapshot B keeps the layer of snapshot A, but the live ranges of that layer do
 not hold page 2. Both snapshots share blob 0. Only page 2 is a copy.
 
 * A `SnapshotMemory` has the layers of one snapshot, in the sequence of their
-  guest physical address. It also has the index of the layer that gives the
-  page tables for a restore.
+  guest physical address. It also has the page tables for a restore.
 * A `SnapshotLayer` has one blob and the ranges in that blob that the layer
   gives. Each layer has its own ranges. Layers in different snapshots can share
   a blob.
-* A `SnapshotBlob` is one immutable, contiguous block of host memory. It has two
-  parts, guest data and page tables. All snapshots that use this data share the
-  blob. A blob has no data
-  when every mapped page is already in a live range, for example when the host
-  calls `map_region` or `map_file_cow` and then takes a snapshot. Every layer
-  contains page tables, but only the last layer's page tables are needed for a
-  restore.
+* A `SnapshotBlob` is one immutable, contiguous block of host memory. It holds
+  guest data. All snapshots that use this data share the blob. A snapshot adds
+  no blob when every mapped page is already in a live range, for example when
+  the host calls `map_region` or `map_file_cow` and then takes a snapshot.
+* A `SnapshotPageTables` is one immutable block of host memory with the page
+  tables of one snapshot. A restore copies it into scratch.
 
 The data ranges of the blobs do not overlap, thus at most one layer holds each
 address. The layers are sorted by the start address of their data range. A
@@ -86,6 +86,16 @@ The snapshots make a tree. A sandbox can restore any snapshot and then make
 more snapshots from it. Snapshots with the same parent share the blobs of that
 parent.
 
+### Page tables
+
+Each snapshot stores its page tables in a separate block, and blobs hold only
+data. Data pages are shared across snapshots, and every snapshot rebuilds its
+page tables.
+
+A restore copies the page tables into writable scratch. The guest writes them on
+copy on write faults, and x86-64 processors set accessed and dirty flags in
+them. Blobs are read-only. Capture builds the tables for their scratch address.
+
 ### Taking a Snapshot of a Sandbox
 
 1. Find the pages that this snapshot must save. Read the guest page tables to
@@ -95,14 +105,24 @@ parent.
 2. Find a place in the guest address space for the new pages. The new blob must
    not overlap the layers that this snapshot keeps. Use the first unused part
    that is large enough.
-3. Save the new pages in a new blob. Copy them into the blob, then write the
-   page tables after them. A saved page gets a new address in the blob, thus
-   the page tables are built again. A shared page keeps its address.
-4. Make the list of layers. The new blob is one layer, and it gives the page
-   tables for a restore. This snapshot keeps each parent layer that still gives
+3. Save the new pages in a new blob. A saved page gets a new address in the
+   blob, thus the page tables are built again, in their own block. A shared
+   page keeps its address.
+4. Make the list of layers. The new blob is one layer.
+   This snapshot keeps each parent layer that still gives
    at least one page. Such a layer keeps its blob, and its live ranges lose the
    pages that moved into the new blob. This snapshot drops a parent layer that
    gives no page.
+5. Install the snapshot into the sandbox. A later snapshot saves only the pages
+   changed after this one. A page the guest wrote before this snapshot faults
+   again on its next write.
+
+### Regions from `map_region` and `map_file_cow`
+
+A snapshot saves the pages of these regions that the guest page tables map.
+Step 5 removes the regions, and the sandbox stops using their memory. A page
+the guest has not mapped is lost. A later snapshot can place a blob at its
+guest physical address.
 
 ### Restoring a Sandbox to a Snapshot
 
@@ -124,41 +144,42 @@ Mapping or unmapping one is a hypervisor call. A restore changes only the
 mappings that differ, but that can be all of them, so an unbounded number of
 live ranges would make a restore arbitrarily slow. A snapshot therefore has a
 fixed cap on its total mappings, and taking a snapshot above the cap fails.
-Every layer except the one with the restore page tables must give at least one
+Every layer must give at least one
 live range, so the cap bounds the layer count too.
 
-The page tables in the layers before the last waste space, in memory and on
-disk, unless the snapshot that created them is also kept.
-
 ### API
 
-No public API changes, except `PtRootFinder`. It received the flat snapshot
+No public API changes, except `PtRootFinder` and `Sandbox::snapshot`.
+`PtRootFinder` received the flat snapshot
 buffer, which no longer exists, so it now receives a reader that takes a guest
-physical address.
+physical address. `Sandbox::snapshot` installs the snapshot into the sandbox
+and removes the regions of `map_region` and `map_file_cow`.
 
 ### Snapshots on Disk
 
 This builds on the OCI image format that snapshots already use. The image has
 one layer file per blob, named by its sha256. The config lists the layers,
-their live ranges, and which layer gives the restore page tables. The writer
-emits the version 2 config and memory media types. The loader still accepts
-version 1 images, and makes their single blob one layer. Thus old snapshots
-still load.
+their live ranges, and the length of the page tables. A page-table layer and a
+transport layer follow the blobs. The writer emits the version 4 config,
+version 2 memory, and version 1 page-table media types. The loader rejects
+older config versions. Older snapshots must be regenerated.
 
 Each snapshot is one manifest that lists every blob it needs by digest. Two
 snapshots that share a blob name the same digest. The layout stores that file
 once.
 
-Snapshot A and snapshot B saved to one directory:
+Snapshot A and snapshot B saved to one directory, transport layers omitted:
 
 ```
 index.json         tag a -> manifest A, tag b -> manifest B
-blobs/sha256/<mA>  manifest A: config <cA>, layers <b0>
-blobs/sha256/<mB>  manifest B: config <cB>, layers <b0> <b1>
+blobs/sha256/<mA>  manifest A: config <cA>, layers <b0> <pA>
+blobs/sha256/<mB>  manifest B: config <cB>, layers <b0> <b1> <pB>
 blobs/sha256/<cA>  config A: layer 0 live p0 p1 p2 p3
 blobs/sha256/<cB>  config B: layer 0 live p0 p1 p3, layer 1 live p2'
 blobs/sha256/<b0>  blob 0
 blobs/sha256/<b1>  blob 1
+blobs/sha256/<pA>  page tables A
+blobs/sha256/<pB>  page tables B
 ```
 
 Deleting tag a leaves blob 0 in place, because manifest B still names it.
@@ -188,26 +209,19 @@ Deleting tag a leaves blob 0 in place, because manifest B still names it.
     restores with scratch empty. A diff holds a scratch region and restores on
     top of a base.
 
-## Open questions
-
-Should one blob hold both the guest data and the page tables?
-
-For separate blobs: A snapshot carries only the page tables it restores from,
-so it drops the page tables that its earlier layers waste. That saves memory
-and disk, though page tables are small next to the data. Guest data and page
-tables also become separate concerns with separate lifetimes, which makes the
-code simpler. The layer types lose the index of the page table layer, and the
-rule that every other layer must stay live.
+* Store the page tables of a snapshot in its new blob, after the guest data.
+  Rejected because:
 
-For one blob: A future plan is for the guest to read the page tables from the
-mapped blob, skipping the copy that a restore puts in scratch. A separate blob
-would then need its own mapping, which costs one more mapping per snapshot and
-two hypercalls per restore. This cost is hypothetical, because nothing maps the
-page tables yet.
+  * Every layer carries page tables, and a restore uses only those of the
+    newest layer. Shared blobs keep the rest in memory and on disk.
+  * The snapshot needs the index of the layer with the restore page tables.
+    Every other layer must give at least one live range.
+  * The gain is one mapping per snapshot, for a guest that runs on page tables
+    in a mapped blob. Page tables need writable memory, and blobs are read-only.
 
 ## Future work
 
 An API to compact a snapshot would rebuild it as a single layer. The new layer
-holds only the live pages, so the snapshot drops its dead pages and its extra
-page tables. It also needs one mapping instead of many, which gives a long
+holds only the live pages, so the snapshot drops its dead pages.
+It also needs one mapping instead of many, which gives a long
 chain room under the cap. This is out of scope for this HIP.