Skip to content

Write initial snapshot memory in place - #1927

Open
ludfjig wants to merge 1 commit into
ll/1-layered-snapshotsfrom
ll/2-lazy-sandbox-memory
Open

ludfjig wants to merge 1 commit into
ll/1-layered-snapshotsfrom
ll/2-lazy-sandbox-memory

Conversation

@ludfjig

@ludfjig ludfjig commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

New sandboxes skip copying their initial memory. Snapshot::from_env writes the guest code, PEB, init data and page tables in place, and the host populates the rest, mostly the heap, on first guest touch. Sandbox creation gets 33 to 88% faster for small to large heaps.

Benchmarks

Compared to the PR below.

  • MSHV: Linux 6.6 mshv.
  • KVM: Linux 6.17.
Benchmark MSHV KVM
sandboxes/create_initialized/small 4.6 ms → 3.09 ms (−33%) 5.12 ms → 2.92 ms (−43%)
sandboxes/create_initialized/medium 11.8 ms → 5.93 ms (−50%) 9.81 ms → 3.11 ms (−68%)
sandboxes/create_initialized/large 38.5 ms → 11 ms (−71%) 28.8 ms → 3.59 ms (−88%)
snapshot_files/cold_start_via_evolve/large 38.8 ms → 10.7 ms (−72%) 29.1 ms → 4.24 ms (−85%)

No benchmark regresses.

Full results

All benchmarks on both hosts (click to expand)

Before → after (change). Negative is faster. Bold is a regression. ~ is no consistent change. A change needs agreement between a 0.5 s and a 3 s run, or a 3 s run alone.

snapshots

Benchmark Size MSHV KVM
call_and_create default 378 µs → 366 µs (−3%) 360 µs → 379 µs (~)
call_and_create small 425 µs → 411 µs (~) 417 µs → 462 µs (~)
call_and_create medium 789 µs → 777 µs (~) 769 µs → 769 µs (~)
call_and_create large 1.84 ms → 1.82 ms (~) 1.85 ms → 1.87 ms (~)
create default 249 µs → 234 µs (−6%) 262 µs → 254 µs (~)
create small 291 µs → 282 µs (~) 324 µs → 313 µs (~)
create medium 606 µs → 612 µs (~) 606 µs → 629 µs (~)
create large 1.68 ms → 1.68 ms (~) 1.7 ms → 1.68 ms (~)
restore default 69 µs → 71.4 µs (~) 22.2 µs → 24.7 µs (~)
restore small 68.7 µs → 73.9 µs (~) 26.7 µs → 23.5 µs (~)
restore medium 72.3 µs → 77.2 µs (~) 30.8 µs → 28 µs (~)
restore large 113 µs → 110 µs (~) 85.2 µs → 61.7 µs (~)
restore_alternate default 78.6 µs → 82.5 µs (~) 71.8 µs → 92.1 µs (~)
restore_alternate small 82.9 µs → 82.2 µs (~) 93.4 µs → 94.4 µs (~)
restore_alternate medium 86.8 µs → 85.6 µs (~) 97.9 µs → 98.8 µs (~)
restore_alternate large 123 µs → 125 µs (~) 158 µs → 129 µs (~)

guest_calls

Benchmark Size MSHV KVM
call default 45.2 µs → 45.4 µs (~) 17.1 µs → 16.9 µs (~)
call small 45.2 µs → 45.2 µs (~) 17.5 µs → 18.0 µs (~)
call medium 45.4 µs → 45.0 µs (~) 17.5 µs → 17.5 µs (~)
call large 44.7 µs → 45.2 µs (~) 17.6 µs → 17.5 µs (~)
call_with_host_function default 75.2 µs → 73.5 µs (~) 34.5 µs → 31.9 µs (~)
call_with_host_function small 74.1 µs → 74 µs (~) 34.7 µs → 33.4 µs (~)
call_with_host_function medium 75.2 µs → 75.7 µs (~) 32.8 µs → 34.2 µs (~)
call_with_host_function large 75.2 µs → 75.8 µs (~) 32.7 µs → 33.6 µs (~)
call_with_restore default 133 µs → 148 µs (~) 62.9 µs → 62.5 µs (~)
call_with_restore small 140 µs → 139 µs (~) 63.8 µs → 71.9 µs (~)
call_with_restore medium 138 µs → 141 µs (~) 67.9 µs → 67.4 µs (~)
call_with_restore large 166 µs → 166 µs (~) 89.1 µs → 89.8 µs (~)
different_thread - 74 µs → 72.3 µs (~) 42.9 µs → 42.3 µs (~)
interrupt_latency - 48.9 µs → 64.5 µs (~) 35.9 µs → 39.2 µs (~)

guest_functions_with_large_parameters

Benchmark Size MSHV KVM
guest_call_with_large_parameters - 163 ms → 172 ms (~) 916 ms → 890 ms (~)

sample_workloads

Benchmark Size MSHV KVM
24K_in_8K_out_c - 56.5 µs → 57.5 µs (~) 27.6 µs → 27.6 µs (~)
24K_in_8K_out_rust - 56.1 µs → 55.3 µs (~) 27.9 µs → 27.8 µs (~)

sandboxes

Benchmark Size MSHV KVM
create_initialized default 1.72 ms → 1.49 ms (−13%) 2.8 ms → 2.83 ms (~)
create_initialized small 4.6 ms → 3.09 ms (−33%) 5.12 ms → 2.92 ms (−43%)
create_initialized medium 11.8 ms → 5.93 ms (−50%) 9.81 ms → 3.11 ms (−68%)
create_initialized large 38.5 ms → 11 ms (−71%) 28.8 ms → 3.59 ms (−88%)
create_initialized_and_drop default 21.4 ms → 24.3 ms (~) 14.1 ms → 11 ms (~)
create_initialized_and_drop small 28.9 ms → 25.9 ms (~) 13.3 ms → 15.7 ms (~)
create_initialized_and_drop medium 31.5 ms → 28.3 ms (−10%) 24.3 ms → 14.8 ms (−39%)
create_initialized_and_drop large 57.9 ms → 30.7 ms (−47%) 39.6 ms → 12.4 ms (−69%)
sandbox_from_snapshot default 665 µs → 631 µs (−5%) 971 µs → 947 µs (~)
sandbox_from_snapshot small 812 µs → 805 µs (~) 978 µs → 971 µs (~)
sandbox_from_snapshot medium 2.02 ms → 2.04 ms (~) 1.01 ms → 999 µs (~)
sandbox_from_snapshot large 6.06 ms → 6.16 ms (~) 1.22 ms → 1.21 ms (~)

snapshot_files

Benchmark Size MSHV KVM
cold_start_via_evolve default 1.61 ms → 1.58 ms (~) 2.96 ms → 3.05 ms (~)
cold_start_via_evolve small 4.55 ms → 3.22 ms (~) 5.76 ms → 3.14 ms (~)
cold_start_via_evolve medium 12.3 ms → 5.31 ms (~) 11 ms → 3.52 ms (~)
cold_start_via_evolve large 38.8 ms → 10.7 ms (~) 29.1 ms → 4.24 ms (~)
cold_start_via_snapshot default 1.97 ms → 2.01 ms (~) 2.85 ms → 2.73 ms (~)
cold_start_via_snapshot small 8.93 ms → 9.13 ms (~) 8.95 ms → 8.9 ms (~)
cold_start_via_snapshot medium 53.8 ms → 54 ms (~) 50.9 ms → 50.5 ms (~)
cold_start_via_snapshot large 205 ms → 205 ms (~) 194 ms → 196 ms (~)
cold_start_via_snapshot_unverified default 1.4 ms → 1.43 ms (~) 2.2 ms → 1.95 ms (−11%)
cold_start_via_snapshot_unverified small 2.08 ms → 2.21 ms (~) 2.17 ms → 2.3 ms (~)
cold_start_via_snapshot_unverified medium 3.59 ms → 3.52 ms (~) 2.25 ms → 2.1 ms (~)
cold_start_via_snapshot_unverified large 7.73 ms → 7.67 ms (~) 2.59 ms → 2.34 ms (~)
load_snapshot default 701 µs → 703 µs (~) 701 µs → 692 µs (~)
load_snapshot small 6.71 ms → 6.63 ms (~) 6.68 ms → 6.56 ms (~)
load_snapshot medium 50.4 ms → 50.3 ms (~) 48.8 ms → 48.3 ms (~)
load_snapshot large 198 ms → 197 ms (~) 192 ms → 192 ms (~)
load_snapshot_unverified default 112 µs → 116 µs (~) 123 µs → 130 µs (~)
load_snapshot_unverified small 113 µs → 112 µs (~) 119 µs → 116 µs (~)
load_snapshot_unverified medium 114 µs → 123 µs (~) 120 µs → 123 µs (~)
load_snapshot_unverified large 118 µs → 122 µs (~) 125 µs → 127 µs (~)
save_snapshot default 941 µs → 943 µs (~) 43.4 ms → 39.5 ms (~)
save_snapshot small 9.09 ms → 9.11 ms (~) 61.5 ms → 52.2 ms (~)
save_snapshot medium 69.6 ms → 67.7 ms (~) 402 ms → 396 ms (~)
save_snapshot large 271 ms → 266 ms (~) 1.75 s → 1.74 s (~)

@ludfjig ludfjig added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Oct 12, 2026
@ludfjig
ludfjig added this pull request to stack #1929 October 12, 2026 20:01
@hyperlight-gh-bot

This comment has been minimized.

Copilot AI balanced review requested due to automatic review settings October 13, 2026 18:56
@ludfjig
ludfjig force-pushed the ll/2-lazy-sandbox-memory branch from 96fab64 to ab775e0 Compare October 13, 2026 18:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Lazy no-reserve allocations need aggregate admission or commitment accounting to prevent deferred host-wide OOM.

1 open finding
What changed in this PR

Optimizes sandbox creation by constructing initial snapshot memory directly in shared memory and relying on lazy page population.

Changes:

  • Writes guest initialization data in place.
  • Freezes shared memory without copying.
  • Documents the performance improvement.
File Description
src/​hyperlight_host/​src/​sandbox/​snapshot/​mod.rs Builds snapshots directly in shared memory.
src/​hyperlight_host/​src/​mem/​shared_mem.rs Restricts the copying helper to tests and HVF.
CHANGELOG.md Documents lazy memory population.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +608 to +610
let mut memory = ExclusiveSharedMemory::new(
(data_len + pt_bytes.len()).next_multiple_of(host_page_size),
)?;
@ludfjig
ludfjig force-pushed the ll/2-lazy-sandbox-memory branch from ab775e0 to 107a858 Compare October 13, 2026 21:01
@hyperlight-gh-bot

This comment has been minimized.

Snapshot::from_env writes the guest code, PEB and init data straight
into shared memory and freezes it. The host populates the remaining
pages on first guest touch, so sandbox creation cost stays independent
of heap size.

Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>
@ludfjig
ludfjig force-pushed the ll/2-lazy-sandbox-memory branch from 107a858 to aa55340 Compare October 13, 2026 23:00
@hyperlight-gh-bot

Copy link
Copy Markdown

Benchmark Results

Measured commit: aa55340c419c
Baseline commit: 67561dc342b8

kvm / amd (Linux) (🌟 6.05x)

Top improvements

  • sandboxes/create_initialized_and_drop/medium — 🌟 6.05x faster

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 743.75 ns (➖ 1.03x faster)
vec_bytes 580.23 ns (➖ 1.01x faster)
374.31 µs (➖ 1.00x slower)

payload_allocation

slot_pool_segmented
262144 521.58 ns (➖ 1.00x faster)
65536 149.04 ns (➖ 1.05x slower)

sandboxes

create_initialized_and_drop
medium 13.86 ms (🌟 6.05x faster)

slot_pool

alloc_dealloc_1500 alloc_dealloc_4096 alloc_dealloc_128
7.73 ns (➖ 1.03x faster) 7.76 ns (➖ 1.01x slower) 7.72 ns (➖ 1.00x faster)

snapshot_files

load_snapshot_unverified
small 117.78 µs (➖ 1.00x faster)

virtq_readonly

slot_pool_segmented_fragmented slot_pool_segmented
262144 7.04 µs (➖ 1.02x faster) 7.42 µs (➖ 1.06x slower)
65536 2.06 µs (➖ 1.06x slower) 1.93 µs (➖ 1.08x faster)

virtq_readwrite

slot_pool_segmented_fragmented slot_pool_segmented
65536 6.26 µs (➖ 1.05x faster) 6.22 µs (➖ 1.01x faster)
8192 1.09 µs (➖ 1.03x slower) 1.06 µs (➖ 1.01x slower)
262144 27.72 µs (➖ 1.00x slower)
kvm / intel (Linux) (🌟 5.48x)

Top improvements

  • sandboxes/create_initialized_and_drop/medium — 🌟 5.48x faster

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 777.61 ns (➖ 1.03x faster)
vec_bytes 587.81 ns (➖ 1.02x slower)
756.70 µs (➖ 1.05x slower)

payload_allocation

slot_pool_segmented
262144 573.23 ns (➖ 1.01x slower)
65536 155.20 ns (➖ 1.01x slower)

sandboxes

create_initialized_and_drop
medium 14.95 ms (🌟 5.48x faster)

slot_pool

alloc_dealloc_1500 alloc_dealloc_4096 alloc_dealloc_128
7.93 ns (➖ 1.02x slower) 7.89 ns (➖ 1.01x slower) 7.90 ns (➖ 1.01x slower)

snapshot_files

load_snapshot_unverified
small 62.92 µs (➖ 1.05x faster)

virtq_readonly

slot_pool_segmented_fragmented slot_pool_segmented
262144 8.69 µs (➖ 1.01x slower) 8.68 µs (➖ 1.01x slower)
65536 2.47 µs (➖ 1.03x slower) 2.42 µs (➖ 1.00x slower)

virtq_readwrite

slot_pool_segmented_fragmented slot_pool_segmented
65536 8.27 µs (➖ 1.00x faster) 8.22 µs (➖ 1.00x faster)
8192 866.94 ns (➖ 1.01x faster) 845.95 ns (➖ 1.02x faster)
262144 33.62 µs (➖ 1.05x faster)
mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 926.75 ns (➖ 1.00x faster)
vec_bytes 698.76 ns (➖ 1.02x faster)
346.66 µs (➖ 1.27x slower)

payload_allocation

slot_pool_segmented
262144 716.11 ns (➖ 1.00x faster)
65536 190.36 ns (➖ 1.32x faster)

sandboxes

create_initialized_and_drop
medium 61.84 ms (➖ 1.02x slower)

slot_pool

alloc_dealloc_1500 alloc_dealloc_4096 alloc_dealloc_128
9.44 ns (➖ 1.00x faster) 9.59 ns (➖ 1.04x faster) 9.51 ns (➖ 1.00x faster)

snapshot_files

load_snapshot_unverified
small 106.31 µs (➖ 1.00x slower)

virtq_readonly

slot_pool_segmented_fragmented slot_pool_segmented
262144 10.49 µs (➖ 1.06x slower) 10.17 µs (➖ 1.04x slower)
65536 2.53 µs (➖ 1.02x slower) 2.25 µs (➖ 1.00x faster)

virtq_readwrite

slot_pool_segmented_fragmented slot_pool_segmented
65536 8.66 µs (➖ 1.13x faster) 8.21 µs (➖ 1.21x faster)
8192 1.27 µs (➖ 1.05x faster) 1.35 µs (➖ 1.06x slower)
262144 35.91 µs (➖ 1.05x faster)
mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 973.90 ns (➖ 1.04x slower)
vec_bytes 661.08 ns (➖ 1.01x slower)
729.85 µs (➖ 1.19x faster)

payload_allocation

slot_pool_segmented
262144 643.35 ns (➖ 1.01x faster)
65536 170.21 ns (➖ 1.02x slower)

sandboxes

create_initialized_and_drop
medium 70.20 ms (➖ 1.00x slower)

slot_pool

alloc_dealloc_1500 alloc_dealloc_4096 alloc_dealloc_128
9.66 ns (➖ 1.00x slower) 9.49 ns (➖ 1.00x slower) 9.23 ns (➖ 1.01x slower)

snapshot_files

load_snapshot_unverified
small 57.46 µs (➖ 1.13x faster)

virtq_readonly

slot_pool_segmented_fragmented slot_pool_segmented
262144 7.96 µs (➖ 1.01x faster) 8.00 µs (➖ 1.00x slower)
65536 2.29 µs (➖ 1.01x slower) 2.28 µs (➖ 1.00x faster)

virtq_readwrite

slot_pool_segmented_fragmented slot_pool_segmented
65536 7.60 µs (➖ 1.00x faster) 7.57 µs (➖ 1.02x faster)
8192 953.67 ns (➖ 1.00x faster) 936.80 ns (➖ 1.01x slower)
262144 39.92 µs (➖ 1.04x faster)
hyperv-ws2025 / amd (Windows) (❌ 1.83x | 🌟 7.10x)

Top improvements

  • sandboxes/create_initialized_and_drop/medium — 🌟 7.10x faster

Top regressions

  • payload_allocation/slot_pool_segmented/262144 — ❌ 1.83x slower

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 1.19 µs (➖ 1.04x faster)
vec_bytes 792.75 ns (➖ 1.02x slower)
1.88 ms (➖ 1.01x slower)

payload_allocation

slot_pool_segmented
262144 1.43 µs (❌ 1.83x slower)
65536 233.72 ns (➖ 1.01x slower)

sandboxes

create_initialized_and_drop
medium 11.24 ms (🌟 7.10x faster)

slot_pool

alloc_dealloc_1500 alloc_dealloc_4096 alloc_dealloc_128
10.46 ns (➖ 1.02x faster) 10.48 ns (➖ 1.01x slower) 10.15 ns (➖ 1.01x slower)

snapshot_files

load_snapshot_unverified
small 826.93 µs (➖ 1.15x slower)

virtq_readonly

slot_pool_segmented_fragmented slot_pool_segmented
262144 9.13 µs (➖ 1.02x faster) 9.23 µs (➖ 1.02x slower)
65536 2.34 µs (➖ 1.02x faster) 2.42 µs (➖ 1.02x slower)

virtq_readwrite

slot_pool_segmented_fragmented slot_pool_segmented
65536 8.90 µs (➖ 1.02x faster) 8.72 µs (➖ 1.01x faster)
8192 1.31 µs (➖ 1.03x faster) 1.30 µs (➖ 1.01x slower)
262144 39.68 µs (➖ 1.04x slower)
hyperv-ws2025 / intel (Windows) (🌟 8.33x)

Top improvements

  • sandboxes/create_initialized_and_drop/medium — 🌟 8.33x faster

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 1.19 µs (➖ 1.02x faster)
vec_bytes 780.18 ns (➖ 1.03x faster)
2.95 ms (➖ 1.00x slower)

payload_allocation

slot_pool_segmented
262144 726.24 ns (➖ 1.02x faster)
65536 210.38 ns (➖ 1.00x slower)

sandboxes

create_initialized_and_drop
medium 13.69 ms (🌟 8.33x faster)

slot_pool

alloc_dealloc_1500 alloc_dealloc_4096 alloc_dealloc_128
10.28 ns (➖ 1.02x slower) 10.15 ns (➖ 1.01x slower) 10.07 ns (➖ 1.01x slower)

snapshot_files

load_snapshot_unverified
small 580.86 µs (➖ 1.06x faster)

virtq_readonly

slot_pool_segmented_fragmented slot_pool_segmented
262144 7.83 µs (➖ 1.02x slower) 7.82 µs (➖ 1.01x slower)
65536 2.54 µs (➖ 1.09x slower) 2.34 µs (➖ 1.00x faster)

virtq_readwrite

slot_pool_segmented_fragmented slot_pool_segmented
65536 8.56 µs (➖ 1.01x faster) 8.50 µs (➖ 1.01x faster)
8192 1.15 µs (➖ 1.08x faster) 1.10 µs (➖ 1.00x faster)
262144 40.99 µs (➖ 1.18x faster)

Reported by cargo ci bench-report --candidate run:38481552023 --config-file bench_report.toml.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/enhancement For PRs adding features, improving functionality, docs, tests, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants