Repository navigation
perf: reserve each function's operand stack on entry - #59
Merged
explodingcamera merged 6 commits intoSep 24, 2026
Merged
explodingcamera merged 6 commits into
explodingcamera merged 6 commits into
Conversation
The parser records the highest operand stack each function body reaches in each lane (WasmFunction::max_stack), and enter_locals reserves it on top of the locals. A push inside a function body then traps at capacity instead of growing, so the instruction handlers no longer call into the growth path. Pushes from outside a body (host arguments and results) still grow dynamic stacks. WasmFunction gained a field, so the archive version moves to 06.
This was referenced Sep 24, 2026
explodingcamera
pushed a commit
that referenced
this pull request
Sep 24, 2026
First of a three-PR stack on `next` (`785be0e`), each on top of the previous one with individual wins that combine nicely: 1. #57 **perf: grow the value stack out of line** (this PR) 2. #58 perf: inline the fused binop and compare helpers 3. #59 perf: reserve each function's operand stack on entry `Stack::push` now checks `len == capacity` itself and hands the full case to a `#[cold] #[inline(never)] push_grow`, which keeps the existing overflow check and growth. After that check `Vec::push` can no longer reach its own growth path, so the allocator call leaves the instruction handlers. It was inlined into every handler that pushes, and the call made those handlers save and restore four pairs of callee-saved registers on every execution, even though the default config's stacks are fixed and never grow. `LocalGet32`'s hot path goes from 57 to 37 instructions on aarch64. No behavior change: fixed stacks still trap at capacity, and dynamic stacks still grow up to `max_size`. **iPhone 12 efficiency cores** (A14 Icestorm, `.utility` QoS). XCode 27 PMU template for low-level CPU stats. 16 benchmark rows, 5 launches per build, interleaved with the other builds of the stack; cycles and instructions per call, geomean against `next` as of `b45a98a`: | | cycles | instructions | rows faster | |---|---:|---:|---:| | this PR vs `next` | **−4.1 %** | −3.5 % | 16/16 | **Checks** on this branch: tinywasm's CI matrix run on my local M4 Mac: `cargo test --workspace` and `cargo test --workspace --examples` with `--features tinywasm/nightly-tail-calls`, each with and without default features
explodingcamera
pushed a commit
that referenced
this pull request
Sep 24, 2026
Second of a three-PR stack on `next` (`785be0e`): 1. #57 perf: grow the value stack out of line 2. #58 **perf: inline the fused binop and compare helpers** (this PR) 3. #59 perf: reserve each function's operand stack on entry This branch includes #57 below it; the last commit is new here. `#[inline(always)]` on `exec_binop_32/64` and `exec_cmp_32/64`. LLVM kept them out of line, so every fused `BinOp*` / compare handler called out for the operator `match` and spilled registers around the call. Inlined, the `match` runs inside the handler. The binary grows by about 16 KB, and I didn't see a notable rise in L1 cache misses on Apple A14 efficiency cores in my benchmarks. No behavior change. **iPhone 12 E-cores**, same setup as the first PR; this PR against the first: | | cycles | instructions | rows faster | |---|---:|---:|---:| | this PR vs the first | **−1.3 %** | −2.0 % | 12/16 | | first two PRs vs `next` | −5.3 % | −5.5 % | 16/16 | The four rows that didn't get faster moved by 0.5 % or less. **Checks** on this branch: tinywasm's CI matrix run on my local M4 Mac: `cargo test --workspace` and `cargo test --workspace --examples` on rust 1.98 and with `--features tinywasm/nightly-tail-calls`, each with and without default features.
Owner
|
thanks for the prs! I also measured some nice performance gains on x86 with these changes. |
explodingcamera
pushed a commit
that referenced
this pull request
Sep 24, 2026
First of a three-PR stack on `next` (`785be0e`), each on top of the previous one with individual wins that combine nicely: 1. #57 **perf: grow the value stack out of line** (this PR) 2. #58 perf: inline the fused binop and compare helpers 3. #59 perf: reserve each function's operand stack on entry `Stack::push` now checks `len == capacity` itself and hands the full case to a `#[cold] #[inline(never)] push_grow`, which keeps the existing overflow check and growth. After that check `Vec::push` can no longer reach its own growth path, so the allocator call leaves the instruction handlers. It was inlined into every handler that pushes, and the call made those handlers save and restore four pairs of callee-saved registers on every execution, even though the default config's stacks are fixed and never grow. `LocalGet32`'s hot path goes from 57 to 37 instructions on aarch64. No behavior change: fixed stacks still trap at capacity, and dynamic stacks still grow up to `max_size`. **iPhone 12 efficiency cores** (A14 Icestorm, `.utility` QoS). XCode 27 PMU template for low-level CPU stats. 16 benchmark rows, 5 launches per build, interleaved with the other builds of the stack; cycles and instructions per call, geomean against `next` as of `b45a98a`: | | cycles | instructions | rows faster | |---|---:|---:|---:| | this PR vs `next` | **−4.1 %** | −3.5 % | 16/16 | **Checks** on this branch: tinywasm's CI matrix run on my local M4 Mac: `cargo test --workspace` and `cargo test --workspace --examples` with `--features tinywasm/nightly-tail-calls`, each with and without default features
explodingcamera
pushed a commit
that referenced
this pull request
Sep 24, 2026
Second of a three-PR stack on `next` (`785be0e`): 1. #57 perf: grow the value stack out of line 2. #58 **perf: inline the fused binop and compare helpers** (this PR) 3. #59 perf: reserve each function's operand stack on entry This branch includes #57 below it; the last commit is new here. `#[inline(always)]` on `exec_binop_32/64` and `exec_cmp_32/64`. LLVM kept them out of line, so every fused `BinOp*` / compare handler called out for the operator `match` and spilled registers around the call. Inlined, the `match` runs inside the handler. The binary grows by about 16 KB, and I didn't see a notable rise in L1 cache misses on Apple A14 efficiency cores in my benchmarks. No behavior change. **iPhone 12 E-cores**, same setup as the first PR; this PR against the first: | | cycles | instructions | rows faster | |---|---:|---:|---:| | this PR vs the first | **−1.3 %** | −2.0 % | 12/16 | | first two PRs vs `next` | −5.3 % | −5.5 % | 16/16 | The four rows that didn't get faster moved by 0.5 % or less. **Checks** on this branch: tinywasm's CI matrix run on my local M4 Mac: `cargo test --workspace` and `cargo test --workspace --examples` on rust 1.98 and with `--features tinywasm/nightly-tail-calls`, each with and without default features.
explodingcamera
added a commit
that referenced
this pull request
Sep 24, 2026
Third of a three-PR stack on `next` (`785be0e`): 1. #57 perf: grow the value stack out of line 2. #58 perf: inline the fused binop and compare helpers 3. #59 **perf: reserve each function's operand stack on entry** (this PR) This branch includes #58 and #57 below it; the last three commits are new here. The first PR moved stack growth out of the instruction handlers but kept a cold call to it, and that call still costs the handlers their register saves. This PR removes growth from `push` entirely: - The parser already tracks the operand stack per lane (`lane_counts` in `visit.rs`). It now also records the highest point each function body reaches, as `WasmFunction::max_stack`. - `enter_locals` reserves `locals + max_stack` in each lane when a function is entered, and grows a dynamic stack there if needed. It already grew for the locals at that point. - `Stack::push`, used by the instruction handlers, then traps at capacity instead of calling a growth path: a full stack inside a body means the limit. Typed guest operand handlers now use reserved capacity; host value marshaling can still grow. - Host argument and result marshaling through `push_dyn` retains `push_or_grow`, including host calls made by a guest. - Catch landing reservations now include tag payload lanes and optional exception references, including function-label catches and cross-frame throws. Self-calls preserve the actual value-stack overflow trap; the WAST exhaustion assertion accepts either internal stack reaching its limit first. GC field reads and exception payload injection use reserved no-growth pushes. Changes to know about: - **Public API**: `tinywasm_types::WasmFunction` gains a `pub max_stack: ValueCounts` field. - **Archive format**: the layout of `WasmFunction` changed, so `TWASM_VERSION` goes to `06` (the first format change since 0.11.0 shipped with `05`). - **Stack limits**: overflow is detected when a function is entered, not at the push that crosses the limit. A function whose deepest operand stack does not fit traps on entry, even if a particular call would not have reached that depth. Deep recursion runs out slightly earlier, by at most one function's operand stack. - `tests/value_stack_reservation.rs` runs a function with a deep operand stack in all three lanes, recursively, on dynamic stacks that start empty or tiny and on a fixed stack. It also checks that a limit smaller than the body's needs still traps with `ValueStackOverflow`. **iPhone 12 E-cores**, same setup as the first PR, with XCode 27 PMU template for low-level CPU stats. This PR against the second, which ittself is against `next`, and against an upper bound: the second PR plus a `push` with no growth path and no reservation, which behaves the same on fixed stacks (the default) but would trap on dynamic ones. | | cycles | instructions | rows faster | |---|---:|---:|---:| | this PR vs the second | **−2.8 %** | −1.9 % | 15/16 | | the whole stack vs `next` | **−8.0 %** | −7.2 % | 16/16 | | upper bound vs the second | −2.7 % | −2.4 % | 16/16 | | this PR vs the upper bound | −0.1 % | +0.5 % | 8/16 | The reservation keeps the whole upper-bound gain on the geomean. Its extra work on function entry shows up on the most call-heavy rows. The EH parser (exnref) is the one row that is slower than with the first two PRs: +1.3 %, consistently across reps. Against the upper bound, it is +2.3 %, and the tail-call FSM +1.2 %. A few rows come out about 3 % faster than the upper bound (sieve, xmrsplayer), which is most likely code layout rather than this change. Cycles per call on `next`, and every build as a ratio to it (median of 5 launches): | row | `next` Mcycles | first PR | first two | whole stack | upper bound | |---|---:|---:|---:|---:|---:| | fib(30) | 303.8 | 0.964 | 0.931 | 0.913 | 0.905 | | matmul relaxed-simd FMA | 8.17 | 0.960 | 0.951 | 0.889 | 0.879 | | audio DSP | 3089 | 0.960 | 0.926 | 0.881 | 0.888 | | call_indirect | 63.36 | 0.970 | 0.975 | 0.952 | 0.953 | | xmrsplayer | 45.52 | 0.943 | 0.946 | 0.909 | 0.938 | | vtable_poly4 | 128.5 | 0.984 | 0.961 | 0.937 | 0.934 | | graphql-validation (AssemblyScript) | 36.61 | 0.975 | 0.945 | 0.936 | 0.926 | | graphql-validation (Porffor) | 31.03 | 0.970 | 0.958 | 0.957 | 0.952 | | sieve (scalar) | 2.04 | 0.909 | 0.908 | 0.857 | 0.889 | | crc32 (scalar) | 14.75 | 0.915 | 0.904 | 0.866 | 0.869 | | convolution (scalar) | 40.93 | 0.935 | 0.933 | 0.882 | 0.887 | | bulk_memory (scalar) | 30.59 | 0.936 | 0.907 | 0.869 | 0.877 | | tail-call FSM | 14.59 | 0.978 | 0.967 | 0.953 | 0.942 | | EH parser (exnref) | 42.01 | 0.980 | 0.981 | 0.994 | 0.972 | | GC binary trees | 284.4 | 0.989 | 0.990 | 0.984 | 0.984 | | call_ref twin (call_indirect) | 55.03 | 0.983 | 0.970 | 0.962 | 0.954 | | **geomean** | | **0.959** | **0.947** | **0.920** | **0.921** | **Checks** on this branch: tinywasm's CI matrix , run on my local M4 MacBook: `cargo test --workspace` and `cargo test --workspace --examples` . I went the distance and made sure the whole suite passes with the default value stacks switched to dynamic stacks that start empty and grow to exactly each function's reservation (a local stress configuration, not in this PR), where a function whose recorded maximum undercounted its pushes would trap. Last, I did a coverage-driven fuzzer pass overnight while I slept, and didn't get any anomalous exceptions or state tainting. --------- Co-authored-by: Henry Gressmann <mail@henrygressmann.de>
explodingcamera
pushed a commit
that referenced
this pull request
Sep 29, 2026
With the tail-call dispatch, a handler that contains a call saves and restores a frame record on every instruction it executes, even when the call is on a path that validated code never takes. On `next` all but three of the 615 handlers had such a call: the `instruction_handler_mismatch` and `stack_underflow` panics, the bounds-check panics of value-stack and global indexing, or the conversion of the `Trap::ValueStackOverflow` a push could return into an `ExecError`. The handlers now `become` the mismatch and instruction-pointer panics instead of calling them. Value-stack and global accesses that validation rules out stop through `invariant_violated`, which with `nightly-tail-calls` in release builds is `core::intrinsics::abort` (an inline trap instruction) and otherwise the panic it was before. A push inside a function body no longer returns a `Result`: `enter_locals` reserves the function's whole operand stack or traps before the body runs (#59). Only modules that skip validation can break these invariants, and archives, which do, are already documented as trusted input. The second commit inlines the five memory helpers that were not `#[inline(always)]`, so a load or store through a local address no longer calls `exec_load_local` and similar. 448 of the 615 handlers are now frameless, and `i32.add` is 23 instructions instead of 31. Memory, call and return handlers still keep a frame for their traps and slow paths. Change in cycles per call against `next` (a0ea681), on the efficiency cores of an iPhone 12 (A14), iPhone XS Max (A12) and iPhone SE (A13), median of five interleaved launches per build: | benchmark | A14 | A12 | A13 | |---|---:|---:|---:| | xmrsplayer (1024-frame buffer) | −5.5% | −4.0% | −12.3% | | audio DSP (1000 frames × 512) | −3.8% | −4.3% | −9.7% | | graphql-validation (AS) | −4.6% | −5.5% | −7.7% | | multi-memory twin: one memory | −9.2% | −8.7% | −11.1% | | crc32 (64 KB) | −5.5% | −10.4% | −7.7% | | convolution 256×256 | −3.5% | −5.9% | −6.1% | | sieve (10000) | −14.1% | −8.6% | −13.9% | | bulk_memory (memory.copy/fill) | −6.7% | −8.3% | −7.6% | | matmul relaxed-simd FMA | −6.0% | −8.4% | −9.7% | | GC binary trees (~130K struct.new) | −1.3% | −2.5% | −1.9% | | fib(30) | −8.7% | −6.4% | −7.6% | | tail-call FSM (65536 return_call) | −8.3% | −6.5% | −7.7% | | call_indirect (200K) | −2.9% | −5.9% | −3.9% | | call_ref (200K) | −4.3% | −3.8% | −5.0% | | vtable_poly4 (200K) | −4.3% | −4.1% | −5.3% | | EH parser, exnref (4096 stmts, 25% throw) | −2.8% | −1.4% | −3.2% | | **geomean, cycles** | **−5.8%** | **−6.0%** | **−7.6%** | | geomean, instructions | −7.2% | −7.2% | −7.2% | | geomean, wall time | −5.7% | −6.0% | −7.7% | | rows faster (cycles) | 16/16 | 16/16 | 16/16 | On an M4's efficiency cores these two commits give −8.6% instructions and −5.6% cycles, and the loop dispatch (without `nightly-tail-calls`) −3.5% instructions and −1.1% cycles. Some gory details: * xmrsplayer cycles per call fall from 39.5M to 37.4M, graphql from 32.3M to 30.8M. * branch mispredicts stay flat, within ±5%. * the main one cost is memory-order flushes: 7.7k → 13.2k per xmrsplayer call, roughly 0.3–0.4% of cycles against the 5.5% gain. I elaborated on the memory-order flushes in a separate PR. The good news is that the extra inlining isn't busting the cache, and the call avoidance isn't unraveling some other CPU performance in a way I didn't predict. we'll keep slamming into this memory-order flush stall pileup until we deal with it explicitly, but it's still net positive on wall clock and benchmarks even on the cut-down efficiency cores :)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third of a three-PR stack on
next(785be0e):This branch includes #58 and #57 below it; the last three commits are new here.
The first PR moved stack growth out of the instruction handlers but kept a cold call to it, and that call still costs the handlers their register saves. This PR removes growth from
pushentirely:lane_countsinvisit.rs). It now also records the highest point each function body reaches, asWasmFunction::max_stack.enter_localsreserveslocals + max_stackin each lane when a function is entered, and grows a dynamic stack there if needed. It already grew for the locals at that point.Stack::push, used by the instruction handlers, then traps at capacity instead of calling a growth path: a full stack inside a body means the limit. Typed guest operand handlers now use reserved capacity; host value marshaling can still grow.push_dynretainspush_or_grow, including host calls made by a guest.Changes to know about:
tinywasm_types::WasmFunctiongains apub max_stack: ValueCountsfield.WasmFunctionchanged, soTWASM_VERSIONgoes to06(the first format change since 0.11.0 shipped with05).tests/value_stack_reservation.rsruns a function with a deep operand stack in all three lanes, recursively, on dynamic stacks that start empty or tiny and on a fixed stack. It also checks that a limit smaller than the body's needs still traps withValueStackOverflow.iPhone 12 E-cores, same setup as the first PR, with XCode 27 PMU template for low-level CPU stats. This PR against the second, which ittself is against
next, and against an upper bound: the second PR plus apushwith no growth path and no reservation, which behaves the same on fixed stacks (the default) but would trap on dynamic ones.nextThe reservation keeps the whole upper-bound gain on the geomean. Its extra work on function entry shows up on the most call-heavy rows. The EH parser (exnref) is the one row that is slower than with the first two PRs: +1.3 %, consistently across reps. Against the upper bound, it is +2.3 %, and the tail-call FSM +1.2 %. A few rows come out about 3 % faster than the upper bound (sieve, xmrsplayer), which is most likely code layout rather than this change.
Cycles per call on
next, and every build as a ratio to it (median of 5 launches):nextMcyclesChecks on this branch: tinywasm's CI matrix , run on my local M4 MacBook:
cargo test --workspaceandcargo test --workspace --examples. I went the distance and made sure the whole suite passes with the default value stacks switched to dynamic stacks that start empty and grow to exactly each function's reservation (a local stress configuration, not in this PR), where a function whose recorded maximum undercounted its pushes would trap. Last, I did a coverage-driven fuzzer pass overnight while I slept, and didn't get any anomalous exceptions or state tainting.