Repository navigation
feat(onlion): trace where the invocation memory goes - #123
Merged
Merged
Conversation
The 2026-10-29 incident has 220 invocations that died at the 120 s ceiling carrying 762-1014 MB, and the log cannot say what that was. Every one of those rows is a killed request, so the sample is conditioned on the outcome: it cannot separate a container baseline from a heap that grew inside a warm instance, and a narrow band across killed requests is exactly what a retry loop plateaued by GC would look like too. Three fields are what separate those cases: - `baseline` is read when the module is evaluated, before SQLite exists. If it is already large, the platform figure includes the container and the V8 reservation, and every allocation we could optimise is noise against it. - `seq` counts requests in this instance, so a trend is read against request number rather than wall time: a leak only means anything inside one warm instance. - the breakdown separates `heapUsed` (JS objects: a manifest, a parsed body) from `external`/`arrayBuffers` (byte buffers, where a WASM linear memory that only grows shows up), and `sqlite` reports the live image count, the instance high-water mark and the WASM linear size. Fast requests are sampled on purpose - one in twenty by default, all of them with NMG_MEM_TRACE=1, none with =0 - because a trace that only ever saw hung requests cannot be compared with anything. The slow-request report and /admin/status carry the fields unconditionally, so memory can be watched while a deployment is live. The same change stops retaining a partition image longer than it must: once the bytes are copied into the WASM heap, the SQLite module owns them (FREEONCLOSE), so the JS copy is dropped while the connection is still open instead of at the end of the request. Not an optimisation yet: the in-flight peak the code can account for is about 32 MiB per partition operation, an order of magnitude below the reported figure, so there is nothing to optimise until the baseline is known. Verified: onlion npm run verify 35 PASS (new test-mem-trace), eslint and prettier clean, complexity gate clean for every file touched here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
220 invocations died at the 120 s ceiling on 2026-10-29 carrying 762-1014 MB, and the log
cannot say what that was. Every one of those rows is a killed request, so the sample is
conditioned on the outcome: it cannot separate a container baseline from a heap that grew
inside a warm instance. A narrow band across killed requests is also what a retry loop
plateaued by GC looks like, so "no row below 700 MB" is not evidence for a baseline.
The code can only account for about 32 MiB of in-flight peak per partition operation (8 MiB
image + 8 MiB WASM copy + 8 MiB export + 8 MiB previous image), and partitions are handled one
at a time, so there is nothing to optimise until the baseline is known.
What this adds
baseline(read at module evaluation, before SQLite)seq,uptime_smem.{heapUsed, external, arrayBuffers}sqlite.{live_databases, live_image_mb, high_water_*, wasm_linear_mb}Sampling: one request in twenty by default, every request with
NMG_MEM_TRACE=1, none with=0. Fast requests are sampled on purpose - the incident sample contained only hung ones.The 60 s slow-request report and
/admin/statuscarry the fields unconditionally, so memory canbe watched during a live deployment instead of only after a kill.
Read-out
baseline.rssalready ~800, flatrssasseqgrowsheapUsedclimbing withseq, no fall at exitexternal/arrayBuffersclimbing withseqlive_databases> 1 orhigh_water_image_mb> 16Also
Once the image is copied into the WASM heap the SQLite module owns it (
FREEONCLOSE), so theJS copy is now dropped while the connection is still open rather than at the end of the request.
Verification
onlion npm run verify: 35 PASS (newtest:memasserts the policy: nothing when disabled, onein twenty by default, every request when forced, and that a baseline exists). eslint, prettier
and the complexity gate are clean for every file touched here.