Skip to content

feat(onlion): trace where the invocation memory goes - #123

Merged
wefio merged 1 commit into
mainfrom
feat/memory-trace
Oct 29, 2026
Merged

wefio merged 1 commit into
mainfrom
feat/memory-trace

Conversation

@wefio

@wefio wefio commented Oct 29, 2026

Copy link
Copy Markdown
Owner

Why

220 invocations died at the 120 s ceiling on 2026-10-29 carrying 762-1014 MB, and the log
cannot say what that was. Every one of those rows is a killed request, so the sample is
conditioned on the outcome: it cannot separate a container baseline from a heap that grew
inside a warm instance. A narrow band across killed requests is also what a retry loop
plateaued by GC looks like, so "no row below 700 MB" is not evidence for a baseline.

The code can only account for about 32 MiB of in-flight peak per partition operation (8 MiB
image + 8 MiB WASM copy + 8 MiB export + 8 MiB previous image), and partitions are handled one
at a time, so there is nothing to optimise until the baseline is known.

What this adds

field why it decides something
baseline (read at module evaluation, before SQLite) if it is already large, the platform figure includes the container and the V8 reservation, and our allocations are noise
seq, uptime_s a leak only means anything inside one warm instance; a trend has to be read against request number
mem.{heapUsed, external, arrayBuffers} JS objects vs byte buffers; a WASM linear memory that only grows shows up in the latter
sqlite.{live_databases, live_image_mb, high_water_*, wasm_linear_mb} whether images outlive their part, and how far the linear heap has grown

Sampling: one request in twenty by default, every request with NMG_MEM_TRACE=1, none with
=0. Fast requests are sampled on purpose - the incident sample contained only hung ones.
The 60 s slow-request report and /admin/status carry the fields unconditionally, so memory can
be watched during a live deployment instead of only after a kill.

Read-out

what you see conclusion
baseline.rss already ~800, flat rss as seq grows platform figure includes the container; stop chasing memory, the 120 s kill was the real problem
heapUsed climbing with seq, no fall at exit JS objects accumulate (manifest / parsed body)
external/arrayBuffers climbing with seq WASM linear memory or byte buffers are not released
live_databases > 1 or high_water_image_mb > 16 images outlive their partition
saw-tooth at exit but platform figure still high concurrency, not a leak

Also

Once the image is copied into the WASM heap the SQLite module owns it (FREEONCLOSE), so the
JS copy is now dropped while the connection is still open rather than at the end of the request.

Verification

onlion npm run verify: 35 PASS (new test:mem asserts the policy: nothing when disabled, one
in twenty by default, every request when forced, and that a baseline exists). eslint, prettier
and the complexity gate are clean for every file touched here.

The 2026-10-29 incident has 220 invocations that died at the 120 s ceiling carrying
762-1014 MB, and the log cannot say what that was. Every one of those rows is a killed
request, so the sample is conditioned on the outcome: it cannot separate a container
baseline from a heap that grew inside a warm instance, and a narrow band across killed
requests is exactly what a retry loop plateaued by GC would look like too.

Three fields are what separate those cases:

- `baseline` is read when the module is evaluated, before SQLite exists. If it is already
  large, the platform figure includes the container and the V8 reservation, and every
  allocation we could optimise is noise against it.
- `seq` counts requests in this instance, so a trend is read against request number rather
  than wall time: a leak only means anything inside one warm instance.
- the breakdown separates `heapUsed` (JS objects: a manifest, a parsed body) from
  `external`/`arrayBuffers` (byte buffers, where a WASM linear memory that only grows shows
  up), and `sqlite` reports the live image count, the instance high-water mark and the WASM
  linear size.

Fast requests are sampled on purpose - one in twenty by default, all of them with
NMG_MEM_TRACE=1, none with =0 - because a trace that only ever saw hung requests cannot be
compared with anything. The slow-request report and /admin/status carry the fields
unconditionally, so memory can be watched while a deployment is live.

The same change stops retaining a partition image longer than it must: once the bytes are
copied into the WASM heap, the SQLite module owns them (FREEONCLOSE), so the JS copy is
dropped while the connection is still open instead of at the end of the request.

Not an optimisation yet: the in-flight peak the code can account for is about 32 MiB per
partition operation, an order of magnitude below the reported figure, so there is nothing
to optimise until the baseline is known.

Verified: onlion npm run verify 35 PASS (new test-mem-trace), eslint and prettier clean,
complexity gate clean for every file touched here.
@wefio
wefio merged commit 6e7421a into main Oct 29, 2026
8 checks passed
@wefio
wefio deleted the feat/memory-trace branch October 29, 2026 07:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant