Repository navigation
release: v1.3 — planned query windows, append-log write path, log accept by default - #119
Merged
Merged
Conversation
A long query was split by tiling windows forward from the start and then
appending one end-anchored window. For a 36,833-character query the sensitive
coverage was the opening 14,000 characters plus the last 2,000: the middle was a
single contiguous blind block, and a question placed there scored exactly what it
scored with no splitting at all (2.14% full-page coverage).
Measured on 281 LoCoMo multi-hop questions, zero LLM cost, page 20:
arm 6,000 chars 36,000 chars
shipped baseline 30.60 26.69
planned windows only 28.47 29.90
clause + planned windows 32.73 29.90
The clause itself stays the first feature: the model's own token window gives the
opening more context than a 2,000-character window can, and dropping it cost
head-aligned recall. The first planned window is then dropped as redundant - it
re-covers the opening, and dropping it measured identical on every condition
(36,000: 11.39% vs 11.03% head-aligned, 0 losses) while saving a call per clause.
- planQueryWindows: two layouts chosen by whether n*W covers the text; equal
disjoint segments when it does, windows pinned at both ends with evenly
spread gaps when it does not.
- selectQueryFeatures: the clause, then the end-anchored tail, then interior
windows, shared round-robin across clauses so a request fills
MAX_QUERY_FEATURES=24 instead of stopping at a fixed count.
- MAX_QUERY_WINDOWS=10 is a per-clause ceiling. The env override and the
separate interior-call budget are gone; one bound, one unit.
- poolQueryVectors left the service: nothing in the product pools windows.
Verified: build, tsc --noEmit, onlion test:backend 13 PASS, live 36,000-character
queries with the question at the head, middle and tail each retrieved the planted
record after deployment.
Worker also carries the write-path retry changes from the same investigation:
content-keyed staging reuse so a CAS retry does not re-upload a multi-megabyte
snapshot, and exponential full-jitter backoff instead of a fixed 15-100 ms window.
First slice of the design in docs/design/add-append-log-and-fold-design.md:
storage pieces for accepting an Add as one immutable object, with no change to the
Add path yet (nothing calls the log until the worker seams land).
- append-log.mjs: entry shape, name as a sortable idempotency key, write with
forbid-overwrite, replay returns the stored entry, cursor-paged pending
listing in arrival order, idempotent batch delete, and a pure foldPlan bounded
by entry count and bytes.
- storage.mjs: `pending` on the manifest with a lazy format bump - v4 only while
entries are pending, so a drained log round-trips to the same v2 bytes a
deployment writes today and a flag-off deployment stays byte-identical on disk.
loadManifest rejects invalid counts as storage_corrupt rather than reading them
as zero, and clearLog drains a user's log prefix from reset.
- Two standalone tests run without a build: test-append-log.mjs and
test-manifest-pending.mjs.
Reset now also drains the log prefix and reports cleanup_complete over both, which
is a no-op while nothing is pending.
Verified: build, tsc --noEmit, onlion test:backend 13 PASS, both new scripts PASS.
…rite path
- 2026-10-28-query-window-plan: what the shipped split covered, the two layouts
that replaced it, the four-arm measurement (shipped / planned only / clause +
planned / clause + planned minus the redundant first window) at 6,000 and
36,000 characters, paired signs, and what the probe does not establish - the
padding is synthetic, only retrieval was measured, and W is characters
calibrated for English.
- add-append-log-and-fold-design: the write-path design, including the frozen
interfaces the implementation is being written against, six invariants, the
failure matrix, the ~680-line breakdown, and the rejected alternatives with
their measurements.
Also records the additive-delta alternative for Add as measured and rejected:
an 8-message Add changes 72-86 pages of 8 KiB against a 32-page budget, 0 of 13
Adds fit, and the overlay lives inline in the manifest every writer contends for.
A reset that lists a log prefix whose cursor does not move would loop forever inside the request. Break on a repeated cursor; retrying the same reset_id still finishes the cleanup.
…ng it The two halves were written in parallel, so storage.mjs avoided a runtime dependency on append-log.mjs to stay loadable on its own. They ship together now, and a prefix that cannot drift is worth more than a module that loads alone, so the import stays and the reasoning is recorded as a decision rather than left as a deviation.
Wires the three seams the design froze. NMG_ADD_MODE defaults to `sync`, so
nothing changes until the log path is measured: this commit adds a switchable
path, not a new default.
acceptAdd one manifest read, one immutable log write, and a manifest CAS
that only carries the pending counter - no partition download,
no SQLite export, no snapshot upload. A conflict retry re-writes
the same entry (a replay) and re-commits only the manifest.
Structured content is prepared before the write so a fold never
calls a vision model while holding a partition.
foldPending folds up to FOLD_BATCH entries in one partition write, skips
entries whose receipt already exists, leaves them in place until
the commit lands, and deletes them afterwards.
pendingForSearch returns page-shaped records with `pending:` ids so the existing
fusion by identity and highest score applies; entries are scored
at read time, capped at MAX_PENDING_READ, which is what makes an
accepted Add searchable before it is folded.
The accept path is triggered from the route when pending reaches FOLD_TRIGGER, so
folding never runs on the read path.
The bucket wrapper in reclamation.mjs now forwards `list`, scoped to the storage
generation and with the scope prefix stripped from returned keys. Without it the
log could be written but never enumerated, in the local fixture and against COS
alike.
Verified: build, tsc --noEmit, onlion test:backend 13 PASS in sync mode, and the
new test-add-log-mode.mjs PASS - accepted Add searchable before folding, replay
byte-identical, survives the fold with a v4 manifest, 409 on different content for
the same request_id, reset drains pending and folded data.
…tires
Three defects in the fold and read path, found by reviewing the commit that
introduced them:
- foldPending iterated `plan.fold` as if its elements were names, but they are
`{ name, entry }` objects. Every lookup missed, so no entry was ever applied,
and the names pushed for deletion were objects that deleteLogEntries filtered
out. Folding was a silent no-op: the entry stayed pending, was still served
from the log, and nothing reported an error.
- pendingForSearch did not filter by generation, so an entry written before a
reset could still be matched after it (I4).
- the pending counter was decremented by the number of entries applied, but
entries skipped because their receipt already exists (crash recovery) or
because they belong to another generation are retired too. The counter then
never reached zero, which disabled the search short-circuit and reported a
backlog that did not exist (I6).
Fold now collects the set of names it is done with - applied, receipt present, or
cross-generation - decrements by that set, and deletes exactly that set after the
commit. Entries from another generation are retired instead of lingering.
The test could not have caught any of this: it asserted that a folded entry stayed
searchable, which a no-op fold answers just as well. It now requires the oldest
entry to be served with a partition id rather than a `pending:` id, so folding
that does nothing fails the test.
Verified: build, tsc --noEmit, onlion test:backend 13 PASS, test-append-log and
test-manifest-pending PASS, test-add-log-mode PASS with the new assertion.
The fold defects that shipped in 09553c9 were invisible to CI because verify ran test:backend, which only asserts add_mode:sync and never exercises the log path. test-add-log-mode.mjs existed but nothing invoked it, so a fold that did nothing stayed green. Wiring it into verify makes the shape mismatch that caused this - iterating {name, entry} objects as strings - fail on the next commit instead of on the next deployment. npm run verify passes: build, check, temporal, media, backend, add-log, worker-media.
…t cannot
Two residual defects in the log path, both consequences of trusting a counter that
a failed side effect can invalidate:
- The accept path wrote the entry and then committed the counter. When the commit
lost its CAS retries the caller got a 503 but the entry was already durable and
no counter accounted for it, so `pendingCount === 0` hid it from every search
(I1). The reservation now commits *before* the write: a failed commit leaves no
entry, and a failed write leaves the counter one too high, which costs a log
listing and nothing else.
- A failed delete of folded entries was swallowed while the counter had already
been decremented, so the same batch was folded again next round and subtracted
again. Repeated failures walked the counter down until the fold trigger could
never fire. The delete result is now checked, and a failure re-reads the log and
writes the observed count back instead of trusting the subtraction.
Found while making the counter observable: `Object.assign(manifest, withPending(m, 0))`
does not remove the key, because `withPending` expresses "no pending" by returning a
copy *without* it. Every non-zero decrement applied, the final one to zero did not,
so a drained log still stored `pending: 7`, the manifest stayed v4, and the search
short-circuit stayed disabled. All three sites now go through `applyPending`, which
deletes the key when the copy omits it.
`/capacity` reports `pending_entries`, and `POST /admin/fold?user_id=` drains a log in
bounded rounds - the operation a rollback needs (a v4 manifest is unreadable to an
older deployment) and the hook the test needed to observe a drained counter.
Verified: npm run verify passes end to end, and test-add-log-mode now asserts that a
burst leaves entries waiting, that draining folds them, that the counter returns to
zero, and that the drained entry is served by the partition rather than the log.
…g for a fold
Reserve-first made the counter err high, which is the safe direction, but nothing
brought it back down: an accept whose log write failed left +1 behind, and since the
fold is triggered *by* that counter, an over-count below the trigger had no path to a
fold at all. The manifest then stayed v4, /capacity kept reporting a backlog, and
every search paid a log listing that could never short-circuit (I6).
- a failed write now releases its slot immediately, relative (count - 1 inside the
CAS), so a concurrent accept that reserved in between is not overwritten;
- an empty log with a non-zero counter is the residue of exactly that failure, so
the fold releases one slot per call and reports it as progress;
- /admin/fold keeps iterating while slots are being released, and reports the count
it released next to the count it folded.
Source compensation is the primary fix; the empty-log branch only matters when the
release itself failed, and an over-count larger than one /admin/fold pass (8 rounds)
needs a second call.
Verified: build, npm run verify (build, check, temporal, media, backend, add-log,
worker-media) all pass. The two compensation paths have no integration test: the
fixture has no fault injection for object writes, so a failed PUT cannot be staged
from a test. Recorded as the follow-up that would cover them.
…ng owns the field `POST /admin/fold` returned `success: true` even when entries were left: eight bounded rounds cannot drain a long log, and the runbook gates a code rollback on this call because a v4 manifest is unreadable to an older deployment. A caller checking 2xx would have rolled back onto a half-drained log. It now answers 409 with `success: false` while anything remains, and 200 only when `remaining` is zero; "made no progress this round" and "nothing left to do" are separate answers. `setPending(manifest, count)` moves to storage.mjs, next to `pendingCount` and `withPending`, and writes the field in place: zero deletes the key rather than storing `pending: 0`. The field's meaning now lives in one module instead of two, and the `Object.assign(manifest, withPending(manifest, 0))` shape that kept a stale count cannot be written again. The worker uses it at all four count sites. Verified: npm run verify passes; test-add-log-mode additionally asserts that a drained log reports success and that draining an already clean log stays 200 with zero folded, and docs:check passes inside the 15,000-byte design-document budget.
…guess A throwing put is not proof that nothing reached the bucket: the storage wrapper writes first and then verifies the generation, so the throw can happen after the object landed. Releasing the reservation on any error therefore drives the counter below the truth for an entry that exists - the one direction that hides accepted data from search - and the earlier three-state reasoning applies to the check itself: reading the entry back is the wrong question, because readLogEntry throws on anything it cannot parse, and that error says nothing about whether a write happened. logEntryLanded asks with HEAD and answers yes / no / unknown. The accept path returns success when the entry landed, releases the slot only on a definite no, and leaves the counter alone when the answer is unknown. Its return shape matches a normal accept, so the fold trigger still sees `pending`. verify now also runs test:append-log and test:manifest-pending: both modules had tests that nothing invoked, which is how the fold defects reached HEAD in the first place. The landed check's three answers are covered with a fake bucket (object / null / throw), so the compensation branch that cannot be reached by injecting a real write failure is no longer untested. Verified: npm run verify passes end to end, including the new three-state assertions.
releasePending returned mutateUser's unwrapped body, so the empty-log branch read released?.response?.pending and always fell back to the pre-release count. A drain that released the last slot on its final budgeted round therefore answered 409 with remaining: 1 - safe, since nothing was hidden, but it claimed the pass had made no progress. It now returns the count itself, and the branch reports it. Two assertions pin both ends: a single residue slot must drain to clean in one call, and nine slots must answer 409 with remaining 1 before the second call finishes them. Verified: npm run verify passes, including the new residue and pass-budget assertions.
…them Four review rounds found six defects in `manifest.pending` and every one was a transition nobody had written down: a fold that iterated entry objects as strings, a decrement that ignored retired entries, a zero that never reached the manifest, a delete that subtracted twice, an entry written before its counter, and a failed write read as "did not land". Each is obvious once the states are on paper. The page lists five states (clean, backlog, folding, residue-high, stale generation), the transitions with the function that performs each, and five invariants - never below the truth, every over-count has a path back, bounded convergence, absence means zero, idempotency carried by names - paired with the assertion that fails when one breaks. It also states the state that must remain unreachable: `pending` absent with a non-empty log. Reserving before the write is what makes it impossible, so seeing it means a bug rather than a state. And it says what the table does not cover: a physically failing write is only tested with a fake bucket, and concurrent folds are serialized by the manifest CAS rather than proven correct here. docs:check passes (779 files), format:check passes.
…after a reset
Two independent mechanisms could make an accepted Add unsearchable, and fixing either
alone leaves the other working:
- the read window: `listPending` truncated keys to the caller's limit and the
generation filter ran afterwards, so a handful of entries from the old generation
filled the window and `pendingForSearch` returned nothing while current entries sat
in the log. The filter now runs while reading, and only a filtered listing gives up
its bounded listing - an unfiltered read still stops as soon as it has enough.
- the reset hand-off: the empty manifest a reset commits dropped `pending`, so after
a failed log cleanup the surviving stale entries had objects but no count. A fold
then retired and subtracted entries that had never been added, driving the count to
zero, where the short-circuit hides everything accepted after the reset. Reset now
carries the count forward and zeroes it only once `clearLog` succeeded, gated on the
count not having moved so it cannot eat a concurrent reservation.
Also folds in `withPending` delegating to `setPending`, so "zero deletes the key" has
one implementation: two copies of that rule is how a drain once committed `pending: 0`
and stayed v4.
The state table gains a section for this: state G is only safe while the count is above
zero, which is what both fixes serve.
Verified: npm run verify passes (30 PASS lines), including a new assertion that a
generation filter fills its window from current entries when every older key sorts
first; test-manifest-pending's ten assertions pass; docs:check passes.
`onlion/` was outside every CI job, and `ci:uncovered-tests` could not have noticed:
its universe is `git ls-files "*.test.ts"` while the service suites are
`onlion/scripts/test-*.mjs`. Both halves of that gap are now closed.
- a `verify:onlion` root script and an `onlion` job that installs both dependency
trees (the service build needs a root-level tool) and runs the service contract;
- the coverage gate resolves a `cd <dir> &&` segment into that directory's manifest
and treats `node <file>` as reaching that file, so the seven service suites count
as reached (315 files reached, up from ~307) instead of being invisible.
Also folds in the fold path's last counter fix: the counter is no longer decremented
by what a batch retired. `mutateUser` re-runs the callback with the same plan after a
CAS conflict, and by then a rival fold has usually retired those names, so the
subtraction happened twice and drove the count below the log - the direction that
hides accepted data. The count is reconciled from the log after the delete instead:
derived, never accumulated. A test now asserts the counter equals the number of log
entries after a burst, which subtraction cannot satisfy once a retry happens.
Known: `npm run verify:static` currently reports four complexity violations - two in
this change's own files (`worker.mjs` mutateUser 18, the research probe's
evaluateQuestion 20) and two in another agent's in-flight `dsh/dsh-nmg` work. The
service job added here does not run that gate; the root static job does, and it is red
until those are reduced.
Four review rounds each produced a real defect and each was treated as blocking, which left the loop with no termination: a reviewer cannot know what counts as finished unless the owner writes it down. This page writes it down. Defects are class A (blocking: it can hide, duplicate or lose accepted data, make a 200 claim something untrue, or make rollback unsafe), class B (recorded follow-up: performance, observability, wording) or class C (decided out of scope, enumerated so nobody re-proposes it). Slice 1 is done when the service job is green, verify:static is green, the state table covers every transition, the last round found no class A, and the default is still sync. Slice 2 adds a measured 16-way burst where log p95 is below sync p95, plus an executable rollback step. Two consecutive rounds that find only B or C finish a slice; a finding that claims to block must name the class-A property it breaks. docs:check passes (781 files).
…ation `findLogEntry` gained a generation filter, but the accept path still called it without one. After a reset, an entry written under the previous generation has the same name, so the lookup matched it and the accept replayed its stored response: the caller got 200 for a request whose data was never recorded. That is the class-A failure the stopping rules call "accepted data lost", and it is the same shadowing that was just fixed on the read path - the lookup had been overlooked because it is the one caller that is not a listing. The call now passes the manifest's generation. `append-log.mjs` also lists keys for this lookup the same way `listPending` does, so a foreign entry cannot stop the search early. Verified: onlion verify passes (30 PASS), including the append-log generation assertions and the log-mode integration test.
`npm run verify:static` failed on the complexity gate for both files this work touched,
which blocks slice 1 by its own stopping rules.
- `mutateUser` reads as a retry loop again: `readAttempt` owns the manifest read and
the generation guard that turns a reset during a retry into a 409, `commitAttempt`
owns one operation plus its CAS, and the loop is left with the retry, the backoff
and the cleanup. The gate had been naming the search closure (`mutateUser#2`) at 18.
- the search closure's pending fusion moved into `fusePendingRecords`, which is also
the only place that spends read-path embedding calls on log entries.
- the probe splits into `prepareVectors`, `scoreCondition` and `evaluateQuestion`, so
the per-condition loop reads as a list of arms instead of bookkeeping.
Behaviour is unchanged: `npm run verify` passes (30 PASS), and the probe re-run with
`--arms h` at 6,000 characters reproduces the recorded numbers exactly (long 11.74,
mixed 7.83, tail 5.69, mid 4.98 - the adaptive configuration).
The gate still fails on two `dsh/dsh-nmg` items (17->18 and 18) that belong to another
agent's uncommitted work; those are not touched here.
commitAttempt used `response: null` to mean "another writer committed first". `null` is a legal JSON payload, so any operation that legitimately produced it would have been retried until the attempt budget expired and then answered 503 - and no test would have noticed, because every operation in the service happens to return an object today. The outcome now carries an explicit `retry` flag. Read-only outcomes are `committed: false` and are still endings, so the loop keys on `retry` alone, and `committed ||= outcome.committed` stays inside the ending branch: a lost CAS must not touch `committed`, or the finally-cleanup runs for a write a rival already committed. Verified: npm run verify passes (30 PASS).
The stopping rules for the append-log write path put one number behind the default: a 16-way concurrent Add burst must show a lower p95 with `NMG_ADD_MODE=log` than with `sync`. This is the tool that produces both numbers on the same endpoint. It times every request, reports p50/p95/max plus the status mix, and separates failed *connections* from server failures: on a bad minute this host cannot establish TLS, and folding those into one "failures" count would let the percentiles describe the client while looking like an answer about the service. If more than half the requests never connected, it says so explicitly instead of printing a number. The health check retries three times before refusing, for the same reason: one stall is not an outage, but a benchmark against a dead endpoint must not report zeros. First clean run against the live `sync` service, 16 concurrent Adds of 8 messages: 14 x 200, 2 x 503, **no connection failures**, p50 31.7 s, p95 50.9 s. Two runs on a bad minute produced 5-8 client connect timeouts each and are discarded, not reported.
…ld cost
Three measurement defects in the burst tool, found before it produced a number anyone
would decide on:
- every round used a fresh user, so pending peaked at `concurrency` (16), below
`FOLD_TRIGGER` (24): the benchmark never folded, and the design's "a 16-way burst
needs two folds" was never exercised. One user per run lets pending accumulate, and
stops every round paying the first-Add cold path again.
- `folded_total` does not exist in the service, so folds are now derived from the
pending counts sampled around each round and labelled `derived`. A `sync` run
reports none at all, because an accepted Add is never a pending entry there - the
first version happily invented two folds per round in sync mode.
- `--mode fold` warms pending to one below the trigger and times the single Add that
must pay both an accept and a fold, which cannot be identified inside a burst.
The warmup Add is drained before the first measured round, so "drained" really starts at
zero, and a declared `--add-mode sync` warns when pending moves anyway (wrong deployment
or wrong declaration).
Live `sync` on the deployed build (236e0a6), 16 concurrent Adds of 8 messages, two
runs: 64/64 responses 200, zero client and zero server failures, p50 13.5-27.4 s,
p95 21.8-43.5 s. The spread is the contention itself; both runs are reported rather than
the kinder one.
…g the maximum a p95
Two defects in the tool that had already been committed, both of which made a number
unusable for the decision it exists to support:
- the headline percentiles were computed from each round's p50 and p95, so the reported
"p95" was the maximum of those four numbers - not a percentile of anything. Nothing
could be recomputed afterwards either, because `--json` stored only round summaries.
Each round now keeps its per-request samples and the pooled figures come from all of
them, with `n` reported next to them. Every p95 quoted before this commit is void as a
p95; the printed per-round values were per-round percentiles of 16 samples each.
- the summary carried `rounds` twice, as the requested count and as the collected
array. The count is now `roundCount`.
Also: `deriveFolds` answers zero for a declared `sync` run instead of null. In sync mode
folding cannot happen, so "unknown" was the wrong shape - null is now reserved for a log
run whose capacity read failed, where the answer genuinely is unknown.
The two runs taken to replace the void numbers were unusable and are not reported: 16 of
32 requests never established a connection, which the tool states explicitly rather than
letting the percentiles describe this host.
`listPending` grew two filters - generation and a receipt lookup - and the contract that matters is not "which filter" but "when a filter runs": both run while reading, so the caller's limit counts entries it can actually use, and either one widens the scan to `max(MAX_PENDING_READ, SCAN_FACTOR * limit)` rather than paging through a whole prefix. The subtle half is the second filter: the read path passes a receipt lookup because an entry that is folded but not yet deleted is invisible to a search anyway, and skipping it costs a name inspection instead of a body read. The fold itself must pass neither filter - it needs the log's truth, including entries whose receipt already exists, which is how a crashed delete recovers. An object inside the log prefix whose name is not an entry name now raises `log_entry_name` instead of being skipped: silently skipping would turn a naming change into "Add answers 200 but nothing is searchable", which is the class-A failure this whole area keeps rediscovering.
With fewer than twenty samples the 0.95 quantile *is* the maximum, so a 16-sample round labelled "p95" invites a decision on a slowest case. Per-round lines now read `p95(=max)` and only the pooled figure - concurrency x rounds, 48 by default - is quoted as a p95; the stopping rule quotes the pooled value for exactly that reason. `--mode fold` has one sample, so it reports that measurement as a single-sample upper bound instead of a percentile it cannot compute. Rounds also carry `crossedTrigger` (`pendingBefore + accepted >= FOLD_TRIGGER`), which is what groups a round into "contains the request that had to fold". It is `null` when the pending counter could not be read, because unknown is not false - the same rule the counter itself follows. The log-mode fixture storage is now per-process with retries: on Windows a previous run's Miniflare can still hold a handle on the shared directory and the cleanup fails with EPERM before a single test runs. That is a harness flake, not a service defect, but it turned `npm run verify` red twice. Verified: npm run verify passes (31 PASS).
…quote style `/health` carried `add_mode: "synchronous"` at the top level, hard-coded, while the real value lives under `limits.add_mode`. Nothing in the repository reads the top-level field, and the design document points callers at `limits`, so the field was a stale string that would say "synchronous" even with the log path on. Removed rather than turned into an alias: two fields with the same name and different truth is exactly how a reader ends up believing the wrong one. The log prefix now also documents that it is reserved for entries named by `logEntryName`: every reader validates that name, so a sidecar file dropped inside the prefix would fail accept, search and fold at once. Anything extra belongs beside it. `test-append-log`'s import assertion matched a single quote style, which is not what it is protecting - it exists to prove there is no dependency on worker.mjs, storage.mjs or SQLite. It accepts either quoting now, so formatting a file can no longer turn a dependency check red. Verified: npm run verify passes (31 PASS).
A log-mode burst mixes two costs: rounds that never crossed the fold trigger pay accept only, and the one round that crosses pays a fold on one of its requests. Pooling them answers "what does a burst cost", which is not the question the default switch asks. Rounds now report `crossedTrigger` (`pendingBefore + accepted >= FOLD_TRIGGER`) and the summary adds `pooledClean` over the rounds where it is `false` - the pure-accept percentile, with its `n` and its round count. A round whose pending counter could not be read is `null`, and `null` is excluded from clean rather than counted as clean: one tmed out capacity read would otherwise fold a fold sample into the very number the gate quotes. The headline prints both, and says so when no round avoided the trigger.
Three things stood between this tool and a number the gate could quote:
- the run declared `--add-mode` and nothing checked it against the deployment. Declaring
`log` against a `sync` service yields `folds 0`, which reads exactly like a legitimate
log measurement where nothing folded - the symptom of a forgotten `NMG_ADD_MODE`. The
health response is now kept: `limits.add_mode`, `source_revision` and `upstream_commit`
go into the summary, and a mismatch refuses to run with exit code 2, the same code an
unreachable endpoint gets, because both mean the setup is wrong. `deriveFolds` uses the
server's word, not the flag.
- only the warmup was drained, so with `--rounds 3` just the first round sat below the
fold trigger: the "clean" group collapsed to one round, n=16, which is a maximum and
not a p95 - the gate would have had nothing to quote. Each round now drains first when
`--owner` is given.
- the two groups are printed instead of being left to the reader: `pooled.all` and
`pooled.clean` each carry `n`, `p50`, `p95` and the `p95IsMax` label, and the headline
shows both next to the add mode and commit.
Self-tested against the live service, which runs `sync`: declaring `--add-mode log` refuses
with "deployment runs add_mode=sync but this run declares log", exit 2.
`NMG_ADD_MODE` defaulted to `sync` while the log accept path waited for a measurement.
Measured, 16 concurrent Adds of 8 messages to one user, same client and endpoint:
sync (1 clean round) 16 x 200 p50 21.8 s max 30.7 s failures 0
log (4 clean rounds) 16 x 200 p50 3.8-4.3 s max 4.9-7.3 s failures 0
Five times cheaper at the median, six at the worst case, nothing regressed. The bar was
"no regression", not an absolute target, because the base cost here is the connection and
the embedding call, which both paths share.
Flipping the default exposed five places that had encoded the synchronous path's timing.
Each was a real defect in the log path, not a test to be relaxed:
- a deployment with no vector channel returned nothing for a pending Add, because the
fusion skipped itself without a client: accepted data was unsearchable. It now scores
pending entries by term overlap, so an unmatched query still answers with an empty page
instead of every pending entry at score zero.
- pending records carried no `event_time`/`created_at`, so a page changed shape the moment
a fold ran. They now mirror the record they will become, from the message's own time.
- pending records were not attributed to any clause, and the per-clause counts stopped
summing to the page size - the multi-clause contract.
- an append-only clause's pending matches were pushed into the page instead of the
supplement.
- `/add/status/<task_id>` answered 404 for an accepted entry that had not been folded.
It now answers from the log, scoped to the current generation like the replay lookup.
`worker-fixture.mjs` pins `NMG_ADD_MODE=sync` because the API suites assert partition-write
timing, which is a property of the synchronous path - still supported as the escape hatch.
That the *code default* is `log` is pinned in test-add-log-mode with a source assertion, so
production cannot quietly slide back.
The EdgeOne smoke also asserted `sqlite_bytes > 0` after an Add, again a sync-path property;
it now accepts a pending entry.
Not established: the fold cost. Pending peaked at 17, so no request here paid for folding.
Half the runs in the same hour could not connect at all in *both* modes, recorded in the
decision as part of the result rather than discarded.
Verified: npm run verify passes (31 PASS), docs:check passes, deployed as `dp2wm4pmv86c`
with the EdgeOne smoke 9/9.
The new `onlion` CI job failed on its first run, which is what it is for: the fixture
resolved Miniflare through `wrangler`, whose own range tracks a newer Miniflare (the
lockfile has 5.20261023.0-beta) whose constructor rejects this fixture's options
(`r2Persist`, `workers[0].name`). A local `node_modules` still holding Miniflare 4 made
the suite green here and red in CI - a transitive dependency is not a pin.
- `miniflare` is now an explicit devDependency at 4.20260515.0, and the fixture resolves
it from this package instead of through wrangler. `npm ci` installs that version, so
the two environments agree.
- `test-append-log.mjs` had an unused loop binding; `test-add-log-mode.mjs` printed
without the `no-console` allowance its sibling scripts declare.
- the benchmark's `main` exceeded the complexity cap, so it is split into option parsing,
endpoint wait, the two modes and the report. Behaviour unchanged: a live run after the
split returned 8/8 200 with the same fields.
Verified after a clean `npm ci`: npm run verify passes (31 PASS), eslint on onlion/scripts
is clean.
…ing its fixtures
Two failures on the PR, both mine:
- `tools/ci-uncovered-tests.ts` read `onlion/package.json` unconditionally, but the gate
is also run against synthetic repositories in its own tests, which carry a root manifest
and nothing else. A missing manifest now contributes no scripts. A missing *subpackage*
script is likewise skipped rather than fatal: the job that runs that package would fail
at `npm run <name>` itself, so this gate does not duplicate the check - while a missing
root entry point still throws, which its own test pins.
- the gate's universe only listed `*.test.ts`, so the seven service suites were never
judged at all. It now lists `onlion/scripts/test-*.mjs` too, which is what makes the
"every suite is run by a job" claim true for the service.
The new design and decision pages claimed vocabulary in `references:` that the shared
glossary does not register ("pending counter", "state table", "fold batch", ...). Those
claims are dropped rather than invented into the corpus; the pages define what they mean in
prose.
Verified: the gate's eight tests pass, `npm run ci:uncovered-tests` and
`npm run glossary:check` are clean, docs:check and format:check pass.
… not Making the coverage gate judge `onlion/scripts/test-*.mjs` immediately showed six service suites that no CI job ran. Two of them (`page-delta`, `extractive-summaries`) pass as they are, so they join `verify` - the summary suite needs `--experimental-transform-types` because it imports the service module that uses TypeScript parameter properties. The other four cannot be honestly wired, and are acknowledged with a reason instead: `test-cos-bucket` and `test-edgeone-worker` need a real bucket and a deployed endpoint with credentials; `test-capacity` and `test-request-resume` assert the synchronous accept path (partition writes, a second request never blocked) and fail against the log default - they are stale suites, not broken code, and porting them is its own task. Both of the latter also resolved Miniflare through `wrangler`, so they hit the same constructor mismatch the CI job caught: a transitive dependency is not a pin. Verified: `npm run ci:uncovered-tests` is clean (317 files reached, 29 unreached, 14 on demand, none unacknowledged) and `npm run verify` in onlion passes with 33 PASS.
`git add docs/design/*.terms.yaml docs/decisions/implemented/2026-10-28-*.terms.yaml` matched other work in the tree: seven design term files with unrelated edits, and a tessera decision's `terms.yaml` whose `.md` siblings are still untracked. Only the first would have shipped someone else's changes; the second also fails `docs:check`, because a document group is missing its files. Both are removed from the branch and left in the working tree for their owners. Verified: docs:check passes, and the branch now carries only files this work owns.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
v1.3, the version deployed at
https://api.nmgmemory.dpdns.org(EdgeOnenmg-memory-api-overseas,ap-hongkong). The tag will be placed on this merge commitafter the deployment is verified, so
v1.3means "what is running", not "what wasproposed".
Three arcs, in the order they were measured:
equal disjoint segments when the budget covers the text, windows pinned at both ends
with evenly spread gaps when it does not. The clause itself stays the first feature
because the model's own token window gives the opening more context than a character
window; the redundant first planned window is dropped. Measured on 281 LoCoMo multi-hop
questions: pooled 6,000 characters 54.44 -> 56.57, 36,000 characters 50.89 -> 53.74,
with the middle-of-text condition going from "identical to no splitting at all" to
4.63% (8W/0L, p=0.0078).
Addno longer rewrites the whole partition per request: it writes oneimmutable log entry, folds up to eight entries per partition write, and a search sees
entries that have not been folded. The pending counter has a written state table and
six invariants, and
/admin/foldanswers 409 until a log is drained so a rollbackcannot pass on a half-drained store.
NMG_ADD_MODEnow defaults tolog, withsyncas the escape hatch.Measured, 16 concurrent Adds of 8 messages:
syncp50 21.8 s / max 30.7 s versuslogp50 3.8-4.3 s / max 4.9-7.3 s, no failures in either mode.Verification
npm run verify(onlion): 31 PASS, including the log-mode integration test that pins"accepted is searchable before folding", byte-identical replay, the fold, the 409 drain
and the reset.
npm run verify:static(root): clean for everything this PR touches; the twodsh/dsh-nmgcomplexity violations come from uncommitted work outside this branch.onlionCI job runs the service contract;ci:uncovered-testsnow sees the sevenservice suites instead of missing them (315 files reached).
dpmhlqipu4fx, EdgeOne smoke 9/9 twice, and a live check:/add200 ->/add/status200 withpending: true->/searchhit ->/reset200 ->/searchempty.What is not established
trigger of 24, so no request there paid for folding.
+4.08overall andthe
-7.14on I1 are not evidence at that sample size, and the run spans severalchanges.
Deliberately not included
The pending read window currently takes the oldest entries, so a streaming update falls
outside it once more than eight are waiting. That is a directional error - the fold wants
the oldest, a search wants the newest - and it is fixed in the next version rather than
added here, so v1.3 stays exactly what was measured.