Repository navigation
fix(onlion): bound the embedding path by the invocation, not the provider - #122
Merged
Merged
Conversation
…ider A scored run was killed twice on 2026-10-29. The platform log holds 220 invocations that all ran exactly 120000 ms - the EdgeOne ceiling, already configured as maxDuration 120 - at 762-1014 MB resident. Nothing about them was slow; they were killed. The account balance explains only the second phase: the 21 provider 400s are all inside a 52-second window starting 41 seconds after the last timeout. The 220 hangs happened with credit available. Root cause is a multiplication, not a dependency: - onlion forced a 90 s per-attempt embedding timeout (NMG_EMBED_TIMEOUT_MS ?? "90000"), overriding the client's own 10 s default. - The client retries a timeout, so one logical call cost 90 + 0.15 + 90 s before it failed - already past the ceiling. - The pending-record fusion made one call per feature *and* per pending record: up to forty sequential calls inside a single search. Forty calls that each abort at 90 s within a 120 s invocation is not a slow request, it is an impossible one, and a killed invocation leaves no diagnostics at all. The fix bounds the work instead of the attempt: - 10 s per attempt, the client's own default. - A timeout is not retried; a connection reset still is. - The fusion collects every feature and every pending record into one batch call. The vectors are identical, only the round trips change: measured as 7 calls of 1 input before and 1 call of 8 inputs after, pinned by a test. - A 25 s budget for all query-side embedding in one request. Past it a call rejects and the caller degrades to term overlap, which every call site already handles. Documents stay unbudgeted: they carry the memory itself. - Every COS round trip gets a 20 s timeout, surfaced as a retryable error. Nothing bounded them, so a stalled bucket produced the same death. - A request still running at 60 s now reports itself (trace, route, elapsed, queue), because the 120 s kill leaves nothing behind to diagnose. - /health limits reports the timeout, the budget and the slow-report threshold. Not established, and recorded as such: resident memory at 88-99% of the invocation (762-1014 MB) may be a second cause, and a killed invocation carries no stage timings to separate it from waiting. The slow-request report is what will separate them. Verified: onlion npm run verify 34 PASS; root product tests pass; eslint and prettier clean; complexity gate clean for every file touched here (the two dsh/dsh-nmg items it reports belong to another agent's uncommitted work).
An Add cannot currently finish without the provider: the fold that follows an accepted entry embeds the batch before the partition is rewritten, so a slow provider turns a write into a request that runs until the platform kills it. The timeouts and the batch call that now bound that path are a limit, not a decoupling. The evaluation shape is serial - every Add before the first Search - so the Add phase is the only phase whose failure ends a run, and one failed Add is enough. The design puts the vector behind the durable write: entry format v2 carries an optional vectors array, the accept path embeds best-effort inline under its own deadline, a bounded drain carried by later requests picks up whatever timed out, and the fold uses a stored vector when there is one. A rejection is explicitly not deferred: a run scored entirely on term overlap would be a silently wrong result rather than a visible error. Not implemented. Includes the invariants each slice must turn into a test, the failure matrix, the lazy migration rule that keeps a drained log byte-identical, and the rejected alternatives.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
A scored run was killed twice on 2026-10-29. The platform log holds 220 invocations that all
ran exactly 120000 ms - the EdgeOne ceiling, already configured as
maxDuration: 120- at762-1014 MB resident. Nothing about them was slow; they were killed.
The account balance explains only the second phase: the 21 provider 400s are all inside a
52-second window starting 41 seconds after the last timeout. The 220 hangs happened with
credit available.
Root cause
A multiplication, not a dependency:
onlionforced a 90 s per-attempt embedding timeout (NMG_EMBED_TIMEOUT_MS ?? "90000"),overriding the provider client's own 10 s default.
90 + 0.15 + 90 sbefore failing -already past the ceiling.
sequential calls inside a single search.
Forty calls that each abort at 90 s within a 120 s invocation is not a slow request, it is an
impossible one, and a killed invocation leaves no diagnostics at all.
The fix
/health limitsMeasured: the same search produced 7 calls with 1 input each before and 1 call with 8
inputs after; a test pins it.
Verification
onlion npm run verify: 34 PASS, including the new batch assertion and the/healthlimitsexpectation.
tests/core/openai-embedding.test.ts: +2 tests - a timeout is contacted exactly once, and adestroyed socket is still retried.
docs:checkand the complexity gate are clean forevery file touched here.
Also included
docs/design/deferred-embedding-design.*- the design that would take the provider off thewrite path entirely (entry format v2, inline deadline, bounded drain, fold fallback), with the
invariants each slice must turn into a test. Design only, not implemented.
Not established
stage timings, so the data cannot separate "waiting on the provider" from "GC thrash at the
ceiling". The 60 s report is what will separate them.
message;
NMG_DEBUG_ERRORS=1records it.