Repository navigation
fix(onlion): give the write path its own embedding timeout and a total budget - #125
Merged
Merged
Conversation
…l budget The deployed logs show Adds taking 5-20 s with successful embedding, and 55-120 s when the provider stalls, ending in `dependency_timeout` at `stage: embedding`. The function runs in ap-hongkong and the provider is elsewhere, so every call crosses a region boundary: two hours earlier the same code accepted an Add in 0.4-0.6 s, so the variable is the path, not the code. Two changes follow from that, plus the measurement that was missing: - documents get a 25 s per-attempt timeout instead of the 10 s used for queries. Ten seconds is generous for a query and too tight for a document batch that carries a whole message across a boundary, where a healthy round trip and a stalled one differ by an order of magnitude. A cut document is a failed Add, which is the one failure the write path cannot absorb. - the write path shares a 45 s total budget (`NMG_EMBED_WRITE_BUDGET_MS`). Without it one slow provider turned a single Add into a 55-120 s request the platform killed with no answer; past the budget the Add fails with `embedding_budget_exhausted` and the caller can retry the same request_id. - `embeddingOperation` records the duration in `lastEmbedding`, and the slow-request report carries `last_embedding`. Two incidents were diagnosed without ever knowing how long the cross-region call took. Verified: onlion npm run verify 35 PASS; eslint clean; /health limits reports embed_document_timeout_ms and embed_write_budget_ms.
…arates endpoints Two changes with one cause: the provider is across a region boundary. **A search embedded one feature at a time.** `warmClauseVectors` called `embedQueries([text])` per feature and the retrieval engine does the same, so a five-clause query with windows cost up to twenty-four round trips before the first partition was opened. The query cache was keyed by the input array, which is why warming had to match the engine's shape exactly: keying by text instead lets the warm pass fetch every distinct feature in one call and the engine's per-feature lookups hit it. Measured on the deployed build the same search now makes two calls (one for features, one for pending records) instead of one per feature; the test asserts the call count, not just that a batch happened. **The index identity ignored the endpoint.** Two deployments of `text-embedding-v4` serve the same model name at different endpoints; a vector from one is not comparable with a query embedded by the other, and nothing in the identity - model prefix, dimensions, templates, profile - differed. Switching endpoints therefore left every stored vector in place and mixed two spaces silently. The endpoint now participates (scheme, host, path, normalised so a trailing slash does not change it), which makes a switch mark the index stale so the existing backfill re-embeds. Consequence to expect on deploy: every stored vector becomes stale by construction and is re-embedded on the next write or search for that user, one batch of 32 at a time. Verified: onlion npm run verify 35 PASS; root product tests pass; prettier and eslint clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What the deployed logs establish
Memory is settled, and it is not the problem. From the trace on
6e7421a8:wasm_linear_mbhigh_water_databases/high_water_image_mbheapUsedper requestbaseline_rssacross instancesuptime_s/seqseq: 1withuptime_s: 2600means every invocation is a fresh isolate inside a container thathas been alive 43 minutes. So the 577-710 MB is the container's rss, already there before our
code runs, and our request adds about 10-30 MB. No allocation change inside a request can move a
600 MB container figure - the WASM hypothesis is dead, and so is the "images outliving their
part" one.
The write path is what stalls.
/addshows 5.8 / 7.5 / 9.5 / 17.5 / 19.9 / 120000 ms, withdependency_timeoutatstage: embeddingand a 60 s report whoselive_databases: 0says therequest is not in SQLite - it is waiting on the provider.
The function is in ap-hongkong and the provider is in North China. Two hours earlier the same
code accepted an Add in 0.4-0.6 s, so the variable is that path.
Changes
NMG_EMBED_WRITE_BUDGET_MS)embeddingOperationrecordsms, slow report carrieslast_embedding/health limitsreportsembed_document_timeout_msandembed_write_budget_ms.Verification
onlion npm run verify: 35 PASS. eslint clean. Complexity gate clean for every file here (thetwo
dsh/dsh-nmgitems belong to another agent's uncommitted work).Not addressed here
The root cause is the topology: every Add and every search crosses a region boundary to the
provider, and up to 24 calls per search. Moving the endpoint next to the function - or running it
in the same region - is a configuration change worth more than any timeout, and the new
msfield is what will price it.