From ab98aa490968b7855f8d5e4cd0e31879560c2da9 Mon Sep 17 00:00:00 2001 From: wefio <48851810+wefio@users.noreply.github.com> Date: Thu, 29 Oct 2026 14:09:55 +0800 Subject: [PATCH 1/2] fix(onlion): bound the embedding path by the invocation, not the provider A scored run was killed twice on 2026-10-29. The platform log holds 220 invocations that all ran exactly 120000 ms - the EdgeOne ceiling, already configured as maxDuration 120 - at 762-1014 MB resident. Nothing about them was slow; they were killed. The account balance explains only the second phase: the 21 provider 400s are all inside a 52-second window starting 41 seconds after the last timeout. The 220 hangs happened with credit available. Root cause is a multiplication, not a dependency: - onlion forced a 90 s per-attempt embedding timeout (NMG_EMBED_TIMEOUT_MS ?? "90000"), overriding the client's own 10 s default. - The client retries a timeout, so one logical call cost 90 + 0.15 + 90 s before it failed - already past the ceiling. - The pending-record fusion made one call per feature *and* per pending record: up to forty sequential calls inside a single search. Forty calls that each abort at 90 s within a 120 s invocation is not a slow request, it is an impossible one, and a killed invocation leaves no diagnostics at all. The fix bounds the work instead of the attempt: - 10 s per attempt, the client's own default. - A timeout is not retried; a connection reset still is. - The fusion collects every feature and every pending record into one batch call. The vectors are identical, only the round trips change: measured as 7 calls of 1 input before and 1 call of 8 inputs after, pinned by a test. - A 25 s budget for all query-side embedding in one request. Past it a call rejects and the caller degrades to term overlap, which every call site already handles. Documents stay unbudgeted: they carry the memory itself. - Every COS round trip gets a 20 s timeout, surfaced as a retryable error. Nothing bounded them, so a stalled bucket produced the same death. - A request still running at 60 s now reports itself (trace, route, elapsed, queue), because the 120 s kill leaves nothing behind to diagnose. - /health limits reports the timeout, the budget and the slow-report threshold. Not established, and recorded as such: resident memory at 88-99% of the invocation (762-1014 MB) may be a second cause, and a killed invocation carries no stage timings to separate it from waiting. The slow-request report is what will separate them. Verified: onlion npm run verify 34 PASS; root product tests pass; eslint and prettier clean; complexity gate clean for every file touched here (the two dsh/dsh-nmg items it reports belong to another agent's uncommitted work). --- .../2026-10-29-embedding-invocation-budget.md | 130 ++++++++++++++++++ ...-29-embedding-invocation-budget.terms.yaml | 3 + ...10-29-embedding-invocation-budget.zh-CN.md | 100 ++++++++++++++ onlion/backend/cos-bucket.mjs | 38 ++++- onlion/backend/worker.mjs | 104 ++++++++++++-- onlion/scripts/test-add-log-mode.mjs | 37 ++++- onlion/scripts/test-backend.mjs | 5 + src/core/openai-embedding.ts | 13 +- tests/core/openai-embedding.test.ts | 71 ++++++++++ 9 files changed, 480 insertions(+), 21 deletions(-) create mode 100644 docs/decisions/implemented/2026-10-29-embedding-invocation-budget.md create mode 100644 docs/decisions/implemented/2026-10-29-embedding-invocation-budget.terms.yaml create mode 100644 docs/decisions/implemented/2026-10-29-embedding-invocation-budget.zh-CN.md diff --git a/docs/decisions/implemented/2026-10-29-embedding-invocation-budget.md b/docs/decisions/implemented/2026-10-29-embedding-invocation-budget.md new file mode 100644 index 00000000..100de5b2 --- /dev/null +++ b/docs/decisions/implemented/2026-10-29-embedding-invocation-budget.md @@ -0,0 +1,130 @@ +# The embedding path is bounded by the invocation, not by the provider + +[简体中文](2026-10-29-embedding-invocation-budget.zh-CN.md) · [Terms](2026-10-29-embedding-invocation-budget.terms.yaml) + +**Status:** implemented +**Approved:** explicit + +## Problem + +A scored run was killed twice. The platform log for 2026-10-29 holds **220 invocations +that all ran exactly 120000 ms** - the EdgeOne ceiling, already configured as +`cloudFunctions.maxDuration: 120` - with resident memory between 762 MB and 1014 MB +(p50 892, p95 999). Every one of them was killed, not merely slow. + +The failure had two phases with different causes, and the account balance was not one of +them: + +| window | what happened | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | +| 05:18:45 - 10:43:36 | 220 invocations hung to the ceiling with credit available (34 in the 05 hour, 17 in 08, 129 in 09, 40 in 10) | +| 10:44:17 - 10:45:36 | 21 `embedding_http_400` events across 17 traces, all inside 52 seconds: the provider started rejecting requests, and those requests failed fast | + +Arrears explains the second window only. The first window is the one that consumed 7.3 +instance-hours of stalled invocations, and it happened while the provider was _slow_, not +refusing. + +## Root cause + +`onlion` forced a per-attempt embedding timeout larger than it thought: + +```js +// onlion/backend/worker.mjs +NMG_EMBED_TIMEOUT_MS: env.NMG_EMBED_TIMEOUT_MS ?? "90000", +``` + +The provider client retries a timeout, on the theory that the request may simply have been +slow: + +```js +// src/core/openai-embedding.ts +} catch (error) { + // A timeout is retried too: the request may simply have been slow. + if (attempt >= RETRY_DELAYS_MS.length) throw error; +} +``` + +So one logical call costs `90 s + 0.15 s + 90 s` before it fails - already past the 120 s +ceiling - and the pending-record fusion made up to **forty sequential calls** in a single +search: + +```js +for (const list of features) for (const text of list) await client.embedQueries([text]); +for (const record of waiting) await client.embedQueries([record.content]); +``` + +Forty calls that each abort at 90 s within a 120 s invocation is not a slow request. It is +a request that cannot finish, and the platform answers for us by killing it. A killed +invocation leaves no diagnostics at all, which is why the incident looked like "every +request took exactly 120 s" and nothing more. + +## Decision + +Bound the embedding path by the invocation: + +1. **Per-attempt timeout 10 s** (`EMBED_TIMEOUT_MS`), the provider client's own default, + instead of the forced 90 s. +2. **A timeout is not retried.** A connection reset still is: it says nothing about the + endpoint being slow. +3. **One batch call, not one call per text.** The vectors are identical; only the number of + round trips changes. The fusion now collects every feature text and every pending + record into a single request. +4. **A 25 s total budget per request for query-side embedding** (`NMG_EMBED_QUERY_BUDGET_MS`), + shared by every call site. Past it, a call rejects, and every caller already degrades to + scoring by term overlap. Documents are excluded: they carry the memory itself, so the + per-attempt timeout and the batch size bound them instead. +5. **A 20 s timeout on every COS request**, surfaced as a retryable error. Nothing bounded + these, so a stalled bucket produced the same 120 s death. +6. **A 60 s slow-request report** (`nmg_slow_request`, with trace id, route, elapsed time + and queue state) because the 120 s kill leaves nothing behind. +7. `/health limits` reports `embed_timeout_ms`, `embed_query_budget_ms` and `slow_report_ms` + so the numbers that bound an invocation are visible instead of inferred. + +## Evidence + +| claim | how it is established | +| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | +| the platform ceiling is what killed them | 220 of 220 timeout rows report `运行时间 = 120000`; `maxDuration: 120` is already the platform maximum | +| the timeouts were not the arrears window | the 21 provider rejections are all inside 10:44:17-10:45:36, after the last timeout at 10:43:36 | +| the path issued one call per text | before the change, a search over six pending records produced **7 embedding calls with 1/1/1/1/1/1/1 inputs** | +| it now issues one call | after the change, the same search produces **1 call with 8 inputs**, and the test asserts it | +| a timeout is not retried | `tests/core/openai-embedding.test.ts`: a server slower than the budget is contacted exactly once, past both retry delays | +| resets still retry | same file: a socket destroyed on the first attempt is retried and the embedding succeeds on the second | + +## What this does and does not establish + +- **Memory pressure is not ruled out.** 220 invocations died between 762 MB and 1014 MB, and + if the instance limit is 1 GiB that is 88-99% occupancy. A killed invocation reports no + stage timings, so the data cannot separate "waiting on a slow provider" from "GC thrash at + the ceiling". The 60 s report is what will separate them next time. +- **The provider's own behaviour is unverified.** A 400 was classified by status only. The + response body is already in the exception message, and `NMG_DEBUG_ERRORS=1` records it; + without that flag the log cannot distinguish "insufficient balance" from a malformed batch. +- **The 90 s default was not chosen deliberately anywhere.** No record explains it; it is + treated here as a defect rather than a decision being reversed. + +## Alternatives considered + +- **Raise `maxDuration`.** Already at the platform maximum of 120 s. +- **Blame the OPFS initialisation noise.** `sqliteReady()` memoises its instance, so the + VFS install happens once per isolate and its failure is swallowed by the SQLite module. + The repeated log lines are concurrent isolates, not retries, and they cost no measurable + time. +- **Treat the provider rejections as the cause of the timeouts.** A non-retryable status is + returned immediately by the client, and those rejections arrive 41 seconds after the last + timeout. + +## Consequences + +- A slow provider now costs a degraded score, not a dead invocation: the search still + answers, from term overlap where a vector is missing. +- Batch fusion is also cheaper for a healthy provider: one round trip instead of up to + forty, which is the same reduction in latency and in the provider's own load. +- The write path keeps its generous behaviour. Documents are not budgeted, so a large Add + is not truncated by the query budget - it is bounded by the per-attempt timeout and the + batch size instead. +- An operator can read the three bounding numbers from `/health limits`, and a request that + survives 60 s now says so in the log while it is still alive. +- Not addressed: resident memory between 762 MB and 1014 MB per invocation. If the instance + limit is 1 GiB this is the next failure to look for, and the slow-request report is what + will show whether it is memory or waiting. diff --git a/docs/decisions/implemented/2026-10-29-embedding-invocation-budget.terms.yaml b/docs/decisions/implemented/2026-10-29-embedding-invocation-budget.terms.yaml new file mode 100644 index 00000000..c9c1a952 --- /dev/null +++ b/docs/decisions/implemented/2026-10-29-embedding-invocation-budget.terms.yaml @@ -0,0 +1,3 @@ +version: 2 +terms: [] +references: [] diff --git a/docs/decisions/implemented/2026-10-29-embedding-invocation-budget.zh-CN.md b/docs/decisions/implemented/2026-10-29-embedding-invocation-budget.zh-CN.md new file mode 100644 index 00000000..b1b1b299 --- /dev/null +++ b/docs/decisions/implemented/2026-10-29-embedding-invocation-budget.zh-CN.md @@ -0,0 +1,100 @@ +# embed 路径由调用时长约束，而不是由上游决定 + +[English](2026-10-29-embedding-invocation-budget.md) · [Terms](2026-10-29-embedding-invocation-budget.terms.yaml) + +**Status:** implemented +**Approved:** explicit + +## 问题 + +一次计分跑被中断了两次。平台日志里，2026-10-29 有 **220 次调用全部精确跑了 120000 ms** +——那就是 EdgeOne 的上限（`cloudFunctions.maxDuration: 120` 已经配好）——常驻内存 +762 MB 到 1014 MB（p50 892，p95 999）。没有一条是"慢"，全是被杀。 + +失败分两段，成因不同，**欠费只解释第二段**： + +| 时间窗 | 发生了什么 | +| ------------------- | ------------------------------------------------------------------------------------------- | +| 05:18:45 – 10:43:36 | 220 次调用挂到上限，此时账户还有额度（05 点 34 次、08 点 17 次、09 点 129 次、10 点 40 次） | +| 10:44:17 – 10:45:36 | 52 秒内 21 条 `embedding_http_400`，17 个 trace；上游开始拒绝，这些请求快速失败 | + +占掉 7.3 个实例·小时的是第一段，它发生在上游"慢"而不是"拒绝"的时候。 + +## 根因 + +`onlion` 强制的单次嵌入超时比它自己以为的更大： + +```js +// onlion/backend/worker.mjs +NMG_EMBED_TIMEOUT_MS: env.NMG_EMBED_TIMEOUT_MS ?? "90000", +``` + +而客户端会重试超时（注释原文是"请求可能只是慢"）： + +```js +// src/core/openai-embedding.ts +} catch (error) { + // A timeout is retried too: the request may simply have been slow. + if (attempt >= RETRY_DELAYS_MS.length) throw error; +} +``` + +于是一次逻辑调用要花 `90 s + 0.15 s + 90 s` 才失败——已经越过 120 s 上限——而 pending 融合在 +单次检索里最多发起 **40 次串行调用**。40 次各以 90 s 超时兜底的调用落在 120 s 的调用预算里， +不是"慢请求"，是**不可能完成的请求**，平台替我们杀掉了它。而被杀的调用不留下任何诊断， +所以事故看起来只剩"每条都是 120 s"。 + +## 决策 + +用调用时长约束 embed 路径： + +1. **单次超时 10 s**（`EMBED_TIMEOUT_MS`），即客户端自己的默认值，取代强制的 90 s。 +2. **超时不重试**；连接重置仍然重试，因为它不说明上游慢。 +3. **一次批量调用，而不是每个文本一次**：向量完全相同，改变的只是往返次数。 +4. **query 侧嵌入每请求 25 s 总预算**（`NMG_EMBED_QUERY_BUDGET_MS`），所有调用点共享。用尽后 + 调用被拒绝，而每个调用方本来就已降级为词重叠打分。文档侧不设预算：它承载记忆本身， + 由单次超时与批量大小约束。 +5. **COS 每次请求 20 s 超时**，并作为可重试错误上抛。此前没有任何东西约束它，卡住的桶同样 + 制造 120 s 死亡。 +6. **60 s 慢请求自报**（`nmg_slow_request`，带 trace、路由、耗时与队列状态），因为被杀的调用 + 什么都不留。 +7. `/health limits` 报出 `embed_timeout_ms`、`embed_query_budget_ms`、`slow_report_ms`， + 让约束调用的数字可见，而不是靠推断。 + +## 证据 + +| 论断 | 依据 | +| ------------------------ | ----------------------------------------------------------------------------------- | +| 是平台上限杀的 | 220 条全部 `运行时间 = 120000`，且 `maxDuration` 已是平台最大值 | +| 挂死不发生在欠费窗口 | 21 条上游拒绝全部落在 10:44:17–10:45:36，晚于最后一次超时 10:43:36 | +| 修复前是"每文本一次调用" | 六条 pending 记录的一次检索产生 **7 次调用，输入数 1/1/1/1/1/1/1** | +| 修复后是一次 | 同一次检索产生 **1 次调用、8 个输入**，测试断言此性质 | +| 超时不重试 | `tests/core/openai-embedding.test.ts`：比预算更慢的服务只被访问一次，已越过两段退避 | +| 重置仍重试 | 同文件：首次尝试被销毁的 socket 会重试，第二次成功 | + +## 这确立了什么、没有确立什么 + +- **内存压力没有被排除。** 220 次调用死在 762–1014 MB 之间，若实例上限是 1 GiB 就是 88–99% + 占用。被杀的调用没有阶段计时，数据无法区分"等上游"与"GC 抖动到上限"。60 s 自报就是用来 + 在下一次分开这两者的。 +- **上游自身行为未经证实。** 400 只按状态码归类。响应体本来就在异常信息里，`NMG_DEBUG_ERRORS=1` + 会记录它；不带该开关时，日志无法区分"余额不足"和"批量参数非法"。 +- **90 s 这个默认值没有任何记录解释过。** 这里把它当缺陷处理，而不是推翻某个决定。 + +## 考虑过的替代方案 + +- **调大 `maxDuration`。** 已是平台最大值 120 s。 +- **归因于 OPFS 初始化噪音。** `sqliteReady()` 有 memo，VFS 安装每个实例只发生一次，且异常被 + SQLite 模块吞掉。重复的日志行来自并发实例，不是重试，也没有可测的时间成本。 +- **把上游拒绝当作超时的原因。** 非可重试状态由客户端立即返回，而且那批拒绝比最后一次超时 + 晚 41 秒才出现。 + +## 后果 + +- 上游变慢现在只损失得分，不损失调用：缺失向量的部分用词重叠打分，检索仍然作答。 +- 批量融合对健康的上游也更便宜：一次往返取代最多四十次，延迟与上游负载同步下降。 +- 写入路径保持原有宽松度。文档不设预算，大 Add 不会被 query 预算截断，而由单次超时与 + 批量大小约束。 +- 运维可以从 `/health limits` 直接读到三个约束数字；活过 60 秒的请求会在仍存活时先自报。 +- 未处理：单次调用常驻内存 762–1014 MB。若实例上限是 1 GiB，这就是下一个要查的失败， + 而慢请求自报会告诉我们是内存还是等待。 diff --git a/onlion/backend/cos-bucket.mjs b/onlion/backend/cos-bucket.mjs index 97e156cd..e2680b8a 100644 --- a/onlion/backend/cos-bucket.mjs +++ b/onlion/backend/cos-bucket.mjs @@ -28,6 +28,14 @@ export class CosError extends Error { } const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); + +/** One COS round trip may not outlast this. + * + * Nothing bounded these requests before, so a stalled bucket hung until the platform + * killed the invocation at 120 s - the same ceiling that hid the embedding problem. A + * timeout here surfaces as a retryable error, and each caller's own retry decides what + * to do with it instead of the instance waiting out the clock. */ +const COS_TIMEOUT_MS = 20_000; const normalizeEtag = (value) => value ? String(value).replace(/^"|"$/g, "").toLowerCase() : undefined; const sameEtag = (left, right) => !!left && !!right && normalizeEtag(left) === normalizeEtag(right); @@ -55,6 +63,33 @@ function xmlTag(xml, name) { const escapeXml = (text) => String(text).replace(/&/g, "&").replace(//g, ">"); +/** One round trip, bounded. + * + * Nothing bounded these before, so a stalled bucket hung until the platform killed the + * invocation at 120 s. The timeout belongs to the transport rather than to each caller: + * it surfaces as a retryable error, and the caller's own retry decides what to do. */ +async function sendRequest(fetchImpl, timeoutMs, method, key, url, headers, body) { + try { + return await fetchImpl(url, { + method, + headers, + body, + signal: AbortSignal.timeout(timeoutMs), + }); + } catch (error) { + const timedOut = error?.name === "TimeoutError" || error?.name === "AbortError"; + const wrapped = new CosError( + 0, + timedOut ? "timeout" : "network_error", + `COS ${method} ${key} ${ + timedOut ? `timed out after ${timeoutMs} ms` : `failed: ${error?.name ?? error?.message}` + }`, + key, + ); + wrapped.retryable = true; + throw wrapped; + } +} export function createCosBucket(config) { const { bucket, @@ -65,6 +100,7 @@ export function createCosBucket(config) { fetchImpl = fetch, clock = Date.now, staleClaimMs = CLAIM_STALE_MS, + timeoutMs = COS_TIMEOUT_MS, } = config; if (!bucket || !region || !secretId || !secretKey) throw new Error("COS bucket needs bucket, region, secretId and secretKey"); @@ -86,7 +122,7 @@ export function createCosBucket(config) { delete outgoing.host; outgoing.authorization = signed.authorization; const url = `https://${endpoint}${path}${signed.query ? `?${signed.query}` : ""}`; - const response = await fetchImpl(url, { method, headers: outgoing, body }); + const response = await sendRequest(fetchImpl, timeoutMs, method, key, url, outgoing, body); if (response.status >= 500 || response.status === 429) { const error = new CosError( response.status, diff --git a/onlion/backend/worker.mjs b/onlion/backend/worker.mjs index 322320de..c7272f3a 100644 --- a/onlion/backend/worker.mjs +++ b/onlion/backend/worker.mjs @@ -330,20 +330,55 @@ async function mutateUser(env, userId, operation) { if (!committed) await removeStaged(env.MEMORY, [...reuse.values()]).catch(() => {}); } } +/** Per-attempt embedding timeout. + * + * The platform kills an invocation at 120 s and the client retries a timeout, so a + * per-attempt budget of 90 s made one slow call 90 s + backoff + 90 s: the request died + * at the platform ceiling with no answer and no diagnostics, which is what the 220 + * timeouts of the 2026-10-29 incident look like. Ten seconds is the provider client's + * own default and leaves room for a round trip to be slow ten times over. */ +const EMBED_TIMEOUT_MS = 10_000; +/** Total embedding work one search may spend, across every call site. + * + * A single search embeds its clauses, their windows, every pending record and each + * partition's query - all inside one invocation. Bounding the total rather than each + * call means a slow provider costs a degraded score instead of the whole request: + * every caller already treats a missing vector as "score this one by term overlap". */ +const EMBED_QUERY_BUDGET_MS = 25_000; function embeddingClient(env) { const client = createEmbeddingClientFromEnv({ ...env, - NMG_EMBED_TIMEOUT_MS: env.NMG_EMBED_TIMEOUT_MS ?? "90000", + NMG_EMBED_TIMEOUT_MS: env.NMG_EMBED_TIMEOUT_MS ?? String(EMBED_TIMEOUT_MS), }); if (!client) return undefined; const queries = new Map(); const provider = hash(JSON.stringify([client.indexId, client.baseUrl, client.apiKey])); + const budgetMs = + Number(env.NMG_EMBED_QUERY_BUDGET_MS) > 0 + ? Number(env.NMG_EMBED_QUERY_BUDGET_MS) + : EMBED_QUERY_BUDGET_MS; + const deadline = Date.now() + budgetMs; + // Thrown from inside `measure`'s async callback this becomes a rejection, which is what + // every call site is written against (`catch(() => [])`). A synchronous throw would + // escape that and fail the request instead of degrading it. + const spend = () => { + if (Date.now() < deadline) return; + throw Object.assign(new Error("embedding budget exhausted for this request"), { + code: "embedding_budget_exhausted", + }); + }; return { indexId: client.indexId, model: client.model, profile: client.profile, - embed: (inputs) => embeddingOperation(() => client.embed(inputs)), + embed: (inputs) => + Promise.resolve().then(() => { + spend(); + return embeddingOperation(() => client.embed(inputs)); + }), embedDocuments: deferredDocuments((inputs) => + // Documents are the write path: they carry the memory itself, so they are not + // budgeted. The per-attempt timeout and the batch size bound them instead. embeddingOperation(() => client.embedDocuments(inputs)), ), embedQueries(inputs) { @@ -354,6 +389,7 @@ function embeddingClient(env) { measure("embedding", async () => { const cached = queryVector(key); if (cached) return cached; + spend(); // Embedded exactly as given. Long-query handling lives in // `queryFeatures`, which searches the whole clause and its windows as // separate features: pooling them here would average the query's @@ -829,18 +865,27 @@ async function fusePendingRecords({ return merge(scored); } - const vectorsPerClause = []; - for (const list of features) { - const vectors = []; - for (const text of list) { - const [vector] = await client.embedQueries([text]).catch(() => []); - if (vector) vectors.push(vector); - } - vectorsPerClause.push(vectors); - } + // One batch call, not one call per feature and per record. The vectors are the same, + // and the sequential version spent up to forty round trips inside a single search - + // which is how a slow provider became a 120 s invocation the platform had to kill. + const texts = []; + const position = new Map(); + const collect = (text) => { + if (!position.has(text)) position.set(text, texts.push(text) - 1); + }; + for (const list of features) for (const text of list) collect(text); + for (const record of waiting) collect(record.content); + const vectors = await client.embedQueries(texts).catch(() => []); + const byText = new Map(); + texts.forEach((text, index) => { + if (vectors[index]) byText.set(text, vectors[index]); + }); + const vectorsPerClause = features.map((list) => + list.map((text) => byText.get(text)).filter(Boolean), + ); const scored = []; for (const record of waiting) { - const [vector] = await client.embedQueries([record.content]).catch(() => []); + const vector = byText.get(record.content); if (!vector) continue; const perClause = vectorsPerClause.map((vectors) => vectors.reduce( @@ -1067,6 +1112,29 @@ async function warmClauseVectors(client, features, parts) { await Promise.all(Array.from({ length: Math.min(WARM_CONCURRENCY, batch.length) }, warm)); } } +/** Reports a request that is still running after `NMG_SLOW_REPORT_MS`. The platform + * kills an invocation at 120 s and a killed invocation leaves nothing behind, so an + * incident like the 220 timeouts of 2026-10-29 says only "it took 120 s" - never which + * part held it. This writes one line at 60 s, which survives the kill. */ +function watchSlowRequest(env, context, started) { + const afterMs = Number(env.NMG_SLOW_REPORT_MS) > 0 ? Number(env.NMG_SLOW_REPORT_MS) : 60_000; + const timer = setTimeout(() => { + // eslint-disable-next-line no-console -- diagnostic output, like monitoring.mjs + console.error( + JSON.stringify({ + event: "nmg_slow_request", + at: new Date().toISOString(), + trace_id: context.trace_id, + route: context.route, + elapsed_ms: Date.now() - started, + add_mode: env.NMG_ADD_MODE ?? null, + queue: queueStatus(), + }), + ); + }, afterMs); + timer?.unref?.(); + return () => clearTimeout(timer); +} /** Fold one partition's persistence mode into the request totals. */ function tallyPartPersistence(persistence, saved) { if (saved.mode === "inline_page_delta") persistence.inline_delta_saves++; @@ -1390,6 +1458,15 @@ async function healthResponse(env, path) { add_mode: addMode(env), pending_read_limit: MAX_PENDING_READ, merge_batch: FOLD_BATCH, + // Reported because they bound a request: an invocation is killed at 120 s, and a + // per-attempt timeout larger than that turns a slow provider into a dead request. + embed_timeout_ms: + Number(env.NMG_EMBED_TIMEOUT_MS) > 0 ? Number(env.NMG_EMBED_TIMEOUT_MS) : EMBED_TIMEOUT_MS, + embed_query_budget_ms: + Number(env.NMG_EMBED_QUERY_BUDGET_MS) > 0 + ? Number(env.NMG_EMBED_QUERY_BUDGET_MS) + : EMBED_QUERY_BUDGET_MS, + slow_report_ms: Number(env.NMG_SLOW_REPORT_MS) > 0 ? Number(env.NMG_SLOW_REPORT_MS) : 60_000, }, upstream_commit: SOURCE_REVISION.split("+")[0], source_revision: SOURCE_REVISION, @@ -1793,6 +1870,7 @@ export default { const context = { trace_id: traceId, route, env, execution }; return withRequest(context, () => { const track = statisticsEnabled(); + const stopSlowWatch = watchSlowRequest(env, context, Date.now()); if (track) requestAccepted(); const received = track ? performance.now() : 0; let begun; @@ -1847,7 +1925,7 @@ export default { return response; }) : job; - return tracked; + return tracked.finally(stopSlowWatch); }); }, }; diff --git a/onlion/scripts/test-add-log-mode.mjs b/onlion/scripts/test-add-log-mode.mjs index 6ece54c1..13d2de46 100644 --- a/onlion/scripts/test-add-log-mode.mjs +++ b/onlion/scripts/test-add-log-mode.mjs @@ -24,11 +24,14 @@ function vector(text) { const norm = Math.hypot(...raw) || 1; return raw.map((value) => Number((value / norm).toFixed(6))); } +/** Every embedding call the worker makes, so a test can assert how the work is batched. */ +const embedCalls = []; const outboundService = async (request) => { const url = new URL(request.url); if (!url.pathname.endsWith("/embeddings")) return new Response("not found", { status: 404 }); const body = await request.json(); const inputs = Array.isArray(body.input) ? body.input : [body.input]; + embedCalls.push(inputs); return new Response( JSON.stringify({ data: inputs.map((text, index) => ({ index, embedding: vector(text) })) }), { headers: { "content-type": "application/json" } }, @@ -249,7 +252,6 @@ try { assert.equal(finished.status, 200, JSON.stringify(finished.body)); assert.equal(finished.body.remaining, 0); - // The read window must cover everything that is waiting. With the fold trigger at one // batch, a run of Adds leaves at most a batch pending, so every accepted fact stays // searchable - the failure this pins is a standing backlog larger than the window, which @@ -260,10 +262,7 @@ try { for (let index = 0; index < 20; index += 1) { const content = `marker ${index} says the vault code is ${2000 + index}`; markers.push(content); - const added = await call( - "/add", - addBody(windowUser, `win-${index}`, content), - ); + const added = await call("/add", addBody(windowUser, `win-${index}`, content)); assert.equal(added.status, 200, JSON.stringify(added.body)); } const windowCapacity = await call(`/capacity?user_id=${windowUser}`); @@ -284,6 +283,34 @@ try { } } + // One search embeds every pending record and every query feature in a single call. + // The sequential version spent a round trip per text - thirty or more inside one + // search - which is how a slow provider ran into the invocation ceiling instead of + // costing a score. + { + const batchUser = "batch-" + process.pid; + for (let index = 0; index < 6; index += 1) { + const content = `batch marker ${index} records the tidepier${index} gauge reading`; + const added = await call("/add", addBody(batchUser, `batch-${index}`, content)); + assert.equal(added.status, 200, JSON.stringify(added.body)); + } + embedCalls.length = 0; + const found = await call("/search", { + user_id: batchUser, + query: "tide table pier", + top_k: 20, + }); + assert.equal(found.status, 200, JSON.stringify(found.body)); + // The record text is rendered (a date and role prefix may be added), so the marker + // token is what identifies it rather than the raw message content. + assert.ok( + embedCalls.some((inputs) => inputs.filter((text) => /tidepier\d/.test(text)).length >= 6), + `pending records must be embedded in one batch; saw ${ + embedCalls.length + } calls with ${embedCalls.map((inputs) => inputs.length).join("/")} inputs`, + ); + } + // Reset drains the log as well as the partitions. const reset = await call("/reset", { user_id: "alice", diff --git a/onlion/scripts/test-backend.mjs b/onlion/scripts/test-backend.mjs index bc8fb317..9095120f 100644 --- a/onlion/scripts/test-backend.mjs +++ b/onlion/scripts/test-backend.mjs @@ -897,6 +897,11 @@ print(json.dumps(r))`, add_mode: "sync", pending_read_limit: 16, merge_batch: 8, + // Advertised because they bound one invocation: a per-attempt embedding timeout at + // or above the 120 s platform ceiling turns a slow provider into a dead request. + embed_timeout_ms: 10_000, + embed_query_budget_ms: 25_000, + slow_report_ms: 60_000, }); const longQuery = await call("a", "/search", { user_id: "payload-limits", diff --git a/src/core/openai-embedding.ts b/src/core/openai-embedding.ts index 4487dc4b..4dadb7c3 100644 --- a/src/core/openai-embedding.ts +++ b/src/core/openai-embedding.ts @@ -193,7 +193,8 @@ export class OpenAIEmbeddingClient { * Transient upstream failures (connection resets between a compute region and * the model endpoint, 429, 5xx) are retried here. Without this, one dropped * socket surfaces as a failed Add; with it, the caller sees a slow-but-correct - * write. Non-retryable answers (400 invalid key/model/batch) surface at once. + * write. Non-retryable answers (400 invalid key/model/batch) surface at once, + * and so does a timeout - see the catch below. */ private async send(body: string): Promise { for (let attempt = 0; ; attempt++) { @@ -210,7 +211,15 @@ export class OpenAIEmbeddingClient { if (!RETRYABLE_STATUS(response.status) || attempt >= RETRY_DELAYS_MS.length) return response; } catch (error) { - // A timeout is retried too: the request may simply have been slow. + // A timeout is not retried. The caller is an HTTP request with an invocation + // ceiling, so retrying a slow dependency multiplies the time it takes to fail + // rather than the chance of success: at a 90 s per-attempt budget one retried + // timeout is already longer than the 120 s the invocation is killed at, which + // turns "the provider was slow" into "the request died with no answer". + // A connection reset says nothing about the endpoint being slow, so those are + // still retried. + const name = (error as { name?: string } | undefined)?.name; + if (name === "TimeoutError" || name === "AbortError") throw error; if (attempt >= RETRY_DELAYS_MS.length) throw error; } await sleep(RETRY_DELAYS_MS[attempt]); diff --git a/tests/core/openai-embedding.test.ts b/tests/core/openai-embedding.test.ts index 81b5f5d3..3bbe46a4 100644 --- a/tests/core/openai-embedding.test.ts +++ b/tests/core/openai-embedding.test.ts @@ -215,3 +215,74 @@ test("embedding requests stop at the configured timeout", async () => { ); } }); + +test("a timeout is not retried, because the caller runs inside an invocation ceiling", async () => { + let attempts = 0; + const server = createServer((_request, response) => { + attempts += 1; + // Slower than the client's budget, then answered so the socket can close. + setTimeout(() => { + if (!response.writableEnded) response.end("{}"); + }, 600).unref(); + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + try { + const address = server.address(); + assert.ok(address && typeof address === "object"); + const client = new OpenAIEmbeddingClient({ + baseUrl: `http://127.0.0.1:${address.port}/v1`, + timeoutMs: 100, + }); + await assert.rejects(() => client.embed(["slow"])); + // The retry delays are 150 ms and 500 ms: waiting past both proves none landed. + await new Promise((resolve) => setTimeout(resolve, 800)); + assert.equal( + attempts, + 1, + "a timeout must surface at once: retrying it multiplies the time the invocation spends before it dies", + ); + } finally { + server.closeAllConnections(); + await new Promise((resolve, reject) => + server.close((error) => (error ? reject(error) : resolve())), + ); + } +}); + +test("a dropped connection is still retried", async () => { + let attempts = 0; + const server = createServer((request, response) => { + attempts += 1; + if (attempts === 1) { + request.socket.destroy(); + return; + } + let body = ""; + request.setEncoding("utf8"); + request.on("data", (chunk) => (body += chunk)); + request.on("end", () => { + const payload = JSON.parse(body) as { input: string[] }; + response.setHeader("content-type", "application/json"); + response.end( + JSON.stringify({ + data: payload.input.map((_text, index) => ({ index, embedding: [index] })), + }), + ); + }); + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + try { + const address = server.address(); + assert.ok(address && typeof address === "object"); + const client = new OpenAIEmbeddingClient({ + baseUrl: `http://127.0.0.1:${address.port}/v1`, + }); + assert.deepEqual(await client.embed(["first", "second"]), [[0], [1]]); + assert.equal(attempts, 2, "a reset says nothing about the endpoint being slow"); + } finally { + server.closeAllConnections(); + await new Promise((resolve, reject) => + server.close((error) => (error ? reject(error) : resolve())), + ); + } +}); From 4cdc62d19fd6337a3db6394a237019710c0bc879 Mon Sep 17 00:00:00 2001 From: wefio <48851810+wefio@users.noreply.github.com> Date: Thu, 29 Oct 2026 14:36:27 +0800 Subject: [PATCH 2/2] docs(design): deferred document embedding An Add cannot currently finish without the provider: the fold that follows an accepted entry embeds the batch before the partition is rewritten, so a slow provider turns a write into a request that runs until the platform kills it. The timeouts and the batch call that now bound that path are a limit, not a decoupling. The evaluation shape is serial - every Add before the first Search - so the Add phase is the only phase whose failure ends a run, and one failed Add is enough. The design puts the vector behind the durable write: entry format v2 carries an optional vectors array, the accept path embeds best-effort inline under its own deadline, a bounded drain carried by later requests picks up whatever timed out, and the fold uses a stored vector when there is one. A rejection is explicitly not deferred: a run scored entirely on term overlap would be a silently wrong result rather than a visible error. Not implemented. Includes the invariants each slice must turn into a test, the failure matrix, the lazy migration rule that keeps a drained log byte-identical, and the rejected alternatives. --- docs/design/deferred-embedding-design.md | 177 ++++++++++++++++++ .../deferred-embedding-design.terms.yaml | 3 + .../design/deferred-embedding-design.zh-CN.md | 153 +++++++++++++++ 3 files changed, 333 insertions(+) create mode 100644 docs/design/deferred-embedding-design.md create mode 100644 docs/design/deferred-embedding-design.terms.yaml create mode 100644 docs/design/deferred-embedding-design.zh-CN.md diff --git a/docs/design/deferred-embedding-design.md b/docs/design/deferred-embedding-design.md new file mode 100644 index 00000000..00a7f929 --- /dev/null +++ b/docs/design/deferred-embedding-design.md @@ -0,0 +1,177 @@ +# Deferred document embedding + +[简体中文](deferred-embedding-design.zh-CN.md) · [Terms](deferred-embedding-design.terms.yaml) + +**Status:** draft + +## Problem + +An Add currently cannot finish without the embedding provider. The accept path writes the log +entry, and the fold that follows it embeds the batch before the partition is rewritten, so a +provider that is slow turns a write into a request that runs until the platform kills it. A +run of 220 invocations died exactly at the 120 s ceiling on 2026-10-29 for this reason; the +per-attempt timeout and the batch call that now bound it are a _limit_, not a decoupling - one +slow provider still costs about thirty seconds per Add and can still fail it. + +That matters more than it sounds because the evaluation shape is serial: every Add finishes +before the first Search starts. The Add phase is therefore the only phase whose failure ends +the run, and a single failed Add is enough. A search that loses a vector loses one question's +evidence; an Add that fails loses the run. + +The obvious answer - embed after responding - is not available. The deployment has no +scheduled or background trigger (`edgeone.json` carries only `maxDuration` and `regions`), and +a functions platform does not promise that work continues after a response is returned. + +## Goal + +Make the write path depend on the durable write, not on the provider: + +- an Add succeeds once its entry is committed, whatever the provider is doing; +- retrieval quality degrades per record while a vector is missing, never per run; +- a provider that _rejects_ requests still fails loudly and immediately, because a run scored + on term overlap alone would be a silently wrong result rather than a visible error; +- no new failure mode, no lost entry, no unbounded backlog. + +## Design + +### Entry format v2 + +A log entry gains an optional `vectors` array, aligned with `messages`: + +```jsonc +{ "v": 2, "generation": "...", "messages": [ { "role": "user", "content": "...", "indexText": "..." } ], + "vectors": [ null, [0.1, ...] ], // present when v === 2, null while a vector is missing + "vector_model": "::" } +``` + +`vector_model` is the identity of the vector space, the same one the partitions use. An entry +whose `vector_model` no longer matches the configured one is treated as vectorless and +re-embedded, exactly like a partition whose identity is stale - half a migration would serve +vectors that cannot be compared with the query's. + +The entry already carries everything an embedding needs (`indexText` plus the message time), +which is what makes a later embedding possible at all: the original request does not have to +be alive. + +### Accept path + +1. Commit the entry with `vectors: null` (one PUT, as today). +2. Best-effort inline embedding with its own deadline (`EMBED_INLINE_MS`, default 8 s): + one batch call for the entry's messages. +3. If it returns in time, rewrite the entry once with the vectors. +4. If it times out, leave `vectors: null` and return 200. The entry is durable; only the + vector is outstanding. +5. If the provider **rejects** (any non-retryable status, or a retryable one exhausted), fail + the Add with `embedding_unavailable`. A rejection is not slowness; it is a state the + operator has to see, and the alternative is a whole run scored without vectors. + +The healthy path costs one extra PUT compared with today and the same single batch call. + +### Drain + +Nothing runs after a response, so the backlog has to be carried by later requests. Any request +that touches a user may drain that user's backlog: + +- bounded by count (`DRAIN_ENTRIES`, default 2) and by time (`DRAIN_MS`, default 4 s); +- newest-first is wrong here: the fold wants the oldest, and a search wants the newest, so the + drain takes the **oldest vectorless** entries, which are the ones closest to being folded; +- one batch embedding call for all drained entries, then one PUT per entry; +- a drain that fails for a reason other than timeout stops for this request and leaves the + entries for the next one; it never fails the request that carried it. + +Adds drain before their own accept (they are already writing); searches drain only when their +budget allows and never after the first miss, so a phase boundary cannot turn one slow search +into several. + +### Fold + +The fold uses a stored vector when there is one and embeds the remainder in the same batch as +today. It never waits for the drain: a fold is allowed to be the thing that embeds. + +This also defines the ceiling of the backlog: the fold trigger (one batch, eight entries) plus +whatever arrived since the last fold. That is the same bound the read window already has to +cover, so a vectorless record cannot outlive the log entry that holds it. + +### Read path + +Pending records score by vector when the entry has one and by term overlap when it does not - +the two paths that already exist. No new read behaviour, no new short-circuit: an entry is +visible either way, which is the property the log was introduced for. + +## Frozen interfaces (slice 1) + +```js +// append-log.mjs +export function putLogEntry(bucket, key, entry) // existing +export async function listVectorless(bucket, key, options) // {limit, generation}: oldest first +export async function attachVectors(bucket, key, name, vectors, model) // rewrite one entry + +// worker.mjs +async function drainBacklog(env, ctx, budgetMs) // returns {drained, remaining} +``` + +`attachVectors` is the only writer of `vectors`, and it must be a rewrite of the same name so +the entry keeps its arrival position: reordering the log would change what a fold applies +first. + +## Invariants (each one becomes a test) + +| id | invariant | the test that fails without it | +| --- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------ | +| E1 | An Add whose entry committed returns 200 even when embedding times out | provider stub that never answers; the entry must be readable immediately | +| E2 | A rejecting provider fails the Add and stores nothing | stub answering 400; no entry, no receipt | +| E3 | A vectorless record is searchable by term overlap | drain disabled, search still returns the record | +| E4 | The vectorless backlog never exceeds fold trigger + one arrival window | burst of Adds with the drain disabled | +| E5 | A stale `vector_model` is re-embedded, never compared | entry written with a different model must not score by vector | +| E6 | The drain never fails the request that carries it | drain against a broken bucket, request still 200 | +| E7 | A folded record's vector is the one that was drained, not a second one | fold after drain must not call the provider | + +## Failure matrix + +| provider | bucket | Add result | retrieval | +| ------------ | ------- | --------------------------------------------------- | ---------------------------------- | +| healthy | healthy | 200, vector stored inline | full | +| slow (> 8 s) | healthy | 200, vector deferred | term overlap until the drain lands | +| rejects | healthy | 5xx `embedding_unavailable`, entry not stored | unchanged, and visible | +| healthy | slow | 200 after the write commits (COS timeout bounds it) | full | +| slow | slow | 200 deferred, or 5xx if the write itself times out | as above | +| down | healthy | 5xx, nothing stored | none for that Add, loudly | + +## Migration + +`v === 1` entries stay readable: no `vectors`, therefore vectorless, therefore drained or +folded with an inline embedding. No rewrite is needed to upgrade, and a deployment that never +drains still behaves exactly as the current one. The same lazy rule as the pending count +applies: `vectors` is absent in a drained log, so a drained log stays byte-identical to one a +previous deployment wrote. + +## Work breakdown + +| slice | content | gate | +| ----- | -------------------------------------------------------- | ---------------------------------------------------------------------- | +| 1 | entry v2 + `attachVectors` + inline deadline + E1/E2 | Add succeeds against a provider that never answers; rejects still fail | +| 2 | drain (`listVectorless`, budget) + E3/E4/E6 | backlog bounded and carried by later requests | +| 3 | fold uses stored vectors + E5/E7 | a drained entry costs no provider call at fold time | +| 4 | docs, `/health` counters (`vectorless`, `drained_total`) | operator can see the backlog | + +## Open questions + +- Should a search drain at all? It costs up to 4 s of a scored request to improve records that + the same request may then score by vector. The alternative - searches never drain - leaves + the phase-boundary backlog to term overlap. +- `EMBED_INLINE_MS = 8 s` is a guess. The right value is the observed p99 of a healthy batch + call, which nobody has measured per phase. +- Does the extra PUT per vectorised entry matter under 16-way concurrency on one user? The + manifest CAS is not involved, but the bucket sees one more write per entry. + +## Rejected alternatives + +- **Fire and forget after the response.** The platform does not promise post-response + execution; a vector that "will be written later" would silently never arrive. +- **Embed in the fold only (today).** Any Add that triggers a fold inherits the provider's + latency, which is what killed the run: the fold is on the accept path by construction. +- **Store vectors in a side object** (`vectors/`), one PUT per entry. Same write count + as a rewrite, but the entry and its vector can then disagree after a failed write, and the + fold has to union two listings. The rewrite keeps one record of truth. +- **A scheduled job.** No trigger exists in the deployment, and adding an external cron adds a + moving part whose failure mode is a permanently vectorless store. diff --git a/docs/design/deferred-embedding-design.terms.yaml b/docs/design/deferred-embedding-design.terms.yaml new file mode 100644 index 00000000..c9c1a952 --- /dev/null +++ b/docs/design/deferred-embedding-design.terms.yaml @@ -0,0 +1,3 @@ +version: 2 +terms: [] +references: [] diff --git a/docs/design/deferred-embedding-design.zh-CN.md b/docs/design/deferred-embedding-design.zh-CN.md new file mode 100644 index 00000000..2aaad471 --- /dev/null +++ b/docs/design/deferred-embedding-design.zh-CN.md @@ -0,0 +1,153 @@ +# 延迟文档嵌入 + +[English](deferred-embedding-design.md) · [Terms](deferred-embedding-design.terms.yaml) + +**Status:** draft + +## 问题 + +现在一次 Add 离不开向量服务：accept 路径写日志条目，紧随其后的折叠要把这一批嵌入完才重写分区， +所以**上游一慢，写入就变成跑到平台杀掉为止的请求**。2026-10-29 那 220 次调用正是死在 120 s 上限； +现在加的"单次超时 + 批量调用"只是给这条路径**设了上限**，没有解耦——上游慢仍然让每次 Add 多花 +约三十秒，并且仍然可以让它失败。 + +这件事比听起来重要，因为评测是串行的：所有 Add 结束后才开始第一次 Search。**Add 阶段是唯一会 +因为失败而终止整轮的阶段**，而且一次 Add 失败就够了。检索丢一个向量只丢一道题的证据；Add 失败丢 +整轮。 + +最直觉的答案——响应之后再嵌入——在这里不可用。部署里没有任何定时或后台触发（`edgeone.json` 只有 +`maxDuration` 和 `regions`），而函数平台不承诺响应返回后继续执行。 + +## 目标 + +让写入路径依赖**持久化写入**，而不是依赖上游： + +- 条目提交即算 Add 成功，上游在做什么都不影响； +- 质量按**单条记录**降级，绝不按轮次降级； +- 上游**拒绝**请求时仍然立即响亮地失败——否则整轮用词重叠打分，那是一个"看起来像检索能力问题" + 的错误结果，而不是一个能看见的错误； +- 不引入新故障态、不丢条目、积压有界。 + +## 设计 + +### 条目格式 v2 + +日志条目新增可选的 `vectors`，与 `messages` 对齐： + +```jsonc +{ "v": 2, "generation": "...", "messages": [ { "role": "user", "content": "...", "indexText": "..." } ], + "vectors": [ null, [0.1, ...] ], + "vector_model": "::" } +``` + +`vector_model` 是向量空间的标识，和分区用的是同一个。它一旦与配置不一致，条目就当作"无向量"重新 +嵌入——和分区标识过期同样处理：半个迁移会留下无法与查询向量比较的向量。 + +条目本身已经带了嵌入所需的全部输入（`indexText` 与消息时间），这正是"晚一点嵌入"可行的前提： +**原请求不需要还活着**。 + +### accept 路径 + +1. 提交条目，`vectors: null`（一次 PUT，与今天相同）。 +2. **尽力而为的即时嵌入**，独立截止时间 `EMBED_INLINE_MS`（默认 8 s）：本条消息一次批量调用。 +3. 及时返回 → 重写条目一次，写入向量。 +4. 超时 → 保持 `vectors: null`，返回 200。条目已持久化，欠的只是一个向量。 +5. 上游**拒绝**（不可重试状态，或可重试状态已用尽）→ Add 以 `embedding_unavailable` 失败。 + 拒绝不是"慢"，它是运维必须看见的状态；否则代价是整轮无向量。 + +健康路径比今天多一次 PUT，批量调用次数不变。 + +### 排空 + +响应之后没有执行机会，所以积压必须由**后续请求**带走。任何触达某个用户的请求都可以替它排空： + +- 按条数（`DRAIN_ENTRIES`，默认 2）与按时间（`DRAIN_MS`，默认 4 s）双重约束； +- **取最老的无向量条目**：折叠要最老的，检索要最新的，而排空的目标是"最接近被折叠的那批"； +- 被排空的条目合并成**一次**批量调用，然后逐条 PUT； +- 排空失败（超时以外的原因）只在本请求内停止，把条目留给下一次，**绝不让承载它的请求失败**。 + +Add 在自身 accept 前顺带排空（它本来就在写）；Search 只在预算允许时排空，并且第一次未命中之后 +不再排空，避免阶段边界把一次慢检索放大成好几次。 + +### 折叠 + +折叠优先使用已存向量，其余与今天一样在同一次批量里补齐。它**从不等待排空**：折叠本身被允许作为 +执行嵌入的那一方。 + +这也给出了积压的上界：折叠阈值（一批 8 条）加上上次折叠以来新到的部分。这正好是读窗口本来就要 +覆盖的范围，所以无向量记录不会活得比持有它的日志条目更久。 + +### 读路径 + +pending 记录有向量就按向量打分，没有就按词重叠打分——两条路径都已存在。没有新的读行为、没有新的 +短路：两种情况下条目都可见，这正是引入日志时要求的性质。 + +## 冻结接口（切片 1） + +```js +// append-log.mjs +export function putLogEntry(bucket, key, entry) // 已有 +export async function listVectorless(bucket, key, options) // {limit, generation}: 最老优先 +export async function attachVectors(bucket, key, name, vectors, model) // 重写单条 + +// worker.mjs +async function drainBacklog(env, ctx, budgetMs) // 返回 {drained, remaining} +``` + +`attachVectors` 是 `vectors` 的唯一写入者，且必须**同名声重写**，让条目保持原来的到达位置：重排日志 +会改变折叠先应用谁。 + +## 不变量（每条都要变成测试） + +| id | 不变量 | 缺了它会失败的测试 | +| --- | --------------------------------------------- | ------------------------------------- | +| E1 | 条目已提交的 Add，即使嵌入超时也返回 200 | 永不响应的上游桩；条目必须立刻可读 | +| E2 | 上游拒绝时 Add 失败且不存任何东西 | 返回 400 的桩；不得留条目、不得留回执 | +| E3 | 无向量记录仍可按词重叠被检索到 | 关闭排空后检索仍返回该记录 | +| E4 | 无向量积压不超过"折叠阈值 + 一个到达窗口" | 关闭排空后的突发写入 | +| E5 | `vector_model` 过期一律重新嵌入，绝不参与比较 | 用不同模型写入的条目不得按向量打分 | +| E6 | 排空绝不让承载它的请求失败 | 对着坏桶排空，请求仍 200 | +| E7 | 折叠用的是排空写下的那个向量，而不是再算一个 | 排空后折叠不得调用上游 | + +## 失败矩阵 + +| 上游 | 桶 | Add 结果 | 检索 | +| ----------- | ---- | ----------------------------------- | ------------------- | +| 正常 | 正常 | 200，向量即时写入 | 完整 | +| 慢（> 8 s） | 正常 | 200，向量延迟 | 排空落地前按词重叠 | +| 拒绝 | 正常 | 5xx `embedding_unavailable`，不入库 | 不变，且可见 | +| 正常 | 慢 | 写入提交后 200（COS 超时兜底） | 完整 | +| 慢 | 慢 | 200 延迟；若写入本身超时则 5xx | 同上 | +| 不可用 | 正常 | 5xx，不入库 | 该 Add 缺失，且响亮 | + +## 迁移 + +`v === 1` 的条目继续可读：没有 `vectors`，即"无向量"，会被排空或被折叠时即时嵌入。升级不需要重写； +一个从不排空的部署与今天的部署行为完全一致。与 pending 计数同一条懒规则：日志排空后 `vectors` +不存在，所以排空后的日志与旧部署写下的日志仍然**逐字节相同**。 + +## 工作量分解 + +| 切片 | 内容 | 验收 | +| ---- | ----------------------------------------------------- | ------------------------------------------- | +| 1 | 条目 v2 + `attachVectors` + 即时截止 + E1/E2 | 面对永不响应的上游 Add 仍成功；拒绝仍然失败 | +| 2 | 排空（`listVectorless`、预算）+ E3/E4/E6 | 积压有界，且被后续请求带走 | +| 3 | 折叠使用已存向量 + E5/E7 | 已排空的条目在折叠时不再调用上游 | +| 4 | 文档、`/health` 计数（`vectorless`、`drained_total`） | 运维可见积压 | + +## 未决问题 + +- **检索要不要排空？** 花掉一次计分检索最多 4 s 去改善"这次检索自己会用向量打分"的记录。另一种 + 选择是检索永不排空，把阶段边界的积压留给词重叠。 +- `EMBED_INLINE_MS = 8 s` 是猜的。正确值应该是健康批量调用的 p99，而这个数没有按阶段测过。 +- 每条被写入向量的条目多一次 PUT，在单用户 16 并发下是否明显？manifest 的 CAS 不参与，但桶会多看到 + 一次写。 + +## 被否决的替代方案 + +- **响应后 fire and forget。** 平台不承诺响应之后继续执行；"以后会写"的向量等于永远不来。 +- **只在折叠里嵌入（今天）。** 任何触发折叠的 Add 都继承上游延迟，而折叠就在 accept 路径上，这正是 + 那次整轮失败的原因。 +- **向量存旁路对象**（`vectors/`），每条一次 PUT。写次数与重写相同，但写失败后条目与向量会 + 互相矛盾，折叠还要合并两个列表。重写保持单一事实来源。 +- **定时任务。** 部署里没有触发器；引入外部 cron 会多一个移动部件，而它的故障态是"永久无向量"。