perf(onlion): no request may be killed, and an accept takes no compare-and-swap

docs/decisions/proposed/2026-10-29-recorded-graveyard-for-cleanup.md

@@ -0,0 +1,158 @@
+# A recorded graveyard for cleanup the commit could not finish
+
+[简体中文](2026-10-29-recorded-graveyard-for-cleanup.zh-CN.md) · [Terms](2026-10-29-recorded-graveyard-for-cleanup.terms.yaml)
+
+**Status:** proposed
+
+## Problem
+
+Cleanup of superseded partition snapshots happens **after** the manifest compare-and-swap
+succeeds, because a key may only be deleted once the manifest that stopped referencing it is
+durable. That leaves three ways to leave an orphan with no name:
+
+- the function is killed after the commit and before the delete (the platform ceiling, a
+  restart);
+- a delete fails and the caller's `.catch(() => {})` swallows it;
+- the delete is now batched at the provider's 1000-key limit, so a later batch can fail after
+  earlier ones succeeded - a **partial** cleanup that reports nothing and names nothing.
+
+None of these is visible from the manifest, which is the only thing a later request reads. The
+invariant that matters - every key under `parts/` is referenced by the manifest - holds only
+during a quiet period with no crash and with every delete batch succeeding, which is exactly the
+condition an incident violates.
+
+## Proposal
+
+Record the doomed keys in the manifest before deleting them, so a failed delete has a name that
+survives the request.
+
+1. **Order: record, then delete.** The keys are appended to `manifest.graveyard` inside the same
+   CAS that stops referencing them. Deletion of the previous graveyard happens **before** that
+   CAS, on the keys that are in neither the new `parts` nor the new `graveyard` - and never on
+   keys written by this same commit, or the drain would eat what it just recorded.
+2. **A failed delete stays named.** Whatever could not be deleted remains in the new
+   `graveyard` and is carried to the next commit at no extra round trip. Retries are idempotent,
+   and a rival cannot reference our graveyard keys, so re-deleting is safe.
+3. **Never delete doomed keys before the CAS.** Tempting - they are not in the new manifest - but
+   a retry's content fingerprint can still point at one of them, and the committed manifest would
+   then reference an object that no longer exists. The drain before the CAS is a different shape
+   with the opposite conclusion, and the reason is worth keeping: graveyard keys come from
+   **earlier requests**, so they can never appear in this request's `reuse`, while staged keys
+   **belong to this request** - which is exactly why deleting a staged key also has to remove it
+   from `reuse`. Same appearance, opposite answer, different reason.
+4. **Bounded, newest first.** The field has a cap; over the cap, drop the **oldest** entries and
+   keep the newest, because the newest are the ones a partitioned delete just failed to remove.
+   An unbounded field would grow the manifest without limit under contention.
+5. **A corrupt graveyard never blocks service, and is not blindly overwritten.** On read, a value
+   that is not an array, or one longer than the read cap, is **skipped** (no drain this request)
+   and counted, never sliced - slicing would drop keys from the cleanup list and manufacture the
+   leak this record exists to prevent. Writing does not simply overwrite, because a partially
+   corrupt value can still contain valid keys: a value that is not an array contributes nothing, an
+   array contributes the entries that are non-empty strings, and everything else is dropped and
+   counted as `graveyard_dropped_unparseable`. The salvaged keys, this commit's doomed set and the
+   carried-forward remainder are then unioned, filtered against the new `parts` - a graveyard entry
+   that is also a live part is a contradiction, and the next drain would delete live data -
+   deduplicated, capped newest-first (drops counted as `graveyard_dropped_cap`), and only then
+   written. Overwriting outright would discard keys that were perfectly parseable, which is our own
+   leak rather than the corruption's. `/add`, `/search` and `/capacity` keep serving regardless of
+   what the field contains.
+6. **The counters ride the channel an operator already reads.** `userCapacity` returns
+   `pending_entries` and `manifest_bytes` per user and already raises `warnings` at 80% of a
+   limit, so `graveyard_pending`, `graveyard_dropped_unparseable` and `graveyard_dropped_cap`
+   belong in the same object, and a non-zero drop counter raises a warning ("leaking") there
+   rather than in a field nobody has a reason to look at. `graveyard_pending` measures **only
+   what was recorded**; orphans created before this mechanism existed are invisible to it and
+   must be measured separately.
+
+7. **Reset keeps the names.** `resetManifestState` builds a fresh object literal, so any field it
+   does not name is dropped, and `clearResetPendingKeys` only knows `parts` keys plus
+   `reset.pending_keys`. A graveyard key would therefore survive a reset with nothing referring to
+   it - permanently, since reclamation only sweeps other epochs - and reset is supposed to be the
+   escape hatch. Reset merges the graveyard into `reset.pending_keys` (already de-duplicated there)
+   and drops the field; `clearResetPendingKeys` already deletes in the same 1000-key batches, so
+   this adds no new limit.
+
+## What this does and does not establish
+
+- It gives a **bounded and self-healing** residue, not an **immediate** one: the last batch of a
+  quiet period waits for the next commit, so the invariant holds "after the last commit", not "at
+  any instant".
+- It does not repair orphans that already exist. A local memory-bucket soak or a one-off admin
+  LIST is the only way to see those, and the two numbers must not be compared as one series.
+- It does not make a stalled bucket fast; it makes the residue named.
+
+## Rollback
+
+During a rollback, an older writer does not know about the field and drops it while rewriting the
+manifest, so keys recorded by the newer writer lose their only name. Rolling forward again does
+not restore them. Storage does not shrink; the leak becomes unnamed. This is recorded here so it
+is not later investigated as a new defect.
+
+## Acceptance criteria
+
+- **Empty state is byte-identical**: with an empty graveyard the manifest contains no
+  `graveyard` key at all, so a user who never triggers the mechanism is indistinguishable from
+  one written by today's deployment. `JSON.stringify` must not contain the string.
+- **Availability with a corrupt field**: seed a manifest with `graveyard: "not-an-array"` and one
+  with an array longer than the read cap; assert `/add`, `/search` and `/capacity` all answer
+  normally, that the value is preserved until the next successful write, and that the counter is
+  reported. Without this test nothing prevents a future edit from turning it back into a 503.
+- **The write side is an assertion, not a description.** Feed several corrupt graveyards - a
+  non-array, an array holding non-string entries, one longer than the cap, and one containing a
+  key that is also a live `parts` entry - and after one successful write assert that the field is
+  either absent or (an array of non-empty strings **and** no longer than the cap **and** disjoint
+  from `parts`). Without this, an implementation that forgets the `parts` filter or the cap passes
+  every other criterion here, and the first entry under Risks - a wrong filter deleting live data -
+  stays a paragraph instead of a test. In the live-key case also assert the referenced **object
+  still exists**: an implementation that deletes first and tidies the field afterwards satisfies
+  disjointness while destroying live data. In the over-cap case assert **which** keys survive, not
+  just the length - the newest entries and the newly doomed ones must be kept and the oldest
+  dropped - because a reversed comparison passes a length-only test.
+- **Record then delete**: a failed delete leaves the key in the next manifest's graveyard, and the
+  following successful commit removes it.
+
+## Risks
+
+- **The drain deletes a live key.** The filter is "in neither the new `parts` nor the new
+  `graveyard`". A mistake there destroys a referenced snapshot, which is data loss, not residue:
+  this is the reason the filter is spelled out and tested before the mechanism is enabled.
+- **The cap drops a key that is still referenced.** Newest-first exists to avoid it, but a wrong
+  comparison could drop the newest instead of the oldest; the acceptance test has to pin the
+  direction, not just the length.
+- **An older writer drops the field** (see Rollback): in-flight keys lose their only name and
+  never come back.
+- **Manifest growth under contention.** Bounded by the cap, but the cap has to be small enough
+  that 128 partitions of doomed keys never approach `MAX_MANIFEST_BYTES`.
+
+## Companion pit: a manifest at the byte ceiling also has no way back
+
+`commitManifest` throws when the receipt count exceeds `MAX_RECEIPTS`, and it throws the same way
+when the encoded manifest exceeds `MAX_MANIFEST_BYTES`. Both are the same class as an unparseable
+graveyard - a user that cannot be written to at all, repairable only by `/reset` or an admin
+action - and both deserve to be recorded together rather than one being fixed and the other
+forgotten. This record does not fix them; it notes that the graveyard's cap must be small enough
+that it can never be the reason a manifest crosses that ceiling.
+
+## Alternatives considered
+
+- **A timeout heuristic** ("delete anything under `parts/` older than N") without recording: needs
+  a background sweep for a scenario that has no background execution on this platform, and it
+  guesses at a threshold instead of knowing which keys were doomed.
+- **Delete before the CAS** so the failure window disappears: breaks retries, see rule 3.
+- **Unbounded graveyard**: grows the manifest without limit exactly when writes are contending.
+- **Swift at delete time rather than recorded**: batching turned a whole-batch rejection into a
+  partial one; a retry-driven sweep inside the same request is what actually finishes the job.
+
+## Consequences
+
+- One new manifest field, written only when there is something to record, so a drained manifest
+  stays byte-identical to today's.
+- Cleanup costs at most one extra commit to converge, and nothing extra on the wire.
+- An operator gets a number (`graveyard_pending`) that says how much recorded backlog exists.
+
+## Scope
+
+This is **not** part of the append-log accept slice, and not part of its gate. It changes the
+manifest format, so it gets its own decision entry (this one) and its own commit boundary; it must
+not land quietly inside the add-log ticket, or a later reader cannot tell which intent added a
+field to the manifest.

docs/decisions/proposed/2026-10-29-recorded-graveyard-for-cleanup.terms.yaml

@@ -0,0 +1,3 @@
+version: 2
+terms: []
+references: []

docs/decisions/proposed/2026-10-29-recorded-graveyard-for-cleanup.zh-CN.md

@@ -0,0 +1,105 @@
+# 用可记录的 graveyard 收尾提交完成不了的清理
+
+[English](2026-10-29-recorded-graveyard-for-cleanup.md) · [Terms](2026-10-29-recorded-graveyard-for-cleanup.terms.yaml)
+
+**Status:** proposed
+
+## 问题
+
+被取代的分区快照的清理只能发生在 manifest 的比较交换**成功之后**——只有"不再引用它的 manifest 已经持久化"
+这一条成立，键才可以删。于是有三种情况会留下**没有名字**的孤儿：
+
+- 提交之后、删除之前函数被杀（平台上限、重启）；
+- 删除失败，调用方的 `.catch(() => {})` 把它吞掉；
+- 删除现在按 provider 的 1000 键上限分批，后一批失败时前一批已经成功——**部分清理**，既不报错也不留名。
+
+这三种从 manifest 都看不出来，而 manifest 是后续请求唯一读取的东西。真正关心的不变量——`parts/` 下的每个键都被
+manifest 引用——只在"无崩溃、且每一批删除都成功"的静默期成立，而那正是事故破坏的条件。
+
+## Proposal
+
+在删除之前把待删的键记进 manifest，让失败的删除有一个能活过本次请求的名字。
+
+1. **顺序：先记后删。** 键在"不再引用它们"的那一次 CAS 里追加进 `manifest.graveyard`。旧 graveyard 的删除发生在
+   该 CAS **之前**，且只删"既不在新 `parts`、也不在新 `graveyard`"里的键——**绝不删本次提交刚写进去的键**，
+   否则 drain 会吃掉自己刚记下的东西。
+2. **删失败的留在名字里。** 删不掉的键留在新 `graveyard` 里，被顺路带到下一次提交，没有任何额外往返。重试幂等，
+   对手不可能引用我们的 graveyard 键，所以重删安全。
+3. **绝不在 CAS 之前删待删键。** 看着安全（它们不在新 manifest 里），但重试的内容指纹仍可能指向其中之一，
+   提交后的 manifest 就会引用一个已不存在的对象。CAS 之前的 drain 是形状相同、结论相反的另一件事，理由值得留住：
+   graveyard 里的键来自**更早的请求**，因此永远不可能出现在本次请求的 `reuse` 里；而 staged 键**属于本次请求**
+   ——这正是"删掉 staged 键还必须同时从 `reuse` 移除"那条规则必要的原因。外观相同，结论相反，理由不同。
+4. **有界、新键优先。** 字段有上限；超限丢**最旧**的、保留最新的——最新的恰好是某批删除刚失败的那批。
+   无界会让 manifest 在争用时无限增长。
+5. **坏 graveyard 永不阻塞服务，也不直接覆盖。** 读侧遇到非数组、或长度超过读上限的值时**跳过**（本次不 drain）
+   并计数，**绝不 slice**——截断等于把一部分键从清理清单里悄悄扔掉，正是本条记录要防的泄漏。写侧不是简单覆盖，
+   因为**部分损坏的值里仍可能有合法键**：非数组贡献 0；是数组则贡献其中非空字符串的项，其余丢弃并计入
+   `graveyard_dropped_unparseable`。随后把「抢救出的键 + 本次待删集合 + 从上次带过来的剩余」做并集，
+   按新 `parts` 过滤（graveyard 里的键同时又是 live part 是自相矛盾，下一次 drain 会删到活数据），去重，
+   按新键优先做 cap（丢弃计入 `graveyard_dropped_cap`），最后才写出。直接覆盖等于丢掉本来完全可解析的键，
+   那是我们自己制造的泄漏，不是损坏造成的。无论字段内容如何，`/add`、`/search`、`/capacity` 都正常服务。
+6. **计数器走运维已经在看的那条通道。** `userCapacity` 按用户返回 `pending_entries`、`manifest_bytes`，并且已经有
+   80% 触发 `warnings` 的机制，所以 `graveyard_pending`、`graveyard_dropped_unparseable`、
+   `graveyard_dropped_cap` 应放进同一个对象，且后两者非零时在那里产生一条 warning（"leaking"），
+   而不是放在没人有理由去查的新字段里。`graveyard_pending` 只度量**被记录过的**积压；机理上线前产生的孤儿
+   它看不见，必须单独测。
+
+7. **reset 要保住这些名字。** `resetManifestState` 构造的是一个全新的对象字面量，它没点名的字段会被丢掉，
+   而 `clearResetPendingKeys` 只认识 `parts` 的键和 `reset.pending_keys`。于是 graveyard 里的键会在 reset 之后
+   活下来、却没有任何东西再引用它——而且是**永久**的，因为回收只扫其它代；而 reset 本该是那条逃生通道。
+   所以 reset 要把 graveyard 并进 `reset.pending_keys`（那里已经去重）并丢掉字段；`clearResetPendingKeys`
+   本来就是按 1000 键分批删，所以这不引入新的上限。
+
+## 这确立了什么、没有确立什么
+
+- 给出的是**有界且自愈**的残留，不是**立即**收敛：静默期的最后一批要等下一次提交，所以不变量在"最后一次提交之后"
+  成立，不是"任意时刻"。
+- **不修复已存在的孤儿**。只有本地 memory bucket soak 或一次性 admin LIST 能看到它们，两个数字不能当同一序列比较。
+- **不让卡住的桶变快**，只是让残留有名字。
+
+## 回滚
+
+回滚期间，旧写入者不认识这个字段，重写 manifest 时会把它丢掉——新写入者记录的那批键就此失去唯一的名字；
+再前滚也不会恢复。存储不会因此下降，泄漏从"有名字"变成"没名字"。记在这里，避免以后被当成新缺陷重新排查。
+
+## Acceptance criteria
+
+- **空态逐字节相等**：graveyard 为空时 manifest 里**根本没有** `graveyard` 键，于是从未触发该机理的用户与今天部署写出的
+  完全不可区分；`JSON.stringify` 里不得出现这个字符串。
+- **坏字段下仍可用**：seed 一个 `graveyard: "not-an-array"` 与一个超长数组的 manifest，断言 `/add`、`/search`、
+  `/capacity` 全部正常作答、值在下一次成功写入前被原样保留、且计数被报出。没有这条，将来把行为改回 503 没有任何东西拦得住。
+- **写侧是断言，不是描述。** 分别喂进几种坏 graveyard——非数组、含非字符串元素的数组、超过 cap 的数组、
+  以及含有同时是 live `parts` 键的数组——各做一次**成功写入**后断言：字段要么不存在，要么满足
+  （非空字符串数组 **且** `length <= CAP` **且** 与 `parts` 无交集）。没有这一条，一个忘了按 `parts` 过滤、
+  或忘了封顶的实现能通过这里列出的全部验收，而 Risks 第一条——过滤器写错就是删掉活数据——就永远只是一段话，
+  不是一个测试。**live 键那一档还要断言被引用的对象仍然存在**：一个"先删、后整理字段"的实现能满足"无交集"，
+  同时销毁活数据。**超 cap 那一档要断言活下来的是哪几个键**，而不是只看长度——最新的、以及本次新判死的必须保留，
+  最旧的必须被丢；比较方向写反的实现能通过只测长度的断言。
+- **先记后删**：一次失败的删除会把键留在下一个 manifest 的 graveyard 里，随后一次成功提交把它清掉。
+
+## Risks
+
+- **drain 删掉一个仍在使用的键。** 过滤条件是"既不在新 `parts`、也不在新 `graveyard`"。这里判断错就是删掉被引用的快照——
+  那是数据丢失，不是残留。这也是为什么必须先把过滤条件写清并测过，再启用机理。
+- **cap 丢掉仍被引用的键。** 新键优先就是为了避免它；但比较方向写错就会丢最新的。验收测试要钉住**方向**，不只是长度。
+- **旧写入者把字段丢掉**（见"回滚"）：在飞的那批键失去唯一名字，且永不恢复。
+- **争用下 manifest 增长。** 由 cap 约束，但 cap 必须足够小，让 128 个分区的待删键永远不会逼近 `MAX_MANIFEST_BYTES`。
+
+## 考虑过的替代方案
+
+- **超时启发式**（"删掉 `parts/` 下比 N 更老的"）：需要一个后台清扫，而这个平台没有后台执行；而且是猜阈值，不是知道
+  哪些键已被判死。
+- **在 CAS 之前删**以消掉失败窗口：破坏重试，见规则 3。
+- **无界 graveyard**：恰在写入争用时让 manifest 无限增长。
+- **只在删除时重试而不记录**：分批把"整批被拒"变成了"部分失败"，真正能收尾的是同一次请求内由重试驱动的清扫。
+
+## 后果
+
+- 新增一个 manifest 字段，只在有话要记时出现，所以排空后的 manifest 与今天的逐字节相同。
+- 清理最多多花一次提交来收敛，线上没有任何额外往来。
+- 运维得到 `graveyard_pending` 这个"已记录积压有多少"的数字。
+
+## 边界
+
+这**不属于**加法日志的 accept 切片，也不属于它的门槛判据。它改变 manifest 格式，因此有自己的决策条目（本条）与
+自己的提交边界；不能悄悄落在加法日志那张票里，否则以后没人说得清"manifest 为什么多了个字段"。