fix(storage): repair graphs written by 4.8.1, and stop the build opening the real one

[kevintseng]

What

v4.8.2's fixes for #240 and #241 stop new damage; they did nothing for the rows already written by 4.8.1 hooks, and every upgraded graph has them. This adds three one-shot repairs that run at the first core open after upgrade (CLI / MCP / HTTP — hooks open through their own wrapper and run no migrations), alongside the existing backfills in db.ts:

pass	invariant it owns (scripts/audit/memory-invariants.mjs)	on the maintainer's graph
dedupeSessionObservations	stop-summary-no-duplicate-observations	706 duplicate rows removed
retractZeroEditClaims	stop-summary-does-not-assert-zero-edits-for-bash-sessions	1 false 0 files edited rewritten to say the count was not recorded
splitFusedLessons	explicit-lessons-not-fused-into-other-bucket	35 lessons moved out of 4 -other buckets into lesson-<project>-<slug> entities

Design choices, each written down in the module header:

• Lesson rows are moved, not copied — ids and created_at survive, nothing is re-authored. Every lesson leaves the bucket (keeping the first would leave one lesson whose re-learned copy lands beside its history — review F4); the emptied bucket loses source:explicit and is archived, not deleted, so relations still resolve and a later auto-learned lesson cannot re-trip the invariant (F3).
• severity:* is carried only when the bucket has exactly one; source:explicit only when no source:auto-learned sits beside it. With several, which lesson was critical is recorded nowhere; guessing would write a fact nobody stated.
• FTS is rebuilt whole (rebuildFtsIndex, same call the segmentation migration makes), not patched per row. entities_fts is contentless: a delete must repeat the exact indexed text, and a row the hook wrote without indexing has none — deleting it corrupts the index (database disk image is malformed, reproduced in the first version of the tests).
• Vectors are left in place — sqlite-vec loads after the backfills, so this code cannot touch entities_vec and no longer claims to (the first version had dead DELETEs; review F2). A reindex is marked owed; memesh doctor reports it, memesh reindex clears it.
• lessonSlug moves to src/core/lesson-slug.ts (a leaf) so storage/ and lesson-engine share one definition without an import cycle through db.ts.

The incident this PR also fixes

npm run build ends with scripts/smoke-test.mjs, whose HTTP-server child inherited no database path. It opened the developer's real ~/.memesh and ran every migration in the working tree against it — which is how the repair above ran on the maintainer's real graph from a build, with no pre-run snapshot. The child now gets its own throwaway path (.smoke-test-http.db, cleaned up). Proven: three builds since, real WAL mtime unchanged each time. The repair itself is content-preserving (exact duplicates removed, rows moved), and a post-state backup exists.

Evidence (this tree, this session)

• node scripts/run-tests-isolated.mjs → exit=0 on the final tree (see the last commit's Verified-By for the count)
• npm run verify:release → exit=0 (5 ✓; C1 hit on the new test file triaged in baseline.json: .toBe(0) are exit codes, each preceded by a toBe(1) on the seeded defect)
• eg check → PASS 4/4 CODE_COMPLETE, 12 receipts; failure paths: dedupe query pointed at a non-existent prefix → tests red; lesson move loop emptied → tests red; group loop emptied in dist → real-graph copy keeps its 24 bucket observations; CHANGELOG entry renamed → docs check red. All restored (no unstaged diff).
• Real-graph copy (backup …bak-2026-08-30-0322, markers cleared): one open → 24 bucket observations → 0, 3 markers set, invariants exit 0; second open → identical row count (9236).
• The one session the old invariant still flagged on the real graph had run node -e '… 1<<32 …' — a JS shift, no file write — so its 0 files edited was true. The first version's substring match would have falsified it; the anchored regex leaves it alone.

Simplification

Read-only pass (sonnet, code-simplifier) on the full diff. Applied: lastInsertRowid instead of a re-SELECT after INSERT; removeTempDir from tests/helpers instead of a hand-rolled rmSync; a title assertion the test selected but never checked (the whole title-plumbing path was unprotected). Also removed an unused lead array from groupLessons and the ?? 1.0 on confidence (nullable column, copied as-is now). Declined, with reasons in the review: collapsing the dedupe loop into one full-table GROUP BY (worse on large graphs); extracting runInvariants() to a shared helper (second occurrence, not third). Correctly flagged as not-safe and left alone: the raw embedding_dimension read in db.ts (the singleton helper is null during open).

Adversarial review (opus, fresh context, read-only) — verdict FAIL on the first version, all eight findings fixed

#	finding	fix	test
F1	LIKE '%0 files edited%' also matched 10 files edited and would have rewritten a true sentence into 1files edited through Bash…, irreversibly	anchored on , 0 files edited in both the repair and the invariant	session-ten-summary stays 10 files edited
F2	hasVectorIndex is always false during open (vec loads later) → the vector DELETEs were dead code and the docs claimed otherwise	deletes removed; module header, CHANGELOG and PR say vectors stay and a reindex is owed	—
F3	invariant could go red again: auto-learned lessons append to the same -other bucket and it kept source:explicit	bucket loses source:explicit; auto-learned path already tags source:auto-learned, which the invariant ignores	mixed-source bucket case
F4	first lesson stayed in the bucket → re-learned copy lands beside it	all groups move; empty bucket archived	observations(bucket) == [], status archived
F5	target looked up by name only → a forget-archived entity would swallow the moved lesson out of search	SELECT id, status; archived target revived (same as remember)	archived nameA revived, found by search
F6	706 deletions / 35 moves on real memories with no output	one stderr line per pass that changed something	—
F7	<< / tee substring marks → python3 - <<'PY', | tee /tmp/t.log counted as writes	the hook's own bashEditedPaths regexes (/dev/, /tmp/ excluded), shared with the invariant	bashWritesFiles positive/negative table; session-stdin-summary untouched
F8	bucket without a project: tag skipped by the repair but flagged by the invariant forever	project derived from the name when the tag is missing	lesson-my-app-other split

Also from the review: smoke-test cleanup moved into finally; .smoke-test*.db-* gitignored. Checked and confirmed sound by the reviewer: atomicity (everything inside runOnceMigration's IMMEDIATE transaction), dedupe deletes only exact duplicates, tags vs the unique index, FTS status filter, relations left on the archived bucket, the raw embedding_dimension read during open, the smoke-test env reaching startServer.

Real-graph check after the review fixes: the retraction pass had not run on the real graph (marker absent) and no rewritten row exists there or on the copy — F1 was a defect of the branch, not of the data.

Second adversarial round (opus, fresh context) on the F1–F8 fixes — verdict FAIL, all eight closed

#	finding	fix	test
N1	the invariant's Bash-write test moved into JS but LIMIT 8 stayed in the SQL, bounding candidates instead of violations — 8 honest sessions sorting first hid a real one (reproduced)	no SQL limit; cap after the JS filter	9 honest + 1 violator → exit 1, violator named
N2	bashWritesFiles took the first match only; | tee /tmp/t.log && … | tee CHANGELOG.md was "no write"	every match, fresh g regex per call (hook parity)	three multi-target commands → true
N3	source:explicit dropped even when a stray row kept the bucket non-empty → invisible to the invariant	tag drop and archive both require an emptied bucket	stray Note: row: tag kept, status active
N4	"hooks do not run migrations" — two hook paths import dist/db.js and do	wording in header + CHANGELOG	—
N5	"relations still resolve" — the work-layer view filters archived ends	wording	—
N6	pending_reindex: vectors-missing on a vec-less machine → doctor says "run reindex" forever, reindex refuses	doctor skips that row when hasVectorIndex is false; dimension-change still shown	tests/core/doctor-unpayable-reindex.test.ts (mocks hasVectorIndex)
N7	split-out lessons had no signal_score and dropped the bucket's metadata (trust/provenance)	metadata copied, signal_score computed at insert	trust: untrusted travels; score is a number
N8	an exception in the new post-filter printed as a benign "schema skip"	own try, exit 2	—

Also from that round: the metadata parse guard in the split is break-tested (catch → rethrow makes the unreadable-metadata test red). Re-check of the real graph with the fixed detector (no limit, every candidate examined): all three invariants ok, exit 0.

Third adversarial round (opus, scoped to the round-2 fixes) — verdict FAIL on one finding, all closed

#	finding	fix	test
R3-1	copying the bucket's metadata also copied an accepted guard object onto every split-out lesson — N identical warnings, N× fires, one acceptance asserted N times, and not re-runnable	guard, evidence_for, previous_namespace are deleted from the copy (per-entity facts)	bucket with guard/evidence_for/previous_namespace → none on the split entity; trust still travels
R3-4	an error slugging to other targets a bucket (its own, or another after a project rename) — rows "moved" into a bucket, moved inflated	such a group is skipped and stays where the invariant sees it	Error: other group stays; the other lesson still splits
R3-5	contract comment in memory-invariants.mjs missed the new exit-2 case	comment	—
R3-6	db.prepare inside the post-filter loop, now unbounded	hoisted, one prepare	—
R3-7	the 8-row cap was untested	—	9 violations → 8 printed + (first 8)

Cleared by the reviewer with evidence: the payableDebt branch matrix (no unintended change; hasVectorIndex()===false ⇔ sqlite-vec did not load, by construction of ensureVecTable); regex sources byte-identical to the hook, g the only flag, no backtracking risk; slug collisions merge exactly as live learn() does (same contract, not a new merge); both changed doc claims verified.

Fourth adversarial round (opus, scoped to the round-3 fixes) — verdict FAIL on one finding, closed

#	finding	fix	test
R4-1	name.endsWith('-other') also skipped a real lesson whose error merely ends in the word ("could not reach the other") — permanently, with the invariant green	predicate is lessonSlug(error) === 'other', the only slug that can name a bucket	that lesson splits out
R4-2	the round-3 test for the skip could not fail (a self-move is a no-op)	replaced by the project-rename fixture (lesson-old-other carrying project:new must not mint lesson-new-other) — red without the guard	—
note	the invariant had the same suffix confusion (LIKE '%-other')	when a project tag exists, the name must be exactly lesson-<project>-other	5-observation lesson named …-the-other is not a violation
note	"(first 8)" printed on exactly eight violations	queries fetch 9, the line prints only when a ninth exists	exactly 8 → no line

Reviewer-confirmed: the three metadata deletes are the complete per-entity set (with compacted_into noted as the next one if lesson_learned ever becomes compactable); bucket logic when every group is skipped is correct; the round-3 metadata-delete test is a real break-test.

Fifth adversarial round (opus, scoped to the round-4 fixes) — verdict FAIL on the invariant clause, closed

#	finding	fix	test
R5-1	the name-vs-project: tag clause made the invariant blind to a real fused bucket after kg rename-project (tags are rewritten, names are not) — a regression against the previous predicate	the invariant now counts distinct Error: lines (HAVING lessons > 1) — the same line groupLessons cuts on, so detector and fixer agree by construction; the name-shape clause is gone	lesson-old-other + project:new, 2 lessons → flagged (2 lessons)
R5-2	the same clause still false-flagged a tag-less re-learned lesson named …-the-other	same fix	5-row re-learned lesson, no project tag → ok
R5-3	the SQL LIMIT MAX_ROWS+1 was unpinned (silent under-report if reverted)	—	nine duplicate-summary entities → 8 printed + (first 8)
R5-4	the bucketsTouched guard was unpinned	—	stderr note asserts out of 1 "-other" bucket(s) with a second, skipped-only bucket present

Stated trade-off (reviewer's own words adopted): a bucket holding two lessons whose errors both slug to exactly other stays permanently red — visible-and-unfixable beats invisible-and-unfixable.

Sixth adversarial round (opus, scoped to the invariant rewrite) — verdict FAIL on one finding, closed

#	finding	fix	test
R6-1	counting distinct Error: lines flagged a legitimate 4.8.2 entity: two different error texts sharing one 8-word slug that ends in "other" are ONE lesson by learn()'s contract; the two "not a bucket" tests were green only because their fixtures used byte-identical text	the invariant now asks the repair's own question in a JS post-filter: cut on Error: exactly as groupLessons does, slug exactly as learn() does (function mirrored, like the Bash shapes), fused ⇔ more than one slug or a slug the name does not end with	lesson-proj-the-agent-could-not-talk-to-the-other with port 3000/4000 texts → not flagged
R6-2	says still said "(4 fields each)" from the old row-count proxy	reworded; also tells the reader that a lesson whose error is literally "other" is left in place and how to fix it	—
R6-3	LIKE is case-insensitive, startsWith is not	the post-filter decides with startsWith, the SQL only narrows	—
R6-4	stderr spy without try/finally	try/finally	—
R6-5	a group targeting the bucket's own name would self-move and print a move	name === bucket.name guard	—

Decision recorded here (reviewer asked that it be explicit): a -other bucket whose lessons all have the error text "other" is now consistent with its own name and not reported — the repair leaves such rows in place either way.

Seventh adversarial round (opus, scoped to the post-filter and self-move guard) — PASS_WITH_CONCERNS, both concerns landed

#	concern	done
R7-1	the self-move guard was deletable with a green suite (reachable: lesson-proj-the-other + project:proj, whose own lesson slugs to the-other)	test: no "moved" note, rows untouched, no pending_reindex; also recorded as the eg failure path
R7-2	says claimed "lessons that belong under other names" while the SQL keeps the deliberate single-lesson carve-out (> 1)	wording back to "more than one explicit lesson"; comment names the carve-out and the test that pins it
R7-4	the mirrored lessonSlug had no drift detector (differential run over 20,025 inputs: 0 divergences today)	test compares the two function bodies whitespace-normalised

Reviewer cleared: endsWith cannot let a fused bucket through (the slug count fires first, and names are minted with one slug); post-filter bounded (46 ms on 104k observations); audit baseline new=0 ×7. Disclosed limitation, unchanged: two unrelated incidents whose error texts both slug to other stay on one entity, unreported — the says text tells the reader how to escape it.

No eighth round: this last commit changes a comment and two tests only, and the round-7 reviewer had already produced the red/green pair for the self-move fixture that was adopted verbatim.

Not in this PR

• pending_reindex is not set when a graph has entities_vec but no embedding_dimension row. ensureVecTable always writes the row when it creates the table, so this looks unreachable; noted rather than guarded.