Skip to content

Store each space in its own file (schema v5) - #13

Open
jonascript wants to merge 8 commits into
mainfrom
feat/per-space-files
Open

Store each space in its own file (schema v5)#13
jonascript wants to merge 8 commits into
mainfrom
feat/per-space-files

Conversation

@jonascript

Copy link
Copy Markdown
Owner

Why

Everything ike knows lived in one `tasks.json`: one corrupt write or bad hand edit put every space at risk, and moving a single space between machines required the export/import dance. This splits storage so each space is its own file — corruption in one space cannot touch the others, and a space becomes a copyable unit.

What changed

Layout (data format v5). `tasks.json` becomes a small manifest (`version`, `current`, both consent flags — deliberately no `spaces` key, so a v4 binary refuses it rather than reading an empty matrix). Spaces live in `tasks.json.spaces/`, one self-describing file each, with its own rolling `.bak`. Filenames are derived from the space name via a minimal percent-encoding (`/`, `%`, leading dot); the embedded name is canonical.

Write path. `mutateFile` is still the one write path, same single flock. The commit step (`writeTree`) dirty-checks per file — a mutation rewrites only the spaces it touched — and orders creates/updates → manifest → deletes so every crash window leaves a state the reader already repairs. Deleting a space renames its file to `.bak` (removal + backup in one atomic step). A case-only rename is guarded by `os.SameFile` so macOS's case-insensitive FS can't delete the file just written.

Degraded open. A space file that fails to parse costs that one space: the others keep working, the broken one is listed as unreadable everywhere (CLI, TUI picker, MCP `list_spaces`), nothing ever writes over its file, and `ike space rm --force` retires it to `.bak` as the recovery affordance. A `Current` naming a corrupt space fails loudly instead of being silently repaired away. Consent status keeps answering from the manifest even with every space corrupt.

Migration. On first write, never on read: the monolith (any version back to v1) is served in memory indefinitely; the first mutation splits it, keeps the original as `tasks.json.pre-v5.bak` (permanent), clears crash debris, and commits via the manifest write. Older binaries mid-window still see a valid v4 file.

Portability. `ike space export` now writes exactly the space's own file — export and a hand copy are byte-identical, and consent flags structurally cannot travel. A single exported space still opens with `--file` for reading and editing (lifecycle ops refuse). Import accepts all three shapes: v≤4 envelope, v5 space file, whole v5 tree with `--all`.

Fixed along the way: `ike space rename` used to strand the space's plans under the old name; it now moves the plans dir under the same lock, and `ike plan --prune` sweeps plan dirs of spaces that no longer exist (sparing unreadable ones).

ModTime (the TUI's poll signal) is now two stats — manifest + spaces dir — which covers every kind of write.

Testing

  • Full suite green at every commit; the readTree refactor commit passes with zero test edits, proving the legacy read path byte-for-byte.
  • New tests: codec round-trip/injectivity, migration (split, pre-v5.bak permanence, crash-debris clearing), ModTime bumps per write kind, degraded open (healthy spaces work, corrupt bytes untouched, rm --force recovery, corrupt Current stays), lifecycle on-disk mechanics (rename moves file+plans, case-only rename, rm-keeps-bak), export≈copy, standalone editing, three import shapes.
  • Durability tests re-pinned per file (modes incl. tightening on skipped writes, per-space .bak, temp patterns, lock).
  • Manual end-to-end run: v4 fixture → read stays v4 → first write migrates → corrupt a space → others fine, listing marks it, rm --force recovers → export == space file → standalone edit → import.

🤖 Generated with Claude Code

jonascript and others added 8 commits August 7, 2026 07:52
Groundwork for one-file-per-space storage: a percent-encoding that maps
any legal space name to a safe filename (covering only /, %, and a
leading dot), and the two on-disk shapes of the coming v5 layout — a
document-level manifest and a self-describing space file whose type has
no field for consent flags to travel in.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…files

readFile becomes readTree, gaining the (still unreachable) version-5
branches: a manifest read that enumerates the .spaces sidecar directory,
a standalone single-space-file shape for --file on an export, and
reconstruction of a document whose manifest is missing. File carries
read-state (raw bytes, filenames, corrupt spaces, on-disk version) so
the coming write side can confine writes to what changed.

The v1-v4 path is byte-for-byte the old behavior; the whole suite
passing without a single test edit is the check.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tasks.json becomes a small manifest (version, current, consent flags —
and deliberately no spaces key, so a v4 binary refuses it rather than
reading an empty matrix). Spaces live in tasks.json.spaces/, one
self-describing file each, written through the same lock and the same
atomic helpers: mutateFile is still the one write path.

The commit step is dirty-checked per file — a mutation rewrites only the
spaces it touched, so a bug or corruption in one space can no longer
clobber the others — and ordered creates/updates, manifest, deletes so
every crash window leaves a state the reader already repairs. Each space
file keeps its own rolling .bak; deleting a space is one atomic rename
to .bak. A skipped write tightens file modes by hand, since it no longer
replaces the inode.

Migration off the monolith happens on the first write, never on read,
with the original file kept as tasks.json.pre-v5.bak and the manifest
write as the commit point; debris of a crashed migration is cleared
before the split is retried. Consent readers now read the manifest
alone, so mcp/agent status keeps answering whatever state the space
files are in. Export writes the space-file shape, which import accepts
alongside v4 envelopes and whole v5 trees.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Task mutations now land as renames inside tasks.json.spaces/, which the
old single stat of tasks.json never saw — an open TUI would have gone
stale forever. Two stats cover every kind of write: renames into the
spaces directory bump its mtime (task edits, space create/remove/
rename), and document-level changes rewrite the manifest itself.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The point of the split: one corrupt space file costs that one space.
Every other space keeps loading and mutating; the broken one is listed
— in the TUI picker, `ike space list`, and list_spaces — as unreadable
rather than silently missing, and resolving it says what is wrong and
what to do, not "no space named X".

No write path can touch an unreadable file: it is absent from the
dirty-write and deletion sets by construction. A Current naming an
unreadable space is deliberately not repaired away — the user must be
told loudly, not shown some other space. The consent readers answer from
the manifest even when every space is corrupt.

The one sanctioned way at a corrupt file is `ike space rm --force`,
which renames it to .bak in case it can still be recovered by hand.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Renaming a space now moves its plan sidecar directory under the same
lock — renaming historically left the plans stranded under the old name
— and, on disk, retires the old name's file to .bak while the write
side's case-only guard keeps a recased name from deleting the file it
just wrote. Removing a space renames its file to .bak: removal and
backup in one atomic step. PrunePlans additionally sweeps plan
directories whose space no longer exists, while leaving an unreadable
space's plans strictly alone — they may be the only part of it still
readable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… shapes

Export and a hand copy of the space's file are asserted byte-identical;
a single exported space opened with --file edits in place (and undoes),
while every lifecycle operation refuses; import accepts a v4 envelope, a
v5 space file, and a whole v5 tree.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
docs/data.md gains the on-disk tree, the damaged-space section, and the
migration note; README's storage blurb follows; CLAUDE.md's invariants
are rewritten where the split changed them (write path, durability
properties, degraded open, migration, export=copy, ModTime); CHANGELOG
records the change under Unreleased.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant