D-WAL: LanceCycleWriter vs MemWAL measured; capped session cache, version clock, copy-free temporal projections

[AdaWorldAPI]

Plan: .claude/plans/wal-shootout-v1.md. Board entry: .claude/board/entries/2026-10-15-d-wal-shootout.md.

Measured

D-id	What was measured	Result
D-WAL-1	Validation of both arms	Both write paths are crash-atomic per cycle and lose nothing they acknowledged. MemWAL needs durable_write = true for that. Its acknowledged writes are visible only to the writing handle until a flush or a WAL replay.
D-WAL-2	Commit cost	With the default 100 ms flush, every MemWAL write waits about 95 ms. With a 10 ms flush it is faster than LanceCycleWriter from 1,024 casts per cycle.
D-WAL-3	Soak (stopped at 27.5 min by a container restart)	MemWAL commit p50 stayed flat at 2.0–2.8 ms; L grew from 5.8 to about 10.5 ms. MemWAL's disk use was 1.57× L's. L's RSS grew from 95 to 762 MiB.
D-WAL-4	Where L's RSS growth comes from	It is Lance's session metadata cache, not our writer. Each commit caches a clone of that version's manifest, and in this append-only store manifest v lists v fragments, so the cache grows as V²/2. At 1,600 versions: 628 MiB with the default session, 7.8 MiB with an 8 MiB cap. RSS 570 vs 52 MiB.

Changes

• D-WAL-5 — capped session. LanceCycleWriter now opens and writes on a Lance session whose metadata cache is capped at 8 MiB (SESSION_METADATA_CACHE_BYTES).
  • Written test-first. 400 commits reached 42.6 MB uncapped. With the cap the cache stays under 8 MiB, after reopening as well.
  • Removing the session from the create write turns the test red, and so does removing it from open.
• D-WAL-6 — version clock. New contract type temporal_pov::{VersionClock, ClockError}.
  • It records the row count at each version of an append-only lane, eight bytes per version.
  • rows_at(&TemporalPov) returns one half-open row range: 4.2 ns per call on an AVX-512 host, against 127 µs for a per-row scan of 400k rows.
  • Tested against a per-row oracle over 200 versions, including skipped versions and versions that add no rows.
  • Three disable runs each turn a test red: no gap fill, an off-by-one start, no base guard.
  • Not yet wired into a writer or reader.
• D-WAL-7 — copy-free projections. temporal.rs gains deinterlace_indices, local_trajectories_indices and local_trajectory_indices_of. They return positions into the input and clone no row. The copying functions keep their signatures and wrap the index forms.
• D-LNC-7 — decision. L stays the default with the capped session; MemWAL stays an option. The 60-minute soak is deferred.

Verification

• Every build ran with debug info 0.
• Tests: lance-graph-contract 1483, lance-graph-planner 458, lance-graph 1058 (lib), all passing.
• Clippy -D warnings passes on every changed target. It still fails on two existing files this PR does not touch: planner/examples/probe_r2il_real_episodes.rs and planner/tests/moore_nars16_probe.rs.

Still open

• Seeding the version clock from storage when a writer opens an existing store.
• Compaction or deletes break the prefix property the clock relies on, so they need a fragment-id remap.
• The full-length soak (deferred).

🤖 Generated with Claude Code

https://claude.ai/code/session_011BMJnDBAfzcuQh5HeT39Rh

---

Generated by Claude Code

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 2c821d27-163b-4f45-b1c6-ec96009684e1

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-15T06:33:41.055183Z	12319fe	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 12319fef8b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".