fix: reopen the bridge for chunks a committed node never agreed to shed - #226
Conversation
Once a node reaches `Committed`, the copier moves only the keys the rank check refuses to shed. A chunk that turns up in the legacy environment after that, and that this node never agreed to give up, is on no such list: nothing copies it, the shed gate correctly refuses to prove a single copy exists elsewhere, and the startup reconciliation only returns a node to `Bridging` when the file store holds less than it recorded keeping, which the file store never does. The node is terminal in that state. Free disk does not change it and neither does a restart. Seen on one node of 38 in the 2026-09-08 beta cohort: it committed with "nothing has to be shed", then pre-retirement verification found 317 chunks in the legacy environment that were in neither view and re-queued them. It has refused retirement every 30 seconds since. Once WithAutonomi#218 removes the LMDB reader those 317 stop being servable, and on the evidence they are the network's only copy. The fix is a third arm in the reconciliation block, next to the two of the same shape: when the marker says `Committed` and there are more legacy-only keys than the node ever approved for shedding, go back to `Bridging`, clear `committed_at_unix` and reset `rebuilds_since_commit`. `open_legacy` already computes the legacy-only set before the marker is read, so this is a comparison against a number that is already in memory. No new I/O, no marker format change. The count, not emptiness, is the test. A node that legitimately shed keeps exactly the keys it is giving up in the legacy environment until they stop being answerable, so a non-empty legacy-only set is its normal state; bouncing it to `Bridging` on every restart would reset its retention clock each time and a regularly restarted node would never retire. The arm fires only on `legacy_only > shed_key_count`. It logs `migration_event = "back_to_bridging"` with both numbers so the case is visible in the beta watch. The upgrade's own restart is enough to take an affected node out of the loop. Tests: `cargo test --lib` 1104 passed, 0 failed; fmt clean; clippy with `-D clippy::panic -D clippy::unwrap_used -D clippy::expect_used` clean on lib and tests. Two tests added, and both were mutation-checked: with the discriminator replaced by `legacy_only > 0`, the new regression test and the existing `the_migration_marker_survives_a_restart` both fail; with the arm disabled, the fire-case test fails. Closes V2-1232 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UrjwPBwPqMi3F2sT1yVYEa
dirvine
left a comment
There was a problem hiding this comment.
Reviewed exact head 3b2b7093ae806a9ecc77b07b93e83eb860ff8621.
The restart reconciliation is correctly scoped: it reopens Committed only when the current legacy-only cardinality exceeds the persisted shed allowance, while preserving the legitimate equal-count shed state. Destructive retirement remains protected by the exact approved-key-set checks.
Verification:
cargo test --lib: 1104 passed, 0 failed- both added regression tests passed
cargo fmt --all --checkandgit diff --checkpassed
Non-blocking: an equal-cardinality/key-substitution retirement-refusal test would document the identity-level safety backstop.
CI watch-outs unrelated to this diff: Security Audit is failing on newly disclosed rustls RUSTSEC-2026-0285 with unchanged Cargo.lock; the btrfs reclaim threshold missed marginally. Platform test jobs were still running at review time.
Once a node reaches
Committed, the copier moves only the keys the rank check refuses to shed.A chunk that turns up in the legacy environment after that, and that this node never agreed to
give up, is on no such list: nothing copies it, the shed gate correctly refuses to prove a
single copy exists elsewhere, and the startup reconciliation only returns a node to
Bridgingwhen the file store holds less than it recorded keeping, which the file store never does. The
node is terminal in that state. Free disk does not change it and neither does a restart.
Seen on one node of 38 in the 2026-09-08 beta cohort: it committed with "nothing has to be shed",
then pre-retirement verification found 317 chunks in the legacy environment that were in neither
view and re-queued them. It has refused retirement every 30 seconds since. Once #218 removes the
LMDB reader those 317 stop being servable, and on the evidence they are the network's only copy.
This must land before or with #218.
The fix
A third arm in the reconciliation block in
ChunkStore::new, next to the two of the same shape:when the marker says
Committedand there are more legacy-only keys than the node ever approvedfor shedding, go back to
Bridging, clearcommitted_at_unixand resetrebuilds_since_commit.open_legacyalready computes the legacy-only set before the marker is read, so this is acomparison against a number that is already in memory. No new I/O, no marker format change.
The count, not emptiness, is the test. A node that legitimately shed keeps exactly the keys it
is giving up in the legacy environment until they stop being answerable, so a non-empty
legacy-only set is its normal state; bouncing it to
Bridgingon every restart would reset itsretention clock each time and a regularly restarted node would never retire. The arm fires only
on
legacy_only > shed_key_count. It logsmigration_event = "back_to_bridging"with bothnumbers so the case is visible in the beta watch.
The upgrade's own restart is enough to take an affected node out of the loop.
Linear issue
Closes V2-1232
Risk tier
Compatibility
shed_key_countis already in the marker. The only stores whosebehaviour changes are those in the state described above, which move from
CommittedtoBridgingon their next open and run the copier again. Every other store keeps its phase,clock and counts across the upgrade.
legacy_only_count.Semver impact
Test evidence
Local, on
5c7f942plus this branch:cargo test --lib1104 passed, 0 failed.cargo fmt --all --checkclean;cargo clippy --all-features --tests -- -D clippy::panic -D clippy::unwrap_used -D clippy::expect_usedclean.a_node_that_committed_with_nothing_to_shed_resumes_the_copy_for_chunks_only_it_holdsshed = 0, a chunk appears in the legacy environment only, reopen →Bridging, clock cleared, copier moves ita_node_that_shed_stays_committed_across_a_restartshed = 1, one legacy-only key, must stayCommittedwith its clock untouchedBoth mutation-checked: with the discriminator replaced by
legacy_only > 0, the regression testand the existing
the_migration_marker_survives_a_restartboth fail; with the arm disabled, thefire-case test fails.
Not covered here: a dev testnet run. The condition takes a node into a state that needs hours
of wall clock plus a specific failure to reach, which is why the tests use the store harness.
New dependency
none
ADR
https://github.com/jacderida/ant-node/blob/3b2b7093ae806a9ecc77b07b93e83eb860ff8621/docs/adr/ADR-0014-file-based-chunk-store-and-lmdb-retirement.md
Mitigation / rollback
Revert the commit. A node this arm moved back to
Bridgingthen re-commits through the normalpath on its next pass, with
shed_key_countrecomputed from what is still legacy-only; nothingon disk differs otherwise. The affected node returns to the loop it is in today.
🤖 Generated with Claude Code
https://claude.ai/code/session_01UrjwPBwPqMi3F2sT1yVYEa