Skip to content

fix: reopen the bridge for chunks a committed node never agreed to shed - #226

Merged
jacderida merged 1 commit into
WithAutonomi:rc-2026.9.2from
jacderida:v2-1232
Sep 14, 2026
Merged

jacderida merged 1 commit into
WithAutonomi:rc-2026.9.2from
jacderida:v2-1232

Conversation

@jacderida

@jacderida jacderida commented Sep 14, 2026

Copy link
Copy Markdown
Member

Once a node reaches Committed, the copier moves only the keys the rank check refuses to shed.
A chunk that turns up in the legacy environment after that, and that this node never agreed to
give up, is on no such list: nothing copies it, the shed gate correctly refuses to prove a
single copy exists elsewhere, and the startup reconciliation only returns a node to Bridging
when the file store holds less than it recorded keeping, which the file store never does. The
node is terminal in that state. Free disk does not change it and neither does a restart.

Seen on one node of 38 in the 2026-09-08 beta cohort: it committed with "nothing has to be shed",
then pre-retirement verification found 317 chunks in the legacy environment that were in neither
view and re-queued them. It has refused retirement every 30 seconds since. Once #218 removes the
LMDB reader those 317 stop being servable, and on the evidence they are the network's only copy.
This must land before or with #218.

The fix

A third arm in the reconciliation block in ChunkStore::new, next to the two of the same shape:
when the marker says Committed and there are more legacy-only keys than the node ever approved
for shedding, go back to Bridging, clear committed_at_unix and reset rebuilds_since_commit.
open_legacy already computes the legacy-only set before the marker is read, so this is a
comparison against a number that is already in memory. No new I/O, no marker format change.

The count, not emptiness, is the test. A node that legitimately shed keeps exactly the keys it
is giving up in the legacy environment until they stop being answerable, so a non-empty
legacy-only set is its normal state; bouncing it to Bridging on every restart would reset its
retention clock each time and a regularly restarted node would never retire. The arm fires only
on legacy_only > shed_key_count. It logs migration_event = "back_to_bridging" with both
numbers so the case is visible in the beta watch.

The upgrade's own restart is enough to take an affected node out of the loop.

Linear issue

Closes V2-1232

Risk tier

  • T0 — docs / tooling / CI / pure UX-output. Repo CI only.
  • T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
  • T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
  • T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Compatibility

  • Wire: none.
  • Storage: no format change; shed_key_count is already in the marker. The only stores whose
    behaviour changes are those in the state described above, which move from Committed to
    Bridging on their next open and run the copier again. Every other store keeps its phase,
    clock and counts across the upgrade.
  • API: none. One private helper, legacy_only_count.

Semver impact

  • breaking
  • feature
  • fix

Test evidence

Local, on 5c7f942 plus this branch: cargo test --lib 1104 passed, 0 failed.
cargo fmt --all --check clean; cargo clippy --all-features --tests -- -D clippy::panic -D clippy::unwrap_used -D clippy::expect_used clean.

Test What it catches
a_node_that_committed_with_nothing_to_shed_resumes_the_copy_for_chunks_only_it_holds the reported state: commit with shed = 0, a chunk appears in the legacy environment only, reopen → Bridging, clock cleared, copier moves it
a_node_that_shed_stays_committed_across_a_restart the naive discriminator: a node with shed = 1, one legacy-only key, must stay Committed with its clock untouched

Both mutation-checked: with the discriminator replaced by legacy_only > 0, the regression test
and the existing the_migration_marker_survives_a_restart both fail; with the arm disabled, the
fire-case test fails.

Not covered here: a dev testnet run. The condition takes a node into a state that needs hours
of wall clock plus a specific failure to reach, which is why the tests use the store harness.

New dependency

none

ADR

https://github.com/jacderida/ant-node/blob/3b2b7093ae806a9ecc77b07b93e83eb860ff8621/docs/adr/ADR-0014-file-based-chunk-store-and-lmdb-retirement.md

Mitigation / rollback

Revert the commit. A node this arm moved back to Bridging then re-commits through the normal
path on its next pass, with shed_key_count recomputed from what is still legacy-only; nothing
on disk differs otherwise. The affected node returns to the loop it is in today.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UrjwPBwPqMi3F2sT1yVYEa

Once a node reaches `Committed`, the copier moves only the keys the rank check refuses to shed.
A chunk that turns up in the legacy environment after that, and that this node never agreed to
give up, is on no such list: nothing copies it, the shed gate correctly refuses to prove a
single copy exists elsewhere, and the startup reconciliation only returns a node to `Bridging`
when the file store holds less than it recorded keeping, which the file store never does. The
node is terminal in that state. Free disk does not change it and neither does a restart.

Seen on one node of 38 in the 2026-09-08 beta cohort: it committed with "nothing has to be shed",
then pre-retirement verification found 317 chunks in the legacy environment that were in neither
view and re-queued them. It has refused retirement every 30 seconds since. Once WithAutonomi#218 removes the
LMDB reader those 317 stop being servable, and on the evidence they are the network's only copy.

The fix is a third arm in the reconciliation block, next to the two of the same shape: when the
marker says `Committed` and there are more legacy-only keys than the node ever approved for
shedding, go back to `Bridging`, clear `committed_at_unix` and reset `rebuilds_since_commit`.
`open_legacy` already computes the legacy-only set before the marker is read, so this is a
comparison against a number that is already in memory. No new I/O, no marker format change.

The count, not emptiness, is the test. A node that legitimately shed keeps exactly the keys it
is giving up in the legacy environment until they stop being answerable, so a non-empty
legacy-only set is its normal state; bouncing it to `Bridging` on every restart would reset its
retention clock each time and a regularly restarted node would never retire. The arm fires only
on `legacy_only > shed_key_count`. It logs `migration_event = "back_to_bridging"` with both
numbers so the case is visible in the beta watch.

The upgrade's own restart is enough to take an affected node out of the loop.

Tests: `cargo test --lib` 1104 passed, 0 failed; fmt clean; clippy with `-D clippy::panic
-D clippy::unwrap_used -D clippy::expect_used` clean on lib and tests. Two tests added, and
both were mutation-checked: with the discriminator replaced by `legacy_only > 0`, the new
regression test and the existing `the_migration_marker_survives_a_restart` both fail; with the
arm disabled, the fire-case test fails.

Closes V2-1232

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrjwPBwPqMi3F2sT1yVYEa
@jacderida
jacderida changed the base branch from main to rc-2026.9.2 September 14, 2026 23:12

@dirvine dirvine left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head 3b2b7093ae806a9ecc77b07b93e83eb860ff8621.

The restart reconciliation is correctly scoped: it reopens Committed only when the current legacy-only cardinality exceeds the persisted shed allowance, while preserving the legitimate equal-count shed state. Destructive retirement remains protected by the exact approved-key-set checks.

Verification:

  • cargo test --lib: 1104 passed, 0 failed
  • both added regression tests passed
  • cargo fmt --all --check and git diff --check passed

Non-blocking: an equal-cardinality/key-substitution retirement-refusal test would document the identity-level safety backstop.

CI watch-outs unrelated to this diff: Security Audit is failing on newly disclosed rustls RUSTSEC-2026-0285 with unchanged Cargo.lock; the btrfs reclaim threshold missed marginally. Platform test jobs were still running at review time.

@jacderida
jacderida merged commit 2cf904c into WithAutonomi:rc-2026.9.2 Sep 14, 2026
22 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants