Skip to content

fix(storage): stop comparing allocated blocks of unsynced encoded artifacts - #1930

Merged
DecisionNerd merged 1 commit into
mainfrom
fix/1928-encoded-revalidate-allocation
Oct 9, 2026
Merged

DecisionNerd merged 1 commit into
mainfrom
fix/1928-encoded-revalidate-allocation

Conversation

@DecisionNerd

@DecisionNerd DecisionNerd commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Closes #1928

Cause

An encoded artifact is not synced when it is captured. ext4 delayed allocation changes st_blocks afterwards with no change to inode or content, and four places compared a fresh allocation measurement with one taken earlier:

Site Kind Change
CapturedEncodedArtifact::revalidate identity check on an unsynced file allocation compare removed; inode, named link, length and link authority remain
CapturedEncodedInventory::open (ledger vs fresh) identity check ledger presence admits the inode; allocation not compared
authenticate_encoded_artifact_identities (supersession sweep) identity check same
record_encoded_active_artifacts (retried encode) accounting an existing ledger entry keeps the allocation observed first

Content is still authenticated where it was: the commit boundary admits every installed object by exact length and XXH64, and the install checks the SHA-256 address. Nothing in this change weakens that.

Left alone, deliberately: ledger vs receipt comparisons in supersession, shape and progress (both sides are recorded values, not a fresh measurement), the gc.rs and private_storage_ownership.rs same-pass comparisons (not unsynced encoded artifacts), and restore_encoded_ledger, whose checkpoint digest covers recorded allocations (see below).

Tests (encoding_publication/tests/unsynced_allocation.rs)

  • captured_encoded_source_survives_allocation_change_with_unchanged_content: fallocate(KEEP_SIZE) past EOF changes st_blocks of a captured artifact with length, content and inode unchanged; revalidation, a fresh capture, the install, the supersession sweep and a retried encode record all hold.
  • captured_encoded_source_still_refuses_content_change_after_capture: with the allocation moved as well, an append is refused by revalidation and a same-length edit is refused by commit-boundary admission.

Mutation proof (each reverted afterwards):

  • Old encoding_publication.rs restored: test 1 fails with captured encoded source identity or length changed (the issue's message).
  • Old supersession.rs only: test 1 fails at the reclaim sweep.
  • Old io_evidence.rs only: test 1 fails with encoded artifact identity allocation changed.
  • Old ledger compare in open only: test 1 fails with captured encoded source allocation identity changed.
  • Length check removed from revalidate: test 2 fails (append installed).
  • XXH64 comparison in admit_checksum_file disabled: test 2 fails (same-length edit admitted).

Verification

  • make check: exit 0.
  • make test-rust ARGS="-p graphforge-storage": 1819 passed, 9 skipped.
  • make test-rust ARGS="-p graphforge-api": 1664 of 1665 passed. algorithm_runs::tests::subprocess_kill_reopens_as_exactly_one_interrupted_event failed with "child did not publish the start generation" at 192 s while my S20 loops and a sync loop were running (host load 20 to 30); it passes alone in 8.7 s.
  • S20 import-session builds (gf import-session begin, register nodes, register edges, validate, commit; 2^20 nodes, 2^24 edges, UUIDv7 Parquet inputs, release gf, separate target dirs, binaries differ by sha256), alternating base (03a39dd) and candidate: 12 plain pairs and 10 pairs with a concurrent sync -f loop forcing writeback. Candidate: 22 runs, 0 refusals. Base: 22 runs, 0 refusals.

Not reproduced on current main

The 5-in-6 failure was measured on 171120c, before #1910. I could not reproduce a refusal on 03a39dd with this driver, even with forced writeback. #1910 replaces the copy with a link, which shortens the capture-to-revalidate window, so the S20 count does not by itself discriminate this fix; the deterministic test does (it fails on main).

Open

restore_encoded_ledger (session reopen) still compares freshly measured allocations with the checkpoint's encoded-ledger digest, which records allocations. The same drift would refuse a reopen of a session whose encoded artifacts were not yet written back. I did not change it: the checkpoint evidence totals depend on those recorded values, so fixing it is a format and accounting decision. It did not fire in 44 S20 runs.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

…ifacts (#1928)

An encoded artifact is not synced when it is captured, and ext4 delayed
allocation changes st_blocks after capture with no change to its inode or
content. CapturedEncodedArtifact::revalidate, the capture in
CapturedEncodedInventory::open, the supersession sweep and the retried
encode record each compared a fresh allocation measurement with one taken
earlier, so bulk initial builds refused intermittently with "captured
encoded source identity or length changed".

Identity is now the inode, the named link, the length and link authority;
content stays authenticated by the XXH64 and SHA-256 checks the consumers
make against the inventory. The allocation ledger entry is accounting
recorded at encode: its presence admits the inode, and a re-record keeps the
value observed first.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Repository: CurateLabs/graphforge/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 02d8185e-83fb-4c54-b770-93fe161e587f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the core Core source code changes label Oct 9, 2026
@DecisionNerd
DecisionNerd added this pull request to the merge queue Oct 9, 2026
Merged via the queue into main with commit d48e6bb Oct 9, 2026
15 checks passed
@DecisionNerd
DecisionNerd deleted the fix/1928-encoded-revalidate-allocation branch October 9, 2026 00:52
DecisionNerd added a commit that referenced this pull request Oct 9, 2026
…facts

restore_encoded_ledger (session reopen) compared freshly measured allocated
bytes of each encoded artifact with the checkpoint's recorded allocations, via
encoded_ledger_sha256. Under ext4 delayed allocation st_blocks changes while
inode, length and content do not, so a reopen of a session whose encoded files
had not been written back could refuse. This is the reopen-side counterpart of
#1928 / #1930, which removed the same comparison from capture, open, the
supersession sweep and retried encodes.

- encoded_ledger_sha256 moves to the `graphforge-construction-encoded-ledger-v2`
  domain and hashes only the physical identity keys. Allocation is observed
  accounting.
- Reopen authenticates each artifact's inode (via the digest) and its length
  against the pinned inventory. It does not re-read payloads and does not judge
  links: content is authenticated at commit-boundary admission and the CAS
  install, links at supersession reclaim, as before.
- Reopen records the allocation it observes and reconciles the
  construction-staging category total and the historical peaks; without this
  the restored evidence fails "category authority differs from identity union".
- Pre-v1, in place: a checkpoint carrying an allocation-bearing (v1) digest is
  refused at reopen and left unmodified. Only unfinished constructions that had
  already omitted their encoded entries are affected; they restart from their
  original inputs. Documented in construction-supersession.md.

Tests: allocation drift (fallocate KEEP_SIZE) is accepted and the observed
allocation recorded; an append is refused with and without allocation drift;
a legacy digest is refused with the checkpoint unchanged; the digest ignores
allocation. One message expectation is updated, the old text named allocations.

Part of #1881

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core source code changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(storage): bulk initial builds fail intermittently because encoded-artifact revalidation compares allocated blocks

1 participant