Repository navigation
fix(storage): stop comparing allocated blocks of unsynced encoded artifacts - #1930
Conversation
…ifacts (#1928) An encoded artifact is not synced when it is captured, and ext4 delayed allocation changes st_blocks after capture with no change to its inode or content. CapturedEncodedArtifact::revalidate, the capture in CapturedEncodedInventory::open, the supersession sweep and the retried encode record each compared a fresh allocation measurement with one taken earlier, so bulk initial builds refused intermittently with "captured encoded source identity or length changed". Identity is now the inode, the named link, the length and link authority; content stays authenticated by the XXH64 and SHA-256 checks the consumers make against the inventory. The allocation ledger entry is accounting recorded at encode: its presence admits the inode, and a re-record keeps the value observed first. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Warning Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…facts restore_encoded_ledger (session reopen) compared freshly measured allocated bytes of each encoded artifact with the checkpoint's recorded allocations, via encoded_ledger_sha256. Under ext4 delayed allocation st_blocks changes while inode, length and content do not, so a reopen of a session whose encoded files had not been written back could refuse. This is the reopen-side counterpart of #1928 / #1930, which removed the same comparison from capture, open, the supersession sweep and retried encodes. - encoded_ledger_sha256 moves to the `graphforge-construction-encoded-ledger-v2` domain and hashes only the physical identity keys. Allocation is observed accounting. - Reopen authenticates each artifact's inode (via the digest) and its length against the pinned inventory. It does not re-read payloads and does not judge links: content is authenticated at commit-boundary admission and the CAS install, links at supersession reclaim, as before. - Reopen records the allocation it observes and reconciles the construction-staging category total and the historical peaks; without this the restored evidence fails "category authority differs from identity union". - Pre-v1, in place: a checkpoint carrying an allocation-bearing (v1) digest is refused at reopen and left unmodified. Only unfinished constructions that had already omitted their encoded entries are affected; they restart from their original inputs. Documented in construction-supersession.md. Tests: allocation drift (fallocate KEEP_SIZE) is accepted and the observed allocation recorded; an append is refused with and without allocation drift; a legacy digest is refused with the checkpoint unchanged; the digest ignores allocation. One message expectation is updated, the old text named allocations. Part of #1881 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Closes #1928
Cause
An encoded artifact is not synced when it is captured. ext4 delayed allocation changes
st_blocksafterwards with no change to inode or content, and four places compared a fresh allocation measurement with one taken earlier:CapturedEncodedArtifact::revalidateCapturedEncodedInventory::open(ledger vs fresh)authenticate_encoded_artifact_identities(supersession sweep)record_encoded_active_artifacts(retried encode)Content is still authenticated where it was: the commit boundary admits every installed object by exact length and XXH64, and the install checks the SHA-256 address. Nothing in this change weakens that.
Left alone, deliberately: ledger vs receipt comparisons in supersession, shape and progress (both sides are recorded values, not a fresh measurement), the
gc.rsandprivate_storage_ownership.rssame-pass comparisons (not unsynced encoded artifacts), andrestore_encoded_ledger, whose checkpoint digest covers recorded allocations (see below).Tests (
encoding_publication/tests/unsynced_allocation.rs)captured_encoded_source_survives_allocation_change_with_unchanged_content:fallocate(KEEP_SIZE)past EOF changesst_blocksof a captured artifact with length, content and inode unchanged; revalidation, a fresh capture, the install, the supersession sweep and a retried encode record all hold.captured_encoded_source_still_refuses_content_change_after_capture: with the allocation moved as well, an append is refused by revalidation and a same-length edit is refused by commit-boundary admission.Mutation proof (each reverted afterwards):
encoding_publication.rsrestored: test 1 fails withcaptured encoded source identity or length changed(the issue's message).supersession.rsonly: test 1 fails at the reclaim sweep.io_evidence.rsonly: test 1 fails withencoded artifact identity allocation changed.openonly: test 1 fails withcaptured encoded source allocation identity changed.revalidate: test 2 fails (append installed).admit_checksum_filedisabled: test 2 fails (same-length edit admitted).Verification
make check: exit 0.make test-rust ARGS="-p graphforge-storage": 1819 passed, 9 skipped.make test-rust ARGS="-p graphforge-api": 1664 of 1665 passed.algorithm_runs::tests::subprocess_kill_reopens_as_exactly_one_interrupted_eventfailed with "child did not publish the start generation" at 192 s while my S20 loops and a sync loop were running (host load 20 to 30); it passes alone in 8.7 s.gf import-sessionbegin, register nodes, register edges, validate, commit; 2^20 nodes, 2^24 edges, UUIDv7 Parquet inputs, releasegf, separate target dirs, binaries differ by sha256), alternating base (03a39dd) and candidate: 12 plain pairs and 10 pairs with a concurrentsync -floop forcing writeback. Candidate: 22 runs, 0 refusals. Base: 22 runs, 0 refusals.Not reproduced on current main
The 5-in-6 failure was measured on 171120c, before #1910. I could not reproduce a refusal on 03a39dd with this driver, even with forced writeback. #1910 replaces the copy with a link, which shortens the capture-to-revalidate window, so the S20 count does not by itself discriminate this fix; the deterministic test does (it fails on main).
Open
restore_encoded_ledger(session reopen) still compares freshly measured allocations with the checkpoint's encoded-ledger digest, which records allocations. The same drift would refuse a reopen of a session whose encoded artifacts were not yet written back. I did not change it: the checkpoint evidence totals depend on those recorded values, so fixing it is a format and accounting decision. It did not fire in 44 S20 runs.🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.