perf(api): read import sources in place instead of copying them into the session

[DecisionNerd]

Closes #1898

Slice of #1881, decision 2. register_parquet no longer copies or fsyncs the source into the import session. It records the canonical path, device/inode, size, mtime and Parquet footer identity, and writes nothing under import-sessions/<uuid>/sources/. Both construction paths read the file where it is.

What changed

• In place. Every read re-establishes identity (name and open handle), size and mtime, checks again before each batch and at the end of the pass, and refuses a missing file (GF_NOT_FOUND) or a replaced, resized or modified one (GF_IDENTITY_CONFLICT). Existing registration refusals (symlink, .., non-regular, limits) are unchanged.
• One read for decode and digest. A Parquet ChunkReader offers every range the decoder asks for (footer included) to the source's SHA-256 as it is read. In-order bytes are hashed at once, early bytes are held (bounded; runs coalesce), and only bytes no decode asks for (the offset index) are read afterwards. The digest therefore names the bytes that were decoded. It is recorded in the session manifest and the receipt (ImportConstructionEvidence::source_provenance, new ImportSourceProvenance).
• Ordered task claim in the bulk builder (graphforge-storage): workers now take the lowest unclaimed task (claim_in_order) in the node, edge and scratch-edge passes instead of a static split. A static split starts each worker at a far-apart offset, so an in-order consumer would have to hold or re-read most of the file. Results are returned in task order. This replaces an earlier revision of this branch that digested the bulk path with a separate thread reading the whole file a second time.
• Historical sessions keep working. Manifest format 3 is written by new sessions; format 2 (a copy under sources/, no external identity or digest) still resumes, validates and appends, reading the copy it owns. Registering an in-place source into a format-2 session raises it to 3.
• Node binding doc comment and docs/book/architecture/resumable-import.md updated.

Evidence

• Counters, not prose: each_source_byte_is_read_once_for_hashing_and_decoding (public facade, both routes) asserts that the digest's own read (reread_bytes) equals exactly the unread index bytes computed from the Parquet metadata, and the decode's observed_bytes is within 2% of the file size. Mutation: making observe a no-op fails it and the staged unit test.
• No copy: registering_and_building_write_no_source_bytes uses the region write counters (registration wrote < 128 KiB for 5 MB of source; Arrow control writes the input into the session) and a content scan of the session tree. The same assertion run against origin/main fails: registering 5462214 bytes wrote 5463904.
• Refusals: deleted, replaced (inode), truncated, appended, touched, same-size rewrite (digest), at registration-to-validate (facade, both routes) and mid-build (hooked unit tests, both routes). Mutations removing the mtime and identity checks fail them.
• Historical session: resume, validate, append (in-place + Arrow), commit and a further append, on both routes, with the original file deleted.
• Claim order: tasks_are_claimed_in_index_order_on_every_pass fails against a static split (first starts were [0, 32, 16, 24, ...]).
• Same answers as an Arrow import of the same rows on both routes.

Review follow-up (P1 findings, each verified by a test first)

• Bulk resume, digest/graph mismatch: confirmed. A stop after the encoded inventory is pinned and before the digests are recorded, then a rewrite that keeps inode, size, mtime and footer, published graph A with the digest of B. Fix: an initial bulk build restarts (epic(storage): build initial graphs in parallel passes derived from the published generation #1881 decision 1) unless the digest of every in-place source is already recorded; a sealed construction session without one is discarded and rebuilt. A reused inventory keeps its recorded digests and the file is not hashed again.

• Staged resume, prefix/suffix mix: confirmed only for a rewrite that preserves the whole pin. A normal edit between stop and resume (deleted, replaced, truncated, appended, touched) is already refused before any staged or encoded progress is reused, on both routes: now covered by a_normal_edit_between_a_crash_and_resume_is_refused_before_progress_is_reused. The pin-preserving rewrite is not detected by design (pin = detector, digest = provenance); the receipt field and docs now say exactly that.

• Digest coverage: confirmed. A range the held-byte bound drops, and bytes no decode asks for, are read from the file when the digest completes. ImportSourceProvenance::sha256, the module docs and resumable-import.md now state the guarantee: SHA-256 of the file content read under the identity pin.

• Should-fix: registration opens the file once without following links and records from that handle (no symlink swap window); a read that fails because the file changed under it keeps the typed GF_IDENTITY_CONFLICT/GF_NOT_FOUND error; the held-byte bound claim was wrong (a slow task lets the others run ahead), so the claim-order and pending_limit docs now say it helps in the usual case and the bound is enforced by dropping ranges.

• Re-review residue. A source is pinned until it is fully consumed: the frame that marks a source staged also carries its complete SHA-256, so a fully staged source edited or deleted before resume publishes its original rows with its original digest, and commit after validate reads no source (tests for both routes). A footer corrupted after the open is reported as GF_IDENTITY_CONFLICT when the pin no longer matches.

• CI follow-up. Windows compiles (the footer-corruption test no longer uses std::os::unix). The certify runner accepts exactly the new receipt contract: construction.source_provenance (closed source kind, lowercase-hex SHA-256 digests, no path or name; nested or unknown shapes still refused) and the region work units observed_bytes and reread_bytes, in the validator and certification-evidence.json. The ownership-growth check treated the import session as a data owner that follows the input, true only while the input was copied; it now requires that owner to grow by less than half of proportionally and refuses a copy of the input.

S20 / S22 measurements (all CONTENDED)

Full commands, binary sha256s, loads and logs: #1898 (comment). Base c3551e1c4 vs candidate 7cfafbf20, separate release builds (cmp differs), inputs regenerated with the Graph500 generator (digests match the earlier recorded ones), two alternating pairs per scale.

• Region counters at S20: registering wrote 805,553,689 + 16,785,693 bytes into the session on base and 1,539 + 1,073 on the candidate; import-sessions/ held 822,338,612 bytes after registration on base and 1,539 on the candidate. The candidate's digest read 157,752 bytes of 822,337,680 itself.
• register-parquet (nodes+edges), contended: S20 4.290 -> 0.102 s and 2.935 -> 0.045 s; S22 27.065 -> 0.075 s and 14.472 -> 0.051 s.
• Five-command total, contended: S20 -16.1 s and -1.0 s; S22 -155.9 s and -40.8 s. These are within the host's noise (identical commit code varied 87-156 s between base runs) and are not attributable beyond the registration saving.

[greptile-apps]

Your trial has ended. Reactivate Greptile to resume code reviews.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Repository: CurateLabs/graphforge/.coderabbit.yaml
• Review profile: CHILL
• Plan: Advanced
• Run ID: ce637d8e-15f4-44e6-8d9d-acb93611b8d0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}