Audit: live-byte source-pin drift after merged replay-speed tooling (no new kappa)

[navinor]

Audit: live-byte source-pin drift after merged replay-speed tooling

Aaron Wang, with help from opencode (substantial AI assistance). Apache-2.0. No new kappa; no certificate, pin, frozen package or selected-frontier change.

Scope

Drawn card replay-runtime (track verifier-engineering, chooser OS nonce 5985f17ee9293abc98cbb69a85efb569), scope refined: the card asks whether one measured replay bottleneck can be sped up without dropping coverage. The ExclusionCircuit.verify speedups are already merged, so this audits their coverage-retention side effect and supplies the missing explicit dependency invalidation as a deterministic drift ledger with negative controls. Package: research/live-pin-drift-audit-seed5985/. Base 0fdd7d82de03e43c9e5a1000dfbcd380d1b687d2. Task card, scope status and reasons are in TASK.md and TASK.json.

What a pin means here

The audit enumerates every JSON {repo path: sha256} entry (exact rule recorded in RESULT.json) and separates two populations that are usually conflated:

• Live pins, consumed by a checker that hashes the current file: shape L, check_sources-style "files": {path: sha256} maps (for example research/skip-clones/SOURCE.json), and shape R-input, scripts/research_package.py SOURCES.json records with usage: "input", which that validator checks against working bytes ("working source drift").
• Receipts, which record a historical digest and are never compared to live bytes: usage: "provenance" records (shape R-provenance, bound to the immutable blob at the package's base commit) and all other {path: sha256} entries (shape P, for example the catalogue SHA-256 recorded in claims and ledgers). Drift in this population is expected and harmless.

Established (all recomputed by verify.py; full ledger in RESULT.json)

Shape	Pins	Match	Drift	Unresolved	Ignored
L (check_sources live)	20,737	19,735	127	46	829
R-input (research_package live)	1,841	1,814	22	0	5
R-provenance (historical blob)	1,363	806	517	0	40
P (provenance receipts)	2,457,086	2,379,080	16,673	2,215	59,118

Total 2,481,027 entries across 17,525 JSON manifests (2 manifests unreadable). The 149 live-drift pins touch 48 distinct files.

1. Smallest falsifying instance, reproduced exactly: research/skip-clones/clone_io.py:check_sources raises ValueError: Pinned inherited source changed: scripts/exclusion_circuit.py. Its manifest recorded b253f9d7318b..., equal to the file's blob at the manifest's own last commit c02590dd; the current bytes hash f9fbcd6e85e4.... The bytes moved in exactly two later commits: d8029ee0 ("Tooling: faster ExclusionCircuit.verify leave-two-out supports", blob 40bdb64670...) and d7d8bfcb ("Tooling: fold ExclusionCircuit.verify_embedding reverse masks into PR 2001", blob f9fbcd6e85..., Tooling: faster ExclusionCircuit and DisjointTensor support masks with identical receipts #2001 ). The package's make skip-clones-verify replay and tests/test_skip_clones.py cannot run at base. The speedups updated the shared certificate pins they touched (certificates/dyadic-central-factor.json and three siblings now record f9fbcd6e85... and verify clean) but left older frozen packages' live pins stale.
2. Registered rejections, reproduced exactly: scripts/research_package.py --check-only fails research/clean-selector-embedding and research/complex-frame-plane-rail-seedca04cc with working source drift: docs/research/agent-task-catalog.json (recorded 6686ad58f1..., current d46eb6d5b3..., changed by priority revision 4) and research/complex-cartesian-grid-screen-seed506af with working source drift: docs/research/bottleneck-brief.md (recorded a5e3165696..., current bbc9396c5c...). Of the 22 drifted R-input pins, 10 pin evolving documents (task catalogue, bottleneck brief or PR ledger) as usage: "input", so any later document update disables those replays; the other 12 pin tool scripts changed by the speedups above or later tooling.
3. Provenance is recorded for every live-drift pin in RESULT.json: for each (manifest, pinned path) pair, the manifest's last commit, the pinned file's blob digest at that commit, and every later commit touching the pinned file with its blob digest. Of the 149 pairs, 68 record a digest equal to the file's blob at the manifest's last commit; 70 record an older state (typical of vendored review snapshots); 11 pairs have no blob at that commit.

Reproduce

python3 -I -B research/live-pin-drift-audit-seed5985/verify.py

Expected and observed under the registered Python 3.12 replay:

PASS_LIVE_PIN_DRIFT_AUDIT pins=2481027 manifests=17525 L(match=19735,drift=127,unresolved=46,ignored=829) R-input(match=1814,drift=22,unresolved=0,ignored=5) R-prov(match=806,drift=517,unresolved=0,ignored=40) P(match=2379080,drift=16673,unresolved=2215,ignored=59118) live_drift_paths=48 controls=7 skip_clones=ValueError
PASS_REGISTERED_RESEARCH_PACKAGE research/live-pin-drift-audit-seed5985 outcome=completed

The run recomputes the whole ledger and requires exact equality with RESULT.json. Seven controls pass: fixed headline digest; a matching pin reports clean (the refreshed certificates/dyadic-central-factor.json pin of scripts/exclusion_circuit.py); a corrupted digest registers as drift; a missing target is unresolved; a non-repository key is ignored; a malformed "files" map is not treated as a live manifest; and the --write mode's independent second enumeration is byte-identical. python3 -B -m unittest discover -s tests -p 'test_research_*.py' passes (65 tests, OK, 5 skipped).

What this does not establish

No pin is repaired here: refreshing a frozen package's recorded digest would rewrite the evidence the pin protects, and vendoring freeze-time bytes would change frozen files. Both are maintainer decisions, so this PR stops at the exact ledger and the two reproductions. research/shared-point-digest and research/mixed-point-scalar-digest already record the decision as an open question ("Pin repair is outside this tooling change"), and this ledger is the data that follow-up needs. Drift disables replay checks; it does not falsify any mathematical certificate. The audit covers JSON {path: sha256} pins only, not Makefile targets, environment pins or non-JSON manifests. Two of 17,525 manifests were unreadable JSON and are counted, not audited. Drift counts for the receipt populations (P, R-provenance) are context, not defects.

Open next questions (full versions in the package README)

1. Pin-repair policy for the 149 live-drift pins: refreshed digest with old digest retained, vendored freeze-time bytes, or a replay declared historical and bound to its recorded commit. First step: one option on research/skip-clones alone, rerun tests/test_skip_clones.py. Blocked by: maintainer authorization to touch frozen pins or bytes.
2. Should usage: "input" be reserved for genuinely required live tools instead of evolving documents? Ten packages pin the catalogue or brief as live input, so every priority revision disables them. First step: one distinguishing sentence in docs/research/research-package-guide.md.
3. Run this deterministic audit in CI against a recorded baseline so only new live drift fails (the audit recomputes in about 100 seconds).

Overlap, credit and limits

Entry overlap and a pre-submission refresh (2026-10-15 02:17-02:19 UTC, 482 open PRs, 961 discussion records, complete pagination, terms "source pin drift", "working source drift", "pin repair", "Pinned inherited source changed", "exclusion_circuit") found no open PR or claim on this object; matches were this package, the predecessor tooling packages and the two recorded pin-repair open questions. Coverage is not a novelty certificate.

Predecessors: Chafik Boukhalfa (research/skip-clones, whose pinned replay is the smallest falsifying instance); Douglas Colkitt (scripts/research_package.py and its input/provenance distinction); the authors of research/exclusion-support-masks, research/exclusion-embedding-masks (#2001 ) and research/disjoint-tensor-support-masks (the merged speedups under audit, which already updated the shared certificate pins they touched); the authors of research/shared-point-digest and research/mixed-point-scalar-digest (the recorded "Pin repair is outside this tooling change" open questions); woklerx (research/git-provenance-stream, pin-validation tooling precedent). No predecessor code was imported; the verifier only executes research/skip-clones/clone_io.py and scripts/research_package.py as black-box reproductions of their own checks. Not run: the full historical replay suite, any cold supplier replay, remote CI, and any repair of the drifted pins.

[Mister-Meeseeks]

Maintainer review at 15836362a3cf17d245db8a01d930a12868c043bd: the complete frozen checkout reproduces the pin counts, drift lists and reproduction results, but the saved census fails equality: ledger.manifests_seen is 17,503, versus 17,525 recorded. All 17,507 tracked JSON files are present; excluding this package leaves 17,503. Please bind the inventory to tracked paths (or explicitly retain the additional 22 inputs) and refresh the count consistently. The README also alternates between 17,525 and 17,528.

There is a separate scope issue: JSON shape alone does not establish whether a consumer checks live bytes. For example, at this same frozen head, scripts/verify_pr_reconciliation.py:207 hashes every current path in ledger["source_sha256"]. Those entries fall into your P class, despite being enforced live pins. scripts/verify_replay_coverage.py:98 is another example. Please describe L/R/P as syntactic categories, distinguish the actually replayed failures from candidate drift, and remove the universal “never compared to live bytes”/“harmless” claims unless individual consumers have been audited. The reproduced failures are useful; this is a request to make the census reproducible and its interpretation precise, not a rejection of the drift finding.

[navinor]

Thank you for the review. Both findings were reproduced independently before any edit. New head: 9a067f8 (replaces 1583636).

What I reproduced at the reviewed head

The saved-census failure is real, and its root cause is case matching in the manifest enumeration, not untracked files. Running the package's own census twice against the saved RESULT.json (one run as written, one run with a case-sensitive .json suffix, which is what a case-sensitive platform enumerates):

A) verify.py walk (Path.rglob, case-insensitive on Windows): manifests_seen=17525 pins_seen=2481027 decode_failures=2 saved_equality=PASS differing_keys=none
B) case-sensitive .json suffix (reviewer platform equivalent): manifests_seen=17503 pins_seen=2481027 decode_failures=2 saved_equality=FAIL differing_keys=['manifests_seen']
  manifests_seen recomputed=17503 vs recorded=17525
tracked JSON manifests (case-insensitive extension): 17529
  of which this package (excluded from census): 4
  tracked manifests with case-variant .JSON extension: 22

So recomputation gives manifests_seen 17,503 versus 17,525 recorded exactly as you report, differing in that one field only; pins_seen 2,481,027, every count row and every drift list recompute equal.

The 22 inputs are not untracked: they are tracked manifests with an uppercase .JSON extension. Path.rglob('*.json') matches them on Windows but not on case-sensitive platforms, and a git ls-files -- '*.json' pathspec misses them everywhere. They are PREMERGE.JSON and SELECTION.JSON in each of research/community-efficient100-audit/ through research/community-efficient1000-audit/ (20 files), plus ENDING-OPEN.JSON and NEXT-QUEUE-SUMMARY.JSON in research/community-efficient500-audit/ (2 files). git ls-files research/community-efficient100-audit/ lists PREMERGE.JSON and SELECTION.JSON, and git check-ignore reports no rule for them. All 22 contribute zero pins, which is why the pin counts reproduce without them. Your tracked total of 17,507 uses the case-sensitive *.json query; the case-insensitive tracked total is 17,529, and 17,529 minus this package's 4 gives the 17,525 the census recorded.

The README's two numbers: 17,525 is the tracked external count above. The stray 17,528 equals 17,525 plus 3, and 3 is what this package itself carried at the time (RESULT.json, SOURCES.json, TASK.json; VERIFY.json is written last), consistent with a prose count that included part of this package. The frozen record cannot prove that decomposition, so the README now uses 17,525 throughout and records both the decomposition and the correction.

What I changed

1. Inventory bound to tracked paths (your preferred option), which here also explicitly retains the 22 inputs: walk_json() now enumerates git ls-files with the .json extension matched case-insensitively (this package still excluded). The census is now identical in every complete checkout on every platform. An eighth control requires the inventory to retain a tracked case-variant .JSON manifest and to equal manifests_seen.
2. Counts refreshed consistently: README 17,528 corrected to 17,525 with the decomposition above; RESULT.json regenerated, and its only content changes are the new control entry and the enumeration_rule text — every count (pins_seen, manifests_seen, all shape rows, drift lists) is unchanged, since the 22 retained manifests contribute zero pins; TASK.json and VERIFY.json refreshed to match (eight controls).
3. Scope fix for your second paragraph. L, R-input, R-provenance and P are now described as syntactic shapes only. The universal claims ("never compared to live bytes", "expected and harmless") are removed. Both of your counterexamples are confirmed at this head and cited in the package: scripts/verify_pr_reconciliation.py:207-208 hashes every current path in ledger["source_sha256"] and fails with Stale source: <path>, and scripts/verify_replay_coverage.py:98-99 hashes every current path in selected["source_sha256"] and fails with Selected coverage producer version differs: <name>. Drift outside the replayed set is now labelled candidate drift: 4 of the 149 live-drift pins are demonstrated failures (the pins named in the reproduced rejections: the skip-clones pin of scripts/exclusion_circuit.py and the three registered R-input pins). The other 145 live-drift pins and all P/R-provenance drift are recorded as candidate drift: no consumer is replayed against them and no harmlessness is claimed.
4. Wording demotions: "which checkers consequently reject" became "which checker rejections can be reproduced from those mismatches"; "which exact commits invalidated each pin" became "which later commits touched each pinned file after its manifest froze". "reproduced exactly" is kept only for the two reproduced items and the git-derived two-commit attribution of the scripts/exclusion_circuit.py bytes.

Fresh verification at the new head (CPython 3.12 on Windows)

Registered replay, python3.12 -I -B research/live-pin-drift-audit-seed5985/verify.py, in the working checkout:

PASS_LIVE_PIN_DRIFT_AUDIT pins=2481027 manifests=17525 L(match=19735,drift=127,unresolved=46,ignored=829) R-input(match=1814,drift=22,unresolved=0,ignored=5) R-prov(match=806,drift=517,unresolved=0,ignored=40) P(match=2379080,drift=16673,unresolved=2215,ignored=59118) live_drift_paths=48 controls=8 skip_clones=ValueError

The same replay in a complete frozen checkout (detached worktree at 9a067f8, 43,088 tracked files, clean status — the scenario from your review):

PASS_LIVE_PIN_DRIFT_AUDIT pins=2481027 manifests=17525 L(match=19735,drift=127,unresolved=46,ignored=829) R-input(match=1814,drift=22,unresolved=0,ignored=5) R-prov(match=806,drift=517,unresolved=0,ignored=40) P(match=2379080,drift=16673,unresolved=2215,ignored=59118) live_drift_paths=48 controls=8 skip_clones=ValueError

That run recomputes the whole ledger and requires exact equality with the saved RESULT.json, so the census equality holds in the frozen checkout: manifests_seen 17,525 recomputed against 17,525 recorded.

python3.12 -I -B scripts/research_package.py --package research/live-pin-drift-audit-seed5985 --check-only:

PASS_PACKAGE_METADATA_AND_PINS (not a mathematical acceptance check)

python3.12 -I -B tests/test_skip_clones.py (the recorded obstruction, unchanged by this fix and not a regression): 8 tests run, 1 error in setUpClass, ValueError: Pinned inherited source changed: scripts/exclusion_circuit.py.

Aaron Wang, with help from opencode.