Tooling: cross-platform byte/codec reproducibility of the make verify surface (no new kappa)

[navinor]

Tooling: cross-platform byte/codec reproducibility of the make verify surface

Status: tooling improvement (verification reproducibility). Track 85–94 (verification engineering), OS random seed 85. Base commit 91a2094f2e022930069e8cfdc810a3ea1b81cd09. No new kappa; no mathematical claim, certificate value, patch payload, or the selected frontier changes. Prepared with help from opencode.

Findings at the base commit (Windows 11, CPython 3.13.15, cp1252)

1. ranked-pair-check crashes: research/ranked-pair-verification/verify.py decoded lean/CappedMass.lean / lean/ConcreteAssembly.lean with Path.read_text() and no encoding=, so the locale codec applied (UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d). The check passed on UTF-8 hosts and crashed elsewhere — an outcome that depended on the host locale.
2. machine-transfer-check crashes: same root cause in research/machine-transfer-verification/next-proof/analytic/check.py (row_region.count("⟨") counted 0 against cp1252 mojibake → Unexpected literal row constructors). The 240-file package is byte-frozen by its own publication-manifest.json, so its sources cannot be edited without invalidating the reviewed publication bytes.
3. Every regenerated certificate was byte-different: scripts/ generators wrote certificates through unpinned text writes, so '\n' was translated to os.linesep. 39 certificates regenerated by the audited checks differed from their tracked bytes in every line, while being equal after newline normalization (39/39 verified). Byte-level receipts (sha256 of a regenerated artifact) diverge from the Linux reference this way.

A static inventory found 7157 locale-dependent text I/O sites repo-wide at the base commit; 150 write sites are in the scripts/ make surface (recorded in research/agent-08-verifeng-seed85/evidence/locale-sites.json).

Change

• Makefile: export PYTHONUTF8=1 for all recipes. Identical behavior on Linux; this is what lets machine-transfer-check pass without touching its publication-frozen sources.
• research/ranked-pair-verification/verify.py: explicit encoding='utf-8' on its three text reads (not hash-pinned anywhere), so direct invocation also works.
• scripts/: encoding='utf-8', newline='\n' on 139 text writes across 126 files, applied mechanically by the committed ast codemod research/agent-08-verifeng-seed85/pin_text_writes.py --fix (one keyword pair per call; nothing else changes).
• 27 certificates regenerated; machine-checked digest-only diffs (see below).
• tests/test_locale_reproducibility.py: four regression guards.

Frozen sources are not touched

The codemod refuses to modify a script named by path in certificates/selected-result.json, certificates/checkpoints/*.json, or SELECTED-SOURCE.json (9 frontier drivers), or whose sha256 appears in a frozen docs/research/*.json receipt. The second rule caught three first-pass edits, which were reverted: scripts/audit_community_candidate.py (pinned by docs/research/community-followup-arithmetic.json, which audit_followup_candidate.py --check compares against — the check failed until the revert) and scripts/verify_frontier617.py / verify_frontier617_bindings.py (tested-source digests in docs/research/community-cold617-integration.json). 43 scripts are skipped as frozen in total. docs/research/preserved-research-manifest.json records original_sha256 — bytes as of its historical base_commit — and does not freeze current sources.

Reproduce and verify

python3 -B -m unittest discover -s tests -p 'test_locale_reproducibility.py' -v

Expected: Ran 4 tests ... OK. The guards run ranked-pair-check under a hostile locale (LC_ALL=C, no UTF-8 mode) and require PASS ranked pair; run machine-transfer-check the way make does (UTF-8 mode pinned) and require PASS machine-transfer offline checkpoint plus the Makefile pin; regenerate certificates/cube-core-family.json with a real generator (scripts/audit_cube_cores.py) and require byte identity; and require pin_text_writes.py --check to report 0 unpinned text writes.

python3 -B research/agent-08-verifeng-seed85/pin_text_writes.py --check
python3 -B research/agent-08-verifeng-seed85/locale_inventory.py

Measured before/after (same host, same inputs)

Check	Before	After
ranked-pair-check (hostile locale)	exit 1 UnicodeDecodeError	exit 0 PASS ranked pair (1.3 s)
machine-transfer-check (PYTHONUTF8=1)	exit 1 Unexpected literal row constructors	exit 0 PASS machine-transfer offline checkpoint
Regenerated certificates	39/39 byte-different (CRLF every line)	27 changed, digest-only diffs (recorded source_sha256 of edited generators); 0 content, 0 newline differences
Isolated test suite	15-module environmental failing set (recorded at base by PR #921's evidence)	identical 15-module set; no test outcome changed

Timing evidence for the audited checks is in research/agent-08-verifeng-seed85/evidence/ (survey-baseline.json pre-fix, survey-after-fix.json post-fix; regeneration-receipt.json records the diff analysis and commands).

Scope and unresolved obligations

• Only generator text-write keywords, three read call sites, one Makefile export, regenerated source-digest fields, and new tests changed. No certificate value, constraint, kappa, or proof source differs.
• machine-transfer-check invoked directly (bypassing the Makefile) still fails on non-UTF-8 hosts: its checker sources are frozen by publication-manifest.json. make verify runs it correctly.
• Residual by design: 14 open() calls in scripts/ with non-literal modes (reported, left alone), the 43 frozen scripts, and research/-side package generators, which can still emit CRLF-translated output on Windows. The repo-wide inventory lists the sites.
• Not re-run here (no numpy/sympy/scipy or C++ toolchain on this host): gen4-bank-verify, the frontier741/799-verify chains, prime_field_network and audit_positive_skip_exact (compile C++ producers), Lean builds, and the 15 environmental Windows test modules (e.g. resource imports). None of their bytes changed.

Overlap and predecessor work

Surveyed 716 open PRs before submission; none targets locale/codec/newline reproducibility (the closest, PR 819, makes one gzip artifact portable by a different method on a different object). Concurrent tooling PR 849/#853/PR 903/#921 and the earlier PR 752/PR 778/PR 808/#824 accelerate other checks and do not touch these surfaces; PR #921's benchmark evidence supplies the base failing set compared against here (credit: Aaron Wang), and the doctrine follows Maxime Fleury's gzip-container reproducibility audit ("compare canonical payloads across environments"). The repository's verification suite, generators and frozen packages are the predecessors; this change only pins how their text is decoded and encoded.

Prepared for the coordinated research sprint by agent-08, with help from opencode. Apache-2.0. No independent human review is claimed. Finite tooling checks do not discharge the inherited all-size/compiler, hosting, precision/tape and analytic hypotheses, and no full Lean multiplication theorem is claimed.