Skip to content

CI: parallelize the recycled-bit frame-pair checks (~2x on the critical path) - #218

Closed
GamingPuzzled wants to merge 1 commit into
CrocSwap:mainfrom
GamingPuzzled:ci/parallel-recycled-bit
Closed

GamingPuzzled wants to merge 1 commit into
CrocSwap:mainfrom
GamingPuzzled:ci/parallel-recycled-bit

Conversation

@GamingPuzzled

Copy link
Copy Markdown
Contributor

CI speedup only. No mathematical change and no new κ.

Why

verify-recycled-bit is the critical path of the verify workflow. On a recent run of #210 it took 1,009 s (3.11), 842 s (3.14) and 725 s (3.13), while every other group finished in under 10 minutes. Profiling research/recycled-bit-integration/verify.py --all shows setup takes 3 s and almost all the remaining time is two single-threaded loops:

  • 4,878 new frame pairs: nesting and nondegeneracy (about 176 s serial);
  • every distinct inherited ledger move: nesting (about 9 minutes serial).

Change

Each check is a pure function of the shared Exact frames. check_moves splits the moves across forked worker processes and returns every move that fails; the verifier asserts that list is empty.

  • The checks are the same exact.inside and exact.nondegenerate calls as before. Only which process evaluates them changes.
  • Moves are grouped by target key, so each worker builds each annihilator basis once.
  • VERIFY_JOBS sets the worker count; the default is the CPU count (4 on GitHub's Linux runners).
  • With one job, or where fork is unavailable, it runs the original serial loop.

certificate.json changes only in the pinned SHA-256 of verify.py, which the verifier hashes into its own certificate:

-    "research/recycled-bit-integration/verify.py": "25a31677…e218",
+    "research/recycled-bit-integration/verify.py": "a5508f42…a0b9",

docs/research/community-round7-validation.json records the old certificate hash as a historical validation record. No script reads it, so it is left as written.

Measured locally (Python 3.11, 4 workers)

verify.py --all
before (serial) 12 min 23 s
after 5 min 59 s

The --all --write run passed every check and reproduced the certificate apart from the hash line. test_recycled_bit_integration.py passes (3 tests). On CI this should bring the group down to about 6 minutes, making the research group and formal (historical) (about 10–11 minutes each) the new critical path, for an estimated ~17 → ~11 minutes wall time.

I also tried precomputing each distinct frame basis once across workers instead. It was slower (7 min 14 s), because the basis keys depend on each other and don't split evenly, so the simpler chunking is used.

Possible follow-ups, for the maintainers to decide: caching the project's Lean build when formal/lean is unchanged (567 s of the 671 s formal (historical) job), and running independent verify-research targets in parallel after checking they don't write the same files.

Prepared by Joel Pulikkan with Anthropic Claude assistance.

🤖 Generated with Claude Code

The verify-recycled-bit group is the critical path of the verify workflow
(up to ~17 minutes). Its time is two single-threaded loops of exact
frame-pair checks. Each check is a pure function of the shared exact
frames, so split the moves across forked workers (VERIFY_JOBS, default
the CPU count; serial without fork or with one job). Every check is
unchanged. Locally with 4 workers: 12m23s -> 5m59s. certificate.json
changes only in the pinned sha256 of verify.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants