Tooling: schedule isolated test modules concurrently with identical receipts (3.3-3.9x faster verify-tests, no new kappa)

research/agent-07-verifeng-seed88/README.md

@@ -0,0 +1,142 @@
+# Concurrent scheduling for the isolated test runner (verification engineering)
+
+Track 85–94 (verification engineering), assignment seed 88. Base commit
+`f567884aeb8851b6e794af1184119a3bfbe1e631`. Prepared with help from opencode.
+No new kappa claim; this is a tooling improvement only.
+
+## What changed
+
+`scripts/run_isolated_tests.py` (run by `make verify-tests`, hence by
+`make verify`) ran each test module in a fresh interpreter, sequentially. A
+timing survey of the `make verify` checks on the base commit found this
+runner to be by far the most expensive single check: it exceeded a 250 s
+sweep budget and measured 298.5 s end to end, while every other individual
+check surveyed was at most 144 s (`scripts/audit_quadratic_affine.py`),
+most below 10 s.
+
+The runner now schedules the same per-module commands concurrently, one
+isolated interpreter per module, and replays each module's captured
+stdout/stderr bytes in sorted module order. The point of the original
+design — one fresh interpreter per test module, so research packages with
+colliding module names (`verify`, `skip_graph`, `data_recovery`, ...) cannot
+bind each other's imports — is unchanged: process isolation is exactly what
+makes the modules independent and safely concurrent. A test-module audit
+found no writes outside `tempfile.TemporaryDirectory()` blocks (see
+"Exactness" below), so concurrent modules share no mutable state.
+
+A `--workers N` flag pins the concurrency width (default: CPU count). With
+no arguments the command line, the per-module commands, the summary lines
+and the exit behavior are those of the pre-change runner.
+
+## Measured result (same host, same input)
+
+| Run | Modules | Wall time |
+|---|---|---|
+| Pre-change runner, base commit, end to end | 87 (its test file not yet added) | 298.5 s |
+| Benchmark sequential pass (pre-change schedule, captured) | 88 | 250.6 s |
+| Benchmark concurrent pass (`run_modules`, 16 workers) | 88 | 75.9 s (3.30x) |
+| Post-change runner end to end | 88 | 76.7 s (3.9x vs. the base run) |
+
+The sequential baseline differs from the shipped old script only in pipe
+capture; commands and ordering are identical. The end-to-end comparison
+covers 87 vs. 88 modules because `tests/test_isolated_test_runner.py` is
+part of this change; the extra module accounts for roughly a second on
+either side. Host: 16 logical CPUs, CPython 3.13.15 on Windows; the
+speedup is bounded by the slowest single module (tail effect), so it scales
+with core count up to that floor.
+
+## Exactness
+
+The change alters scheduling only. Per module, the command is
+byte-identical to the pre-change one:
+
+    python -m unittest discover -s <tests> -p <module> -v
+
+with the same interpreter, working directory and environment. The runner
+replays each module's captured bytes in sorted module order, so the
+combined stdout and stderr streams equal a sequential run module for
+module.
+
+Equivalence is machine-checked by `benchmark.py`: it runs every module with
+the pre-change sequential schedule and again with the concurrent scheduler,
+then compares per-module exit codes and stdout/stderr SHA-256. Measured on
+88 modules:
+
+- exit codes: identical for all 88; the failing-module sets of the two
+  schedules are equal (the same 15 modules fail in both);
+- stdout: byte-identical for all 88;
+- stderr: byte-identical for 87 after normalizing the wall-clock field
+  `unittest -v` embeds in its `Ran N tests in T.TTTs` line (raw stderr
+  differs in every module for exactly that field);
+- `test_frontier799_bindings.py` is inherently nondeterministic across any
+  two runs: its failing traceback embeds a machine-random temporary
+  directory path (`...\Temp\tmpXXXX\ancestor\bit`). The benchmark proves
+  this automatically by re-running any mismatched module twice under the
+  same schedule; `scheduling_driven_mismatches` is empty.
+
+Coverage is unchanged: the same glob (`tests/test*.py`), the same command
+per module, and every module's exit code still contributes to the shared
+`Failed modules: ...` failure summary and the
+`PASS all N test modules in isolated interpreters` success line.
+
+## Negative controls
+
+`tests/test_isolated_test_runner.py` runs the real runner end to end in a
+sandbox (a copied runner plus a tiny `tests/` directory):
+
+1. Two passing and one deliberately failing module: every module runs and
+   the failure is reported in the shared summary (failure detection intact).
+2. A barrier test: three modules each write a marker and then wait (8 s)
+   for three markers to exist. They only all pass if at least three module
+   processes run concurrently. Against the pre-change runner this test
+   fails with "barrier timeout: test modules did not run concurrently"; it
+   passes with `--workers 3` after the change.
+3. Default-invocation compatibility: unchanged summary lines and exit
+   code 1 on failure.
+
+The benchmark itself is a negative control on exactness: any exit-code or
+scheduling-driven receipt change makes it fail.
+
+## Reproduce
+
+    python3 -B research/agent-07-verifeng-seed88/benchmark.py
+
+Expected final line (numbers above it may vary with core count):
+
+    PASS identical exit codes (88 modules); receipts identical for 87 modules modulo the unittest timing field; 1 module(s) inherently nondeterministic across same-mode runs: ['test_frontier799_bindings.py']
+
+    python3 -B scripts/run_isolated_tests.py
+
+Expected on a host with the full Unix test surface: per-module
+`Checking ...` lines and `PASS all 88 test modules in isolated interpreters`.
+Measured on this Windows host: exit 1 with the pre-existing 15-module
+failing set (`Failed modules: ...`), 88 `Checking` lines, 76.7 s; the same
+set fails at the base commit.
+
+## Scope and inherited obligations
+
+- The 15 failing test modules on this Windows host are environmental and
+  present identically before and after this change (for example
+  `ModuleNotFoundError: No module named 'resource'`, a Unix-only
+  standard-library module, in the `copied_*` tests). This package does not
+  fix Windows portability of the test suite; the claim is that the
+  concurrent runner reproduces the sequential runner's per-module receipts,
+  exit codes and failing set exactly.
+- This changes no mathematical content: no certificate, patch, generator,
+  proof supplement or pinned source file is touched.
+  `run_isolated_tests.py` is not hash-pinned by
+  `certificates/selected-result.json`,
+  `research/community-frontier799-audit/SELECTED-SOURCE.json` or any
+  checkpoint record.
+- Unresolved: live interleaved output during a run is replaced by
+  per-module replay when each module finishes (still in sorted order). A
+  hung module stalls the printed stream where a sequential run would, but
+  earlier finished modules wait for their turn to print.
+
+## Attribution
+
+Written with help from opencode. The runner's original design and the
+test-suite content are repository predecessors; this package only changes
+scheduling and adds evidence. Related concurrent work on other targets
+includes PR #853 (concurrent pinned stages for the #741 replay); this
+result does not touch or depend on it.

research/agent-07-verifeng-seed88/SPRINT-CLAIM.md

@@ -0,0 +1,19 @@
+# Sprint claim — agent-07 (recorded locally; no pinned coordination issue exists)
+
+- Track: 85–94 Verification engineering
+- Assignment seed: 88 (OS randomness via Python `secrets.randbelow(100)`)
+- Base commit: f567884aeb8851b6e794af1184119a3bfbe1e631 (origin/main of CrocSwap/integer-mult-bounds)
+- Branch: research/agent-07-verifeng-seed88
+- Object under investigation: one costly `make verify` check not covered by any open
+  verification-engineering PR (survey taken 2026-10-13; open tooling PRs already cover
+  data-corner replay #903, sparse-fusion #900, Leech minors #898, joint-frame #871,
+  #741 concurrent stages #853, PR799 dyadic fusion #849, stage-pair #824,
+  signed-sparse #833, R420 RREF #818, cyclic routing #808, PR626 gzip portability #819,
+  frontier741 receipts #847/#838, PR799 evidence coverage #865).
+- Method: profile the remaining checks; pick the slowest untargeted one; optimize its
+  hot loop with exact arithmetic preserved (identical stdout/receipts), or make it
+  reproducible; benchmark before/after on the same input; keep exactness and
+  failure-detection coverage.
+- Expected deliverable: a tooling PR with before/after wall times and byte-identical
+  receipts, or a scoped reproducibility improvement.
+- Expected finish: same session (2026-10-13).

research/agent-07-verifeng-seed88/benchmark.py

@@ -0,0 +1,140 @@
+#!/usr/bin/env python3
+"""Before/after benchmark and receipt-equivalence check for run_isolated_tests.
+
+Before: runs every test module with the exact sequential command the
+pre-change runner used, capturing each module's stdout/stderr bytes.
+After: runs the concurrent scheduler from scripts/run_isolated_tests.py
+(run_modules) over the same modules with the same commands.
+
+Then it compares per-module exit codes and byte streams. unittest -v
+embeds a wall-clock field ("Ran N tests in T.TTTs") in stderr, so stderr
+is compared both raw and with that single field normalized; stdout is
+compared raw. The failing-module sets must match exactly.
+
+Usage: python3 -B research/agent-07-verifeng-seed88/benchmark.py
+Writes evidence/benchmark.json next to this file.
+"""
+from hashlib import sha256
+from importlib.util import module_from_spec, spec_from_file_location
+import json
+import os
+from pathlib import Path
+import re
+import subprocess
+import sys
+import time
+
+ROOT = Path(__file__).resolve().parents[2]
+HERE = Path(__file__).resolve().parent
+TIMING = re.compile(r'in \d+\.\d+s')
+
+
+def load_runner():
+    spec = spec_from_file_location('run_isolated_tests', ROOT / 'scripts' / 'run_isolated_tests.py')
+    module = module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+def digest(data):
+    return sha256(data).hexdigest()
+
+
+def normalize(data):
+    return TIMING.sub('in T', data.decode('utf-8', errors='replace')).encode()
+
+
+def record(results):
+    out = {}
+    for path, code, stdout, stderr in results:
+        out[path.name] = {
+            'code': code,
+            'stdout_sha256': digest(stdout),
+            'stderr_sha256': digest(stderr),
+            'stderr_normalized_sha256': digest(normalize(stderr)),
+        }
+    return out
+
+
+def main():
+    runner = load_runner()
+    files = sorted((ROOT / 'tests').glob('test*.py'))
+    workers = min(len(files), os.cpu_count() or 1)
+
+    sequential = []
+    start = time.perf_counter()
+    for path in files:
+        result = subprocess.run(
+            runner.module_command(sys.executable, ROOT / 'tests', path.name),
+            cwd=ROOT, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        sequential.append((path, result.returncode, result.stdout, result.stderr))
+    seq_wall = time.perf_counter() - start
+
+    start = time.perf_counter()
+    concurrent = runner.run_modules(files, workers=workers)
+    par_wall = time.perf_counter() - start
+
+    seq_rec, par_rec = record(sequential), record(concurrent)
+    mismatches = []
+    code_mismatches = []
+    raw_equal = normalized_equal = 0
+    for path in files:
+        s, p = seq_rec[path.name], par_rec[path.name]
+        if s['code'] != p['code']:
+            code_mismatches.append(path.name)
+        if s['stdout_sha256'] == p['stdout_sha256'] and s['stderr_sha256'] == p['stderr_sha256']:
+            raw_equal += 1
+        if s['stdout_sha256'] == p['stdout_sha256'] and s['stderr_normalized_sha256'] == p['stderr_normalized_sha256']:
+            normalized_equal += 1
+        else:
+            mismatches.append(path.name)
+
+    # A module whose receipts differ between schedules is acceptable only if its
+    # output is nondeterministic across two runs of the SAME schedule as well
+    # (for example a failing traceback embedding a random temporary path).
+    inherent = []
+    scheduling_driven = []
+    for name in mismatches:
+        runs = []
+        for _ in range(2):
+            result = subprocess.run(
+                runner.module_command(sys.executable, ROOT / 'tests', name),
+                cwd=ROOT, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+            runs.append((result.returncode, result.stdout, result.stderr))
+        same = (record([(Path(name),) + runs[0]])[name] == record([(Path(name),) + runs[1]])[name])
+        (inherent if not same else scheduling_driven).append(name)
+
+    evidence = {
+        'runner': 'scripts/run_isolated_tests.py',
+        'baseline': 'per-module sequential command of the pre-change runner (identical commands, sequential scheduling)',
+        'python': sys.version.split()[0],
+        'cpu_count': os.cpu_count(),
+        'workers': workers,
+        'modules': len(files),
+        'sequential_wall_seconds': round(seq_wall, 2),
+        'concurrent_wall_seconds': round(par_wall, 2),
+        'speedup': round(seq_wall / par_wall, 2) if par_wall else None,
+        'raw_equal_modules': raw_equal,
+        'normalized_equal_modules': normalized_equal,
+        'code_mismatches': code_mismatches,
+        'receipt_mismatches': mismatches,
+        'inherently_nondeterministic': inherent,
+        'scheduling_driven_mismatches': scheduling_driven,
+        'sequential_failing': sorted(n for n, r in seq_rec.items() if r['code']),
+        'concurrent_failing': sorted(n for n, r in par_rec.items() if r['code']),
+        'per_module_sequential': seq_rec,
+        'per_module_concurrent': par_rec,
+    }
+    (HERE / 'evidence').mkdir(exist_ok=True)
+    (HERE / 'evidence' / 'benchmark.json').write_text(json.dumps(evidence, indent=1) + '\n')
+    print(json.dumps({k: v for k, v in evidence.items()
+                      if not k.startswith('per_module')}, indent=1))
+    if code_mismatches or scheduling_driven:
+        raise SystemExit('scheduling changed behavior: %r %r' % (code_mismatches, scheduling_driven))
+    print('PASS identical exit codes (%d modules); receipts identical for %d modules '
+          'modulo the unittest timing field; %d module(s) inherently nondeterministic '
+          'across same-mode runs: %r' % (len(files), normalized_equal, len(inherent), inherent))
+
+
+if __name__ == '__main__':
+    main()