Tooling: 1.88× faster production joint-dual replay via containment masks with identical receipts

certificates/joint-dual-kappa.json

@@ -835,14 +835,14 @@
     "scripts/experiments/binary_frame_math.py": "1f96e8aafc8ae89f8eeec5d03800131d93d58e01bf8a7fca8feb03702d1740d0",
     "scripts/experiments/binary_frame_profile_prepare.py": "16dea3d3ef3dc90db928ed3e35d002f6926d2e7388da4766fc2fd725550f739e",
     "scripts/experiments/binary_frame_profiles.cpp": "8e2e391d3ee57cbb1372c68e4aa031e1b861321536eabd642ada54e12e678669",
-    "scripts/experiments/binary_frame_replay.py": "1a572dbb33faba6695b3293768a7e61392d11cb4c2d5f708eedcd2cb95d0aff9",
+    "scripts/experiments/binary_frame_replay.py": "a1cbab2900440baa9b32f482f1531518fd95d55de2fe30e40e7a5249420b33a0",
     "scripts/experiments/joint_dual_compiler.py": "853d3fc4a740c5b74c23b3a6af5d4dc9b41fb38e9b55678539a7e0ee71947c13",
     "scripts/experiments/joint_dual_compose.py": "3554e957570e2d3f65c5bd7c945338c9e2fc5f3e79efa2b436a4dfc61899c394",
     "scripts/experiments/joint_dual_reclaim_compiler.py": "edab206b83cfe3f7a08f1b605b182e354c87cc24e2f3ad81a95f7a083ff342fb",
     "scripts/experiments/pin_joint_dual_sources.py": "5d41535a73ad62fae7acb5365c5b7f40d7c99e52b6bc4266e44572018080acc5",
     "scripts/experiments/verify_joint_dual.py": "7a31bba1e9c0ded32ae285df2ca9bc91ef20c330f6789fdba361876f3c7a1985"
   },
-  "source_manifest_sha256": "330fac72bad557ef4ae999298121c7d63ce237e23bb3184222c9bd6b54264687",
+  "source_manifest_sha256": "6d37a7c0df19a8b16e3b0d6501d0b92f9985704fe988a94517a50f85141af202",
   "sources": [
     {
       "files": {

certificates/skip-frame-kappa.json

@@ -825,7 +825,7 @@
     "scripts/experiments/binary_frame_math.py": "1f96e8aafc8ae89f8eeec5d03800131d93d58e01bf8a7fca8feb03702d1740d0",
     "scripts/experiments/binary_frame_profile_prepare.py": "16dea3d3ef3dc90db928ed3e35d002f6926d2e7388da4766fc2fd725550f739e",
     "scripts/experiments/binary_frame_profiles.cpp": "8e2e391d3ee57cbb1372c68e4aa031e1b861321536eabd642ada54e12e678669",
-    "scripts/experiments/binary_frame_replay.py": "1a572dbb33faba6695b3293768a7e61392d11cb4c2d5f708eedcd2cb95d0aff9",
+    "scripts/experiments/binary_frame_replay.py": "a1cbab2900440baa9b32f482f1531518fd95d55de2fe30e40e7a5249420b33a0",
     "scripts/experiments/skip_frame_compiler.py": "8ffd079fd2e4f055b5befefecc5f33e686de9898219e9252f5d3913df8b195d8",
     "scripts/experiments/skip_frame_compose.py": "d48e688ebb9a490f4eaf50d9ab0a4d4e3e76b6588b55d213df14c703c27a7f66",
     "scripts/experiments/verify_skip_frame.py": "fc42dc3d59239ee3908d0e3e95402eb53fd6815a100521911fde1b541070c292"

research/joint-dual-replay-production/NOTICE.md

@@ -0,0 +1,19 @@
+# Attribution and license
+
+Prepared by Wentao Li ([VictorLiwentao](https://github.com/VictorLiwentao),
+[@Is__Victor](https://x.com/Is__Victor), wentaol4@illinois.edu) with substantial
+AI assistance for orientation, overlap search, profiling, production wiring,
+packaging, benchmarking and writing. Not independent human peer review.
+Apache-2.0; see the repository root `LICENSE`.
+
+The exact containment-mask acceleration of the per-output triple scans was
+introduced by **Aaron Wang** in the opt-in adapter
+`research/joint-dual-replay-fast/` (see that package's README and TASK.json).
+This package wires an equivalent check into the production
+`scripts/experiments/binary_frame_replay.py:replay` path used by
+`make joint-dual-verify`.
+
+The original joint-dual serialized-word verifier, PR64 scatter multiset control,
+compiler receipts and scientific certificate are by **Chafik Boukhalfa** and the
+retained frame-compiler lineage. Upstream material retains its own source
+notices and license.

research/joint-dual-replay-production/README.md

@@ -0,0 +1,105 @@
+# Production containment masks for joint-dual word replay
+
+**Tooling improvement; no new kappa.** At base
+`0fdd7d82de03e43c9e5a1000dfbcd380d1b687d2`,
+`scripts/experiments/binary_frame_replay.py:replay` still scanned every
+weight-three triple for each paid output. Aaron Wang's opt-in adapter
+`research/joint-dual-replay-fast/` already proved an exact containment-mask
+reduction with identical receipts. This package wires that equivalent check
+into the **production** source used by `make joint-dual-verify`.
+
+Compiler replay receipts in `certificates/joint-dual-compiler.json` stay
+**byte-identical**. Only the source digests for
+`scripts/experiments/binary_frame_replay.py` change inside
+`research/joint-dual/SOURCE.json`, `certificates/joint-dual-kappa.json` and
+`certificates/skip-frame-kappa.json`. Every assertion, `check_word` control
+(including the PR64 scatter multiset), both full-basis orientations and the
+histogram/rank-mass fields are unchanged. Selected frontier untouched.
+
+## Reproduce
+
+Python **3.12**, standard library only, from the repository root:
+
+```sh
+python3.12 -I -B research/joint-dual-replay-production/verify.py
+python3.12 -I -O -B research/joint-dual-replay-production/verify.py
+python3.12 -B scripts/research_package.py --package research/joint-dual-replay-production
+python3.12 -I -B research/joint-dual-replay-production/benchmark.py --rounds 5
+```
+
+Expected:
+
+```text
+PASS_JOINT_DUAL_REPLAY_PRODUCTION words=[23, 25] outputs=12261 receipts=2 no_new_kappa
+PASS_JOINT_DUAL_REPLAY_PRODUCTION_O_POLICY refused_by_assert_gate=2 no_new_kappa
+PASS_REGISTERED_RESEARCH_PACKAGE research/joint-dual-replay-production outcome=completed
+PASS_IDENTICAL_JOINT_DUAL_REPLAY_PRODUCTION_RECEIPTS
+```
+
+Default verify is read-only. Timing is host-descriptive and is not a
+correctness gate.
+
+On Linux / CPython 3.12.3, five alternating fresh-process pairs replaying both
+pinned words (`h=23` and `h=25`) gave median wall time **5.873 → 3.120 s
+(1.882×)**. Retained samples are in [`RESULT.json`](RESULT.json).
+
+## Why the masks are equivalent
+
+For `triples = list(combinations(range(h), 3))`, bit `i` of `center[c]` is set
+iff `c in triples[i]`, and bit `i` of `pair[c][x]` is set iff
+`{c, x} ⊆ triples[i]`. Then
+
+```text
+expected(common, triple) = center[common]
+    & ~pair[common][x] for each x in set(triple) - {common}
+```
+
+equals the historical scan
+
+```text
+sum(1 << i for i, t in enumerate(triples)
+    if common in t and not (set(t) & (set(triple) - {common})))
+```
+
+Destination lists for copied centres are the same index lists as
+`[i for i, t in enumerate(triples) if common in t]`. Frame lookup preserves
+first-match `F.index` semantics. This is the same argument as
+`research/joint-dual-replay-fast/README.md`.
+
+## Overlap and credit
+
+Closest priors:
+
+- [Aaron Wang / joint-dual-replay-fast](../joint-dual-replay-fast/) — opt-in
+  containment-mask adapter; this package is the production wiring.
+- [CrocSwap/integer-mult-bounds#756](https://github.com/CrocSwap/integer-mult-bounds/pull/756)
+  — identical-receipt tooling quality bar.
+- Open leave-alone tooling on other objects (#1922 joint frames, #2054/#2150
+  shared/mixed-point, #2100 ternary, #2117 DisjointTensor.frame_data, #2194
+  production verify_factor, #2210 production leech_minor, #2231 production
+  sparse-fusion orbits, #1861 parity-side) was not reopened. Fresh overlap
+  refresh over open PRs found no competing production rewrite of
+  `binary_frame_replay.replay`.
+
+Chafik Boukhalfa authored the joint-dual verifier and certificates.
+Aaron Wang authored the containment-mask algorithm. Prepared by Wentao Li
+([VictorLiwentao](https://github.com/VictorLiwentao),
+[@Is__Victor](https://x.com/Is__Victor)) with substantial AI assistance, under
+Apache-2.0.
+
+## Open next questions
+
+1. **Question:** Should `research/joint-dual-replay-fast/` note that production
+   now ships the same containment-mask scan?
+   **Why it matters:** That package's source-admission pins expect the
+   historical pairwise enumeration block.
+   **First step:** Add a one-line status note without editing pinned evidence.
+   **Blocked by:** Maintainer preference on touching incorporated historical
+   packages.
+
+2. **Question:** After mask wiring, is the remaining full-basis word simulation
+   the next productive production pass?
+   **Why it matters:** With the per-output scans reduced, dual-orientation XOR
+   simulation dominates residual replay time.
+   **First step:** Profile production `replay` with masks merged.
+   **Blocked by:** No open identical-receipt adapter yet for that stage.

research/joint-dual-replay-production/RESULT.json

@@ -0,0 +1,86 @@
+{
+  "ai_assistance": "Substantial AI assistance for orientation, overlap search, profiling, production wiring, packaging, benchmarking and writing. Not independent human peer review.",
+  "base_commit": "0fdd7d82de03e43c9e5a1000dfbcd380d1b687d2",
+  "both_words_replay": {
+    "median_baseline_seconds": 5.87312843299992,
+    "median_optimized_seconds": 3.1202947950000635,
+    "median_ratio": 1.882235115224041,
+    "rounds": 5,
+    "samples": [
+      {
+        "baseline_seconds": 5.888532680000026,
+        "optimized_seconds": 3.4563439109999763,
+        "order": [
+          "baseline",
+          "optimized"
+        ],
+        "ratio": 1.7036882994367242,
+        "round": 0
+      },
+      {
+        "baseline_seconds": 5.776117523000039,
+        "optimized_seconds": 3.1633046890001424,
+        "order": [
+          "optimized",
+          "baseline"
+        ],
+        "ratio": 1.8259757092276037,
+        "round": 1
+      },
+      {
+        "baseline_seconds": 5.87312843299992,
+        "optimized_seconds": 3.08962101599991,
+        "order": [
+          "baseline",
+          "optimized"
+        ],
+        "ratio": 1.9009219585785246,
+        "round": 2
+      },
+      {
+        "baseline_seconds": 5.924514798000018,
+        "optimized_seconds": 3.085845221999989,
+        "order": [
+          "optimized",
+          "baseline"
+        ],
+        "ratio": 1.9199001802690023,
+        "round": 3
+      },
+      {
+        "baseline_seconds": 5.381213688000116,
+        "optimized_seconds": 3.1202947950000635,
+        "order": [
+          "baseline",
+          "optimized"
+        ],
+        "ratio": 1.7245850285117073,
+        "round": 4
+      }
+    ],
+    "words": [
+      "certificates/joint-dual-word-23.json.gz",
+      "certificates/joint-dual-word-25.json.gz"
+    ]
+  },
+  "engines": {
+    "baseline": "research/joint-dual-replay-production/baseline.py",
+    "optimized": "scripts/experiments/binary_frame_replay.py"
+  },
+  "environment": {
+    "cpu_affinity_set": false,
+    "executable": "/usr/bin/python3.12",
+    "implementation": "CPython",
+    "os_caches_flushed": false,
+    "platform": "Linux-6.12.94+-x86_64-with-glibc2.39",
+    "python": "3.12.3 (main, Aug 31 2026, 10:18:26) [GCC 13.3.0]"
+  },
+  "no_new_kappa": true,
+  "predecessor": "research/joint-dual-replay-fast/",
+  "source_sha256": {
+    "baseline.py": "1a572dbb33faba6695b3293768a7e61392d11cb4c2d5f708eedcd2cb95d0aff9",
+    "research/joint-dual-replay-fast/fast_replay.py": "dd96fe7e6f9403a519a06b93fec6420532858b5857762d567793c671dd62f2d0",
+    "scripts/experiments/binary_frame_replay.py": "a1cbab2900440baa9b32f482f1531518fd95d55de2fe30e40e7a5249420b33a0"
+  },
+  "status": "PASS_IDENTICAL_JOINT_DUAL_REPLAY_PRODUCTION_RECEIPTS"
+}

research/joint-dual-replay-production/SOURCES.json

@@ -0,0 +1,60 @@
+{
+  "schema_version": 1,
+  "base_commit": "0fdd7d82de03e43c9e5a1000dfbcd380d1b687d2",
+  "files": [
+    {
+      "path": "docs/research/agent-task-catalog.json",
+      "usage": "provenance",
+      "git_blob": "e8870e5917057d59b730eb14721a778639eca050",
+      "sha256": "d46eb6d5b3e279428753d770653663fbdf1063e55c255ea073d768f344927ef7"
+    },
+    {
+      "path": "certificates/selected-result.json",
+      "usage": "provenance",
+      "git_blob": "ae6a99f28aa468649eb66a361e764d769363bf11",
+      "sha256": "10b9e340fd8e354a37a0c4813c90e5de1dd42044b5f6aca4a3fc5a42b685fa32"
+    },
+    {
+      "path": "certificates/joint-dual-compiler.json",
+      "usage": "input",
+      "git_blob": "919fff2e6dac36adecf8ab4503a0d6fd65605e61",
+      "sha256": "516c4d85158553dabd0a98216a064b2a7397957a9d5dd0df1d6e5aec53a4e805"
+    },
+    {
+      "path": "certificates/joint-dual-word-23.json.gz",
+      "usage": "input",
+      "git_blob": "143f2684b069c9c6bc034b91fa5deb3ddab3abf7",
+      "sha256": "634da1cb39634be15c6d4070132c6bdf4f5a927331cac3b4f13dc995dd799b6b"
+    },
+    {
+      "path": "certificates/joint-dual-word-25.json.gz",
+      "usage": "input",
+      "git_blob": "fc67bf3a5789da29392ca8f82ff0fbed906af2e3",
+      "sha256": "dd2223d7ee0b91c4445f5acd0b7cad945f56f4e16456b6818f53ddef64a0cb16"
+    },
+    {
+      "path": "scripts/experiments/binary_frame_replay.py",
+      "usage": "provenance",
+      "git_blob": "a4897e5ecb5c92404257e85c8558e7264f0ef0bb",
+      "sha256": "1a572dbb33faba6695b3293768a7e61392d11cb4c2d5f708eedcd2cb95d0aff9"
+    },
+    {
+      "path": "tests/test_joint_reclaim.py",
+      "usage": "provenance",
+      "git_blob": "c94302af4c27e6797baec4fa8fe3147dc088f61d",
+      "sha256": "c4e2817bf6e699f462aa3c589ee45e24ab0dcad6bcdb6152ca2a1e16ac7bffeb"
+    },
+    {
+      "path": "research/joint-dual-replay-fast/fast_replay.py",
+      "usage": "input",
+      "git_blob": "a14c466421138b500eb85798c68bfeafe72d29d0",
+      "sha256": "dd96fe7e6f9403a519a06b93fec6420532858b5857762d567793c671dd62f2d0"
+    },
+    {
+      "path": "research/joint-dual-replay-fast/README.md",
+      "usage": "provenance",
+      "git_blob": "a3863aa62d35ee03ab867a105782014fffd6448f",
+      "sha256": "532d668b38e66167f8b6db77ae2d004964db05272d4596364ff1f55d80f4b5de"
+    }
+  ]
+}

research/joint-dual-replay-production/TASK.json

@@ -0,0 +1,70 @@
+{
+  "schema_version": 1,
+  "task_id": "replay-runtime",
+  "priority_revision": 4,
+  "base_commit": "0fdd7d82de03e43c9e5a1000dfbcd380d1b687d2",
+  "chooser_seed": "3518c0ee1ad394869a4a6efb529a5503",
+  "original_card": {
+    "id": "replay-runtime",
+    "question": "Can one measured replay bottleneck be sped up without dropping coverage?",
+    "starting_points": [
+      "scripts/run_isolated_tests.py",
+      "scripts/verify_replay_coverage.py",
+      "certificates/selected-result.json"
+    ],
+    "methods": [
+      "Profile one isolated replay and optimize the measured bottleneck.",
+      "Design source-bound reuse with explicit dependency invalidation and negative controls."
+    ],
+    "acceptance_gate": "Report before/after wall time on the same fixture, retained checks and invalidation behavior.",
+    "first_step": {
+      "command": [
+        "python3",
+        "-B",
+        "scripts/research_baseline.py"
+      ],
+      "expected": "PASS_RESEARCH_BASELINE_BINDINGS_ONLY",
+      "inspect": "Read the selected reproduction command and choose a single replay stage for profiling.",
+      "scope": "Positive control / source-bound orientation only; this is not a new construction or full cold replay."
+    },
+    "first_experiment": "Measure one unchanged fixture, preserve check coverage, and test one optimization with invalidation controls.",
+    "bottleneck_link": "Faster trustworthy replays increase review throughput and let mechanism candidates be checked sooner.",
+    "allowed_variations": [
+      "Profile one stage or design one source-bound cache boundary."
+    ],
+    "scope_change_triggers": [
+      "Skipping coverage or comparing different fixtures to claim speedup."
+    ],
+    "overlap_terms": [
+      "replay runtime",
+      "cache invalidation",
+      "isolated tests"
+    ]
+  },
+  "scope_status": "refined",
+  "actual_scope": "Can the production joint-dual serialized-word replay (scripts/experiments/binary_frame_replay.py:replay, exercised by make joint-dual-verify on certificates/joint-dual-word-{23,25}.json.gz) adopt the exact containment-mask scan from research/joint-dual-replay-fast/ so the shipped verify path is faster while every assertion, check_word control, full-basis orientation and compiler replay receipt stays identical?",
+  "scope_change_reason": "The catalogue card asks for one measured replay bottleneck with before/after wall time and retained checks. Expertise override selected verifier-engineering after profiling make verify poles and scanning proved opt-in adapters still pairwise/slow on the shipped path. Leave-alone objects (#1922 joint frames, #2054/#2150 shared/mixed-point, #2100 ternary, #2117 DisjointTensor.frame_data, #2194 production verify_factor, #2210 production leech_minor, #2231 production sparse-fusion orbits, #1861 parity-side, quadratic-affine deferred for #1076) were not reopened. Aaron Wang's incorporated joint-dual-replay-fast adapter already proved an opt-in identical-receipt accelerator; production still used the O(outputs x triples) literal scans.",
+  "bottleneck_link": "Side result on review throughput: joint-dual-verify is part of make verify-joint. Faster identical replay receipts do not change kappa or the binding complex supplier.",
+  "ai_assistance": "Substantial AI assistance for orientation, overlap search, profiling, production wiring, packaging, benchmarking and writing",
+  "outcome": "completed",
+  "evidence": "PASS_JOINT_DUAL_REPLAY_PRODUCTION via research/joint-dual-replay-production/verify.py (also -O policy control); PASS_IDENTICAL_JOINT_DUAL_REPLAY_PRODUCTION_RECEIPTS via benchmark.py; compiler replay receipts unchanged; SOURCE/kappa/skip-frame pins updated only for the production source digest.",
+  "author": "Wentao Li",
+  "email": "wentaol4@illinois.edu",
+  "github": "VictorLiwentao",
+  "budget_minutes": 60,
+  "stop_rule": "Ship an equivalence-preserving production binary_frame_replay.replay change with frozen baseline, identical word receipts, rejection/temp-corruption controls, updated source pins only, registered replay and a measured before/after median. Do not invent kappa or reopen leave-alone objects.",
+  "open_questions": [
+    {
+      "question": "Should research/joint-dual-replay-fast/ note that production now ships the same containment-mask scan?",
+      "why_it_matters": "That opt-in package's source-admission pins expect the historical pairwise enumeration block in scripts/experiments/binary_frame_replay.py.",
+      "first_step": "Add a one-line status note to that package README without editing its pinned evidence.",
+      "blocked_by": "Maintainer preference on touching incorporated historical packages."
+    },
+    {
+      "question": "After mask wiring, is the remaining full-basis word simulation the next productive production pass?",
+      "why_it_matters": "With the per-output scans reduced, the dual-orientation XOR word dominates residual replay time.",
+      "first_step": "Profile production replay with masks merged and name any further exact reuse.",
+      "blocked_by": "No open identical-receipt adapter yet for the full-basis stage."
+    }
+  ]
+}