Tooling: exact eligibility masks accelerate historical reclamation

research/reclamation-priority-performance/CLAIM.json

@@ -0,0 +1,80 @@
+{
+  "track": "verification engineering",
+  "assignment_seed": 85,
+  "randomness": "python3 secrets.randbelow(100), operating-system randomness, drawn once",
+  "base_commit": "66849e1475c930fd7732b1a0067ffee7e1e95f60",
+  "started_utc": "2026-10-13T04:51:11Z",
+  "expected_finish_utc": "2026-10-13T05:36:00Z",
+  "verification_reserve_starts_utc": "2026-10-13T05:24:45Z",
+  "coordination": "No pinned issues and no sprint coordination issue; only issue 650 requesting Discussions. Local claim recorded before implementation.",
+  "candidates": [
+    "Avoid repeated exact data_corners.cpp checks in five historical producers",
+    "Replace repeated sorting of retired slots in joint_dual_reclaim_compiler.acquire with an exact rank/slot index",
+    "Accelerate quadratic_affine.rank_witness orthogonality-row construction"
+  ],
+  "selection": "85 mod 3 = 1, zero-based in the above list; inspect the ranked-reclamation bottleneck identified but not changed by PR154",
+  "object": "scripts/experiments/joint_dual_reclaim_compiler.py, retired-slot iteration in acquire; h=23 and h=25 PR55 graph",
+  "hypothesis": "Per-coordinate core/union membership bitsets can exclude exactly the ineligible retired slots before ordered rank/slot iteration, avoiding both the base's repeated sorts and PR175's repeated scans of ineligible candidates.",
+  "method": "Optional source-pinned in-memory adapter; preserve matching, eligibility, span elimination, clearing XORs, frames, dirty-basis checks and generated words. Benchmark reference and indexed engines sequentially on identical complete axes.",
+  "deliverable": "One tooling PR with exact word/receipt comparisons, an index invariant argument, seed-85 state-machine controls, source pins and before/after timing evidence. No new kappa.",
+  "stopping_rule": "Complete both shipped axes; stop or report a negative performance result if the exact index fails equivalence or does not reduce cost. At most one implementation pivot. No new mathematical search.",
+  "initial_overlap": {
+    "open_prs_read": 544,
+    "metadata_refreshed_through_pr": 732,
+    "relevant_pr154_head": "f571bcbd6e06618aeae0b8279da48060032dc74b",
+    "pr154_scope": "Reports repeated sorting in acquire as an unchanged bottleneck; optimizes concurrency and paired-cube frame algebra instead.",
+    "other_distinct_tooling_prs": [
+      367,
+      452,
+      662,
+      667,
+      672,
+      704,
+      707,
+      709,
+      721,
+      727
+    ],
+    "ledger": "PR154 partially reviewed; PR367 and PR452 unreviewed at base. These proposals are background, not accepted performance premises or code dependencies.",
+    "negative_result_check": "Read preserved-research.md and searched the research index and archived negative-result summaries; this replays a fixed historical construction, without reopening a closed mathematical search."
+  },
+  "scope_refinement": {
+    "reason": "The closed-PR ledger identified incorporated PR175 after the initial open-PR scan. Its ordered buckets already solve sorting, so sorting alone is not a new contribution.",
+    "predecessor": "Chafik Boukhalfa, PR175 at f1081fabbfadaaaaf7292a40d428dbbb865ce920, compiler SHA256 f0696953d057239ed5a75e28d857b814d43ca43deaaf7b7b8fd191aa115112ff",
+    "distinct_work": "Exact coordinate-membership filtering, with per-slot frame updates; compare against both the unoptimized base and the exact PR175 compiler. The complete eligible candidate sequence must match a sorted-set oracle at every acquisition in an untimed audit."
+  },
+  "status": "Candidate Ready",
+  "result": {
+    "kind": "tooling improvement",
+    "new_kappa": false,
+    "core": "Exact coordinate-membership filtering before unchanged retired-slot selection",
+    "compile_speedup_over_pr175": [
+      4.525399396610507,
+      4.871103587001165
+    ],
+    "benchmark_runs": 18,
+    "additional_public_reproduction_runs": 6,
+    "all_query_audits": 71519,
+    "focused_tests": 12,
+    "unchanged_historical_tests": 16,
+    "selected_record_check": "PASS",
+    "review_map": "README.md"
+  },
+  "final_overlap": {
+    "open_prs_read": 586,
+    "through_pr": 772,
+    "new_distinct_tooling": [
+      736,
+      752,
+      756
+    ],
+    "finding": "No competing coordinate-membership eligibility index found. PR175 is the incorporated predecessor and an exact comparator, not a new claim.",
+    "later_arrivals_checked": [
+      768,
+      769,
+      770,
+      771,
+      772
+    ]
+  }
+}

research/reclamation-priority-performance/NOTICE.md

@@ -0,0 +1,36 @@
+# Attribution and license
+
+Prepared for **Peter Carpenter (PeterCarp320)** with substantial assistance
+from **OpenAI Codex (GPT-6 Astra)**, including experiment selection,
+implementation, equivalence argument, testing, measurements and documentation.
+No independent human mathematical review is claimed. New material is supplied
+under the repository's Apache-2.0 license.
+
+The original source files and their notices remain unchanged:
+
+- **eumemic** supplied the PR57 joint compiler and identified repeated sorting
+  in `acquire` as an unchanged bottleneck in PR154.
+- **Chafik Boukhalfa (chafreaky)** supplied descending-rank reclamation and its
+  exact integration, with OpenAI Codex assistance. His **PR175** already
+  replaced whole-set sorting with ordered per-rank lists and correctly moved
+  retired anchors when clearing changes their frames. That contribution is
+  incorporated in an archived prerequisite, as the review ledger records.
+  This package does not claim that ordering optimization as new.
+- `reference-pr175.py` preserves the exact compiler bytes from PR175 commit
+  `f1081fabbfadaaaaf7292a40d428dbbb865ce920`, SHA256
+  `f0696953d057239ed5a75e28d857b814d43ca43deaaf7b7b8fd191aa115112ff`.
+  Its original compiler credits and the root Apache-2.0 license apply.
+- **Rohan Gupta** supplied the PR55 dual-skip producer. The PR48 fixed-basis
+  geometry and preceding compiler/graph contributors retain their existing
+  attribution in `research/joint-dual/` and `references/frame-compiler/`.
+- **rfu08's PR64 audit** identified the cancelling-scatter failure covered by
+  the retained independent replay and regression controls.
+
+The new contribution is exact coordinate-membership filtering before ordered
+iteration, its equivalence argument and complete same-input comparison against
+both the base compiler and PR175. PR367/452/662/667/672/707/709/721/727 address
+other verification workloads and are not dependencies.
+
+The accepted selected result at the base commit is #626/#543, conditional
+κ = 51110230636731/31250000000000000. This package concerns the historical
+joint-dual verifier; neither the selected frontier nor `upstream/` is edited.

research/reclamation-priority-performance/PROOF.md

@@ -0,0 +1,100 @@
+# Exact geometry filtering and retired-slot order
+
+This is an equivalence argument for the source-pinned finite compiler in
+`SOURCE.json`. It changes its implementation cost, not its mathematical
+construction or any conditional exponent. It assumes the pinned compiler's
+frame and control-flow contracts. The runtime tests are finite corroboration,
+not a formal proof of the full multiplication theorem.
+
+## Membership invariant
+
+Let `S` be the retired physical slot set. A slot `s` has current rank `r_s`
+and current original-envelope frame `(c_s, u_s)`, where `c_s` and `u_s` are
+coordinate subsets of `{0, ..., h-1}`. The index maintains:
+
+- `all_mask`: exactly the slots in `S`;
+- `masks[r]`: exactly the slots in `S` whose current rank is `r`;
+- `core_masks[i]`: exactly the slots in `S` for which `i` belongs to `c_s`;
+- `union_masks[i]`: exactly the slots in `S` for which `i` belongs to `u_s`.
+
+Each bit position is a physical slot ID, not a coordinate ID. All masks start
+empty. Adding and removing a retired slot sets or clears its rank bit and
+toggles exactly its frame-coordinate membership bits. An idempotent duplicate
+add leaves the masks alone. Removal of an absent member raises `KeyError`.
+
+The adapter inserts `move` immediately before the original `raise_` updates
+`frames[s]`. Thus the callback still sees the old frame. For a retired slot,
+the rank masks change if the rank changes, and coordinate bits toggle exactly
+on the symmetric differences of the old and new core/union. Nonretired slots
+do not enter the index. Changes within the same rank still update geometry.
+The original frame-containment assertion remains before this hook.
+
+In particular, clearing changes **both** the chosen destination's frame and
+each retired source anchor's frame. Both changes pass through the same
+`raise_` hook. Removing the chosen slot after clearing uses its new frame.
+This is the subtle update obligation already identified by Chafik Boukhalfa's
+PR175 ordered-bucket implementation.
+
+## Exact eligible set
+
+For a requested frame `(c, u)`, the original predicate is
+
+```
+not (c & ~c_s) and not (u_s & ~u)
+```
+
+Equivalently, `c` is a subset of `c_s`, and `u_s` is a subset of `u`.
+Therefore the eligible slot set is precisely
+
+```
+S intersect (intersection of core_masks[i] for i in c)
+  intersect (intersection of complements of union_masks[i] for i outside u).
+```
+
+The index computes this expression with integer bit operations. Starting from
+`all_mask` bounds every complement to existing slots. `outside` uses the finite
+`h`-coordinate universe; all caller frames come from the pinned constructor.
+Empty intersections impose no extra restriction. No rank, span, source-access
+or paid-frame criterion is approximated by this expression.
+
+## Same order, same clearing, same word
+
+Intersecting the eligible set with rank masks in descending rank order, and
+extracting each integer mask's least significant set bit, enumerates exactly
+the eligible subsequence of `sorted(S, key=(-current_rank, slot_ID))`.
+
+The original `acquire` builds the same anchor basis first. An ineligible slot
+is skipped **before** `ins(s)`, so omitting just those slots cannot change the
+basis. Each eligible slot reaches the unchanged elimination in the same order.
+Consequently the first dependent candidate, its exact clearing expression,
+every clearing XOR, and any decision to allocate a new slot are identical.
+
+The loop performs no frame or set mutation while searching. Once it finds a
+dependent candidate, it clears the slot, removes it and returns immediately.
+The generator is never resumed after that mutation. Its partially consumed
+mask therefore cannot affect a later query. The standalone index does not
+promise general mutation-during-iteration semantics.
+
+Induction over the unchanged compiler operations gives identical slots,
+frames, paid events, XORs and final receipts. The original per-slot containment
+test, rank/width accounting, full input and dirty-basis replay in both
+orientations remain enabled. No witness, helper or recursive child is removed.
+
+## What is tested
+
+The package checks complete equality to the frozen decompressed words at
+`h=23` and `h=25`, not just summary counts. Independent serialized-word replay
+and event reconstruction consume the newly generated word. An untimed run
+also compares **every complete eligible candidate sequence** against an
+ordinary sorted-set oracle using the current original frames.
+
+Separate tests exhaust all pairs of three-coordinate envelope frames, then
+exercise additions, removals and frame changes with seed 85, including sparse
+physical IDs as large as 40,337. Stale frame updates, stale removals, wrong core
+or union masks, and wrong rank masks have explicit negative controls. Source
+drift, ambiguous edit sites, disabled assertions and reused output paths fail.
+
+The `h=23/25` tests do not assert a theorem for every future compiler or graph.
+The adapter refuses changed source pins. Inherited all-size compilation,
+physical realization, hosting, residual, precision, tape, recovery and analytic
+transfer obligations remain outside this tooling contribution.

research/reclamation-priority-performance/README.md

@@ -0,0 +1,126 @@
+# Exact retired-slot eligibility masks
+
+**Tooling improvement, no new κ.** This optional adapter accelerates the
+historical joint-dual compiler by using coordinate-membership bitsets to
+exclude ineligible retired slots before iterating in the unchanged priority
+order. All compiler operations, paid frame changes, helper choices, dirty
+restoration checks and emitted word bytes must remain identical.
+
+Chafik Boukhalfa's [PR175](https://github.com/CrocSwap/integer-mult-bounds/pull/175)
+already removes repeated sorting using ordered rank buckets. The new step is
+geometry filtering before iteration. The comparison includes PR175's exact
+archived compiler, as well as the slower compiler still used by the root
+historical verifier. This does not claim PR175's ordering improvement as new.
+
+Base commit: `66849e1475c930fd7732b1a0067ffee7e1e95f60`.
+OS-random track assignment: **85**, verification engineering. `CLAIM.json`
+records the shortlist, scope refinement, overlap checks and stopping rule.
+There was no sprint coordination issue, so the claim is local.
+
+## Measured result
+
+Three cyclic-order rounds per axis on CPython 3.14.6 / macOS 26.5.2 arm64,
+18 fresh processes in total. All generated words and all receipt fields agree.
+Medians in seconds:
+
+| Workload | Base sorting | PR175 buckets | Coordinate masks |
+|---|---:|---:|---:|
+| h=23 compilation | 54.310 | 19.668 | 4.346 |
+| h=25 compilation | 93.476 | 31.894 | 6.548 |
+| h=23 complete measured process | 57.422 | 22.499 | 7.443 |
+| h=25 complete measured process | 97.542 | 35.837 | 10.875 |
+
+Compilation is **4.53–4.87× faster than PR175**, and **12.50–14.28× faster
+than the base compiler**. The complete measured process includes imports,
+source checks, compilation, serialization, exact comparison, independent
+word replay and transition reconstruction. Its improvement relative to PR175
+is 3.02–3.30×. These are local measurements under ordinary machine load, not
+a cross-platform guarantee. CPU timings and every raw sample are retained in
+[`evidence/BENCHMARK.json`](evidence/BENCHMARK.json), together with exact hashes
+of all executed adapter/runtime files. No timing result is an admission check.
+
+The full public command was then run once separately and passed, including
+six more matched timing processes. Its complete log and raw timings are in
+`evidence/PUBLIC-REPRODUCTION*`. The two independent sequence audits cover
+30,770 and 40,749 requests (71,519 total); `AUDIT-23.json` and `AUDIT-25.json`
+record their coverage. Their `unfiltered_candidates` counts sum whole retired
+set sizes, **not** actual predecessor visits before early return.
+
+All **12 focused tests** and **16 unchanged historical control tests** pass.
+The latter include omitted XORs, nonnested frames, missing/aliased outputs,
+unpaid cancelling scatter, omitted transitions and understated width. Local
+temporary fixtures were retained for inspection; the assertions were unchanged.
+Reproduce those existing controls separately with
+`python3 -B -m unittest discover -s tests -p 'test_joint_reclaim.py' -v`.
+The selected-record/source-inventory check and staged whitespace check pass.
+This is local evidence; remote CI is not asserted to have run.
+
+## Reproduce
+
+From a full checkout, using Python 3.11+ and only its standard library:
+
+```sh
+python3 -B research/reclamation-priority-performance/verify.py --out /tmp/reclamation-review
+```
+
+Use a fresh output directory. Expected final marker:
+`PASS_RECLAMATION_PRIORITY`. The command runs focused tests, audits every
+eligible candidate sequence on both shipped axes, then runs a sequential
+three-engine timing comparison. Every generated word and receipt is retained
+under the output directory, including failed-run files for inspection. It
+never regenerates a repository certificate.
+
+For the recorded three-round benchmark alone:
+
+```sh
+python3 -B research/reclamation-priority-performance/benchmark.py --rounds 3 --out /tmp/reclamation-timings
+```
+
+Expected: `PASS_RECLAMATION_BENCHMARK`. There is no timing threshold. Runtime
+depends on machine load; the comparison runs sequential fresh processes and
+rotates engine order. OS caches are not flushed.
+
+## Evidence and implementation
+
+`priority.py` maintains rank and coordinate-membership bitsets. `adapter.py`
+checks the frozen source closure, then applies exactly three checked
+in-memory substitutions: create the index, update it before a frame change,
+and use its eligible ordered iterator. The original containment test and
+elimination remain in place. Default commands and all existing sources,
+manifests, certificates and `upstream/` are unchanged.
+
+`SOURCE.json` pins the base manifest, compiler, consumer verifiers and the
+exact PR175 compiler bytes. The original manifest checks and `check_sources()`
+also run. Each measured process rebuilds one complete `h=23` or `h=25` axis,
+compares the **entire decompressed physical word** to the frozen input, and
+compares every compiler-receipt field. The independent serialized-word checker
+then replays the fresh word's complete input/dirty basis in both orientations;
+the transition consumer independently reconstructs paid events from its XORs.
+
+The untimed `audit_index.py` compares the full eligible sequence at **every**
+acquisition to an ordinary sorted-set oracle using the original current
+frames. It checks candidates after the first successful one too. Unit controls
+exhaust three-coordinate frame pairs and exercise 2,400 seed-85 set/frame
+changes with sparse IDs up to 40,337. Five deliberately stale or incorrect
+index mutations are detected. Source tampering, duplicated substitution sites,
+disabled assertions and reused outputs are rejected.
+
+The exact invariant and control-flow argument is in [PROOF.md](PROOF.md).
+[NOTICE.md](NOTICE.md) credits eumemic, Chafik Boukhalfa, Rohan Gupta, rfu08
+and inherited contributors, and discloses substantial OpenAI Codex assistance.
+
+## Limits
+
+This measures historical axis compilation and its replay/transition process.
+It is not a speedup measurement of integer multiplication, the selected
+frontier's cold replay, or the full joint-dual verifier with C++ CRT profiles
+and assembly. The selected #626/#543 certificate remains
+κ = `51110230636731/31250000000000000`.
+
+The source-pinned adapter is opt-in and sequential. Future compiler versions
+require a new source review; mutation during a partially consumed iterator is
+only supported under the pinned acquire-and-return control flow. No global
+optimality or formal multiplication theorem is asserted. Inherited all-size
+compiler, physical realization, hosting, residual, precision, tape, recovery
+and analytic obligations remain. The repository-wide `make verify`, supplier
+cold rebuild, C++ profile computation and Lean are outside this isolated run.

research/reclamation-priority-performance/SOURCE.json

@@ -0,0 +1,13 @@
+{
+  "base_commit": "66849e1475c930fd7732b1a0067ffee7e1e95f60",
+  "files": {
+    "research/joint-dual/SOURCE.json": "e60f1f78c9729700ef9606a6576163de0b553792e5e8f24591dce5f51597158c",
+    "research/reclamation-priority-performance/reference-pr175.py": "f0696953d057239ed5a75e28d857b814d43ca43deaaf7b7b8fd191aa115112ff",
+    "scripts/experiments/binary_frame_profile_prepare.py": "16dea3d3ef3dc90db928ed3e35d002f6926d2e7388da4766fc2fd725550f739e",
+    "scripts/experiments/binary_frame_replay.py": "1a572dbb33faba6695b3293768a7e61392d11cb4c2d5f708eedcd2cb95d0aff9",
+    "scripts/experiments/joint_dual_compiler.py": "853d3fc4a740c5b74c23b3a6af5d4dc9b41fb38e9b55678539a7e0ee71947c13",
+    "scripts/experiments/joint_dual_reclaim_compiler.py": "9984aaccf880e4840402322d76678bfddb33ce7e494af78430a43429f51b85fd"
+  },
+  "pr175_head": "f1081fabbfadaaaaf7292a40d428dbbb865ce920",
+  "source_closure": "The pinned joint-dual manifest and original check_sources() validate the inherited PR48/55/58 dependencies and all finite word inputs."
+}