Skip to content

Verify that pr-ledger.md matches pr-ledger.json (539 stale rows, attribution intact) - #3695

Open
muatfannajm-dot wants to merge 1 commit into
CrocSwap:mainfrom
muatfannajm-dot:verify-pr-ledger-markdown
Open

muatfannajm-dot wants to merge 1 commit into
CrocSwap:mainfrom
muatfannajm-dot:verify-pr-ledger-markdown

Conversation

@muatfannajm-dot

@muatfannajm-dot muatfannajm-dot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

scripts/find_overlap.py reads docs/research/pr-ledger.md as its local-ledger
overlap corpus. scripts/verify_pr_reconciliation.py checks only
docs/research/pr-ledger.json. scripts/reconcile_pr_queue.py writes both files
but never verifies the committed .md. Nothing in the repository checks that the
two ledgers agree.

The committed .md is currently stale relative to the committed .json.

Result

Base commit 20ecf2891d80b5acf18d980b2164fb44ea7faa90, priority revision 6
(verified again at submission time against canonical main).

Attribution is not affected: the contributor and disposition cells are
identical in all 539 cases, so recorded credit is intact. The divergence is
confined to the evidence-record column.

Root cause (539/539, no exceptions)

For every one of the 539, reconciliation_evidence in pr-ledger.json contains
docs/research/pr-intake-decisions.json, and the Markdown row lacks it.

Example, PR #1365:

json  reconciliation_evidence:
      research/community-efficient500-audit/DISPOSITIONS.json
      research/community-efficient2000-audit/DISPOSITIONS.json
      research/community-efficient2500-audit/DISPOSITIONS.json
      research/community-efficient2600-audit/DISPOSITIONS.json
      docs/research/pr-intake-decisions.json
      docs/research/pr-intake-evidence/queue-oct18-20261018.json      (6 records)

md    [Exact-head intake](../../docs/research/pr-intake-evidence/queue-oct18-20261018.json)
                                                                       (1 record)

So the intake-queue tooling updates pr-ledger.json in place, adding
pr-intake-decisions.json to reconciliation_evidence, while pr-ledger.md is
only written by reconcile_pr_queue.py, which is not re-run. The drift is
permanent and grows with every intake round.

Why it matters, stated honestly

Impact is bounded and modest, not a scientific result:

  • no change to κ, to certificates/selected-result.json, or to any mathematical claim
  • credit and attribution are intact — the two credit-critical fields agree everywhere
  • the real effect is a weaker/partial overlap signal for 539 PRs in the exact
    corpus the overlap process depends on, in a repository that documents recurring
    duplicate work

find_overlap.py currently reports a local-ledger hit with no freshness or
binding information, so a stale ledger is indistinguishable from a current one.

The verifier

scripts/verify_pr_ledger_markdown.py re-derives the Markdown projection that
reconcile_pr_queue.py (lines 363–371) emits — disposition counts plus one row per
PR with number, contributor, disposition and ordered evidence records — and reports
a bounded set of differences instead of a bare boolean.

python3 scripts/verify_pr_ledger_markdown.py           # FAIL, exit 1 (539 differences)
python3 scripts/verify_pr_ledger_markdown.py --json    # full structured report
python3 -B tests/test_verify_pr_ledger_markdown.py      # 13 tests, OK
python3 -B tests/test_research_assignment.py            # OK (skipped=2)
python3 -B tests/test_research_coordination.py          # OK

13 tests: one positive fixture plus eight failure controls (dropped evidence
record, disposition drift, attribution drift, missing row, count drift, duplicate
row, absent counts table, JSON counts self-drift), one evidence dedup/ordering
control and one repository-shape assertion that does not hard-code the current
(known-stale) state.

The derivation is corroborated by 2489 of 3028 rows matching it exactly.

What I deliberately did not do

I did not regenerate pr-ledger.md. python3 scripts/reconcile_pr_queue.py
would rewrite it, but that rewrites a tracked evidence artifact and is a
maintainer decision — and it is not clear that regenerating is even the intended
fix (see open questions). I also did not wire the verifier into
scripts/verify_pr_reconciliation.py, because
#3686 currently
modifies that file; a maintainer can decide the integration point.

If regenerating is intended, the one-line fix is:

python3 scripts/reconcile_pr_queue.py && python3 scripts/verify_pr_ledger_markdown.py

Overlap check

Live checks were completed for this PR (authenticated REST + GraphQL):

Adjacent but not duplicative:
#3686
("Preserve optional intake metadata in ledger verification", by @roger-sato) fixes a
KeyError: 'closure' crash in the JSON-side checker for 24 intake decisions. It is
the same ledger family and the same canonical base, and it explicitly cites
PR1365 — one of the 539 rows above. It does not compare the Markdown ledger to
the JSON ledger. Its diff touches scripts/verify_pr_reconciliation.py and
tests/test_pr_reconciliation.py; this PR adds only new files, so there is no
conflict.

Predecessor credit

  • @roife — #704,
    the original review-ledger metadata binding and its rejection controls
  • Douglas R. Colkitt — the reconciliation infrastructure (reconcile_pr_queue.py,
    verify_pr_reconciliation.py)
  • @Dennis / @antunishdPursuit —
    #2026 and
    #2859, historical replay maintenance
  • #2624 — precedent for
    fixing a stale blocker in a generated ledger
  • @roger-sato — #3686,
    whose failure analysis surfaced the same ledger family

I found no prior work binding pr-ledger.md to pr-ledger.json. I dropped an
earlier candidate (auditing selected-result.json hash bindings) because
scripts/verify_selected_result.py already covers it.

Remaining obligations / not claimed

  • I did not determine whether the .md/.json divergence is intentional or a
    bug. A reviewer should decide that before regenerating.
  • I did not run any cold replay, Lean check, or arithmetic. This is an
    attribution/coverage binding check only; it establishes nothing about κ.
  • The derivation reproduces my reading of reconcile_pr_queue.py:363-371. The
    2489 exact row matches corroborate it, but a second reader is welcome.
  • Only the last 60 comments of each discussion were scanned, not all 574.
  • No live GitHub PR state was audited; that is verify_pr_reconciliation.py's job.

Open next questions

  1. Is the divergence intentional? If pr-ledger.md is meant to be a frozen
    human snapshot rather than a projection of the JSON, the right fix is to state
    that contract and stop emitting both from the same inputs.
  2. Which tool appends pr-intake-decisions.json to reconciliation_evidence?
    Closing the loop at the writer is more robust than checking at the reader.
  3. Should find_overlap.py report ledger freshness? It can currently be
    pointed at a stale ledger with no signal.
{
  "schema": "integer-mult-coordination/v1",
  "kind": "FINDING",
  "task": "overlap-receipt",
  "scope": "Check pr-ledger.md, the find_overlap.py local-ledger corpus, against pr-ledger.json: 539 stale rows, attribution intact",
  "base": "20ecf2891d80b5acf18d980b2164fb44ea7faa90",
  "expiry": null,
  "artifact": "https://github.com/CrocSwap/integer-mult-bounds/pull/3695"
}

Prepared by an AI agent (OpenCode / Space Bunny) with substantial AI assistance.
Apache-2.0.

find_overlap.py reads docs/research/pr-ledger.md as its local-ledger overlap
corpus, but verify_pr_reconciliation.py checks only pr-ledger.json and
reconcile_pr_queue.py writes both files without verifying the committed .md.
A stale or hand-edited .md therefore misreports disposition and recorded
review coverage with no failing check.

scripts/verify_pr_ledger_markdown.py re-derives the Markdown projection that
reconcile_pr_queue.py emits (disposition counts plus one row per PR) and
reports bounded differences. On current main it finds 539 of 3028 rows stale:
contributor and disposition cells agree in every case, but the evidence-record
column has lost the docs/research/pr-intake-decisions.json record for all 539.

Source-bound attribution/consistency check only. It does not judge whether a
recorded review was correct and does not contact GitHub.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant