[IanFrelinger]

Accuracy study of the github.blockers answer. It covers 44 real pull requests in 5 public repositories, graded against ground truth computed from GitHub's API at one recorded snapshot. The brick arm and a native raw-REST arm both run on claude-sonnet-5-5 and replay the same fixtures.

Rebased onto current master (01366adc8, after 0.4.0). Head 0b8970c01. The only conflict was CHANGELOG.md: the study stays under [Unreleased], above the token-benchmark follow-up and the dogfood entry; [0.4.0] is unchanged.

Headline (first snapshot, brick 1.0.0)

Answer	Verdict right	Blockers found	Overcalls	Fabricated	Cost
brick (raw payload and + Sonnet, identical)	44/44	21/111	16	0	$0.27 total
native + Sonnet	42/44 (2 truncated at output cap)	52/111	0	0	$4.61 total

The study found three brick bugs, F1–F3. All have one cause: required checks and approvals came only from admin-only branch protection.

• F1. Overcalls. 16 non-required failing or pending checks were reported as blockers on 10 PRs.
• F2. Missing reviews. A required review was never named, on 24 of 24 PRs.
• F3. Never-run required checks. All 66 were missed, or reported as "an unidentified branch rule".

Re-run after the fix (second snapshot, brick 1.0.0 vs 1.1.0 back to back)

The fix is #874 (merged as 043a37a5). On a second snapshot (CI run 40218094404, about 18:00 UTC), each PR was asked of both bricks:

Answer	Verdict right	Blockers found	Overcalls	Unattributed	Every blocker found	Cost
brick 1.0.0	44/44	19/109	13	6	3 of 28 PRs	$0
brick 1.1.0	44/44	109/109	0	0	28 of 28 PRs	$0
brick 1.1.0 + Sonnet	44/44	109/109	0	0	28 of 28 PRs	$0.33

The doc states the limit on this result. Brick 1.1.0 now reads the GitHub sources the rubric is computed from, so the full agreement shows it applies them without error; it is not independent evidence. The native arm was not re-run.

Spend: $5.10 for the study and $0.33 for the re-run. Each time the key was used only as an env var and unset afterwards.

What changed

• docs/benchmarks/blockers-accuracy.md: the written rubric, method, results, findings with their fix status, the re-run, limits and the per-PR table.
• tools/blockers-accuracy/: the harness (capture, fixtures, truth, run, score), both snapshots (fixtures/, fixtures-rerun/) and the raw runs.
  • capture.py --extra asks a second host build.
  • run.py --arms runs only the arms named.
  • score.py --fixtures/--brick-dir grades another snapshot or brick build.
• docs/DocsIndex.md, CHANGELOG.md (Unreleased), and the regenerated knowledge graph.

Verification

• scripts/ci/run-repo-gates.sh: all 44 gates pass on the rebased tip.
• verify-doc-anchors.py: passes.
• The staged diff was scanned for key and token patterns; none found.
• python3 tools/blockers-accuracy/score.py … re-grades offline.