Skip to content

docs(benchmarks): blockers accuracy scale — 100 PRs / 10 repos, brick 1.1.0 CLEAN - #879

Merged
cursor[bot] merged 1 commit into
masterfrom
cursor/blockers-accuracy-scale
Oct 26, 2026
Merged

cursor[bot] merged 1 commit into
masterfrom
cursor/blockers-accuracy-scale

Conversation

@IanFrelinger

@IanFrelinger IanFrelinger commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

Summary

Scale accuracy pass for github.blockers 1.1.0 (post-#874) on 100 public PRs across 10 repositories, including IanFrelinger/Ashlar and other .NET-heavy hosts. Brick-raw grading only; Sonnet skipped because it adds no rubric signal beyond the payload.

Rebased onto current master after #871 merged. One commit on top of master.

Headline (CLEAN): verdicts 100/100, blockers 211/211, wrong claims 0, Anthropic spend $0 / $6 budget. No new brick bugs. Re-scored after the rebase: same numbers.

Changes

  • docs/benchmarks/blockers-accuracy-scale.md — published scale result
  • Pointer from docs/benchmarks/blockers-accuracy.md; CHANGELOG + harness README
  • tools/blockers-accuracy/targets.json — 100 PRs / 10 repos
  • fixtures-scale/ + results/2026-10-25-scale/ — snapshot and brick-raw scores
  • score.py — strip github.blockers: quote text read from GitHub so it reaches the answer as data #872 inline-code backticks on check names when mapping brick reasons (without this, every required-check claim scored as fabricated)

Testing

cd tools/blockers-accuracy
python3 score.py results/2026-10-25-scale --fixtures fixtures-scale --brick-dir brick
# → verdict_correct 100/100, found 211/211, overcall/fabricated/unattributed 0

Re-score of the post-#874 44-PR snapshot after the quote strip: still 109/109. Re-run after rebase onto master: still 100/100 and 211/211.

Testing strategy (blast radius)

Docs + harness only. No product code change.

  • N/A kernel / prod-style gates for this docs+fixtures PR

Checklist

Records / docs touched (for drift audit)

  • docs/benchmarks/blockers-accuracy-scale.md
  • docs/benchmarks/blockers-accuracy.md
  • CHANGELOG.md
  • tools/blockers-accuracy/**
Open in Web Open in Cursor 

…brick 1.1.0 CLEAN

Expand tools/blockers-accuracy beyond the 44-PR sample: capture fixtures-scale for 100
public PRs across 10 repos (incl. IanFrelinger/Ashlar), grade brick-raw only.
Verdicts 100/100, blockers 211/211, no wrong claims, $0 Anthropic spend.
score.py strips #872 inline-code quotes on check names so brick reasons match truth keys.

Co-authored-by: icfrelinger <icfrelinger@gmail.com>
@cursor
cursor Bot force-pushed the cursor/blockers-accuracy-scale branch from 4d2e432 to 05238e2 Compare October 26, 2026 04:21
@cursor
cursor Bot merged commit 3717068 into master Oct 26, 2026
34 of 35 checks passed
@cursor
cursor Bot deleted the cursor/blockers-accuracy-scale branch October 26, 2026 04:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants