fix(evalboard): Path-to-GA shows only still-tagged tasks, scored on runs that executed

[uipreliga]

Why

Two independent bugs made /path-to-ga overstate GA readiness, plus a pricing-mirror drift found on the way.

1. Stale tag rows. A run.json task row carries tags as a historical stamp written at execution time; the board never re-reads the skills repo. getTagTaskBreakdown unioned every task that carried the tag in any run in the 30-day window, so seven tasks de-tagged upstream on 2026-07-16 kept rendering as GA candidates for a month. The live board showed 15 tasks; 8 were current.

2. Mature carry-forwards counted as measured passes. The nightly "matures" a task that has passed 5× consecutively: it skips execution and carries the row forward as status: SUCCESS, weighted_score: 1.0, mature_skipped: true. On 2026-07-31, 551 of 1005 rows were such skips — and a GA-readiness page was reporting them as passes.

What changed

Data layer — lib/overview.ts

Extracted a pure buildTagTaskRows(perRun, tag) (mirroring lib/trends.ts::aggregate); getTagTaskBreakdown is now a 9-line IO wrapper.

• The de-tag rule is proof, not a heuristic. A task is dropped only when it appears in a newer run in the window whose rows for it don't carry the tag. A task that merely stopped appearing (retired / renamed / skip: true) is unknowable from run data, so it is kept and dated — hence the new "Last seen" column.
• One predicate, applied symmetrically. Split taskCarriesRepoTag out of taskMatchesTag and used it on both the accumulate and de-tag paths. Review tags (review_index.json) are post-hoc annotations from an effectively disjoint namespace — a spike over 84 runs found 100 review tags vs 451 YAML tags with 2 incidental collisions and zero carrying path-to-ga — so using them for the de-tag signal would make an unreviewed newest run (the normal case) look like a de-tag. taskMatchesTag itself is unchanged, so the front-page rails keep filtering on review tags.
• passRate is now over executed appearances only, null when nothing ran, alongside matureSkips and latestMatureSkipped.

Deliberate, page-local divergence. lib/trends.ts and app/runs/[id] count a mature skip as a pass and exclude it only from cost/duration averages. This page excludes it from both terms, because GA readiness must report measured passes. That difference is intentional and commented as such — please don't "harmonise" it. Authoritative P/R/F1 still comes from the mature-inclusive surfaces.

Presentation — app/path-to-ga/

Table extracted into task-table.tsx (pure, no "use client", no fetching) so it is render-testable; page.tsx is now the IO shell.

• New Last seen column; (n mature) beside Appearances; MaturePill + — for the score when the latest appearance was a skip (never a Mature pill beside an inherited 1.00); — rather than NaN%/0% for an unmeasured rate.
• A caveat paragraph under the tiles: the headline rate and chart keep their mature-blind, union-over-window semantics (they feed the front page and every tag-filtered view), so they legitimately read higher than the table. The distinct-tasks tile now says "still tagged".

Pricing mirror resync — lib/pricing.ts

pricing-parity.test.ts had been failing on main for a while. pricing.py is authoritative and is untouched; only the mirror moves.

• Opus 4.5+ carried the pre-repricing $15/$75 instead of $5/$25 — every Opus cost on the board read 3× high. Same 3× error on gemini-3-flash-preview.
• Five models that appear in run data rendered — for cost: claude-sonnet-5 (~32k rows in runs-remote/), gpt-5.6-terra (~17k), gpt-5.6-sol / gpt-5.6-luna (~2k each), claude-opus-5 (~3k). Four of them sat in the parity test's DELIBERATELY_UNMIRRORED set under "the evalboard never runs them" — the escape hatch was silencing the guard on a live bug. That set is now trimmed to ids genuinely absent from the corpus, with a note to grep before adding one.
• Dropped claude-opus-4-6-20250514 / claude-sonnet-4-6-20250514 — Claude 4's release date pasted onto 4-6 keys, ids the backend never knew. The date-strip fallback resolves them to the undated keys, so nothing regresses.

Verification

Verified against the real 32-run window (2026-07-09 … 2026-08-07) in evalboard/runs-remote/:

• Exactly the 7 documented tasks drop; exactly the 8 still-tagged remain.
• Paired control (so the result isn't just runs ageing out): 2026-07-16_04-24-15 is still inside the window with 24 runs newer than it, so the drop is the de-tag rule doing its job.
• Predicted rate changes reproduce exactly: resolution-writer 75% → 73% measured (6 of 32 appearances were skips), invoice-lookup → 71%, hitl-invoice-approval → 59%.
• Confirmed in the browser: 7 columns fit inside TableScroll, three rows show (n mature).

pnpm exec vitest run 456/456 (green for the first time — the 3 pricing-parity failures were pre-existing on main), tsc --noEmit clean, next build clean. No Python change: git diff -- src/ is empty.

Tests: 19 new for buildTagTaskRows, 11 for TagTaskTable / fmtRunDate. The de-tag drop is asserted directly, modelled on the real ipe-drive-to-slack case. Two tests were caught passing for the wrong reason during review (a downstream rule masked the mutation they claimed to catch) and replaced with mutation-verified versions.

Known gaps (not fixed here — they exist on the Python side too, so they aren't drift)

• virtuoso-1-5 is the single most common model in the corpus (46,547 rows) and neither table prices it. It's registered via the plugin register_pricing seam in coder_eval_uipath, which the frontend can't see. Real architectural gap.
• eu.anthropic.claude-haiku-4-5-20251001-v1:0 (833 rows) — neither normalizeModel nor Python's _normalize_model strips the Bedrock -v1:0 suffix. One-line fix, best made in pricing.py first so the mirror stays a mirror.
• Four TS-side invariants this change surfaced are recorded in .claude/harness-candidates.md — all need a TypeScript lint harness evalboard/ doesn't have (the CExxx runner is Python-only over src/coder_eval/). Notably: run-view.tsx:283 still inlines taskCarriesRepoTag's body, deliberately not adopted because importing from lib/overview.ts would drag server-only code into a client bundle.

🤖 Generated with Claude Code

[github-actions]

Claude finished @uipreliga's task in 1m 26s —— View job

---

Code Review in Progress

I'm reviewing this PR following the guidelines in .github/code_review.md.

Todo List:

• Read code review guidelines and project conventions
• Get full diff from origin/main
• Review each changed file with full context
• Check cross-file consistency
• Perform design-level scrutiny
• Provide structured feedback

[uipreliga]

Review: coder_eval — pr:94

Scope: pr:94 · branch fix/evalboard-path-to-ga-accuracy · 0f94307 · 2026-08-08T04:22Z · workflow variant

Change class: complex — introduces new de-tagging control flow in buildTagTaskRows, changes the passRate denominator/type semantics (number -> number|null), extracts a new client component, and resyncs a duplicated pricing table across a Python/TS boundary; correctness requires reasoning about run-ordering, replicate collapsing, and cross-file parity.

The harness core is in excellent shape — clean security and error-handling axes, no critical or high findings in Python, and a genuinely discriminating new test block (7 of 8 planted mutants killed) — but the real risk is concentrated in evalboard, where 478 lines of vitest guards including the pricing-parity drift check are run by no CI job, Makefile target or hook (they were red on main and a 3x-wrong Opus rate plus 17 unmirrored models shipped anyway), so the bottom line is: merge-ready once the TypeScript surface is actually gated, since today its numbers-on-a-GA-board are unenforced by construction.

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	9.3 / 10	0	0	1	2	buildTagTaskRows carries a redundant seenInRun Set and an extractable third pass (92 lines, ~17 decision points)
2. Type Safety	9.9 / 10	0	0	0	1	New pass-count path compares the untyped status: string against a raw "SUCCESS" literal instead of the shared isPassStatus predicate
3. Test Health	8.5 / 10	0	1	1	0	The pricing drift guard — and the 478 lines of vitest this PR adds — is executed by no CI job, no Makefile target and no pre-commit hook, so it was red on main and the drift shipped anyway
4. Security	10 / 10	0	0	0	0	—
5. Architecture & Design	9.8 / 10	0	0	0	2	The executed-appearance denominator is computed twice — once as the value, once as the tooltip that describes it
6. Error Handling & Resilience	10 / 10	0	0	0	0	—
7. API Surface & Maintainability	9.8 / 10	0	0	0	2	Path-to-GA table copy describes state it no longer matches (per-run MATURE_TOOLTIP on a 30-day Appearances cell; empty state after a de-tag)
8. Evaluation Harness Quality	9.4 / 10	0	0	1	1	Pricing-parity drift guard's DELIBERATELY_UNMIRRORED opt-out list is hand-maintained, unexpiring and unverified — it hid four in-use models

Overall Score: 9.6 / 10 · Weakest Axis: Test Health at 8.5 / 10
Totals: 🔴 0 · 🟠 1 · 🟡 3 · 🔵 8 across 8 axes.

Blockers

1. [Axis 3] The pricing drift guard — and the 478 lines of vitest this PR adds — is executed by no CI job, no Makefile target and no pre-commit hook, so it was red on main and the drift shipped anyway (evalboard/lib/__tests__/pricing-parity.test.ts:14) — The file's own contract is a lie today: line 9-14 say "every model priced in lib/pricing.ts must exist in pricing.py with identical rates (a frontend rate that disagrees with the backend, or prices a model the backend doesn't, fails the build)" and lib/pricing.ts:18-20 says "fail the build on drift". Nothing builds it. grep -rn "pnpm\|vitest" .github/workflows/ returns only one hit — a comment at .github/workflows/pr-checks.yml:102 ("(evalboard/pnpm-lock.yaml, template node_modules) are") excluding the lockfile from osv-scanner; the quality-gate job runs ruff/pyright/pytest/bandit only, and grep -rn evalboard Makefile .pre-commit-config.yaml returns nothing. Proof this is not theoretical: I ran the parity test against origin/main in a scratch worktree — Test Files 1 failed (1) / Tests 3 failed | 1 passed (4), reporting 17 models priced in pricing.py and missing from pricing.ts (claude-opus-5, claude-haiku-4-5, gpt-5.1-codex-max, gemini-3.6-flash, …) plus the 3x Opus rate divergence ($15/$75 vs the authoritative $5/$25 at src/coder_eval/pricing.py:29-34). The guard detected the drift and the drift still reached the dashboard, which is what commit 0f94307 is now cleaning up. Fix: add a evalboard job to .github/workflows/pr-checks.yml (actions/setup-node@v4 + corepack enable + pnpm install --frozen-lockfile + pnpm verify, path-filtered on evalboard/** and src/coder_eval/pricing.py so a Python-side reprice also trips it). Until that exists, every assertion added by this PR is documentation, not enforcement.

Non-blocking, but please consider before merge

1. [Axis 1] buildTagTaskRows carries a redundant seenInRun Set and an extractable third pass (92 lines, ~17 decision points) (evalboard/lib/overview.ts:715) — overview.ts:715-725 builds two Sets per run but only one is load-bearing:

const seenInRun = new Set<string>();
const taggedInRun = new Set<string>();
for (const t of overview.tasks) {
    seenInRun.add(t.taskId);
    if (taskCarriesRepoTag(t, tag)) taggedInRun.add(t.taskId);
}
for (const taskId of seenInRun) {
    if (!newestTagged.has(taskId)) {
        newestTagged.set(taskId, taggedInRun.has(taskId));
    }
}

seenInRun exists only to de-duplicate replicate rows before the second loop — but newestTagged.has(taskId) (L722) already does exactly that de-duplication, and taggedInRun is fully populated over ALL rows before the second loop starts, so first-row-wins is already correct. The Set and its loop collapse to:

for (const t of overview.tasks) {
    if (!newestTagged.has(t.taskId)) newestTagged.set(t.taskId, taggedInRun.has(t.taskId));
}

More broadly, the function spans overview.ts:686-777 (92 lines) and contains 16 branch/loop/nullish points (L708 for, L713 if, L717 for, L719 if, L721 for, L722 if, L727 for, L728 if, L730 if, L742 ??, L747 if, L749 else if, L756 for, L762 if + ??, L769 ternary) → cyclomatic complexity ≈ 17. The newest-tagged computation (L715-725 + the L706 map + the L762 drop) is a self-contained concern; extracting it as function newestTaggedByTask(sorted: PerRun[], tag: string): Map<string, boolean> would drop the host function to ~60 lines / CC ≈ 11 and make the de-tag rule directly unit-testable instead of only observable through row output.
2. [Axis 3] New Path-to-GA logic ships with unasserted/uncovered paths (measured-0% boundary, passRateTooltip branches, getTagTaskBreakdown at 0%) (evalboard/lib/overview.ts:769) — Line 769 is passRate: executed > 0 ? (e.executedPasses / executed) * 100 : null, — the gate that turns a gap into a score on a GA-readiness page. I mutated it to passRate: e.executedPasses > 0 ? (e.executedPasses / executed) * 100 : null, and ran the suite: Tests 60 passed (60) — SURVIVED. Under that mutant a task whose every executed appearance FAILED renders "—" (via task-table.tsx:150 r.passRate != null ? ... : "—"), i.e. the worst task on the page reads as "no data" instead of a red 0%. The new describe("buildTagTaskRows") block covers 66.67% (mixed) and null (all-mature) but never the all-fail boundary. Add a test to evalboard/lib/tests/overview.test.ts: two runs of the same tagged task with status: "FAILURE", asserting rows[0].passRate is 0 (not null, not NaN) and matureSkips is 0. For calibration, seven other mutants I planted on this function WERE killed (de-tag drop removal, mature-inclusive denominator, mature-counts-as-pass, latestMatureSkipped pinned false, last-write-wins on newestTagged, reversed run sort, dropped output sort), so this is a single hole in an otherwise discriminating suite.
3. [Axis 8] Pricing-parity drift guard's DELIBERATELY_UNMIRRORED opt-out list is hand-maintained, unexpiring and unverified — it hid four in-use models (evalboard/lib/__tests__/pricing-parity.test.ts:87) — The final commit of this PR (0f94307) exists because four in-use models sat in the guard's own escape hatch: the diff removes claude-sonnet-5, gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna from DELIBERATELY_UNMIRRORED (pricing-parity.test.ts:87), and the added comment concedes they "appear ~32k / ~2k / ~17k / ~2k times in runs-remote/, so every one of those runs rendered '—' for cost with nothing failing." The remedy shipped is prose only — // KEEP THIS SET HONEST. at line 80. The mechanical check is unchanged: const missing = Object.keys(py).filter((m) => !(m in PRICING) && !DELIBERATELY_UNMIRRORED.has(m)); (line 102) still lets the next engineer silence a real omission with one line, and nothing asserts the set members even still exist in pricing.py, so a stale exemption never fails either. Two concrete strengthenings, both a few lines: (1) expect([...DELIBERATELY_UNMIRRORED].filter((m) => !(m in py))).toEqual([]) — a stale exemption becomes a build failure instead of a permanent silencer; (2) since "absence from run data is the ONLY justification" is not statically checkable, close the loop at the consumption end instead — scan evalboard/runs-remote/*/run.json for distinct model_used values when the directory is present (the test already reads the filesystem for pricing.py) and assert every one resolves via resolvePricing, skipping when the corpus is absent so CI stays green. Failing that, make an unpriced model visible in the UI rather than an anonymous "—", so the symptom is reportable.

Nits

1. [Axis 1] TagTaskTable's window prop shadows the DOM global window, and no lint gate exists to catch it (evalboard/app/path-to-ga/task-table.tsx:44) — task-table.tsx:41-50 destructures a prop literally named window:

export function TagTaskTable({
    rows,
    tag,
    window,
    harness,
}: {
    rows: TagTaskRow[];
    tag: string;
    window: Window;

Inside this component body window is now the string "30d", not the browser global. The file's own header comment (L11-13) contemplates the "use client" boundary, and sibling client components in this app do use the real global (app/_components/scroll-table.tsx:54 window.addEventListener("resize", measure);, app/_components/search-box.tsx:48 new URLSearchParams(window.location.search)), so a future edit here hits a confusing type error at best. Rename the prop to windowLabel (or range). Secondary nit in the same file: task-table.tsx:40-41 has no blank line between the end of passRateTooltip and export function TagTaskTable({. Neither is mechanically caught — evalboard/package.json defines "verify": "tsc --noEmit && vitest run && next build" with no eslint/biome/prettier step at all. Adding eslint with next/core-web-vitals plus no-restricted-globals/no-shadow to pnpm verify is the durable fix and is the TS-side equivalent of this repo's CEnnn harness rule.
2. [Axis 1] Comment block added to buildTagTaskRows outweighs the code and hardcodes unverified cross-file line numbers (evalboard/lib/overview.ts:680) — overview.ts:680-688 pins design rationale to line numbers in another file:

// MATURITY — a DELIBERATE, page-local divergence. lib/trends.ts:158-171 (and
// app/runs/[id]/run-view.tsx) count a mature carry-forward as a pass and exclude
...
    // Run ids are date-shaped, so a lexical sort is chronological — the same
    // assumption trends.ts:90 and the previous implementation already make.

Both references are accurate at this SHA (trends.ts:158-171 is the if (t.status === "SUCCESS") { ... if (!t.matureSkipped) { ... } } block; trends.ts:90 is const sorted = [...perRun].sort((a, b) => b.id.localeCompare(a.id));) but nothing keeps them accurate — any insertion in trends.ts silently makes them point at unrelated code, and a stale pointer is worse than none because a reader trusts it. Cite the symbol (lib/trends.ts::aggregate) rather than the line, as the same block already does correctly on overview.ts:658-659 ("mirrors lib/trends.ts::aggregate"). Context for the volume: overview.ts:656-777 is 122 lines of which 48 are comment-only (sed -n '656,777p' | grep -c '^\s*//' → 48), i.e. 48 comment lines to 74 code lines. Most of that prose is genuinely load-bearing and is backed by the new describe("buildTagTaskRows") block, so this is a pointer-rot nit rather than a comments-instead-of-tests problem.
3. [Axis 2] New pass-count path compares the untyped status: string against a raw "SUCCESS" literal instead of the shared isPassStatus predicate (evalboard/lib/overview.ts:749) — The new executedPasses accumulator reads } else if (t.status === "SUCCESS") {. RunOverviewTask.status is typed string | null (evalboard/lib/runs.ts:854), so the compiler cannot catch a typo or a widened pass set. evalboard/lib/status.ts is documented as the "Single source of truth for coder_eval task status categorization" and exports isPassStatus(status: string | null): boolean (evalboard/lib/status.ts:22-24), already adopted by evalboard/app/runs/[id]/task-grid.tsx:522-523. Semantics are identical today (statusCategory maps only SUCCESS to passed), so nothing is broken; the risk is that path-to-ga's pass rate silently diverges from every other surface if a second passing FinalStatus is ever added. Noting that the raw literal is a pre-existing repo pattern (evalboard/lib/trends.ts:158, evalboard/lib/overview.ts:871 and :980, evalboard/lib/watchlist.ts:87), so this is carried-forward debt the rewrite reproduced rather than new debt. Fix: } else if (isPassStatus(t.status)) {, and consider narrowing RunOverviewTask.status to a TaskStatus string-literal union so === "SUCESS" becomes a compile error.
4. [Axis 5] The executed-appearance denominator is computed twice — once as the value, once as the tooltip that describes it (evalboard/app/path-to-ga/task-table.tsx:25) — buildTagTaskRows computes the denominator at evalboard/lib/overview.ts:763 (const executed = e.appearances - e.matureSkips;) but does not put it on the row; task-table.tsx:25 re-derives the identical expression (const executed = r.appearances - r.matureSkips;) to render Measured over ${executed} executed appearance.... The tooltip is an independent restatement of the aggregator's rule, so if the exclusion set ever widens (e.g. also dropping ERROR rows, or rows with no recorded status) the percentage moves and the tooltip keeps asserting the old sample size — a caption that silently lies about a number sitting next to it. Add executed: number (or executedPasses) to TagTaskRow in overview.ts:632-654 and have the tooltip read it, so the number and its explanation come from the same computation.
5. [Axis 5] A second page's aggregation grew inside lib/overview.ts (now 1010 lines) instead of the per-page module convention it claims to mirror (evalboard/lib/overview.ts:686) — The new comment at overview.ts:656-659 says buildTagTaskRows is "Pure over the PerRun[] it is handed ... mirrors lib/trends.ts::aggregate" — it adopts trends.ts's shape but not its placement. Both siblings that do this put the page's model in its own module reusing only the PerRun loader: lib/trends.ts:1-12 ("Per-task trend aggregations ... Reuses the PerRun loader from overview.ts") and lib/watchlist.ts:1-12 ("Pure aggregation for the Watchlist page. Input: the PerRun[] already loaded ... No I/O, no React — fully unit-testable"). Here the whole /path-to-ga model — TagTaskRow (632-654), buildTagTaskRows (686-777), getTagTaskBreakdown (783-795) — stays in lib/overview.ts, which this PR pushes to 1010 lines and four concerns (windowed blob fetch + cache, front-page chart/tiles/rails, run + ad-hoc listings, path-to-ga breakdown). Not painful yet, and the placement is inherited rather than invented, but the extraction is mechanical: move those ~145 lines to lib/path-to-ga.ts, importing PerRun, loadWindowData/normalizeHarness, and taskCarriesRepoTag — the same seam trends.ts and watchlist.ts already use.
6. [Axis 7] Path-to-GA table copy describes state it no longer matches (per-run MATURE_TOOLTIP on a 30-day Appearances cell; empty state after a de-tag) (evalboard/app/path-to-ga/task-table.tsx:118) — The new mature annotation on the Appearances column reuses the shared per-run string:

                                    {r.matureSkips > 0 && (
                                        <span
                                            className="text-gray-500"
                                            title={MATURE_TOOLTIP}
                                        >
                                            {" "}
                                            ({r.matureSkips} mature)
                                        </span>
                                    )}

MATURE_TOOLTIP (lib/pills.tsx) reads: "Mature: skipped this run to save cost and carried forward as a pass (re-validated about weekly on its fixed slot). It was not executed this run." That copy was written for a single-run surface (the MaturePill on /runs/[id]), and it is correct there. On /path-to-ga the cell is a window aggregate — hovering (3 mature) on a 24-appearance row returns text about "this run" twice, on a page that renders no single run at all. Its use two cells to the right on the MaturePill (task-table.tsx:153) IS correct, because that pill does describe one row, so the same string reads right in one place and wrong in the other within the same <tr>.

The adjacent pass-rate tooltip shows the right register for an aggregate — app/path-to-ga/task-table.tsx:34-38: `Measured over ${executed} executed appearance` … ` (${r.matureSkips} mature carry-forward${...} excluded).` Give the Appearances annotation a matching aggregate-voice string (e.g. "3 of these 24 appearances were mature carry-forwards — skipped to save cost and carried forward as a pass, not executed."), either inline or as a sibling export in lib/pills.tsx next to MATURE_TOOLTIP / matureLinkTooltip / MATURE_NO_SOURCE_TOOLTIP, which is already the established shape for per-context variants of this message.
7. [Axis 7] taskCarriesRepoTag extracted as the single repo-provenance predicate, but the duplicate copy at run-view.tsx:283 survives (and cannot adopt it from a server-only module) (evalboard/lib/overview.ts:523) — The new export states the DRY intent in its own doc comment (overview.ts:521-524):

// post-hoc annotations, and an unreviewed run carries none. Split out so
// buildTagTaskRows can use exactly this half without inlining a second copy.
export function taskCarriesRepoTag(task: RunOverviewTask, tag: string): boolean {
    return task.skill === tag || task.tags.includes(tag);
}

But it lands in lib/overview.ts, whose line 5 is import { unstable_cache } from "next/cache"; plus the blob readers from ./runs — so no "use client" module can import it. I verified the one remaining duplicate the PR's own notes call out: app/runs/[id]/run-view.tsx:283 is exactly (tag) => t.tags.includes(tag) || t.skill === tag, inside a "use client" component. The extraction is therefore structurally unable to reach the only call site that was already duplicating it, and any future client-side tag filter will duplicate it again for the same reason.

.claude/harness-candidates.md defers this honestly and diagnoses it correctly ("importing the predicate there would drag server-only code into the client bundle… extracting the predicate into a dependency-free module (e.g. lib/tags.ts)"), and concludes it is "worth doing next time either file is touched, not worth a standalone change." That reasoning is sound for a standalone change — but this PR is already creating the symbol, so the placement is a decision inside the diff, and the cost of getting it right now is a new ~6-line lib/tags.ts holding the two-line pure function plus a re-export from overview.ts (zero behaviour change, covered by the existing describe("taskCarriesRepoTag / taskMatchesTag") tests). Doing it here also lets run-view.tsx:283 be de-duplicated the next time it is touched, instead of requiring the extraction first. With no eslint in evalboard/, no mechanical gate will ever prompt this later.
8. [Axis 8] "Last seen" can name a run in which the task did not execute (evalboard/app/path-to-ga/task-table.tsx:132) — {fmtRunDate(r.latestRunId)} (task-table.tsx:132) is titled Newest run in the window that carried this task under ${tag} (line 130), and fmtRunDate exists per its own doc comment "for columns that need to answer 'how old is this'" (format.ts:19-20). But latestRunId is the newest TAGGED appearance, which may be a mature carry-forward that never ran: on the current corpus skill-flow-devcon-billing-resolution-writer shows Last seen 2026-08-07 while its newest executed run is 2026-08-05, and skill-bpmn-e2e-live-debug shows 2026-08-07 against 2026-08-05. The MaturePill in the Latest-status column is the only cue, and it is two columns away. Gap is 0-2 days today but is bounded only by the mature re-validation cadence ("about weekly" per MATURE_TOOLTIP). Either relabel the column "Last appearance" or carry a separate latestExecutedRunId so the age signal on a GA board means "last actually measured".

What's Missing

Parallel paths:

• 🟠 /trends?tag=path-to-ga still reproduces the exact defect this PR's own test names "the de-tag bug": lib/trends.ts::aggregate accumulates tagSet as a UNION over the window and trendMatchesTag (lib/trends.ts:257-259) matches on that union, so a task de-tagged upstream keeps rendering until it ages out — and its passRate (b.successCount / b.totalCount) still counts mature carry-forwards as passes. The fix landed only in buildTagTaskRows, and only /path-to-ga got the caveat paragraph, so two adjacent pages now answer "which tasks carry this tag, and how are they doing" with different rules and no cross-reference. Either apply the newest-run de-tag rule to the trends tag filter or state the divergence on /trends the way page.tsx does. (trigger: evalboard/lib/overview.ts)
• 🔵 A third inline copy of the repo-provenance predicate went unrecorded: lib/trends.ts:257-259 (trendMatchesTag) opens with trend.skill === tag / trend.tags.includes(tag) over a TaskTrend. The PR's harness-candidates deferral names only app/runs/[id]/run-view.tsx:283, so the proposed lib/tags.ts extraction as scoped would still leave this one behind — the shared module needs a shape-based signature ({skill, tags}) rather than a RunOverviewTask-typed one to absorb both. (trigger: evalboard/lib/overview.ts) (restates: Axis 7: taskCarriesRepoTag extracted as the single repo-provenance predicate, but the duplicate copy at run-view.tsx:283 survives)

Daily/nightly:

• 🟡 The Path-to-GA pass rate now depends on a run.json field this repo does not produce: mature_skipped has zero occurrences in src/coder_eval/ (it is stamped by the external nightly eval_runner), and buildTagTaskRows reads it as t.matureSkipped ?? false. If the producer renames or drops it, the page silently reverts to counting carry-forwards as executed passes — the rate inflates, the (N mature) annotation and Mature pills disappear, and nothing errors. The PR documents the intra-board divergence meticulously but never states this cross-repo contract dependency or the nightly blast radius; a one-line note plus a schema assertion on the field would close it. (trigger: evalboard/lib/overview.ts)

Display & mapping dicts:

• 🟡 The new de-tag drop has no rendered affordance: dropped rows simply vanish and the "distinct tasks still tagged" tile just shrinks, so a genuine upstream de-tagging, a producer that stopped stamping tags, and an empty window all render identically. That is precisely the fail-closed hazard .claude/harness-candidates.md records as knowingly unguarded — a "N task(s) dropped as de-tagged in this window" line next to the caveat paragraph would make the new rule observable without adding the taggedInRun.size === 0 guard they deliberately rejected. (trigger: evalboard/app/path-to-ga/task-table.tsx)
• 🔵 evalboard/README.md's Layout section — the only prose inventory of the dashboard's pages — still lists just /, /trends, /runs/latest, /runs/<id> and /runs/<id>/<task-id>; /path-to-ga (whose semantics this PR materially redefined: executed-only pass rate, newest-run de-tag drop, maturity annotations) and /watchlist are absent, so the page's rules exist only inside code comments. Add a /path-to-ga bullet naming the two rules that differ from every other surface. (trigger: evalboard/README.md)

Tests:

• 🟡 The parity guard's Python parser has no row-count assertion: parsePythonTable matches only the single-line "id": ModelPricing(a, b, c, d) form (all 56 entries happen to be single-line today) and the only sanity check is keys.length > 10. A ruff format reflow or a switch to keyword args in pricing.py silently shrinks the parsed table, after which the reverse-direction test stops reporting exactly the class of omission commit 0f94307 had to fix by hand. Add expect(Object.keys(py).length).toBe(<count of /ModelPricing\(/ matches in the source>) so a parse regression fails loudly instead of narrowing the guard. (Adjacent to, but mechanically distinct from, the DELIBERATELY_UNMIRRORED finding.) (trigger: evalboard/lib/tests/pricing-parity.test.ts)
• 🟡 The IO wrapper's own contract is untested even though it carries a load-bearing claim: getTagTaskBreakdown states "Harness scoping happens HERE, before the pure function sees the runs, so a newer run on a different harness cannot de-tag a row in a harness-scoped view", plus the !r.adhoc filter. Every new test drives buildTagTaskRows directly and bypasses both. A fake loadWindowData fixture where the newest run is ad-hoc / on another harness and carries the task untagged would pin that the row survives — today that invariant rests on prose. (trigger: evalboard/lib/overview.ts) (restates: Axis 3: New Path-to-GA logic ships with unasserted/uncovered paths)
• 🔵 The server shell left behind by the extraction is still untested, despite the split being made "purely so it is render-testable": nothing covers the headline tile math overview.runs.reduce((sum, r) => sum + (r.successRate ?? 0), 0) / runsInWindow (a run with a null successRate silently dilutes the average toward 0 rather than being excluded from the denominator), the new caveat paragraph, or the "distinct task(s) still tagged" pluralization/count. Hoisting avgPassRate into a tiny exported pure helper would make the null-successRate rule assertable. (trigger: evalboard/app/path-to-ga/page.tsx)

Downstream consumers:

• 🟡 The pricing resync retroactively changes every already-rendered cost figure for the repriced models — Opus 4.6/4.7/4.8 and the dated claude-opus-4-6-20250514 fall 3x ($15/$75 → $5/$25 via the dated→undated fallback), claude-haiku-4-5-20251001 rises 25% (0.8/4 → 1/5), gemini-3-flash-preview falls 3x — across lib/runs.ts::messageCostUsd, the Tokens↔USD toggle in app/runs/[id]/task-grid.tsx and [...task]/_sections.tsx, and the thinking-cost simulator (lib/thinkingSim.ts:317). Task/run totals keep coming from run.json's recorded total_cost_usd, and nothing asserts the message-level sum now agrees with it; add a fixture test summing messageCostUsd against a recorded task total, and state the retroactive display change in the commit body. (trigger: evalboard/lib/pricing.ts)
• 🔵 Plugin-registered rates sit entirely outside the mirror: pricing.py exposes register_pricing as a first-class seam for out-of-tree agents (the coder_eval_uipath Delegate SDK agent is the worked example), and the parity test parses only the static literal table, so a model priced solely by a plugin renders "—" for cost on the board with nothing failing — the identical silent symptom as the four ids this PR just un-exempted. Either mirror the plugin rates or make an unpriced model visibly flagged in the UI so the symptom is reportable. (trigger: evalboard/lib/tests/pricing-parity.test.ts) (restates: Axis 8: Pricing-parity drift guard's DELIBERATELY_UNMIRRORED opt-out list is hand-maintained, unexpiring and unverified)

Harness & Lint Improvements

Static checks (lint / type):

• [ce-lint] CE034 — pricing mirror parity, enforced from the Python side. New @pytest.mark.lint class in tests/test_custom_lint.py (+ helper tests/lint/pricing_mirror.py), following the CE026-CE031 whole-tree precedent (those already reason over Markdown/YAML rather than one .py AST). It imports coder_eval.pricing._PRICING directly, regex-parses the "<model>": p(a, b, c, d) rows out of evalboard/lib/pricing.ts, parses the DELIBERATELY_UNMIRRORED set out of evalboard/lib/__tests__/pricing-parity.test.ts (do NOT duplicate the list — read the existing one so there stays exactly one exemption SSOT), then asserts: (a) every TS-priced model exists in _PRICING, (b) all four rates match exactly, (c) every _PRICING model is either mirrored or exempt, (d) every exemption id still exists in _PRICING (stale-membership guard). I prototyped this in ~20 lines and ran it against both trees: on main it is RED — 2 orphans (claude-opus-4-6-20250514, claude-sonnet-4-6-20250514), 6 rate mismatches including the 3x Opus divergence (claude-opus-4-8 $15/$75 vs the authoritative $5/$25), 28 unmirrored models; at pr-94 HEAD it is GREEN with exactly the 7 documented exemptions. Critically it runs inside pytest tests/test_custom_lint.py, which pr-checks.yml quality-gate already executes ('Custom architectural lint (CE001+)'), so it needs no Node toolchain and no new CI job — it closes the pricing half of the high finding today at zero infra cost. Prevents: Axis 3 high — evalboard/lib/__tests__/pricing-parity.test.ts is executed by no CI job / Makefile target / pre-commit hook, so the guard was red on main and the drift shipped anyway (commit 0f94307 is the cleanup). Axis 8 medium — DELIBERATELY_UNMIRRORED (pricing-parity.test.ts:87) is hand-maintained and unverified; assertion (d) turns a stale exemption into a build failure. Also covers the reverse direction: a reprice in src/coder_eval/pricing.py now trips a Python gate immediately.
• [ce-lint] CE039 — every test suite in the repo must be executed by some CI job. Whole-tree lint class: for each package.json declaring a test/verify script (or each vitest.config.* / pytest suite root), assert its directory is named in at least one .github/workflows/*.yml run: body or paths: filter. Verified today: grep -rn "pnpm\|vitest" .github/workflows/ returns exactly one hit and it is a comment (pr-checks.yml:102, excluding evalboard/pnpm-lock.yaml from osv-scanner); grep -rn evalboard Makefile .pre-commit-config.yaml returns nothing. This fails the class ('a guard exists that nothing runs') rather than the instance, and keeps firing if a future docs/ or plugin sub-package lands its own unrun suite. Prevents: Axis 3 high — the 478 lines of vitest added by this PR (task-table.test.tsx, overview.test.ts, pricing-parity.test.ts, …) are documentation, not enforcement, because evalboard/package.json's "verify": "tsc --noEmit && vitest run && next build" is invoked by nothing automated.
• [ce-lint] CE035 — no raw status === "SUCCESS" outside evalboard/lib/status.ts. Text scan over evalboard/{lib,app}/**/*.ts{,x} forbidding \.?status\s*===\s*"SUCCESS"; lib/status.ts is the only allowed definition site, every other call site must use isPassStatus(). Current violations (verified): lib/trends.ts:158, lib/overview.ts:749 (the new code), lib/overview.ts:871, lib/overview.ts:980, lib/watchlist.ts:87, app/trends/trends-view.tsx:142, plus lib/pills.tsx:47 which also tests "Completed" and may need an explicit exemption. Land the rule together with the six mechanical rewrites (~10-line diff) so it starts clean. Prevents: Axis 2 low — the new executedPasses accumulator (evalboard/lib/overview.ts:749) compares an untyped status: string | null against a raw literal, so path-to-ga's pass rate silently diverges from every other surface the day a second passing FinalStatus appears. Also retires the five pre-existing copies the rewrite reproduced.
• [ce-lint] CE036 — the repo-provenance tag predicate may exist in exactly one place. Regex scan over evalboard/{lib,app} forbidding an inline tags.includes(<x>) || <y>.skill === <x> (either operand order) anywhere except the module defining taskCarriesRepoTag. .claude/harness-candidates.md already proposes this guard ('a lint rule ("no inline tags.includes(x) || skill === x") would catch future copies') and defers it for want of a TS harness — but it needs none: it is a two-line regex in the existing Python lint runner. Pair it with the finding's fix (move the predicate into a dependency-free lib/tags.ts so the "use client" copy can actually import it, re-export from overview.ts), otherwise the rule has no compliant home to point run-view.tsx at. Prevents: Axis 7 / 5 / 1 low (grouped) — taskCarriesRepoTag was extracted as the single predicate (evalboard/lib/overview.ts:523) but the identical body survives at app/runs/[id]/run-view.tsx:283, and the extraction landed in a next/cache-importing server module no client component can adopt, so the next client-side tag filter duplicates it again.
• [ce-lint] CE037 — component props/params must not shadow DOM globals. Scan evalboard/app/**/*.tsx for destructured prop or parameter identifiers in {window, document, location, history, navigator, screen, name, status}. evalboard/app/path-to-ga/task-table.tsx:44 declares a prop literally named window (value "30d") inside a client-component tree whose siblings genuinely use the global (app/_components/scroll-table.tsx:54 window.addEventListener, app/_components/search-box.tsx:48 window.location.search). Rename to windowLabel/range. A regex over destructuring patterns suffices here; the scope-aware version is the eslint no-restricted-globals / no-shadow pair in the harness bucket. Prevents: Axis 1 low — TagTaskTable's window prop shadows the DOM global with no mechanical gate to catch it (evalboard/package.json has no eslint/biome/prettier step at all; verified: no config file of any of those exists in evalboard/).
• [ce-lint] CE038 — no hardcoded cross-file line-number citations in comments. Repo-wide scan (src/**/*.py, tests/**/*.py, evalboard/**/*.ts{,x}) forbidding a comment matching [\w./-]+\.(py|ts|tsx):\d+(-\d+)?; cite the symbol instead (lib/trends.ts::aggregate), the form the same comment block already uses correctly at overview.ts:658-659. Exactly two violations exist in evalboard today and both are in the new block (lib/overview.ts:680 → lib/trends.ts:158-171, lib/overview.ts:688 → trends.ts:90), so the rule lands clean after a two-line edit. Keep the allowlist narrow (e.g. review artifacts under .claude/) — a stale pointer is worse than none, because a reader trusts it. Prevents: Axis 1 / 7 low (grouped) — the new buildTagTaskRows rationale block pins design intent to line numbers in another file that nothing keeps accurate; any insertion in trends.ts silently repoints them at unrelated code.

Harness improvements (not statically reachable):

• Add a path-filtered evalboard job to .github/workflows/pr-checks.yml: checkout → actions/setup-node (already present 5x in this file, lines 283/376/540/779/848, today only to npm install -g @anthropic-ai/claude-code) → corepack enable → pnpm install --frozen-lockfile → pnpm verify (tsc --noEmit && vitest run && next build), with paths: on evalboard/** and src/coder_eval/pricing.py so a Python-side reprice also trips the TS parity guard. No new action is needed, only the two pnpm steps. Why not static: Running tsc, 456 vitest tests and next build requires a Node/pnpm toolchain and actual execution; the Python lint harness can assert the job exists (CE039) but can never substitute for running the suite. Prevents: Axis 3 high — the entire evalboard suite, including the pricing drift guard that was red on main, is executed by no CI job, no Makefile target and no pre-commit hook.
• Make the evalboard suite reachable locally: add make evalboard-verify (cd evalboard && pnpm install --frozen-lockfile && pnpm verify), list it in the CLAUDE.md 'Development Commands' block next to make verify, and add a pre-commit hook scoped to files: ^evalboard/.*\.(ts|tsx)$ running pnpm -C evalboard test. Today grep -rn evalboard Makefile .pre-commit-config.yaml returns nothing, so a Python-focused contributor has no signal the directory even has tests. Why not static: Workflow and discoverability wiring — a lint rule can require the target to exist, but the value is that a human's habitual make verify actually exercises the JS side. Prevents: Axis 3 high — same root cause on the local-development loop; the pricing drift reached the dashboard because no routine command ever ran the guard.
• Stand up the TypeScript lint harness now — the deferral condition is met. .claude/harness-candidates.md (added by this same PR) parks four TS-side invariants with 'promote if a fourth TS-side invariant appears, and stand up the harness once for all of them'; this review adds at least four more (raw "SUCCESS" literal, DOM-global shadowing, inline tag predicate, aggregate-voice copy reuse). Add eslint + typescript-eslint + next/core-web-vitals to evalboard/, enable no-restricted-globals, no-shadow, @typescript-eslint/switch-exhaustiveness-check and a no-restricted-syntax entry for the inline tag predicate, and append eslint . to the verify script. Ship it with the type-level half of the Axis 2 finding: narrow RunOverviewTask.status (evalboard/lib/runs.ts:854) from string | null to a TaskStatus literal union so === "SUCESS" becomes a compile error. Then flip the four deferred entries in harness-candidates.md to 'promoted'. Why not static: The Python CE runner can regex TS text (which is why CE035-CE038 are worth landing first, ahead of this), but it cannot do scope-aware shadowing analysis, union exhaustiveness or type narrowing — those need the TS type-checker and eslint's scope manager. Prevents: Axis 1 low (window prop shadowing), Axis 2 low (raw "SUCCESS" against an untyped status), Axis 7/5/1 low (duplicated tag predicate), plus the four invariants already parked in harness-candidates.md.
• Close the boundary/mutation gap on the new aggregation. (a) Add the missing test to evalboard/lib/__tests__/overview.test.ts: two runs of the same tagged task, both status: "FAILURE", asserting rows[0].passRate === 0 (not null, not NaN) and matureSkips === 0. (b) Add coverage thresholds for lib/ in evalboard/vitest.config.ts — @vitest/coverage-v8 is already a devDependency but the config declares no coverage block at all, so getTagTaskBreakdown sits at 0% with nothing complaining. (c) Add a periodic (nightly, not per-PR) mutation run over the evalboard/lib/*.ts aggregation functions — Stryker, or a checked-in scripted mutant list — since hand-mutation is how both this hole and the two wrong-reason tests in the prior commit were found. Why not static: Mutation survival and coverage are properties of executing the suite against modified source; no AST rule can tell that passRate: executed > 0 ? … : null has an unasserted false-branch boundary. Prevents: Axis 3 medium — the mutant passRate: e.executedPasses > 0 ? … survives (60/60 file-local and 456/456 suite-wide stay green), which would render the worst task on the GA-readiness page as '—' instead of a red 0% (app/path-to-ga/task-table.tsx:148-150); plus getTagTaskBreakdown at 0% coverage.
• Close the pricing loop at the consumption end, out-of-band from PRs. Add a nightly/manual job (or a coder-eval analysis subcommand) that scans the blob run corpus for distinct model_used values and asserts each resolves via resolvePricing, and change the UI to render the unpriced model id instead of an anonymous '—' so the symptom is reportable by whoever sees it. Record the verified caveat with it: evalboard/runs-remote/ is gitignored (0 files tracked by git; 192 run dirs exist only as a local blob cache), so a 'skip when absent' corpus test is CI-inert — it must run where the corpus lives, or pull it. Why not static: 'Is this model actually in use?' is a property of production run data that does not exist in the repo; no lint or type check can see it, and the exemption justification ('no harness runs this model') is exactly the part that expires silently. Prevents: Axis 8 medium — four in-use models (claude-sonnet-5, gpt-5.6-sol/terra/luna; ~32k/~2k/~17k/~2k occurrences in runs-remote/) sat in DELIBERATELY_UNMIRRORED under an expired justification, so every one of those runs rendered '—' for cost with nothing failing. CE034(d) catches stale membership; only this catches a stale justification.
• Give the aggregate surfaces their own copy and pin it with tests. Add MATURE_AGGREGATE_TOOLTIP beside MATURE_TOOLTIP / matureLinkTooltip / MATURE_NO_SOURCE_TOOLTIP in lib/pills.tsx (the established shape for per-context variants) and use it for the Appearances annotation; put executed/executedPasses on TagTaskRow so task-table.tsx:25's tooltip reads the aggregator's denominator instead of re-deriving it; and either relabel 'Last seen' → 'Last appearance' or carry a separate latestExecutedRunId. Cover all three in app/path-to-ga/__tests__/task-table.test.tsx so the copy is pinned to the number it describes. Why not static: Whether a string's voice ('this run') matches the aggregate it labels, and whether a column title means 'last appeared' or 'last measured', are semantic judgments; the one mechanizable slice (an import-scope ban on MATURE_TOOLTIP outside app/runs/**) is only worth adding once the aggregate variant exists. Prevents: Axis 7 low (grouped) — per-run MATURE_TOOLTIP reused on a 30-day aggregate cell (app/path-to-ga/task-table.tsx:118); Axis 5 low — the executed denominator computed twice, once as the value (lib/overview.ts:763) and once as the caption describing it (task-table.tsx:25); Axis 8 low — 'Last seen' naming a run in which the task did not execute (observed 2026-08-07 vs a newest executed 2026-08-05 on two live tasks).
• Split the page model out of lib/overview.ts and unit-test the de-tag rule directly. Move TagTaskRow / buildTagTaskRows / getTagTaskBreakdown (~145 lines) into lib/path-to-ga.ts, importing only PerRun + loadWindowData/normalizeHarness + taskCarriesRepoTag — the seam lib/trends.ts and lib/watchlist.ts already use — and extract newestTaggedByTask(sorted, tag): Map<string, boolean> so the de-tag rule is testable in isolation rather than only observable through row output (this also drops the redundant seenInRun Set, whose de-duplication the newestTagged.has() guard already performs). Why not static: Module placement and function-size/complexity budgets are conventions, not mechanically-derivable invariants; a blanket file-length or cyclomatic-complexity gate over evalboard/ would fire on unrelated existing code and is not worth landing for this. Prevents: Axis 1 medium — buildTagTaskRows at 92 lines / CC ≈ 17 with a redundant seenInRun Set (lib/overview.ts:715); Axis 5 low — a second page's aggregation grown inside lib/overview.ts (now 1010 lines, four concerns) against the per-page module convention its own comment claims to mirror.

Top 5 Priority Actions

1. Add a path-filtered evalboard job to .github/workflows/pr-checks.yml (corepack enable + pnpm install --frozen-lockfile + pnpm verify, filtered on evalboard/** and src/coder_eval/pricing.py) — setup-node is already present 5x for the Claude CLI, and until this exists every assertion in evalboard/lib/tests/pricing-parity.test.ts:14 is documentation, not enforcement, which is exactly how the 3x Opus mispricing reached the dashboard.
2. Harden the DELIBERATELY_UNMIRRORED escape hatch at evalboard/lib/tests/pricing-parity.test.ts:87 — the four ids just removed (claude-sonnet-5, gpt-5.6-sol/terra/luna) made ~53k run appearances render '—' for cost with nothing failing, and the shipped remedy is a // KEEP THIS SET HONEST. comment; assert every exemption still exists in pricing.py and make an unpriced model render visibly instead of an anonymous '—' so the symptom is reportable even when the guard is silenced.
3. Close the surviving-mutant hole at evalboard/lib/overview.ts:769 by adding an all-FAIL case to evalboard/lib/tests/overview.test.ts asserting passRate === 0 (not null, not NaN) — mutating executed > 0 to e.executedPasses > 0 leaves all 456 evalboard tests green while making the single worst task on the GA-readiness page render '—' (no data) instead of a red 0% via task-table.tsx:150.
4. Replace the raw literal at evalboard/lib/overview.ts:749 (t.status === "SUCCESS") with the shared isPassStatus predicate from evalboard/lib/status.ts:22 and narrow RunOverviewTask.status (evalboard/lib/runs.ts:854) from string | null to a TaskStatus union, so path-to-ga's pass rate cannot silently diverge from every other surface if a second passing FinalStatus is added.
5. Make the Path-to-GA copy match what it measures: carry executed on TagTaskRow (evalboard/lib/overview.ts:632-654) instead of re-deriving the denominator in the tooltip at evalboard/app/path-to-ga/task-table.tsx:25, relabel or re-source the 'Last seen' column at task-table.tsx:132 (which today can name a run in which the task never executed, e.g. 2026-08-07 vs a true 2026-08-05), and give the Appearances cell an aggregate-voice tooltip rather than the per-run MATURE_TOOLTIP at task-table.tsx:118.

---

Stats: 0 🔴 · 1 🟠 · 3 🟡 · 8 🔵 across 8 axes reviewed.

[Mihaiii]

Luna price is much lower now (terra to be checked if it's the new price or the old one). Also, all the OpenAI models displayed here have the price for cache_write == input, which is incorrect (same pattern for gemini below - to be checked if it's correct).