Skip to content

fix(evalboard): Path-to-GA shows only still-tagged tasks, scored on runs that executed #94

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
uipreliga wants to merge 5 commits into
base: main
Choose a base branch
from
Open

fix(evalboard): Path-to-GA shows only still-tagged tasks, scored on runs that executed #94

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
4e5bbc4
fix(evalboard): 1/3 — drop de-tagged tasks and score only executed runs
uipreliga Aug 7, 2026
ecba70a
feat(evalboard): 2/3 — surface last-seen and maturity on the Path-to-…
uipreliga Aug 7, 2026
dd5f7e9
fix: code review fixes for evalboard path-to-ga stale tags and mature…
uipreliga Aug 7, 2026
f6fc9b8
docs(harness): defer four TS-side evalboard invariants from the path-…
uipreliga Aug 7, 2026
0f94307
fix(evalboard): resync lib/pricing.ts with the authoritative pricing.…
uipreliga Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions .claude/harness-candidates.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,3 +127,51 @@ Deferred lint/test guardrails surfaced during reviews. Promote to a `CExxx` rule
that environment injects the criterion. Until then CE030 stays scoped to the four
top-level models; the `command_pattern`/`exclude_pattern` contract this PR changed is
documented in the Field descriptions and TASK_DEFINITION_GUIDE regardless.

## From the evalboard Path-to-GA de-tag / mature-passes fix (4e5bbc4…dd5f7e9) — TS-side guards deferred

Context: the CExxx harness is a **Python** AST runner over `src/coder_eval/`, so none
of the invariants below are mechanizable in it. Each would need a TypeScript lint
harness (eslint config + custom rules) that `evalboard/` does not have today —
standing one up for three call sites fails the KISS/YAGNI gate. Deferring rather
than dropping; promote if a fourth TS-side invariant appears, and stand up the
harness once for all of them.

- [ ] **"Every consumer of `RunOverviewTask.matureSkipped` must decide explicitly
whether a carry-forward row counts."** Four consumers now, and they deliberately
DISAGREE: `lib/trends.ts` and `app/runs/[id]/run-view.tsx` count a mature skip as
a pass; `lib/overview.ts::buildTagTaskRows` excludes it from both terms
(`/path-to-ga` is a GA-readiness page). A new consumer silently inheriting either
convention is a real hazard. Guard shape: flag a file that reads `.matureSkipped`
without a nearby comment naming its convention — weak, hence the deferral. Closed
for now by unit tests that assert the exclusion from BOTH numerator and denominator
(`lib/__tests__/overview.test.ts` → `describe("buildTagTaskRows")`).

- [ ] **`taskCarriesRepoTag` is the single repo-provenance tag predicate — but one
duplicate survives.** `app/runs/[id]/run-view.tsx:283` still inlines its exact body
(`(tag) => t.tags.includes(tag) || t.skill === tag`). NOT adopted deliberately:
`run-view.tsx` is `"use client"` and `lib/overview.ts` imports `next/cache` plus the
blob readers, so importing the predicate there would drag server-only code into the
client bundle. Fixing it properly means extracting the predicate into a
dependency-free module (e.g. `lib/tags.ts`) — worth doing next time either file is
touched, not worth a standalone change. A lint rule ("no inline `tags.includes(x) ||
skill === x`") would catch future copies.

- [ ] **The de-tag rule fails CLOSED on a newest run that loads fine but stamps no
`tags`** (`lib/overview.ts::buildTagTaskRows`): every tagged task would read as
de-tagged and the table would empty, rendering an empty state indistinguishable from
a genuine full de-tagging. Its sibling failure mode (`overview == null`, a transient
blob read failure) IS guarded, with exactly this rationale. Currently unreachable —
0 of ~116k date-shaped non-ad-hoc task rows in `runs-remote/` lack `tags`, and the
six zero-tag runs found are all ad-hoc (filtered upstream by id shape + `meta.adhoc`)
— so the barrier is two upstream filters rather than a check at the seam. Left
unguarded on purpose: a `if (taggedInRun.size === 0) skip the de-tag signal` guard
would also mask a real, total de-tagging. Revisit if the pipeline ever stops
stamping tags, or if a non-ad-hoc run legitimately carries zero tagged rows.

- [ ] **Discriminating-test discipline for predicate narrowings.** Two tests in this
change passed for the wrong reason — a downstream rule (the de-tag drop) masked the
mutation they claimed to catch — and the plan leaned on a `grep` acceptance criterion
that CI never runs. Both were found by mutation-testing the suite and fixed. No
mechanizable guard; the durable lesson is: when a test names a narrowing, construct
the fixture so the row SURVIVES every other rule, or the assertion proves nothing.
117 changes: 117 additions & 0 deletions evalboard/app/path-to-ga/__tests__/task-table.test.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
import { describe, expect, test } from "vitest";
import { render, screen } from "@testing-library/react";
import type { TagTaskRow } from "@/lib/overview";
import { TagTaskTable } from "../task-table";

// TagTaskTable is a pure props-in/JSX-out component — no router hooks, so unlike
// run-view.render.test.tsx this needs no next/navigation stub.

function row(overrides: Partial<TagTaskRow> = {}): TagTaskRow {
return {
taskId: "skill-flow-coded-agent",
skill: "uipath-maestro-flow",
appearances: 20,
matureSkips: 0,
passRate: 90,
latestStatus: "SUCCESS",
latestScore: 1.0,
latestRunId: "2026-07-31_04-38-51",
latestMatureSkipped: false,
...overrides,
};
}

function renderTable(
rows: TagTaskRow[],
harness: string | null = null,
) {
return render(
<TagTaskTable
rows={rows}
tag="path-to-ga"
window="30d"
harness={harness}
/>,
);
}

describe("TagTaskTable", () => {
test("shows a Mature pill instead of Passed when the latest run skipped the task", () => {
renderTable([row({ latestMatureSkipped: true })]);
expect(screen.getByText("Mature")).toBeInTheDocument();
expect(screen.queryByText("Passed")).not.toBeInTheDocument();
});

test("shows Passed and the numeric score for an ordinary executed row", () => {
renderTable([row({ latestScore: 0.75 })]);
expect(screen.getByText("Passed")).toBeInTheDocument();
expect(screen.getByText("0.75")).toBeInTheDocument();
});

test("dashes out the latest score on a mature row", () => {
// 1.0 on a carry-forward row is inherited, not measured — showing it
// beside a Mature pill would read as a fresh result.
renderTable([row({ latestMatureSkipped: true, latestScore: 1.0 })]);
expect(screen.queryByText("1.00")).not.toBeInTheDocument();
// Exactly one cell dashes — the score. Every other column on the default
// row is populated, so this pins WHICH cell went un-measured.
expect(screen.getAllByText("—")).toHaveLength(1);
});

test("annotates Appearances with the mature count, and only when non-zero", () => {
renderTable([row({ appearances: 24, matureSkips: 3 })]);
expect(screen.getByText("(3 mature)")).toBeInTheDocument();
// The raw count stays plain beside it (getByText matches an element's own
// direct text nodes, so this is the cell's "24", not "24 (3 mature)").
expect(screen.getByText("24")).toBeInTheDocument();
});

test("no mature annotation when nothing was skipped", () => {
renderTable([row({ appearances: 24, matureSkips: 0 })]);
expect(screen.queryByText(/mature\)/)).not.toBeInTheDocument();
});

test("renders an em dash for an unmeasured pass rate", () => {
// Every appearance was a carry-forward → nothing executed → no rate.
// Must not read as NaN% or a measured 0%. `latestMatureSkipped` is true
// by construction here: buildTagTaskRows reads latest* off one of the
// counted appearances, so matureSkips === appearances forces it — the
// score dashes too, hence two dashes rather than one.
renderTable([
row({
appearances: 4,
matureSkips: 4,
passRate: null,
latestMatureSkipped: true,
}),
]);
expect(screen.queryByText(/NaN/)).not.toBeInTheDocument();
expect(screen.queryByText("0%")).not.toBeInTheDocument();
expect(screen.getAllByText("—")).toHaveLength(2);
});

test("Last seen shows the date half of the latest run id", () => {
renderTable([row({ latestRunId: "2026-07-16_04-24-15" })]);
expect(screen.getByText("2026-07-16")).toBeInTheDocument();
});

test("empty rows render the empty state naming the tag and window", () => {
// Newly reachable: a tag whose every task was de-tagged yields [] where
// it previously yielded stale rows.
renderTable([]);
expect(
screen.getByText(/No tasks tagged path-to-ga in the last 30d\./),
).toBeInTheDocument();
});

test("the pooled-across-harnesses note tracks the harness prop", () => {
const { unmount } = renderTable([row()], null);
expect(screen.getByText(/pooled across harnesses/)).toBeInTheDocument();
unmount();

renderTable([row()], "claude-code");
expect(
screen.queryByText(/pooled across harnesses/),
).not.toBeInTheDocument();
});
});
123 changes: 20 additions & 103 deletions evalboard/app/path-to-ga/page.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,14 @@
import Link from "next/link";
import {
getOverview,
getTagTaskBreakdown,
listRecentHarnesses,
} from "@/lib/overview";
import { parseHarnessScope } from "@/lib/harness";
import { humanizeTaskId } from "@/lib/format";
import { passClass } from "@/lib/pass-rate";
import { HarnessSelector } from "../_components/harness-selector";
import { harnessShortLabel } from "../_components/harness-badge";
import { type Window } from "@/lib/reviews-types";
import { DailySuccessChart } from "../_overview/daily-chart";
import { TableScroll } from "../_components/scroll-table";
import { StatusPill } from "@/lib/pills";
import { TagTaskTable } from "./task-table";

export const dynamic = "force-dynamic";

Expand Down Expand Up @@ -98,10 +94,22 @@ export default async function PathToGaPage({
{taskRows.length}
</div>
<div className="text-xs text-gray-500">
distinct task{taskRows.length === 1 ? "" : "s"}
distinct task{taskRows.length === 1 ? "" : "s"} still
tagged
</div>
</div>
</div>
{/* The tile above and the chart below keep their original
mature-blind, union-over-the-window semantics (they feed the
front page and every tag-filtered view); the table does not.
Say so, rather than let the two silently disagree. */}
<p className="text-xs text-gray-500">
The rate above and the chart cover every run that carried a{" "}
<span className="font-mono">{TAG}</span> task at the time it
ran, counting mature carry-forwards as passes. The table
below is narrower: only tasks still carrying the tag, scored
on runs that actually executed.
</p>
{runsInWindow > 0 ? (
<DailySuccessChart
data={overview.runs}
Expand All @@ -116,103 +124,12 @@ export default async function PathToGaPage({
)}
</section>

<div className="space-y-2">
<div className="flex flex-wrap items-baseline gap-2">
<h2 className="text-sm font-semibold text-gray-900">
Tasks
</h2>
{/* Unscoped, a task's appearances span harnesses, so its rate
pools regimes that aren't strictly comparable. Say so
rather than let the number read as one harness's. */}
{!harness && (
<span className="text-xs text-gray-500">
pooled across harnesses · pick one above to separate
them
</span>
)}
</div>
<TableScroll>
<table className="w-full text-sm">
<thead>
<tr className="bg-gray-50 border-b border-gray-200 text-left text-gray-600">
<th className="py-3 px-4 font-medium">
Task
</th>
<th className="py-3 px-4 font-medium">
Skill
</th>
<th className="py-3 px-4 font-medium text-right">
Appearances
</th>
<th className="py-3 px-4 font-medium text-right">
Pass rate
</th>
<th className="py-3 px-4 font-medium">
Latest status
</th>
<th className="py-3 px-4 font-medium text-right">
Latest score
</th>
</tr>
</thead>
<tbody>
{taskRows.map((r) => (
<tr
key={r.taskId}
className="border-b border-gray-100 last:border-b-0 hover:bg-gray-50 transition-colors"
>
<td className="py-3 px-4">
<Link
href={`/runs/${r.latestRunId}`}
className="text-gray-900 hover:text-studio-blue font-medium"
>
{humanizeTaskId(r.taskId)}
</Link>
<div className="font-mono text-[11px] text-gray-400">
{r.taskId}
</div>
</td>
<td className="py-3 px-4 text-gray-700">
{r.skill ?? "—"}
</td>
<td className="py-3 px-4 text-right tabular-nums text-gray-700">
{r.appearances}
</td>
<td className="py-3 px-4 text-right tabular-nums">
<span
className={`font-medium ${passClass(r.passRate)}`}
>
{r.passRate.toFixed(0)}%
</span>
</td>
<td className="py-3 px-4">
<StatusPill
status={r.latestStatus}
relabel
/>
</td>
<td className="py-3 px-4 text-right tabular-nums text-gray-700">
{r.latestScore != null
? r.latestScore.toFixed(2)
: "—"}
</td>
</tr>
))}
{taskRows.length === 0 && (
<tr>
<td
colSpan={6}
className="py-6 px-4 text-center text-sm text-gray-500"
>
No tasks tagged {TAG} in the last{" "}
{WINDOW}.
</td>
</tr>
)}
</tbody>
</table>
</TableScroll>
</div>
<TagTaskTable
rows={taskRows}
tag={TAG}
window={WINDOW}
harness={harness}
/>
</div>
);
}
Loading
Loading