feat(evals): port HardBenchmark suite onto the harness wave - #2866
Open
miguelg719 wants to merge 1 commit into
Open
feat(evals): port HardBenchmark suite onto the harness wave#2866miguelg719 wants to merge 1 commit into
miguelg719 wants to merge 1 commit into
Conversation
46-task residual hard core of WebTailBench + Online-Mind2Web with audited provenance (source_suite/failure_mode/capability_axis) and verbatim precomputed rubrics. Registered as agent/hardbenchmark across discovery, planner, external-harness plan, TUI shorthand (b:hardbenchmark) and config.
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
4 issues found across 10 files
Confidence score: 3/5
packages/evals/suites/hardbenchmark.tslabels all rows asagent/hardbenchmark, dropping OnlineMind2Web site-scoping metadata; external harnesses may apply generic guidance and run those tasks incorrectly. Preserve the dataset-specific contract in the generated rows.packages/evals/evals.config.jsonexposesbenchmarks.hardbenchmark.limit, but the runner does not consume it, so configured limits have no effect. Wire the value into suite limit resolution or remove the setting.packages/evals/framework/benchPlanner.tshas no focused coverage for the newagent/hardbenchmarkroute, dataset routing, or generated testcase shape, leaving regressions able to disable the suite silently. Add planner tests for this route.packages/evals/framework/externalHarnessPlan.tsomitsagent/hardbenchmarkfrom the unsupported-task fallback, producing incomplete diagnostics when that suite is requested. Update the fallback message.
Not reviewed (too large): packages/evals/datasets/hardbenchmark/HardBenchmark_data.jsonl (~46 lines) - if these are generated or fixture files, add them to ignored paths to exclude them from future reviews.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/evals/evals.config.json">
<violation number="1" location="packages/evals/evals.config.json:22">
P2: `benchmarks.hardbenchmark.limit` is not consumed by the runner, so this setting cannot control the suite and creates a false configuration surface. Wire the config value into the suite's limit resolution, or remove this entry until benchmark limits are read from config.</violation>
</file>
<file name="packages/evals/framework/benchPlanner.ts">
<violation number="1" location="packages/evals/framework/benchPlanner.ts:303">
P2: Add a focused planner test for `agent/hardbenchmark`; this new route currently has no coverage for dataset routing or generated testcase shape, so regressions can silently disable the suite.
(Based on your team's feedback about unit tests for new behavior.)</violation>
</file>
<file name="packages/evals/framework/externalHarnessPlan.ts">
<violation number="1" location="packages/evals/framework/externalHarnessPlan.ts:51">
P3: When an unsupported external task is requested, the error now omits the newly supported `agent/hardbenchmark` suite. Add `agent/hardbenchmark` to the fallback message so the diagnostic matches this branch.</violation>
</file>
<file name="packages/evals/suites/hardbenchmark.ts">
<violation number="1" location="packages/evals/suites/hardbenchmark.ts:116">
P2: OnlineMind2Web tasks lose their site-scoping contract when run through this suite. Because every row is labeled `agent/hardbenchmark`, external harnesses use generic guidance and cannot distinguish the 13 OnlineMind2Web rows; preserve the source suite in the plan/input or add source-aware guidance, and retain the source URL when it is available.</violation>
</file>
Architecture diagram
sequenceDiagram
participant TUI as TUI/CLI
participant Discovery as Discovery
participant Planner as Bench Planner
participant Builder as HardBenchmark Builder
participant Dataset as HardBenchmark_data.jsonl
participant Config as Config/TaskConfig
participant Harness as External Harness Plan
participant Eval as Eval Runner
Note over TUI,Eval: HardBenchmark Suite Registration Flow
TUI->>Discovery: b:hardbenchmark shorthand
Discovery->>Discovery: Register agent/hardbenchmark as external_agent_benchmark
Discovery-->>TUI: Suite discovered
TUI->>Planner: Run agent/hardbenchmark
Planner->>Planner: Generate suite testcases
Planner->>Builder: buildHardBenchmarkTestcases(models)
Builder->>Dataset: Read HardBenchmark_data.jsonl
Dataset-->>Builder: 46 rows with precomputed_rubric + provenance
alt EVAL_HARDBENCHMARK_IDS set
Builder->>Builder: Select tasks by explicit IDs
else EVAL_HARDBENCHMARK_MODE set
Builder->>Dataset: Filter by audited failure_mode
Builder->>Builder: Apply sampling/limit
else Default (all 46)
Builder->>Builder: Use all tasks (EVAL_MAX_K or EVAL_HARDBENCHMARK_LIMIT)
end
Builder->>Config: Look up task categories
Config-->>Builder: categories
Builder->>Builder: Build Testcase with provenance metadata (source_suite, failure_mode, capability_axis)
Builder-->>Planner: 46 testcases
Planner->>Harness: Build external harness plan for each task
Harness->>Harness: Map agent/hardbenchmark -> dataset: "hardbenchmark"
Harness->>Eval: Execute with startUrl, instruction, precomputed_rubric
alt Task runs successfully
Eval-->>Harness: Result with rubric match
Harness-->>Planner: Testcase outcome
else Task fails (expected at ~0% pass rate)
Eval-->>Harness: Failure with trajectory
Harness-->>Planner: Failure attributed to capability_axis
end
Planner-->>TUI: Aggregate results with provenance-aware reporting
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| "limit": 25 | ||
| }, | ||
| "hardbenchmark": { | ||
| "limit": 46 |
Contributor
There was a problem hiding this comment.
P2: benchmarks.hardbenchmark.limit is not consumed by the runner, so this setting cannot control the suite and creates a false configuration surface. Wire the config value into the suite's limit resolution, or remove this entry until benchmark limits are read from config.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/evals.config.json, line 22:
<comment>`benchmarks.hardbenchmark.limit` is not consumed by the runner, so this setting cannot control the suite and creates a false configuration surface. Wire the config value into the suite's limit resolution, or remove this entry until benchmark limits are read from config.</comment>
<file context>
@@ -18,6 +18,9 @@
"limit": 25
},
+ "hardbenchmark": {
+ "limit": 46
+ },
"odysseysbench": {
</file context>
| "agent/webvoyager": (models) => buildWebVoyagerTestcases(models), | ||
| "agent/onlineMind2Web": (models) => buildOnlineMind2WebTestcases(models), | ||
| "agent/webtailbench": (models) => buildWebTailBenchTestcases(models), | ||
| "agent/hardbenchmark": (models) => buildHardBenchmarkTestcases(models), |
Contributor
There was a problem hiding this comment.
P2: Add a focused planner test for agent/hardbenchmark; this new route currently has no coverage for dataset routing or generated testcase shape, so regressions can silently disable the suite.
(Based on your team's feedback about unit tests for new behavior.)
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/framework/benchPlanner.ts, line 303:
<comment>Add a focused planner test for `agent/hardbenchmark`; this new route currently has no coverage for dataset routing or generated testcase shape, so regressions can silently disable the suite.
(Based on your team's feedback about unit tests for new behavior.) </comment>
<file context>
@@ -299,6 +300,7 @@ export function generateSuiteTestcases(
"agent/webvoyager": (models) => buildWebVoyagerTestcases(models),
"agent/onlineMind2Web": (models) => buildOnlineMind2WebTestcases(models),
"agent/webtailbench": (models) => buildWebTailBenchTestcases(models),
+ "agent/hardbenchmark": (models) => buildHardBenchmarkTestcases(models),
"agent/odysseysbench": (models) => buildOdysseysBenchTestcases(models),
};
</file context>
| for (const modelEntry of normalizeAgentModelEntries(models)) { | ||
| for (const row of rows) { | ||
| const input: EvalInput = { | ||
| name: "agent/hardbenchmark", |
Contributor
There was a problem hiding this comment.
P2: OnlineMind2Web tasks lose their site-scoping contract when run through this suite. Because every row is labeled agent/hardbenchmark, external harnesses use generic guidance and cannot distinguish the 13 OnlineMind2Web rows; preserve the source suite in the plan/input or add source-aware guidance, and retain the source URL when it is available.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/suites/hardbenchmark.ts, line 116:
<comment>OnlineMind2Web tasks lose their site-scoping contract when run through this suite. Because every row is labeled `agent/hardbenchmark`, external harnesses use generic guidance and cannot distinguish the 13 OnlineMind2Web rows; preserve the source suite in the plan/input or add source-aware guidance, and retain the source URL when it is available.</comment>
<file context>
@@ -0,0 +1,154 @@
+ for (const modelEntry of normalizeAgentModelEntries(models)) {
+ for (const row of rows) {
+ const input: EvalInput = {
+ name: "agent/hardbenchmark",
+ modelName: modelEntry.modelName as AvailableModel,
+ agentMode: modelEntry.mode,
</file context>
| } | ||
|
|
||
| if (input.name === "agent/webtailbench") { | ||
| if (input.name === "agent/webtailbench" || input.name === "agent/hardbenchmark") { |
Contributor
There was a problem hiding this comment.
P3: When an unsupported external task is requested, the error now omits the newly supported agent/hardbenchmark suite. Add agent/hardbenchmark to the fallback message so the diagnostic matches this branch.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/framework/externalHarnessPlan.ts, line 51:
<comment>When an unsupported external task is requested, the error now omits the newly supported `agent/hardbenchmark` suite. Add `agent/hardbenchmark` to the fallback message so the diagnostic matches this branch.</comment>
<file context>
@@ -48,13 +48,13 @@ export function buildExternalHarnessTaskPlan(input: EvalInput): ExternalHarnessT
}
- if (input.name === "agent/webtailbench") {
+ if (input.name === "agent/webtailbench" || input.name === "agent/hardbenchmark") {
const instruction = readString(params, "ques");
if (!instruction) {
</file context>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2812 (top of the harness wave). Part of the harness×model campaign.
What
Ports the 46-task HardBenchmark suite — the residual hard core of WebTailBench + Online-Mind2Web (tasks both frontier models failed, audited to genuine model faults, date-rotted tasks dropped) — onto the wave head.
datasets/hardbenchmark/HardBenchmark_data.jsonl(verbatimprecomputed_rubricper row + audited provenance:source_suite,failure_mode,capability_axis)suites/hardbenchmark.tsbuilder; registered asagent/hardbenchmarkin discovery, planner, external-harness plan, TUI (b:hardbenchmark), configNotes
feat/facade-batch-surface.Summary by cubic
Adds the 46-task HardBenchmark residual suite to the eval harness to focus model evaluation on audited failures from WebTailBench and Online-Mind2Web. Unlike the parent suites, it runs all 46 cases by default and preserves each task’s precomputed rubric.
Included
agent/hardbenchmarkacross discovery, planning, external harnesses, config, and theb:hardbenchmarkTUI shorthand.Written for commit 2c93dcf. Summary will update on new commits.