feat(evalboard): count agent time apart from eval overhead, and filter runs by variant - #198
Merged
Merged
Conversation
…all clock The Time tab's headline divided each task's full duration by the passes, so sandbox setup, pre_run, grading and cleanup counted as agent speed. That share is ~4% of claude-code's wall clock and ~20% of codex's, so it skewed every cross-harness comparison and diluted slowdowns on short tasks. The headline now sums the agent's turns only; every past run.json already records them, so the whole history switches at once. The within-expected-time ratios are unchanged: they compare against the runner-stamped expected_seconds, which is still drawn from full durations. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bai-uipath
marked this pull request as ready for review
September 24, 2026 21:06
bai-uipath
requested review from
CarlesUIPath,
akshaylive,
tmatup and
uipreliga
as code owners
September 24, 2026 21:06
|
Claude finished @bai-uipath's task in 1m 26s —— View job Code Review in Progress
|
…iant The run page's Time card summed each task's full duration, so on a two-arm run it compared the arms' setup and grading along with the agents: the 2026-09-24 flow v1 vs v2 run read v2 as ~49 minutes faster where the agent gap is ~9. The card is now "Agent time", from the same per-turn seconds as the overview headline. Multi-arm runs also get a Variant selector (all, or one arm) that scopes the cards and the task grid to that arm, kept in the URL as ?variant=. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…rhead The task page's timing strip listed setup, the turn buckets, grading and one task-wide residual side by side. It now reads as a hierarchy: task total, then agent time (startup, generation, tool exec, teardown, unaccounted) and eval overhead (setup, grading, other), with each group's cells summing to its header. Runs without per-turn durations keep the single task-wide residual. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bai-uipath
changed the title
…groups Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bai-uipath
changed the title
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
tmatup
approved these changes
Sep 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Time tab's headline (time per passed task) counted each task's full duration, which includes sandbox setup, pre_run, grading and cleanup. None of that is agent work, no skill change can move it, and its share differs a lot by harness: about 4% of claude-code's wall clock on the September nightlies versus about 20% of codex's. That skews cross-harness comparisons and hides slowdowns on short tasks.
The overview headline, the overview's window total (was "Compute time") and the run page's time card, both now "Agent time", count only the agent's turns. Every past run already records them, so history switches together instead of stepping down on the merge date. On a two-arm run this matters most: the 2026-09-24 flow v1 vs v2 run read v2 as ~49 minutes faster on full duration, where the agent gap is ~9.
The task page's timing strip is simpler: one bar shows where the task's time went, and under it two groups, agent time (startup, generation, tool exec, teardown) and eval overhead (setup, grading, other), each summing to its total. The unaccounted residual only shows when it is not close to zero.
Multi-arm runs get a Variant selector on the run page that scopes the cards and the task grid to one arm.
The within-expected-time share and the per-task duration cells are unchanged: they still compare full durations against the runner's stamped baseline.
Companion: https://github.com/UiPath/coder_eval_uipath/pull/144 switches the runner's stamped headline and the Flow v1 vs v2 comparison to the same measure.
Screenshots