Repository navigation
[bot] Merge master/f04c9ce9 into rel/dev - #1885
Merged
Merged
Conversation
A report only carried wall_clock_s, a duration with no anchor, so a finished run could not say when it ended. The only calendar time available was the minute-resolution start stamp embedded in the Langfuse run name. EvalReport gains started_at and finished_at (timezone-aware UTC), set around the same span wall_clock_s already times in both run_items and run_agentic_items. When the CLI merges the agentic and single-shot phases the run spans the earliest start to the latest finish. The JSON summary exposes both as ISO 8601 strings, or null for a report that was never timed. The change is additive: no existing key is renamed or removed. jira: GDAI-2613 risk: low Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The report gave one average latency per run, which says nothing about how slow the slowest answers are. A P99 needs every individual run, but only their sum was kept: passing runs left no latency behind, and the agentic kinds timed an item as a whole. Every run that completes is now timed, passing or not, and each item lists its own `run_latencies_s`. Each run's summary and comparison entry carry `latency_stats_s` -- n, mean, min, P50, P70, P90, P99 and max over those runs -- computed once in core/timing.py with linear interpolation, so P50 is the median and the console, the JSON and any later view cannot disagree. The console prints the percentiles under the summary line, and the multi-model comparison gains P50, P90 and P99 columns. The twelve agentic evaluators take a monotonic timestamp before each run's first message and hand the list out on success, on a failed gate and on the all-ungraded path. A structural test fails any kind that stops doing so: a missing list reads as "not timed", so a kind that forgot it would silently drop out of the percentiles. metric_skill's clock-patching tests gain the two extra readings per run. `avg_latency_s` and every other existing field are unchanged. jira: GDAI-2613 risk: low Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The report used an ad-hoc palette that looked like no GoodData product, and showed only an average latency even though the JSON now carries the full per-run distribution. The page follows the product UI: a black 44px header with a lime rule, cyan primary, the #464e56 text and #dde4eb border scale, 3px controls and status chips tinted at 15% alpha. It keeps a system font stack rather than embedding Avenir, so the file stays self-contained and opens offline. The text steps of the status colours and the secondary grey are one step darker than the product's, because the product values fall below 4.5:1 on tinted rows and conversation turns; every text/background pair now clears AA. Each run card gets a Latency per run panel -- P50 (median), P70, P90, P99, average, maximum and the number of runs -- with a strip placing the percentiles between the fastest and slowest run, and the comparison table gains the same columns. All of it is read from `latency_stats_s`; the page computes no percentile, and a report written before that field shows a dash. The comparison table now scrolls inside its own box instead of clipping in a narrow window, and rows and sort headers can be operated from the keyboard. jira: GDAI-2613 risk: low Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The latency percentiles covered every run as one population, which mixed a single chat turn with whole conversations and counted the LLM judge and conversation set-up as GoodData's time. gen-ai starts one trace per chat message, so the unit that describes the agent is the trace. Items now fall into two groups by eval kind. Single-shot kinds send one message per run, so a run is one trace and its latency is that turn's response time -- the main figure. Multi-turn kinds drive a conversation and are reported both per trace (each turn) and per full run (the whole conversation's wall clock). The JSON replaces `latency_stats_s` with `by_mode` on each run's summary and comparison entry, and every item states its `mode` and `trace_latencies_s`. Traces are recorded once, where the chat client completes a turn, into a context-scoped recorder that the single-shot and agentic runners open per item. No evaluator has to remember to hand them out, and concurrent items cannot mix. The summary endpoint is timed the same way. `run_latencies_s` stays and now feeds the multi-turn per-run figure. A guard test fails when a new agentic kind has not been classified. The console prints pass counts and one latency line per population, and the comparison table shows single-shot trace P50, P90 and P99. The HTML report gets a Single-shot section, then a Multi-turn section with both views, each a table with a range strip, plus a mode column and filter on the items table. jira: GDAI-2613 risk: low Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The item table's one "Latency" column was the mean wall-clock of a whole run (chat turns plus judge, simulated user and set-up), sitting under tables that report per-trace response time. Two numbers with different meanings were presented as one, and the drawer chip was a third, unlabelled one. Show two columns instead: Latency / trace (median of the item's chat turns) and Latency / run (mean of its whole runs), each sortable, with a tooltip on the header and on every value listing the samples behind it. The drawer chips say whether they are the item's traces or the one run it describes. A report written before per-trace timing has no trace figure and falls back to avg_latency_s as the run figure. The kind filter is now a multi-select (a details element of checkboxes, with item counts and a clear action), so several kinds can be shown at once. Nothing ticked still means every kind. jira: GDAI-2613 risk: low Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The filters sat above the item table and only narrowed that table, so the run cards, the comparison and the latency tables kept describing the full set while the rows underneath described a few. Move the filter bar to the top of the page, sticky on wide windows, and make it select one set of items that every section covers. Kind, mode and search are properties of the item; status and the expression are read from the run picked in the run selector, which is hidden for a single run. A filtered view states how many items it covers, and Reset clears it. Unfiltered, the page still reads summary, comparison and by_mode from the report, so it cannot disagree with the console or the JSON. Only a narrowed selection is recomputed in the page, from the selected items' own records, by the rules EvalReport uses. An empty selection shows dashes rather than a 0% rate. Status chips and the recomputed counts follow gate_passed instead of pass_at_k, so they agree with the report's own counts under --gate power. jira: GDAI-2613 risk: low Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The single filter bar drove the whole page, so there was no way to look at a few rows of the table without also changing every figure above it, and status and the run picker could only hold one value. Keep the bar at the top for the whole page (kind, mode, status, run) and repeat those filters in the Items section, where they narrow the table only. Search and the expression now live only in the Items section. The table starts from the rows the top filters allow and only narrows them; its menus offer just what survives the top selection, and a ticked value that no longer survives is dropped. Status and run are multi-selects, like kind. The run filter chooses which runs the page (or the table) shows, replacing the single focused run; the item table gains a status and latency column per run shown, and the drawer gets a run switcher. With several runs an item matches a status, or an expression, when any of the runs in play does. jira: GDAI-2613 risk: low Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
P50 is the median, so the average and P50 are the only central figures the latency tables had, and a handful of very slow or very fast samples move the first while the second ignores everything but the middle. Add trimmed_mean to every latency distribution: the average of the samples left after dropping the fastest and slowest 10% from each end (n * 10 // 100 of them, in integer arithmetic). It is null below ten samples, where there is nothing to drop and it would only repeat mean. It is computed in latency_stats, so the JSON and the HTML agree, and the HTML recomputes it by the same rule for a filtered view. The latency tables show it as "Trimmed avg" with a tooltip, and say when a sample is too small. The console line is unchanged: it carries percentiles only. jira: GDAI-2613 risk: low Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The report opened on run cards and a flat table, and spoke in the
vocabulary of the harness (item, kind, trace, run, P50, P99). Lay it out
as a document a reader can take in top to bottom, in plain words.
A masthead names the report and carries a Run setup tooltip: what is the
same in every run and what differs, because results from different
setups are not directly comparable. Results is three headline cards for
the latest run (pass rate, quality, median time per attempt), each with
one change line in words ("down from 86% on Sep 28") coloured by whether
the direction is good, with a chevron and screen-reader text. From three
runs each card has a sparkline; clicking a point compares the latest run
with that one, and a pick that a filter removed or that became the
latest run is ignored instead of compared with itself.
The latency tables read Fastest, Median, P90 and Slowest, newest run
first, with the quickest run in each column green and the slowest red
(each with a chevron) and a caution icon when a run was timed on a
different number of samples. P70, P99, the average and the trimmed
average stay in the JSON and are still recomputed for a filtered view;
they are no longer shown. Seconds drop a redundant ".0".
The test case table lists the latest result, a Since previous run column
(Regressed, Fixed, Flaky, New, Removed; the table opens on regressions),
then earlier runs going back. The kind, status and run filters follow
the product's attribute filter (search, All, Only, Cancel, Apply, staged
until Apply), the expression filter is folded under Advanced, and a
print stylesheet drops the filters and the drawer. An explicit --title
is honoured, and a setup field a redacted report does not carry is not
listed as an empty row.
Faces are Inter and Funnel Display, named but never loaded, so the file
stays self-contained. Tests that pinned the old layout now pin the new
one, and the README describes it.
jira: GDAI-2613
risk: low
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Review of the latency report found two places where the response-time population and the scored runs disagree, three where the HTML report disagrees with the console or labels the wrong run, and one repeated computation. A turn that ended in a ChatError (a turn or item deadline, an error event, an incomplete turn) was never recorded as a trace, although evaluators score its partial result as a run. The slowest turns went missing from the per-response percentiles. send_message now records the turn's wall clock from the partial result before re-raising. An item that raised outside its evaluator's own failure reporting kept the turns it had completed but reported no runs, so it showed up in the per-response table and not in the per-conversation one. Its turns are now dropped (TraceRecorder.discard), along with the runs it never reported. The pass-rate card divided by every test case, skipped ones included, while the console divides by those evaluated, so 10 passed of 14 with 2 skipped read 71% against 83%. The card, and the recomputed figures of a filtered view, now use one passRateOf helper. The page's pass_rate figure was read by nothing and is removed; the JSON field is unchanged. The Since previous run labels described earlier runs when they should describe the latest: a test case skipped in the latest run was labelled Regressed beside a SKIP cell, and one absent from the previous run but present before it was labelled New instead of Fixed or Flaky. A skipped latest run now has no label, and New and Removed look at every earlier run. EvalReport.by_mode is a cached_property, since the console, the comparison and both JSON builders each rebuilt and re-sorted it. The README describes the changed definitions. The HTML rules are tested by running the page's own functions in Node, skipped where Node is absent. jira: GDAI-2613 risk: low Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ML report Add a "View by" switch that lays the same report out by date (a column per run) or by model (a column per model, ranked by the console's winner rule, each model read at its latest run). A report comparing several models, each seen once, opens By model; everything else opens By date as before. By model the page gains outcome bars, a quality-against-speed plot with its frontier, a per-category heat map, a pass@K vs pass^K consistency chart, per-model latency percentile bars, and an "Across models" column in the test case table that opens on the disagreements. Both views gain an Outcomes table (passed / failed / could not be graded / skipped / pass rate) and the latency sections now open with a line chart of the median, P70, P90 and P99 across runs. Tests and README updated to match. jira: GDAI-2613 risk: low Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The Results sparklines used a viewBox stretched with preserveAspectRatio="none", which squashed every dot into a wide ellipse. Drop the viewBox so heights are real pixels and x positions are percentages of the card; the line is now drawn as segments so it can use the same percentages. jira: GDAI-2613 risk: low Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…t-speed chart Drop the dashed frontier line, the filled/hollow "on the frontier" / "beaten" distinction and its legend. Every model is now the same dot with a plain label; the chart only shows where each model sits. Remove the now-unused frontierOf helper, its CSS, test and README description. jira: GDAI-2613 risk: low Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…by start time The report now always opens on the Date view; the Model view is one choice away in the View by dropdown, and a report that compares several models no longer opens on it by itself. Runs are ordered by when they started instead of the order their files were given in, so the oldest run is always on the left of the charts and the latest on top of the tables. A run's place comes from `summary.started_at`, else from the date its file is named for, and a report where any run has neither keeps the order the files were given in rather than sorting only part of it. jira: GDAI-2613 risk: low Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…in outcomes table A run was identified in the Outcomes table by its name alone, and a name such as 2026-09-28 is styled as "Sep 28" elsewhere, so the table could not say when a run started or what it was actually called. The table now has a Run date column, read from summary.started_at and shown in the reader's time zone with the UTC stamp on hover (a report written before started_at existed shows the date its file is named for, without a time), and a Run name column with the name as given. By model, the leader cards are dropped and its outcomes table becomes the Results table, the first thing on the page. jira: GDAI-2613 risk: low
- Introduced `started_at` attribute to `EvalReport` to record the UTC time when the evaluation starts. - Updated `run_items` function to set `started_at` when the evaluation begins. - Enhanced tests to verify the presence and correctness of the `started_at` timestamp in reports. - Modified HTML report generation to include run date and model information in the output. - Adjusted JSON report building to write `started_at` in ISO format and handle cases where it may be null. jira: GDAI-2613 risk: low
…ions Reasoning arrives as markdown with a **bold** title line, and the report printed the asterisks. Render the title as bold in the timeline and in the Reasoning section. The text is escaped first, so the only markup that reaches the page is the <strong> the report adds. jira: GDAI-2613 risk: low
…sons in reports A sparkline shows three kinds of dot (the latest run, the run being compared with, and the rest) and nothing on the page said which was which, or that the white one can be moved. Add a key under the cards, drawn at the sizes and colours of the dots, and a line saying that a click on any other dot moves the comparison. It shows only when a sparkline is drawn. jira: GDAI-2613 risk: low
changeOf checked for New before it checked for a skip in the latest run, so a test case that appeared for the first time and was skipped there got the New label beside a SKIP cell. The skip check now comes before New; Removed stays first, because a case missing from the latest run reads as 'none' and must keep its label. Also annotate the three tests this change set added without a return type, as AGENTS.md asks. jira: GDAI-2613 risk: low Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…troller Add the missing return annotations CodeRabbit flagged on the tests added for the run start/finish times, and type the _FakeController stub in test_cli.py to match WorkspaceModelController. jira: GDAI-2613 risk: nonprod Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Review feedback on the run-latency and HTML report work. - Record a turn's response time only when it hit a turn or item deadline or ended without a final answer. A 503 that outlasted its retries carried the last attempt's ~0s, which put a fast response into the percentiles for a turn that never answered. - With --redact, give a model one alias across all its runs and number them (Run 1 · Model A, Run 2 · Model A), so one model's history is not read as several models. A model with one run stays Model A. - Say in the tooltip of the Median time per attempt card what it covers: it pools single-question and conversation attempts and counts whole attempts, so it matches neither per-response latency table. - Install nodejs in the CI image, so the page's Node tests run instead of skipping. - Document in AGENTS.md that a new agentic kind must also be classified in core/interaction.py, and say in runner.py which path records which turns. jira: GDAI-2613 risk: low Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
feat(gooddata-eval): run start/finish times, per-trace latency percentiles and a restyled HTML report
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## rel/dev #1885 +/- ##
===========================================
+ Coverage 84.95% 85.09% +0.13%
===========================================
Files 336 337 +1
Lines 24125 24331 +206
===========================================
+ Hits 20495 20704 +209
+ Misses 3630 3627 -3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚀 Automated PR to perform merge from master into rel/dev with changes up to f04c9ce (created by https://github.com/gooddata/gooddata-python-sdk/actions/runs/39391880065).