Skip to content

Model routing: cut the experimental page to five essential views - #19014

Closed
SivaKesava1 with Copilot wants to merge 4 commits into
mainfrom
copilot/model-routing-cut-experimental-page
Closed

SivaKesava1 with Copilot wants to merge 4 commits into
mainfrom
copilot/model-routing-cut-experimental-page

Conversation

Copilot AI commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

The experimental Model routing page had 24 views and 82 queries, so it was hard to tell whether routing works. This PR cuts it to five views that answer that question, all declared in Dashboard Language. The run page (routing-run), adapter, canonical/IndexedDB/PostgreSQL schemas, the Routing scope form and the navigation entry are unchanged.

Page model-routing

# View Source Question
1 routing-health (table, locked) new routing-health Is routing on, and is it working?
2 routing-models (bar, by pick model) new routing-weekly-models What is the router choosing?
3 routing-cost-trend (line, per routing state) existing routing-weekly-cost, unchanged Is it paying off?
4 routing-problems (table, locked) new routing-problems What needs attention?
5 routing-runs unchanged Opens the run page
  • Health: one row per repository and workflow with a routed run (selected, selection_failed, harness_reported, configured_unrecorded). Columns:

    • routed runs and the share in state selected;
    • failure or fallback: failure-count > 0 or routing-degraded. The view says no separate fallback field exists;
    • followed the pick: only runs with at least one flagged agent-phase request count. Runs without one are excluded from both numerator and denominator;
    • known success rate: over the four known agent-job outcomes;
    • median AIC per complete run.

    A zero denominator or missing cost shows as unknown (quotient returns null), never 0.

  • Models picked: counts only proxy-selected runs that have a known pick model and a recorded clock, per UTC ISO week. Runs without a recorded effort still count.

  • Cost per run:

    • Uses a line chart so weeks below minimum-sample (null median) are gaps. Bar charts would draw a stub.
    • Labeled observational, not causal.
    • The x-axis is untyped because the validator treats week as non-temporal. The models chart uses ordinal for the same reason.
  • Problems: a union of per-run aggregates, giving one row per run and kind:

    • routing failure: distinct codes, or the event type when no code is recorded;
    • agent-phase deviation: distinct deviated axes, or deviated when none are recorded;
    • resolution failure: distinct causes.

    Each row joins routing-runs for the run link, repository, workflow and time. Order: timed rows newest first, untimed rows last, then run descending.

  • Locked tables: Health and Problems are locked, because the validator (DLS-E013) allows only one unlocked table per page.

  • Known limitation: the cost view's table mode cannot show sample status and complete-run count without JavaScript. The underlying rows still carry sample-status and complete-runs.

Removals

  • 23 views: routing-adoption, routing-decisions, routing-cost and all 20 supplemental views.
  • The 41 queries listed in the issue, re-checked with the query-source resolver. No remaining view, the run page or another fragment uses them, so none had to be kept. The fragment goes from 82 to 49 queries (41 kept plus 8 new).

Tests

  • Contract fixture: views and queries updated; detail-views and states unchanged.
  • model-routing.test.js:
    • acceptance cases 2–5 for health, models picked and Problems, using the existing fixtures and replaying run 101;
    • new routing-weekly-cost test pinning both sample states (passes against main's query);
    • attribution assertions retargeted to routing-attributed-accounting;
    • assertions on removed queries dropped.
  • model-resolution.test.js: page-level coverage, status and alias tests replaced by the Problems resolution-failure rows (990000004 before 990000003, no time, removed by the Engine = codex filter). The run-page drill-down tests are kept.
  • Playwright (model-routing.spec.js):
    • checks the five headings in order, run 101's health row, the Problems rows, both drill-downs and no horizontal overflow, on both backends at 1280 and 390 px;
    • table-mode checks retry with toPass, because the IndexedDB backend re-renders the page (and resets it to chart mode) after data publication.
  • Postgres (model_resolution_sql_test.go):
    • retargeted to routing-resolutions, routing-resolution-failure-records and routing-problems;
    • keeps parity for run 990000003's model_unavailable failure;
    • the seeded run now carries workflowPath so the Problems row has a workflow.

Spec, docs and generated files

  • specs/dashboard-data.md ("Model routing evidence"):
    • describes the five views, the unknown-not-zero rule, the observational cost label and one Problems row per run and kind;
    • failed resolutions now appear in Problems instead of a Model resolution section;
    • alias rules remain only as conditions for any future alias view.
  • docs/dashboard-data-ingestion.md: the sentence about the Model resolution section is updated to match.
  • src/agent/catalog.generated.json and queries.generated.json are regenerated.
  • The page description now starts with "Inspect …", because the view-semantics test requires an allowed leading verb.

Notes for review

  • Not run locally: the full npm run test:e2e suite, npm run test:performance, and the core dashboard document size comparison with main. Please dispatch Dashboard CI on this branch.
  • The default build omits experimental pages, so the check that no removed query name remains in the build output did not cover the Model routing page's own chunk.
  • Root npm test has one failure: tests/unit/landing-page-diagram.test.mjs (5 hero paragraphs, expected 4). This PR doesn't touch the landing page, but I haven't confirmed the test also fails on main.

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor
Dashboard dead views

Dead element views (0):
Dead dashboard pages (0):
Analyzed 8 element renderers and 66 dashboard pages across 1 dashboard.json file(s) from dashboard/site/src/components/ui-elements.js.

⚠️ This reachability report is informational and does not block the pull request.
🔎 View the workflow run and report artifact.

Copilot AI changed the title [WIP] Cut the experimental model routing page to five essential views Model routing: cut the experimental page to five essential views Oct 19, 2026
Copilot AI requested a review from SivaKesava1 October 19, 2026 23:36
@SivaKesava1
SivaKesava1 marked this pull request as ready for review October 19, 2026 23:39

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Correctness issues remain, and cross-backend rendering and performance validation are incomplete.

3 open issues
What changed in this PR

Simplifies CAO’s experimental Model routing dashboard to five declarative views for assessing routing behavior.

Changes:

  • Replaces detailed analytics with health, model picks, cost trends, problems and runs.
  • Removes unused queries while preserving scope filters and run drill-downs.
  • Updates tests, documentation and generated agent metadata.
File Description
specs/​dashboard-data.md Documents the five-view evidence contract.
server/​internal/​server/​model_resolution_sql_test.go Retargets PostgreSQL checks to Problems.
docs/​dashboard-data-ingestion.md Updates resolution-failure presentation guidance.
dashboard/​site/​test/​unit/​model-routing.test.js Tests health, picks, problems and cost sampling.
dashboard/​site/​test/​unit/​model-resolution.test.js Updates failure coverage and preserves drill-down checks.
dashboard/​site/​test/​fixtures/​model-routing-contract.json Updates expected views and queries.
dashboard/​site/​test/​e2e/​model-routing.spec.js Updates responsive, cross-backend browser checks.
dashboard/​site/​src/​agent/​catalog.generated.json Refreshes page and query metadata.
dashboard/​site/​dashboard-fragments/​model-routing.json Declares essential views and prunes queries.

Review effort: Max

Comment on lines +764 to +770
"id": "routing-models", "title": "Models picked over time", "mark": "chart", "chart": "bar", "layout": "half",
"subject": "Proxy-selected runs by UTC ISO week and picked model.",
"objective": "Show what the router is choosing.",
"acceptance": "Only proxy selections with a known model and a recorded run clock count.",
"description": "Proxy selections only; harness-reported, configured-unrecorded and unrouted runs are not counted.",
"data": { "source": "routing-weekly-models", "query-context": false },
"encoding": { "x": { "field": "week", "type": "ordinal", "title": "UTC ISO week start" }, "y": { "field": "runs", "type": "quantitative", "title": "Runs" }, "color": { "field": "pick-model", "type": "nominal", "title": "Picked model" } },
Comment on lines +779 to +780
"data": { "source": "routing-weekly-cost", "query-context": false },
"encoding": { "x": { "field": "week", "title": "UTC ISO week start" }, "y": { "field": "median-aic", "type": "quantitative", "title": "Median AIC per complete run", "unit": "aic" }, "color": { "field": "routing-state", "type": "nominal", "title": "Routing state" } },
Comment thread specs/dashboard-data.md
Comment on lines +89 to +90
The Model routing page currently provides five views. Routing health by
workflow answers whether routing is on and working: per repository and workflow
@pelikhan

Copy link
Copy Markdown
Collaborator

these changes are bing integrated in another session

pelikhan added a commit that referenced this pull request Oct 20, 2026
Integrate the reviewed page reduction from #19014 while preserving run-scoped accounting correlation, authored detail bounds and canonical Audit-index execution. Regenerate query catalogs and retarget contracts to health, models, costs, problems and routing runs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
pelikhan added a commit that referenced this pull request Oct 20, 2026
* Optimize slow dashboard pages through declarative shared queries

Consolidate the six deployed-data performance workstreams into one reviewed implementation. Share page-bound dependency graphs and campaign declarations; retain scoped canonical reads, evidence quality, authored limits, previews, pagination and reactive lifetimes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Simplify Model routing to five essential views

Integrate the reviewed page reduction from #19014 while preserving run-scoped accounting correlation, authored detail bounds and canonical Audit-index execution. Regenerate query catalogs and retarget contracts to health, models, costs, problems and routing runs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Peli de Halleux <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pelikhan

Copy link
Copy Markdown
Collaborator

Superseded by #19015, which consolidates the dashboard performance work and Model routing simplification and is now merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Model routing: cut the experimental page to five essential views

4 participants