Skip to content

Model routing: retain configured-model resolution and add the Model resolution section - #18757

Merged
SivaKesava1 merged 5 commits into
mainfrom
copilot/model-routing-enhancement
Oct 18, 2026
Merged

SivaKesava1 merged 5 commits into
mainfrom
copilot/model-routing-enhancement

Conversation

Copilot AI commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Upstream gh-aw now writes one model.resolution event per role into usage/aw_session.jsonl (T-UAS-074). CAO dropped most of each event: event engine, role, configured model, resolved model and cause were lost. model.resolution was also missing from the routing-session-observations allowlist, so the Model routing page could not show how a configured model resolved, whether an alias moved, or which resolutions failed.

Audit projection

  • adapters/model-routing.js: each model.resolution event becomes one Audit record with the fields below. Every value must pass the existing identifier-shape check; anything else stays unknown.
    • sourceEngine, resolutionRole, provider
    • resolutionConfiguredModel, resolutionResolvedModel: kept exactly as recorded, including prefixes such as copilot/gpt-5-mini. They are not passed through the served-model shortener.
    • status: limited to resolved / exact / failed.
    • code (cause): limited to model_unavailable / catalog_unavailable.
  • A failed resolution's null resolved model stays unknown. Missing or unsafe values never become empty strings, zero, or a fallback model.
  • The new columns are carried through queries/database.json and specification.js TABLE_FIELDS.audits.
  • CANONICAL_SCHEMA_VERSION 31 → 32 and IndexedDB DATABASE_VERSION 41 → 42.
  • server/spec/storage.tsp updated; schema.sql and schema.gen.go regenerated through server/spec. Hosted PostgreSQL needs a fresh schema.

Dashboard Language (dashboard-fragments/model-routing.json)

  • model.resolution added to the routing-session-observations allowlist.
  • 17 new queries. They reuse the page's latest-attempt selection and Routing scope filters, and filter runs before joining events. No JavaScript queries or presenter-side filtering.
  • New collapsible views on the Model routing page:
    • Model resolution coverage: scoped runs, runs with resolution evidence, positively identified routed runs, routed runs without resolution, and non-routed runs without resolution. Absence alone never marks a run as routed or failed.
    • Model resolution status: by event engine and role, with distinct runs and role records counted separately.
    • Alias moves: one row per repository / workflow / engine / role / provider / configured model. Shows earliest → latest model, observed transitions, first and last run, and a run count. Only adjacent runs with recorded session clocks can produce an alias moved row; untimed runs are counted but never create a transition.
    • Alias history: one row per resolved model in each series, with its contributing runs.
    • Resolution failures and Resolution failure runs: grouped by configured model and cause, with engine and role context and links to the runs.
  • routing-run page: a Model resolution drill-down listing role, configured → resolved model, status, cause, engine and provider. Unknown values are shown as unknown. Routed and legacy sessions produce no rows.
  • badge.js: alias moved displays as an attention state.
  • Agent catalog (catalog.generated.json, queries.generated.json) regenerated.

Fixtures and tests

  • test/fixtures/model-resolution-sessions.json: minimized envelopes from the five github/gh-aw smoke runs (39091097441, 39091099406, 39091101641, 39091103540, 39091105929), with head SHA, artifact ID and path, and original line numbers. The Opus move, the two failure causes, unsafe values and the legacy session are clearly labelled controlled replays with test run IDs 9900000xx. Providers for the copilot, claude and codex records aren't documented in the issue, so they are stored as null (shown as unknown) rather than guessed.
  • test/model-resolution-fixture.js is shared by the unit and e2e tests.
  • test/unit/model-resolution.test.js covers the seven acceptance cases through ingestion → processDataRequest → view payload.
  • model-routing-contract.json and model-routing.test.js extended only with the new views, queries and run-page views; the original expectations are unchanged.
  • server/spec/storage.test.mjs covers the new optional columns. TestModelResolutionContractUsesPostgres runs the new queries in hosted Postgres, including null and unknown values and routed absence.
  • e2e/model-routing.spec.js: server startup moved into a shared helper with no behaviour change, plus a new test at 1280px and 390px for both IndexedDB and SQLite. It checks alias and failure text, the Engine scope filter, navigation to the run drill-down, and that the mobile layout doesn't scroll horizontally.

Docs

  • specs/dashboard-data.md ("Model routing evidence" and version tables), docs/dashboard-data-ingestion.md and docs/dashboard-data-model.md now describe:
    • the retained fields and safe-value rules
    • runner provenance
    • routed and legacy absence
    • role versus run counting
    • alias-history limits
    • versions 32 / 42 and the hosted schema refresh
  • The spec no longer lists alias moves as a remaining gap.

Known failures not caused by this PR

  • Three Postgres privilege tests (TestPostgres*PrivilegeBoundary, TestPostgresProvisioningRequiredBeforeRuntime) fail the same way on the base commit because of local role password authentication.
  • Safari and iPhone e2e projects fail here because WebKit isn't installed.

Needs checking before merge

  • Three Chrome e2e failures haven't been investigated or compared with main: lazy-page-chunks and two repository-upgrade-batch tests. lazy-page-chunks checks the core bundle size and this PR adds 17 queries, so that one may be caused by this change.

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor
Dashboard dead views

Dead element views (0):
Dead dashboard pages (0):
Analyzed 8 element renderers and 65 dashboard pages across 1 dashboard.json file(s) from dashboard/site/src/components/ui-elements.js.

⚠️ This reachability report is informational and does not block the pull request.
🔎 View the workflow run and report artifact.

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
Copilot AI changed the title [WIP] Enhance model routing to expose configured model resolution and failures Model routing: retain configured-model resolution and add the Model resolution section Oct 18, 2026
Copilot AI requested a review from SivaKesava1 October 18, 2026 17:50
Copilot AI and others added 2 commits October 18, 2026 18:41
…-enhancement

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
@SivaKesava1
SivaKesava1 marked this pull request as ready for review October 18, 2026 20:52
@SivaKesava1
SivaKesava1 merged commit ded621f into main Oct 18, 2026
44 of 46 checks passed
@SivaKesava1
SivaKesava1 deleted the copilot/model-routing-enhancement branch October 18, 2026 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Model routing: expose configured-model resolution, alias moves, and failures

2 participants