Preserve AWF model inventories with null endpoints and live reasoning metadata (G-7) #69754

Problem

The unified session collector rejects an otherwise usable AWF model inventory when any endpoint has routing_models: null. An unconfigured endpoint therefore discards the configured endpoint's live model list too. In the six supplied runs checked below, usage/aw_session.jsonl contains no model.inventory event and instead records session.collection_warning with code invalid_model_inventory.

This is G-7 of the model routing analytics design in #69507. The analytics implementation is referenced in githubnext/gh-cao#18475. A unified-session-only reader needs this inventory to distinguish a live pick/configured model from an unavailable or unknown one, including which reasoning efforts were live. This issue fixes the collector contract, not the analytics page or routing policy.

Verified against main and recorded artifacts

Checked on October 17, 2026, against github/gh-aw main at 52c5bb8fbf92d13eb0fdd0c86866e25b9c9a368b.

The primary reproduction is the successful smoke-claude-on-copilot run 38995410124, at commit 12ba3ffcbaf5760b6094f52d8f7d61938e4e951e. It was created October 17, 2026, 02:33:46 UTC, which is October 16, 2026, 7:33:46 p.m. PDT; this is the October 16 run referenced in the report.

Its agent artifact 12087091651 contains sandbox/firewall/awf-reflect.json with models_fetch_complete: true and five endpoints:

All 40 Copilot routing entries supply supported_endpoints and supported_reasoning_efforts. Of those entries, 36 supply positive integer context_window_tokens values and four supply null. They also carry candidate_metadata_complete and source.

The unchanged reflection file's SHA-256 is 9d59511015584d359b4159737d1d44ed12ad9426a1690e62148ab53b7543648a. Preserve this exact file as the primary regression fixture; do not reconstruct it from the table or trim away the null endpoints.

The run's usage artifact 12086757726 has zero model.inventory events and one invalid_model_inventory collection warning whose data path is sandbox/firewall/awf-reflect.json.

The same signature was independently checked in the existing usage artifacts of all six supplied runs:

Every warning above names the same reflection path. This establishes recurrence across the supplied routed/unrouted and multi-engine sample, not a measured failure rate for every October 16 run. No new workflow run is needed to reproduce the problem.

Cause

The failure is in normalizeModelInventoryData in unified_session_payload.cjs, lines 469-526:

• normalizeRouting rejects a non-array, including null. Endpoint-local routing_models: null consequently makes the normalizer return undefined for the entire snapshot. The collector, lines 481-498, reports invalid_model_inventory and skips that source.
• A null model collection accompanied by boolean configured is currently converted into an empty array. The projected endpoint does not retain configured, so unknown/unconfigured becomes indistinguishable from an observed empty inventory.
• Routing entries project only modelId and optional supportedEndpoints. The recorded reasoning-effort and context-window metadata are dropped.

The current main normalizer was exercised directly with the unchanged recorded reflection: it rejected the snapshot. A diagnostic-only copy with just the four null routing_models properties removed was accepted, retaining all 40 Copilot models and routing entries, but losing all configured fields and reasoning-effort metadata and converting the other four model collections to empty arrays. This isolates the null-routing failure from the separate lossy-projection bugs; that diagnostic copy must not replace the regression fixture.

Existing null-model tests in unified_session_payload.test.cjs, lines 94-104 explicitly expect the lossy empty-array output. Update those expectations as part of this fix, together with directly affected collector tests.

Possible solution

Keep the existing model.inventory event and metadata allowlist, but preserve endpoint state and distinguish unknown from observed empty:

Readers must treat an absent models or routingModels as unknown, never as evidence of an empty list or model unavailability. This includes unconfigured endpoints and configured endpoints whose fetch is pending. Preserve both configured: true and configured: false; a false boolean is not an absent value.

Keep the projection metadata-only. Do not copy arbitrary endpoint/model objects, credentials, headers, URLs, free-form content, or unrelated reflection fields. candidate_metadata_complete and source are not required additions in this fix. New reasoning-effort values must be identifier-shaped, not arbitrary text.

Do not make genuinely malformed inventories success-shaped: wrong collection types, non-string model IDs, and invalid identifier values must still produce invalid_model_inventory and no inventory event for that malformed source. Recognizing AWF's explicit unknown/null states is not a blanket filter that silently drops corrupt entries. The current helper named isIdentifier checks only for a nonblank string, not a full identifier grammar; document and test any required identifier validation accurately rather than claiming the current implementation already enforces it.

Normalization must accept its own projected unknown/omitted fields, remain idempotent through merging and serialization, and leave the recorded source unchanged. This matters because collection normalizes the snapshot before the shared event projection normalizes it again.

Update directly dependent ModelInventoryData declarations and regenerate the unified session schema using the existing generator. The current generated schema requires endpoint models and rejects the new properties, so changing the normalizer alone is insufficient.

Recheck every actual model.inventory reader in pkg/cli after the prerequisite changes land. If a reader assumes every endpoint has a model array, update it to retain unknown versus observed-empty semantics and add focused Go regressions. No dedicated Go reader of model.inventory was found on the checked main; do not add one or expand audit/analytics behavior just to create work for this issue.

Specification and ordering dependencies

Update docs/src/content/docs/specs/unified-agent-session-specification.md: extend the model.inventory description and essential field list with endpoint configured, optional models/routingModels, supportedReasoningEfforts, and contextWindowTokens; explicitly specify null/omission versus empty-array semantics, validation, provenance, and privacy exclusions.

Add a change-log entry and update the document version/date and version references consistently, following Section 13's semantic-versioning rule. The checked document is 1.10.0. The added capability fields are additive, but allowing omission of previously required endpoint models changes an existing consumer guarantee; account for that incompatibility when choosing the version rather than treating the whole change as editorial. Under the current rule this calls for the next major document version, ordinarily 2.0.0 if the prerequisites remain in 1.x. Keep the independent numeric session.format serialization version unchanged unless the file-format contract itself changes.

Start implementation from a main that includes both #69487 and the change tracked by #69725. This companion work overlaps the unified session specification and collector/projection surface; preserve its fields, requirement IDs, tests, and version history. More precisely, #69487's current diff changes the specification but does not change unified_session_payload.cjs; #69725 targets both the specification and that payload file. At verification time #69487 is an open PR and #69725 is an open issue for the companion change, not two already-merged PRs. Choose the version from the actual merged base, not from this issue's snapshot of main.

Acceptance cases

Name each case in actions/setup/js/unified_session_payload.test.cjs as below. Use the recorded file as the common modern fixture; derived inputs must be explicitly labelled compatibility/pending/malformed variants, not described as new production observations. For collection-warning assertions, collect from a temporary root containing only the stated inventory source, so unrelated sources and valid fallbacks cannot mask the result. Exercise publication/serialization and schema validation, not just a helper's return value.

1. Complete recorded reflection with unconfigured endpoints

Test: preserves the complete recorded 38995410124 inventory with unknown unconfigured endpoints

Input: the unchanged sandbox/firewall/awf-reflect.json from run 38995410124's agent artifact, stored as a fixture with the provenance and SHA-256 above.

Exact observable result: the published session contains exactly one model.inventory event and zero warnings with code invalid_model_inventory. The event retains firewall component, agent phase, source path sandbox/firewall/awf-reflect.json, all five endpoints in source order, and modelsFetchComplete: true. Copilot has configured: true, exactly 40 model IDs and 40 routingModels, with model IDs and supported endpoints equal to the recorded sequences. Each routing entry has supportedReasoningEfforts equal to its recorded array, including any explicitly empty array. The 36 known context windows survive as exactly equal contextWindowTokens numbers; that property is absent for the four recorded nulls. OpenAI, Anthropic, Gemini, and Vertex each have configured: false and neither an own models property nor an own routingModels property. No source credential, URL, or unallowlisted field survives. The stored fixture bytes and parsed input remain unchanged.

2. Configured endpoint with a pending model fetch

Test: keeps a configured pending endpoint unknown instead of inventing an empty model list

Input: an explicitly labelled pending-fetch variant of case 1's recorded Copilot endpoint: retain its provider and configured: true, set models and routing_models to null, and mark the enclosing fetch incomplete. Do not hand-author a different endpoint shape.

Exact observable result: there is one inventory event, zero invalid_model_inventory warnings, and modelsFetchComplete: false. The Copilot endpoint retains configured: true and has no own models or routingModels property. It is not serialized as null or an empty array. Re-normalization and a JSON round trip preserve those absences and the boolean; readers classify the model list as unknown, not empty/unavailable.

3. Recorded legacy reflection

Test: preserves the existing projection of a recorded legacy golden reflection

Input: an unchanged older /reflect copy associated with a case under pkg/cli/testdata/model_routing_golden/, without configured or routing_models; store that real recorded copy with its case/run/artifact/source provenance. Use the same input against the pre-change main normalizer to pin the baseline.

Exact observable result: the projected event data is deeply identical to today's projection of that same recorded input, with the same providers, model ID lists, order, supported existing fields, and omission behavior. There is one inventory event and zero invalid_model_inventory warnings for the valid source. No configured, routingModels, supportedReasoningEfforts, or contextWindowTokens is invented where the input supplied none.

Fixture availability correction: the checked main's committed model-routing goldens contain neither raw /reflect copies nor model.inventory events. Therefore an existing committed copy cannot currently be cited as if it were present. Obtain the genuine older reflection from a retained source download/artifact for a golden case and add that copy as a directly related fixture. The source run for copilot-routed-gpt-responses, 37886511291, returned HTTP 404 during this investigation. If its original reflection is unavailable, identify another retained golden source with the required legacy shape; report the missing evidence rather than inventing legacy JSON or relabelling a modified modern file as recorded.

4. Genuinely malformed model ID

Test: warns and emits no inventory for a recorded reflection with a non-string model ID

Input: case 1's recorded reflection, changing only the first Copilot models entry to numeric 42, explicitly labelled a malformed-input variant. Keep the unconfigured endpoint nulls so acceptance of legitimate nulls cannot hide the bad ID.

Exact observable result: zero model.inventory events and exactly one session.collection_warning with code invalid_model_inventory and path sandbox/firewall/awf-reflect.json; no partial Copilot event and no success-shaped empty inventory. Retain the existing wrong-type/invalid-ID rejection coverage. Add tightly related cases for malformed new capability fields, ensuring they cannot bypass the identifier-only/numeric metadata contract.

5. Golden regression boundary

Test: limits recorded model-routing golden changes to corrected inventory events

Input: the recorded cases under pkg/cli/testdata/model_routing_golden/, with real inventory sources from cases 1 and 3 where available. Reuse the unchanged recorded sources; do not synthesize inventories for historical sessions that never retained them.

Exact observable result: fixture/session diffs are limited to new or corrected model.inventory events and removal of the corresponding now-unwarranted invalid_model_inventory warning/related collection bookkeeping when regenerated from that same source. Historical inputs that genuinely lack inventory remain inventory-free. All unrelated events, routing decisions, costs, tokens, request counts, deviations, subagent attribution, and documented golden discrepancies remain unchanged. Existing expected*.json audit outputs should not change on checked main, which has no dedicated inventory reader.

Run make update-model-routing-golden, review every generated diff, then run make verify-model-routing-golden. Regeneration must not be used to accept unrelated behavior changes or broadly recapture/churn fixtures.

For all five cases, verify schema validity, deterministic/idempotent canonical projection, JSON round-trip behavior, and source immutability where applicable. Keep the named payload tests even if collector-level assertions are also added in unified_session.test.cjs.

If a pkg/cli inventory reader changes after the prerequisites, add focused tests named TestModelInventoryReaderPreservesUnknownEndpointModels, TestModelInventoryReaderPreservesLegacyModelLists, and TestModelInventoryReaderRejectsMalformedModelID, using these same recorded inputs/labelled variants. Continue running existing TestModelRoutingGoldenAudit and the golden verifier; do not introduce unrelated Go reader/rendering changes.

Scope

• Keep the PR to this one fix. Do not change files or behavior unrelated to it. Directly dependent collector tests, declarations/generated schema, specification, affected readers if any, and narrowly affected fixtures belong with the fix. Analytics-page changes, router/AWF changes, other analytics gaps, audit-output expansion, and unrelated cleanup do not.
• Before fixing a failing check, compare it with the same check on main. If it also fails on main, do not fix it in this PR; merge main once main is fixed, and note in the PR which checks are known failures on main.
• This applies even if a bot comment asks to fix a failing check: fix only failures caused by this PR's changes.