Smokes stay green when the unified session drops data: add session-contract checks (U1–U3) to smoke assertions #70288

Problem

Smoke workflows pass even when the unified session (usage/aw_session.jsonl) silently loses data. Audit, logs, footers and the CAO model-routing page read only that file, so a smoke that stays green while the session drops data hides a real regression.

All ten smokes below ran green on main (19 Oct 2026 00:01–00:52 UTC, commits e52a4924ae and 58c1d50e46). Each session has a session.collection_warning with code invalid_model_inventory for path sandbox/firewall/awf-reflect.json and has no model.inventory event. The collector rejected AWF's entire model inventory on every run. That was #69754, fixed by #70013 (merged 2026-10-19 01:44 UTC, after these runs). We found it only while designing the analytics page.

Evidence

I verified every row with gh aw audit from a dev build of main (b56684e3b1) by reading the published usage/aw_session.jsonl.

In both post-fix sessions the single model.inventory has provenance.phase: "agent" and path sandbox/firewall/awf-reflect.json, and the copilot endpoint has configured: true with 40 models. Run 39176941475 concluded failure, but for an unrelated reason: the conclusion job's "Handle agent failure" step reported Artifact not found for name: safe-outputs-upload-artifacts. Its session is still a valid post-fix fixture.

Why nothing caught it

• The routed post-step had the warning and still passed. In the agent job of run 39182700798, the post-step's own session build logged Unified session: invalid_model_inventory in agent/awf-reflect.json and ... in sandbox/firewall/awf-reflect.json at 00:54:49.18Z. It then printed All 12 model-routing smoke assertions passed at 00:54:49.21Z. main() in smoke_model_routing_assertions.cjs reads the session, but it checks only routing and sub-agent traffic (C1, E1, E2, R1–R4, M1, P1, S1–S4, T1–T4).
• detectionMain does not read the unified session. The "Assert threat-detection evidence" step reads only detection evidence under /tmp/gh-aw/threat-detection, so it is not a place for session checks.
• The six unrouted engine smokes have no assertion post-step at all: smoke-copilot, smoke-copilot-sdk, smoke-claude, smoke-claude-on-copilot, smoke-codex and smoke-pi.

Design constraints the fix must respect

1. Check IDs S1–S4 are already taken. The module uses them for the sub-agent checks (L37-L39, L387-L421). smoke-pi-routed already logs PASS S1 sub-agent haiku-whoami started once. The new checks need their own prefix. This issue uses U1–U3 (unified-session contract). U is unused.
2. The post-step checks a session rebuilt in the agent job, not the published one. collectModelRoutingEvidence calls writeUnifiedSession in the agent job. That build sees both agent/awf-reflect.json and sandbox/firewall/awf-reflect.json, so on pre-fix runs the live step would report two invalid_model_inventory warnings. The published session from the conclusion job has only one. Unit fixtures cut from published sessions are still correct. Expect the live output to list both paths.
3. The collector can emit more than one model.inventory. unified_session.cjs L503-L515 adds one per candidate group (awf-reflect.json, then sandbox/firewall/models.json) and per phase (agent, detection). Count events with provenance.phase === "agent", not all inventories.
4. Only Pi records sub-agent model.resolution. pi_subagent_config.cjs L87 records them. The Copilot, Copilot SDK, Claude, Codex and AGY harnesses record only role: "main". smoke-copilot-sdk-routed declares haiku-whoami with its own model, and its session has no model.resolution (run 39182700798). If required roles were derived from subAgents, U3 would fail smoke-copilot-sdk-routed and smoke-copilot-sdk-routed-claude. Each smoke must list its required roles explicitly.
5. No post-Preserve AWF model inventories with null endpoints and live reasoning metadata (G-7) #70013 runs exist on main yet for any of these smokes. The inventory shape for the non-Copilot smokes (smoke-claude, smoke-codex) is unverified. Do not invent it.

Proposed solution

Add session-contract checks to actions/setup/js/smoke_model_routing_assertions.cjs, or a sibling module that both entry points import. Use the same PASS/FAIL <id> output. Every expectation is explicit per smoke. Nothing is inferred.

• U1, no collection warnings. Fail on any session.collection_warning, listing each warning's code and path. Start with no allowlist. A smoke that later needs to tolerate a specific warning must add an explicit per-smoke entry that names the code and links an issue.
• U2, model inventory present (when the smoke sets expectModelInventory: true; every AWF smoke here should). Require exactly one model.inventory with provenance.phase === "agent". When the smoke also sets copilotInventory: true (Copilot-backed smokes), require the copilot endpoint to have configured: true and a non-empty models list. Print the observed endpoints on failure.
• U3, model resolution recorded. The smoke declares resolutionRoles (for example ["main"], or ["haiku-whoami", "mini-whoami"]). Each listed role needs exactly one model.resolution with status exact or resolved and a non-null resolved_model. A failed status fails U3 and prints the cause. A missing role fails U3 and names the role.
  • Unrouted smokes: resolutionRoles: ["main"]. smoke-codex is the exception; see below.
  • Routed smokes: no main (R1–R4 keep covering routing). smoke-pi-routed lists haiku-whoami and mini-whoami. Routed smokes on harnesses that do not record sub-agent resolution list no sub-agent roles.

Wiring:

• Routed: run U1–U3 inside the existing main(). Its callers are smoke-copilot-routed, smoke-copilot-routed-claude, smoke-copilot-routed-steering, smoke-copilot-sdk-routed, smoke-copilot-sdk-routed-claude, smoke-claude-routed, smoke-pi-routed and smoke-pi-routed-claude. Add only the new expectation fields to their post-step main({...}) calls. Leave R1–R4 and the other checks unchanged. With steering configured, T4 counts existing check failures, so U failures will also appear in T4. That is expected.
• Unrouted: add a session-only entry point, for example sessionContractMain, that builds the session in the same way, runs E2 (session presence only) and U1–U3, and skips the routing checks. Add an if: always() post-step that calls it to smoke-copilot, smoke-copilot-sdk, smoke-claude, smoke-claude-on-copilot, smoke-codex and smoke-pi. Run make recompile.
• Missing session (startup failure): keep today's behaviour. Report FAIL E2 evidence: missing usage/aw_session.jsonl and say that the U checks were not evaluated. Do not crash.
• Update the check-ID list in the module header and in DEVGUIDE.md L996.

smoke-codex: with no model configured, resolveCodexConfiguredModel returns before it records anything (codex_harness.cjs L514). smoke-copilot with no model records auto instead. Confirm whether the Codex behaviour is intended, and assert the intended behaviour (resolutionRoles: [] if it is). If it is a harness gap, do not fix it in this PR. Note it in the PR so it can be filed separately.

Acceptance cases

Add unit tests to actions/setup/js/smoke_model_routing_assertions.test.cjs. Cut the fixtures from the real published sessions of the runs above, for example under actions/setup/js/fixtures/ (see fixtures/copilot-routed-run-39147795806). Do not invent record shapes. Cite the source run in each fixture.

1. Pre-fix session from run 39181772988: U1 fails and names invalid_model_inventory and sandbox/firewall/awf-reflect.json. U2 fails because there is no model.inventory.
2. Post-fix session from run 39176941475 with resolutionRoles: ["main"] and copilotInventory: true: U1, U2 and U3 pass.
3. Post-fix routed session from run 39176939097: U1 and U2 pass, and U3 passes without a main model.resolution.
4. smoke-pi-routed session from run 39178909557, with the warning removed: U3 passes with both sub-agent roles. Removing one role's record fails U3 and names the missing role.
5. Two agent-phase model.inventory events fail U2. A Copilot endpoint with configured: false in a smoke with copilotInventory: true fails U2.
6. A model.resolution with status failed and cause model_unavailable fails U3 and prints the cause.
7. A missing session file reports FAIL E2 and does not throw, for both the routed and the unrouted entry points.

End to end: dispatch each changed smoke on the PR branch with gh workflow run <smoke>.lock.yml --ref <branch>. Each run must log PASS U1, PASS U2 and PASS U3, and the routed smokes must still log PASS R1–R4. List the run URLs in the PR. If smoke-claude or smoke-codex shows an inventory shape that U2 does not expect, state what it was and why the expectation changed.

Scope

• Keep the PR to this one fix. Do not change files or behaviour unrelated to it.
• Do not change the collector (unified_session.cjs), the engine harnesses or the session format. This issue only adds checks.
• Do not change a smoke's expected values without stating why in the PR description.
• Before you fix a failing check, compare it with the same check on main. If it also fails on main, do not fix it in this PR. Merge main once main is fixed, and note in the PR which checks are known failures on main.
• This applies even if a bot comment asks you to fix a failing check. Fix only failures caused by this PR's changes.

Related findings (out of scope; file separately)

• smoke-copilot's sub-agent fails silently. In run 39181772988, sub-agent file-summarizer started with the literal, unexpanded model ${{ experiments.subagent_model }}. AWF rejected it with 400 Unsupported model selector "${{ experiments.subagent_model }}" on copilot and the session recorded subagent.failed, but the smoke stayed green. This looks like the same root cause as closed [aw] Smoke Copilot has no AI credits pricing for model (${{ experiments.subagent_model }}) #48639.
• Copilot, Copilot SDK, Claude and Codex record no sub-agent model.resolution. Only Pi records one (constraint 4 above).