Add unified-session contract checks (U1–U3) to smoke assertions

[Copilot]

Smokes passed even when the unified session (usage/aw_session.jsonl) silently dropped data. On every smoke run on 19 Oct 2026 before #70013 merged, the collector rejected AWF's whole model inventory (#69754). The routed post-step logged the warning and still passed, and the six unrouted engine smokes had no assertion step. This PR adds explicit, per-smoke session-contract checks. It does not change the collector, the harnesses or the session format.

Checks (smoke_model_routing_assertions.cjs)

• U1: fails on any session.collection_warning and lists each code in path. There is no allowlist.
• U2 (when expectModelInventory is set): requires exactly one model.inventory with provenance.phase === "agent". Detection-phase inventories are ignored. With copilotInventory, the copilot endpoint must be configured: true with a non-empty models list; on failure the message lists every observed endpoint.
• U3: each role in the smoke's explicit resolutionRoles needs exactly one model.resolution with status exact or resolved and a resolved model. A failed record prints its cause, and a missing role is named. Roles are never derived from subAgents, because only Pi records sub-agent resolutions.
• Routed main(): U1–U3 run only when one of the new fields is set, so existing callers and their output are unchanged. They run before T4, so the steering smoke also reports U failures under T4.
• New sessionContractMain: builds the session the same way as main(), then runs only E2 (session present) and U1–U3.
• Missing session: both entry points report FAIL E2 evidence: missing usage/aw_session.jsonl, log U1–U3 not evaluated: …, and don't throw.

await sessionContractMain({ core, engine: "copilot", expectModelInventory: true, copilotInventory: true, resolutionRoles: ["main"] });

Per-smoke expectations (all set expectModelInventory: true)

Smoke	copilotInventory	resolutionRoles	Rationale
copilot-routed, copilot-routed-claude, copilot-routed-steering, claude-routed, pi-routed-claude	true	[]	Routed through the Copilot gateway; no main resolution is recorded, and R1–R4 cover routing
copilot-sdk-routed, copilot-sdk-routed-claude	true	[]	The SDK harness records no sub-agent resolution (run 39182700798)
pi-routed	true	["haiku-whoami","mini-whoami"]	Pi records both (run 39178909557)
copilot, copilot-sdk, claude-on-copilot, pi	true	["main"]	Copilot-backed; main resolution is recorded
claude	false	["main"]	Anthropic provider; the inventory shape after #70013 is unverified
codex	false	[]	See below

• Routed smokes: the eight main({...}) calls get only the three new fields.
• Unrouted smokes: the six get an if: always() post-step that calls sessionContractMain. Lock files are recompiled.

smoke-codex: treated as intended, not a harness gap. With no configured model, gh-aw passes no model and Codex uses its own default, so there is nothing to resolve. Copilot differs because it always requests auto.

Fixtures and tests

• Fixtures: fixtures/session-contract/*.jsonl holds verbatim session.format, session.collection_warning, model.inventory and model.resolution records from the published sessions of runs 39181772988, 39176941475, 39176939097 and 39178909557. fixtures/session_contract_sessions.cjs cites each source run.
• Derived variants: the failed status, unconfigured endpoint, duplicate inventory and removed role are edits of those records and are labelled in the tests.
• Tests: cover the seven acceptance cases plus a sessionContractMain pass case and a test that T4 counts U1/U2 failures.

Docs

• Check-ID lists in the module header and DEVGUIDE.md now include U1–U3.

Reviewer notes

• Live output: the post-step builds the session in the agent job, which sees both agent/awf-reflect.json and sandbox/firewall/awf-reflect.json. On a build without Preserve AWF model inventories with null endpoints and live reasoning metadata (G-7) #70013, U1 therefore lists two warnings.
• Possible U2 failure: if AWF writes sandbox/firewall/models.json in the agent job, the collector adds a second agent-phase inventory and U2 fails. The failure lists both paths. Nothing in this repo writes that file, so this is unverified.
• E2E not run: each changed smoke still needs to be dispatched on this branch (gh workflow run <smoke>.lock.yml --ref <branch>), with the run URLs added here.

Out of scope (to file separately)

• smoke-copilot's file-summarizer sub-agent starts with an unexpanded ${{ experiments.subagent_model }} and fails silently.
• The Copilot, Copilot SDK, Claude and Codex harnesses record no sub-agent model.resolution.

• Fixes Smokes stay green when the unified session drops data: add session-contract checks (U1–U3) to smoke assertions #70288

---

---

✨ PR Review Safe Output Test - Run 39294456721

💥 [THE END] — Illustrated by Smoke Claude · claude · sonnet46 · 73 AIC · ⌖ 6.24 AIC · ⊞ 922 · ◷
_{Comment /smoke-claude to run again}

[github-actions]

💥 [THE END] — Illustrated by Smoke Claude · claude · sonnet46 · 73 AIC · ⌖ 6.24 AIC · ⊞ 922
_{Comment /smoke-claude to run again}

[SivaKesava1]

End-to-end verification at e0ac41e5e1 (all 14 changed smokes dispatched on this branch, 19 Oct 2026):

Smoke	Run	U1	U2	U3	R1–R4
smoke-copilot	39294445658	PASS	PASS (40 Copilot models)	PASS main auto → claude-sonnet-5.5 (resolved)	—
smoke-copilot-sdk	39294451360	PASS	PASS (40)	PASS main copilot/gpt-5.4 → gpt-5.4 (exact)	—
smoke-claude	39294456721	PASS	PASS	PASS main claude-sonnet-4-6 (exact)	—
smoke-claude-on-copilot	39294460429	PASS	PASS (40)	PASS main haiku → claude-haiku-5.5 (resolved)	—
smoke-codex	39294466143	PASS	PASS	PASS (no roles required)	—
smoke-pi	39294472203	PASS	PASS (40)	PASS main copilot/auto → auto (exact)	—
smoke-copilot-routed	39294477012	PASS	PASS (40)	PASS	PASS
smoke-copilot-routed-claude	39294481306	PASS	PASS (40)	PASS	PASS
smoke-copilot-routed-steering	39294486170	PASS	PASS (40)	PASS	PASS
smoke-copilot-sdk-routed	39294491193	PASS	PASS (40)	PASS	PASS
smoke-copilot-sdk-routed-claude	39294495695	PASS	PASS (40)	PASS	PASS
smoke-claude-routed	39294499601	PASS	PASS (40)	PASS	PASS
smoke-pi-routed	39294504845	PASS	PASS (40)	PASS haiku-whoami, mini-whoami (exact)	PASS
smoke-pi-routed-claude	39294509695	PASS	PASS (40)	PASS	PASS

• Every run found exactly one agent-phase model.inventory, from agent/awf-reflect.json, so the "second agent-phase inventory" risk in the PR description did not occur.
• smoke-claude and smoke-codex (copilotInventory: false) also produced one agent-phase inventory.
• smoke-copilot and smoke-pi concluded failure in the conclusion job's "Handle agent failure" step, for reasons unrelated to this PR: the agents reported report_incomplete ("bash command batch denied (no interactive approval)" and missing_terminal_safe_output). Their U checks passed. Both smokes have failed this way on main before (for example 15 Oct runs 38660888679 and 38660477645).

CI is green (JS shards 1–4, typecheck, lint, stale lock files). Ready to finish.

[github-actions]

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · opus55 · 11.2 AIC · ⌖ 31.9 AIC · ◇ 14.4 AIC · ⊞ 1.9K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

[github-actions]

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

[github-actions]

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · sonnet50 · 13.3 AIC · ⌖ 15.1 AIC · ◇ 6.63 AIC · ⊞ 12.6K · ◷

🧪 Test quality analysis by Test Quality Sentinel

[github-actions]

✅ Ponytail Reviewer completed successfully!

Generated by Ponytail Reviewer for #70294

[github-actions]

⚠️ PR Code Quality Reviewer failed to deliver outputs during code quality review.

🔎 Code quality review by PR Code Quality Reviewer

[github-actions]

L814: shrink: duplicated result reporter. Extract the shared reporter.

net: -10 lines possible.

Generated by ✂️ Ponytail Reviewer for #70294 · codex · gpt56 · 15.4 AIC · ⌖ 5.4 AIC · ⊞ 14.7K
_{Comment /ponytail to run again}

[github-actions]

Comment Memory

Peek at saved memory (pr-code-quality-reviewer)

reviewed_at: 2026-10-19T17:58:47Z
review_event: REQUEST_CHANGES
top_themes:
  - brittle unified-session warning handling
  - false failures on duplicate inventory sources
files_reviewed:
  - actions/setup/js/smoke_model_routing_assertions.cjs
  - actions/setup/js/smoke_model_routing_assertions.test.cjs
  - actions/setup/js/unified_session.cjs
  - .github/workflows/smoke-*.md
comment_count: 2

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 101.6 AIC · ⌖ 6.03 AIC · ⊞ 16K · ◷
_{Comment /review to run again}

[Copilot]

🔵 Needs a closer look

Live U1–U3 checks passed for Smoke Claude, but end-to-end validation of the other 13 smoke variants remains unverified.

1 open issue

• Coalesce repeated warnings while preserving distinct diagnostics · New

What changed in this PR

Adds unified-session contract checks to gh-aw smoke workflows, addressing #70288 without changing the collector, harnesses, or session format.

Changes:

• Adds U1–U3 checks and a session-only assertion entry point.
• Configures explicit expectations across 14 smokes and regenerates their lock files.
• Adds recorded-session regression fixtures, tests, and documentation.

File	Description
DEVGUIDE.md	Documents U1–U3 and unrouted checks.
actions/​setup/​js/​smoke_model_routing_assertions.test.cjs	Tests contract checks and entry points.
actions/​setup/​js/​smoke_model_routing_assertions.cjs	Implements session-contract assertions.
actions/​setup/​js/​fixtures/​session-contract/​smoke-pi-routed-run-39178909557.jsonl	Records Pi sub-agent resolution evidence.
actions/​setup/​js/​fixtures/​session-contract/​smoke-copilot-run-39181772988.jsonl	Records pre-fix collection failure evidence.
actions/​setup/​js/​fixtures/​session-contract/​smoke-copilot-run-39176941475.jsonl	Records successful inventory and resolution evidence.
actions/​setup/​js/​fixtures/​session-contract/​smoke-copilot-routed-run-39176939097.jsonl	Records successful routed inventory evidence.
actions/​setup/​js/​fixtures/​session_contract_sessions.cjs	Loads fixtures and cites source runs.
.github/​workflows/​smoke-pi.md	Adds standalone Pi contract checks.
.github/​workflows/​smoke-pi.lock.yml	Generates Pi contract-check step.
.github/​workflows/​smoke-pi-routed.md	Requires inventory and both sub-agent resolutions.
.github/​workflows/​smoke-pi-routed.lock.yml	Generates routed Pi expectations.
.github/​workflows/​smoke-pi-routed-claude.md	Adds routed inventory expectations.
.github/​workflows/​smoke-pi-routed-claude.lock.yml	Generates Claude-pool Pi expectations.
.github/​workflows/​smoke-copilot.md	Adds standalone Copilot contract checks.
.github/​workflows/​smoke-copilot.lock.yml	Generates Copilot contract-check step.
.github/​workflows/​smoke-copilot-sdk.md	Adds standalone SDK contract checks.
.github/​workflows/​smoke-copilot-sdk.lock.yml	Generates SDK contract-check step.
.github/​workflows/​smoke-copilot-sdk-routed.md	Adds routed SDK inventory expectations.
.github/​workflows/​smoke-copilot-sdk-routed.lock.yml	Generates routed SDK expectations.
.github/​workflows/​smoke-copilot-sdk-routed-claude.md	Adds Claude-pool SDK inventory expectations.
.github/​workflows/​smoke-copilot-sdk-routed-claude.lock.yml	Generates Claude-pool SDK expectations.
.github/​workflows/​smoke-copilot-routed.md	Adds routed Copilot inventory expectations.
.github/​workflows/​smoke-copilot-routed.lock.yml	Generates routed Copilot expectations.
.github/​workflows/​smoke-copilot-routed-steering.md	Enables contract checks alongside steering checks.
.github/​workflows/​smoke-copilot-routed-steering.lock.yml	Generates steering-smoke expectations.
.github/​workflows/​smoke-copilot-routed-claude.md	Adds Claude-pool Copilot inventory expectations.
.github/​workflows/​smoke-copilot-routed-claude.lock.yml	Generates Claude-pool Copilot expectations.
.github/​workflows/​smoke-codex.md	Adds inventory checks without required resolutions.
.github/​workflows/​smoke-codex.lock.yml	Generates Codex contract-check step.
.github/​workflows/​smoke-claude.md	Adds inventory and main-resolution checks.
.github/​workflows/​smoke-claude.lock.yml	Generates Claude contract-check step.
.github/​workflows/​smoke-claude-routed.md	Adds Copilot-backed inventory expectations.
.github/​workflows/​smoke-claude-routed.lock.yml	Generates routed Claude expectations.
.github/​workflows/​smoke-claude-on-copilot.md	Adds Copilot inventory and main-resolution checks.
.github/​workflows/​smoke-claude-on-copilot.lock.yml	Generates Claude-on-Copilot contract step.

_{Review effort: Max}

[github-actions]

Skills-Based Review 🧠

Applied /tdd and /grill-with-docs to the new unified-session contract logic (evaluateSessionContract, U1–U3). Two targeted comments posted inline; nothing blocking.

📋 Key Themes & Highlights

Key Themes

• Branch coverage gap (/tdd): the duplicate-role (forRole.length > 1) and empty-resolved_model branches in U3 aren’t exercised by any test, even though the PR adds thorough coverage for most other paths (missing role, failed status, missing session, two inventories, unconfigured endpoint).
• Silent-skip ambiguity (/grill-with-docs): when a smoke sets only resolutionRoles (no expectModelInventory/copilotInventory), U2 is gated off with no log line at all — worth an explicit INFO note or a DEVGUIDE callout so a quiet U2 isn’t mistaken for a passing one.

Positive Highlights

• ✅ Real, cited fixture sessions (fixtures/session_contract_sessions.cjs) trace each JSONL back to an actual run — strong grounding for the assertion logic, not synthetic guesses.
• ✅ Explicit per-smoke resolutionRoles/copilotInventory config avoids inferring expectations from subAgents, which the PR body correctly notes would be unsound (only Pi records sub-agent resolutions).
• ✅ Clean separation between the routed main() path (U1–U3 folded into existing checks) and the new sessionContractMain for unrouted smokes, with consistent E2/U1–U3 messaging on both.
• ✅ DEVGUIDE.md and the module header check-ID list were both updated alongside the code — no doc drift.

Scope note

As the PR itself flags, no smoke has been E2E-dispatched on this branch yet (reviewer notes section). That’s out of scope for this review but worth confirming before merge given the per-smoke copy-pasted expectations (copilotInventory, resolutionRoles) are easy to get subtly wrong without a live run.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 120.5 AIC · ⌖ 15 AIC · ⊞ 10.1K
_{Comment /matt to run again}

[gh-aw-bot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
2. Review (.github/workflows/smoke-claude.md:155): The post-steps assertion block looks good — the sessionContractMain function provides solid unified-session contract checks (U1-U3). Consider adding error handling for cases where the runner temp path might be unavailable. - Add unified-session contract checks (U1–U3) to smoke assertions #70294 (comment)
3. Review (.github/workflows/smoke-claude.md:163): The resolutionRoles: ["main"] parameter looks correct for limiting checks to the main engine role. It would be valuable to also document what happens when expectModelInventory: true but the inventory file is missing — does it fail gracefully? - Add unified-session contract checks (U1–U3) to smoke assertions #70294 (comment)
4. Review (actions/setup/js/smoke_model_routing_assertions.cjs:814): L814: shrink: duplicated 10-line result-reporting/failure block from main. Extract reportResults(core, results, label) and have both entry points call it, removing the second copy. - Add unified-session contract checks (U1–U3) to smoke assertions #70294 (comment)
5. Review (actions/setup/js/smoke_model_routing_assertions.cjs:612): Optional: coalesce repeated U1 warnings by (code, path). The collector emits a warning per bad line or record (unified_session.cjs:219–225,248–251,274–277), so one broken log can produce thousands of identical entries in both the error and the setFailed summary. Other warning sources become hard to find, and available data.line values are also omitted. Report a count and an example line for each pair, while keeping every distinct pair visible to preserve Smokes stay green when the unified session drops data: add session-contract checks (U1–U3) to smoke assertions #70288's diagnostic requirement. - Add unified-session contract checks (U1–U3) to smoke assertions #70294 (comment)
6. Review (actions/setup/js/smoke_model_routing_assertions.cjs:645): [/tdd] The duplicate-role branch (forRole.length > 1, line 645) and the empty-resolved_model branch (line 651) have no direct test cases — only the single-resolution-per-role and status: "failed" paths are exercised. A smoke that double-emits a role's resolution, or resolves to an empty string, would silently rely on untested message formatting. - Add unified-session contract checks (U1–U3) to smoke assertions #70294 (comment)
7. Review (actions/setup/js/smoke_model_routing_assertions.cjs:616): [/grill-with-docs] hasSessionContract (used to gate whether U1–U3 run at all) treats copilotInventory: false as "configured" (since it checks !== undefined), but the U2 gate itself (expectModelInventory || copilotInventory) is falsy when both are false/unset. So a smoke that sets only resolutionRoles: [] and leaves expectModelInventory/copilotInventory unset still runs U1 and U3 but silently skips U2 with no PASS/FAIL line — which could read as "all green" in the log even though inventory wasn't checked. - Add unified-session contract checks (U1–U3) to smoke assertions #70294 (comment)

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: 796c4d3
Sous-chef work: 0dc7f6501c8f2d8c4c5538220b4869a5e6395a0b5eb5d804a11a393bcffeae03 15a7cffa7c74a5b21b68d439fd06bc9b342ffe158594279f1ff4eaac7ca94036 33d40fc59f65737831c6bc161997aae748a97fd1f0b171391094b59768dea37d 46ad575c90c4d7c67ecc91990e12148dd398b3366cd00efcf0c698801b12c55c d97c712275eb75ae1bcb54480aa52a63fcd89711d3b7909c3f6a993c6709f7da f8305fff1b93de4ac7d0aa989905c03f36fea83421669ae0a2722e1db9be9795
Sous-chef state: 07c3308963f1c5bd33bc8eb1dc35ac4493d1de5a90d225c65b954891180f01c5

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 7.02 AIC · ⌖ 11.7 AIC · ⊞ 1.1K · ◷
_{Comment /souschef to run again}

[Copilot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes....

Merged latest main with make merge-main (f6dae34) and pushed all six review fixes in e18885d. All 112 focused tests, typecheck, make lint, make agent-report-progress, and workflow compilation passed; no CI was triggered. Replied to every linked thread. Thread resolution is not exposed by the available PR tools, so the addressed threads still need marking resolved. Independent review found no significant issues; the standard review backend was unavailable and CodeQL timed out. CI on the new head remains unverified.

[gh-aw-bot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
2. Fix failing check JS Tests (shard 2/4) (FAILURE): https://github.com/github/gh-aw/actions/runs/39311151325/job/118261059898.

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: e18885d
Sous-chef work: 043185828d65a534dd8b2b6b4d00e0d47bf21a2940d3f277c0ed15da7f668648
Sous-chef state: d6484250713fcd5ab23bc497c20b11315a7828485df7632017373637644e77f2

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 3.19 AIC · ⌖ 8.06 AIC · ⊞ 1.1K · ◷
_{Comment /souschef to run again}

[Copilot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes....

Fixed in 95ac4cd. The two CI failures came from a stale nano-whoami/gpt-5-nano fixture; it now matches mini54-whoami/gpt-5.4-mini in the workflow without weakening assertions. make merge-main completed successfully; latest main was already included. All 120 focused tests, make lint, and make agent-report-progress passed. Full shard rerun: 4,580 passed, 2 skipped, with 2 unrelated work-queue performance tests timing out locally. Independent review found no issues. No CI was triggered; CI on the new head remains unverified. Earlier threads already have replies, but resolving them is unavailable through the provided tools.

[SivaKesava1]

Re-verification at 893ffe1377, after the merge with main and the review fixes:

Smoke	Run	Result
smoke-copilot	39325986042	success; PASS U1, U2 (36 Copilot models), U3 (main auto → claude-sonnet-5.5)
smoke-codex (copilotInventory: false)	39325990466	success; PASS U1, U2, U3 (no roles required)
smoke-copilot-routed	39327773561	success; PASS R1–R4, U1, U2, U3
smoke-pi-routed	39325995892	failure, not caused by this PR (details below)

smoke-pi-routed. Pi refused to start because the mini-whoami sub-agent's copilot/gpt-5.4-mini is no longer in the live Copilot catalogue (fatal: Pi model "gpt-5.4-mini" is not advertised by the configured copilot proxy endpoint). Copilot removed it around 18:00 UTC today, and #70367 lists the pin as unresolvable. The same smoke passed on this branch at 17:29 (39294504845). The new checks behaved correctly on the broken run:

• U1 failed on the unrecognized_engine_log warning from the aborted startup.
• R3, R4 and S1–S3 failed because no requests were made.

U3 still passed, because Pi records mini-whoami as exact/gpt-5.4-mini before its liveness check rejects the model. That is a harness recording issue to file separately, not part of this PR.

All review threads are answered and resolved. JS shards 1–4, typecheck, lint and the stale lock-file check pass. Ready to merge.

[github-actions]

🎉 This pull request is included in a new release.

Release: v0.92.6