audit: report model resolution startup failures from the unified session

.github/aw/debug-playbook.md

@@ -35,17 +35,18 @@ Use same-run artifacts, not baselines.
 
 | Codes | Check | Remedy |
 | --- | --- | --- |
-| `workflow_failed` | First failed step; native error. | Route specific cause; conclusion is not root cause. |
+| `workflow_failed` | First failed step; native error. | Route specific cause; conclusion isn't root cause. |
 | `workflow_timeout` | Timed-out step, budget, terminal result. | Remove blocking work/narrow scope; justify timeout changes. |
 | `multiple_errors` | First distinct error versus cascades. | Fix first cause, not repeats. |
 | `mcp_server_failures`, `tools_not_available` | MCP startup, registration, failed calls. | Fix tool/server/config/auth; no blind permission expansion. |
-| `model_unavailable` | Complete same-run matching provider inventory excludes exact model ID. | Follow [model availability](#model-availability); verify advertised candidate/API, edit pin, recompile. |
-| `model_availability_unknown` | Missing, incomplete, malformed, ambiguous, or contradictory reflection. | Inspect discovery/auth; do not declare removal or switch models on missing evidence. |
-| `subagent_model_not_observed` | Spawn, child usage, evidence coverage. | Missing observations do not prove nonexecution. |
+| `model_unavailable` | Complete same-run provider inventory excludes exact model. | [Model availability](#model-availability); verify candidate/API, edit pin, recompile. |
+| `model_availability_unknown` | Missing/incomplete/malformed/ambiguous/contradictory reflection. | Inspect discovery/auth; do not declare removal/switch models without evidence. |
+| `model_resolution_failed` | Failed `model.resolution` lacking same-role startup-stop evidence. | Advisory: verify listed pins; not a proven failure cause. |
+| `subagent_model_not_observed` | Spawn, child usage, evidence coverage. | Missing observations don't prove nonexecution. |
 | `subagent_model_unavailable` | Child model, request, provider policy/catalog. | Verify child separately; follow [model/engine guide](debug-model-engine-misconfiguration.md). |
-| `subagent_failed` | Child native error; parent dispatch. | Fix child cause; parent success does not prove child success. |
+| `subagent_failed` | Child native error; parent dispatch. | Fix child cause; parent success doesn't prove child success. |
 | `blocked_network_requests` | Destination, caller, denial, necessity. | Fix URL/tool or authorize narrow destination; retain firewall. |
-| `threat_detection_job_failed` | Detector startup/runtime versus verdict. | Repair detector; failure is not a clean verdict. |
+| `threat_detection_job_failed` | Detector startup/runtime versus verdict. | Repair detector; failure isn't a clean verdict. |
 | `threat_detected` | Private structured verdict/artifact. | Stop unsafe effects; follow [security guidance](debug-security-review.md). Never publish payloads. |
 | `high_token_usage`, `many_iterations` | Tokens/cache/turns, repeated calls. | [Optimize](optimize-agentic-workflow.md) scope/context/loops before raising caps. |
 | `agentic_resource_heavy_for_domain`, `agentic_overkill_for_agentic` | Assessment, deterministic alternatives, judgment needed. | Script mechanical work; retain necessary LLM judgment. |
@@ -62,10 +63,10 @@ Inventory: [Model availability](#model-availability).
 
 | Codes | Check | Remedy |
 | --- | --- | --- |
-| `inference_access_error`, `inference_access_denied`, `authentication_failed` | Native 401/403, credential names, permissions/policy, proxy guards. | Fix access/config; budget/policy denial does not warrant credential rotation. |
+| `inference_access_error`, `inference_access_denied`, `authentication_failed` | Native 401/403, credential names, permissions/policy, proxy guards. | Fix access/config; budget/policy denial doesn't warrant credential rotation. |
 | `mcp_policy_error` | Blocked MCP server; organization policy. | Use approved tool/authorized policy correction; no bypass. |
-| `model_not_supported_error`, `model_endpoint_mismatch_error`, `model_endpoint_mismatch` | Model, API, tool protocol, same-run inventory/routes. | Follow [model/engine guide](debug-model-engine-misconfiguration.md); endpoint change does not establish availability. |
-| `http_400_response_error` | Provider diagnostic, endpoint, feature/schema. | Fix unsupported request; 400 alone does not prove model removal. |
+| `model_not_supported_error`, `model_endpoint_mismatch_error`, `model_endpoint_mismatch` | Model, API, tool protocol, same-run inventory/routes. | Follow [model/engine guide](debug-model-engine-misconfiguration.md); endpoint change doesn't establish availability. |
+| `http_400_response_error` | Provider diagnostic, endpoint, feature/schema. | Fix unsupported request; 400 alone doesn't prove model removal. |
 | `unsupported_endpoint`, `invalid_selection`, `uncorroborated_endpoint`, `uncorroborated_selection`, `harness_selection_mismatch` | Selection versus harness model/API and metadata. | Align selection/API; retain verification; never invent evidence. |
 | `unsupported_effort` | Effort versus engine/model support. | Use supported effort/mapping; no silent discard. |
 | `missing_model_pricing_error`, `unknown_model_ai_credits` | Proxy pricing table and `missing_model_pricing_model_name`. | Correct supported pricing/version/config; never disable credit accounting. |
@@ -75,10 +76,10 @@ Inventory: [Model availability](#model-availability).
 | `effective_tokens_limit_exceeded` | Effective tokens; prompt/turn growth. | Reduce context/iterations; retain guard. |
 | `permission_denied_limit_exceeded` | Denied calls; allowed tools. | Stop denied calls; use allowed tools, not larger denial budget. |
 | `model_policy_violation`, `content_filter` | Provider/proxy reason; endpoint mismatch separately. | Correct request within policy; no credential swap/bypass. |
-| `awf_api_proxy_blocking_requests` | Specific budget/policy/filter guard. | Address guard; blocking alone does not establish authentication failure. |
+| `awf_api_proxy_blocking_requests` | Specific budget/policy/filter guard. | Address guard; blocking alone doesn't establish authentication failure. |
 | `capi_quota_exceeded_error` | Native 429; exhausted retries. | Respect quota/reset; no retry storm. |
 | `capi_server_error`, `engine_outage`, `infrastructure_error` | Retry-exhausted native 5xx/infrastructure timeline. | Report transient cause; bounded backoff, no unapproved redispatch. |
-| `agentic_engine_timeout` | Step/SDK budget versus post-result `watchdogFired=true`. | Fix genuine stall; post-result watchdog alone does not establish timeout. |
+| `agentic_engine_timeout` | Step/SDK budget versus post-result `watchdogFired=true`. | Fix genuine stall; post-result watchdog alone doesn't establish timeout. |
 | `sandbox_runtime_crash` | Exit/signal, resources, runtime revision. | Fix crash/resources; retain redacted diagnostic/version. |
 | `goal_already_active` | Unfinished goal. | Update/complete existing goal; no duplicate. |
 | `shell_expansion_guard_rejected` | Private denied command; allowed syntax. | Use allowed tools without dangerous expansion; no obfuscation/bypass. |

docs/adr/70653-use-unified-session-as-primary-source-for-audit-model-resolution-findings.md

@@ -0,0 +1,52 @@
+# ADR-70653: Use the Unified Session as the Primary Source for Audit Model-Resolution Findings
+
+**Date**: 2026-10-20
+**Status**: Draft
+**Deciders**: SivaKesava1 (PR author / actor); maintainer review pending
+
+---
+
+### Context
+
+`gh aw audit` only diagnosed model-availability startup failures by matching a message in Pi's `agent-stdio.log`. Other harnesses (for example Copilot and Copilot SDK sub-agent runs 39499460077, 39499464895 and 39508150750) already write fail-closed `model.resolution` records to the unified session (`usage/aw_session.jsonl`) under the T-UAS-074 contract, but audit ignored them, so these failures went undiagnosed. A failed `model.resolution` record does not say whether the failing role was a declared sub-agent or an optional repository agent, so a record alone cannot prove that it caused the run to fail. Routed runs also produce router or classifier traffic and wrapper `state: "started"` markers that can look like agent activity without the native agent ever running.
+
+### Decision
+
+We will read failed `model.resolution` events from the unified session as the primary evidence for model-resolution findings, and keep the Pi `agent-stdio.log` match only as a fallback when the session has no failed resolution events. Resolution records are accepted only with the recorded provenance (`model_resolution` / `agent` / `agent/model-resolution.json`) and fields that satisfy the T-UAS-074 contract. Only the canonical `usage/aw_session.jsonl` is read. It must be a regular file (no symlinks) of at most 256 MiB with a `session.format` v1 header and exactly one `agent.execution` record; anything else is an invalid session. A role's failure is attributed as the startup cause only when the run failed, the single `agent.execution` exited non-zero, there is no native agent activity (`session.start`, `session.init`, `assistant.*`, `tool.*` from component `agent`), and the same run's `agent-stdio.log` affirmatively links that role to the stop: a harness `Refusing to start` line naming the role, or that role's model-specific resolution error. Roles without that evidence get a single low-severity advisory (`model_resolution_failed`). Failures are grouped by engine, provider, model and cause, and each group is classified only against the same run's `awf-reflect.json`. A run therefore gets at most one `model_unavailable` (high) finding and at most one `model_availability_unknown` finding. Recorded causes other than `model_unavailable` are reported as recorded and never confirmed as unavailability.
+
+### Alternatives Considered
+
+#### Alternative 1: Extend log-message matching to each harness
+
+Add per-engine log patterns, like the existing Pi `agent-stdio.log` match, for Copilot and Copilot SDK output. This would follow the existing pattern, but it ties audit to free-form log text that varies by harness and version. It also ignores the structured, contract-validated records the harnesses already write. It was not chosen because the unified session is the cross-engine source of truth for this data.
+
+#### Alternative 2: Report every failed resolution record as a high-severity failure cause
+
+Treat any failed `model.resolution` record as the reason the run failed, without checking startup evidence. This is simpler, but it misattributes failures. An optional repository agent can fail resolution while the main agent continues and the run succeeds, as in the run 39508154330 negative control. It was not chosen because the record cannot tell sub-agents and repository agents apart, and the producer was deliberately left unchanged.
+
+#### Alternative 3: Change the producer to record the role kind (sub-agent vs repository agent)
+
+Add a field to `model.resolution` so audit can attribute failures directly. This would make attribution more precise. It was not chosen in this PR because it changes the T-UAS-074 producer contract and is outside the `pkg/cli` scope. Historical runs would also still lack the field. [TODO: verify whether this is planned as a follow-up]
+
+### Consequences
+
+#### Positive
+- Audit diagnoses model-resolution startup failures for all harnesses that write the unified session, not only Pi.
+- Attribution requires run-level startup evidence plus per-role harness stop evidence and reports a low-severity advisory in every other case, which avoids false "model unavailable" root causes on successful or continuing runs.
+- Classification uses only the same run's reflection inventory, and recorded causes such as `catalog_unavailable` are preserved, so audit does not invent unavailability.
+- Strict provenance, schema and file-type validation (no symlinks, bounded total size and line length, v1 header, a single `agent.execution`) limits trust in artifact contents.
+
+#### Negative
+- Audit now depends on the T-UAS-074 record schema and the provenance constants. Producer changes must be coordinated or records will be silently ignored.
+- Per-role attribution depends on harness `agent-stdio.log` wording; a harness that stops without such a line yields only the advisory.
+- There are two detection paths (session-first, legacy Pi log fallback) to maintain until the legacy path can be removed.
+- A new finding code (`model_resolution_failed`) adds to the audit surface; the debug playbook has a row for it, but `audit.md` still describes these findings as Pi-only.
+- The audit cache schema moves to 10 so reports cached before this analysis are regenerated.
+
+#### Neutral
+- `checkModelAvailability` was extracted from the Pi path and is now shared by both paths, with its messages unchanged.
+- Sanitized replay fixtures from real runs live under `pkg/cli/testdata/model_resolution_startup/`. Derived variants are labelled as derived in the test comments.
+
+---
+
+*ADR created by [adr-writer agent]. Review and finalize before changing status from Draft to Accepted.*

pkg/cli/audit_cache.go

@@ -13,7 +13,7 @@ import (
 
 type auditCacheSource string
 
-const auditSchemaVersion = 9
+const auditSchemaVersion = 10
 
 const (
 	auditCacheSourceFull auditCacheSource = "full"

pkg/cli/audit_cache_test.go

@@ -78,9 +78,10 @@ func TestLoadCachedAuditDataDropsHistoricalOutcomeSnapshot(t *testing.T) {
 func TestLoadCachedAuditDataRejectsOlderSchema(t *testing.T) {
 	t.Parallel()
 
-	// Earlier audits lack captured queue diagnostics and must be regenerated.
-	require.Equal(t, 9, auditSchemaVersion)
-	for _, schemaVersion := range []int{0, 5, 6, 7, 8} {
+	// Earlier audits lack captured queue diagnostics or unified-session model
+	// resolution findings and must be regenerated.
+	require.Equal(t, 10, auditSchemaVersion)
+	for _, schemaVersion := range []int{0, 5, 6, 7, 8, 9} {
 		runDir := t.TempDir()
 		run := WorkflowRun{DatabaseID: 42, Status: "completed", Conclusion: "success"}
 		data, err := json.Marshal(AuditData{

pkg/cli/audit_finding_codes.go

@@ -8,7 +8,7 @@ const auditDebugPlaybookURL = "https://github.com/github/gh-aw/blob/main/.github
 
 func auditFindingPlaybookURL(code AuditFindingCode) string {
 	switch code {
-	case AuditFindingModelUnavailable, AuditFindingModelAvailabilityUnknown:
+	case AuditFindingModelUnavailable, AuditFindingModelAvailabilityUnknown, AuditFindingModelResolutionFailed:
 		return auditDebugPlaybookURL + "#model-availability"
 	case AuditFindingSubagentModelNotObserved, AuditFindingSubagentModelUnavailable, AuditFindingSubagentFailed:
 		return "https://github.com/github/gh-aw/blob/main/.github/aw/debug-model-engine-misconfiguration.md"
@@ -27,6 +27,7 @@ const (
 	AuditFindingToolsNotAvailable        AuditFindingCode = "tools_not_available"
 	AuditFindingModelUnavailable         AuditFindingCode = "model_unavailable"
 	AuditFindingModelAvailabilityUnknown AuditFindingCode = "model_availability_unknown"
+	AuditFindingModelResolutionFailed    AuditFindingCode = "model_resolution_failed"
 	AuditFindingBlockedNetworkRequests   AuditFindingCode = "blocked_network_requests"
 	AuditFindingWorkflowSucceeded        AuditFindingCode = "workflow_succeeded"
 	AuditFindingSubagentModelNotObserved AuditFindingCode = "subagent_model_not_observed"

pkg/cli/audit_model_availability.go

@@ -31,6 +31,24 @@ type auditModelEndpoint struct {
 }
 
 func generateModelAvailabilityFindings(run WorkflowRun) []AuditFinding {
+	if run.LogsPath == "" {
+		return nil
+	}
+	evidence, err := readSessionModelResolutionEvidence(run.LogsPath)
+	if err != nil {
+		auditReportLog.Printf("Could not read model resolution evidence from unified session: %v", err)
+	}
+	if evidence != nil && len(evidence.Failures) > 0 {
+		// Recorded resolution failures are authoritative; the legacy Pi log match must
+		// neither duplicate them nor override their recorded cause.
+		return generateSessionModelResolutionFindings(run, evidence)
+	}
+	return generateLegacyPiModelAvailabilityFindings(run)
+}
+
+// generateLegacyPiModelAvailabilityFindings matches Pi's startup log line. It is a
+// compatibility fallback for runs whose unified session has no failed model.resolution.
+func generateLegacyPiModelAvailabilityFindings(run WorkflowRun) []AuditFinding {
 	if run.LogsPath == "" || run.Conclusion != "failure" {
 		return nil
 	}
@@ -50,42 +68,57 @@ func generateModelAvailabilityFindings(run WorkflowRun) []AuditFinding {
 		return nil
 	}
 	description := fmt.Sprintf("Pi startup rejected model %q on provider %q before inference. ", model, provider)
-	unknown := func(reason string) []AuditFinding {
+	reflection, reflectionErr := readModelAvailabilityReflection(run.LogsPath)
+	confirmed, detail := checkModelAvailability(reflection, reflectionErr, provider, model)
+	if !confirmed {
 		return []AuditFinding{{
 			Code:        AuditFindingModelAvailabilityUnknown,
 			Category:    "configuration",
 			Severity:    "high",
 			Title:       "Model Availability Unverified",
-			Description: description + reason + "; model absence cannot be confirmed from this run's inventory.",
+			Description: description + detail + "; model absence cannot be confirmed from this run's inventory.",
 			Impact:      "Inspect model discovery and authentication evidence before changing the model or runtime.",
 		}}
 	}
-	data, err := readModelAvailabilityArtifact(run.LogsPath, "sandbox/firewall/awf-reflect.json", 8<<20)
+	return []AuditFinding{{
+		Code:        AuditFindingModelUnavailable,
+		Category:    "configuration",
+		Severity:    "high",
+		Title:       "Model Unavailable",
+		Description: description + detail,
+		Impact:      "The agent could not start inference. This evidence does not establish global model retirement or why provider availability changed.",
+	}}
+}
+
+func readModelAvailabilityReflection(runDir string) ([]byte, error) {
+	data, err := readModelAvailabilityArtifact(runDir, "sandbox/firewall/awf-reflect.json", 8<<20)
 	if err != nil {
 		auditReportLog.Printf("Could not read same-run model reflection: %v", err)
-		return unknown("Same-run awf-reflect.json is unavailable or unsafe")
 	}
-	endpoint, models, reason := parseModelAvailabilityInventory(data, provider)
+	return data, err
+}
+
+// checkModelAvailability checks one exact model against the same run's provider inventory.
+// It returns confirmed=true with a confirmation detail only when the complete, unambiguous
+// inventory excludes the model; otherwise it returns the reason absence is unverified.
+func checkModelAvailability(reflection []byte, reflectionErr error, provider, model string) (bool, string) {
+	if reflectionErr != nil {
+		return false, "Same-run awf-reflect.json is unavailable or unsafe"
+	}
+	endpoint, models, reason := parseModelAvailabilityInventory(reflection, provider)
 	if reason != "" {
-		return unknown(reason)
+		return false, reason
 	}
 	requested := modelNameWithoutProvider(strings.ToLower(model))
 	requested, _, _ = strings.Cut(requested, "?")
 	if slices.Contains(models, requested) {
-		return unknown("The exact model is advertised in saved reflection, contradicting the startup rejection")
+		return false, "The exact model is advertised in saved reflection, contradicting the startup rejection"
 	}
-	description += fmt.Sprintf("Same-run awf-reflect.json confirms models_fetch_complete=true; the configured %s endpoint advertises %d model(s), but not %q.", provider, len(models), requested)
+	detail := fmt.Sprintf("Same-run awf-reflect.json confirms models_fetch_complete=true; the configured %s endpoint advertises %d model(s), but not %q.", provider, len(models), requested)
 	if candidates := modelAvailabilityCandidates(requested, models, endpoint); len(candidates) > 0 {
-		description += " Advertised same-family candidates (verify compatibility): " + strings.Join(candidates, "; ") + "."
+		detail += " Advertised same-family candidates (verify compatibility): " + strings.Join(candidates, "; ") + "."
 	}
-	return []AuditFinding{{
-		Code:        AuditFindingModelUnavailable,
-		Category:    "configuration",
-		Severity:    "high",
-		Title:       "Model Unavailable",
-		Description: description,
-		Impact:      "The agent could not start inference. This evidence does not establish global model retirement or why provider availability changed.",
-	}}
+	return true, detail
 }
 
 func parseModelAvailabilityInventory(data []byte, provider string) (*auditModelEndpoint, []string, string) {