diff --git a/.github/aw/debug-playbook.md b/.github/aw/debug-playbook.md index 2e01036f48a..fb550105de0 100644 --- a/.github/aw/debug-playbook.md +++ b/.github/aw/debug-playbook.md @@ -35,17 +35,18 @@ Use same-run artifacts, not baselines. | Codes | Check | Remedy | | --- | --- | --- | -| `workflow_failed` | First failed step; native error. | Route specific cause; conclusion is not root cause. | +| `workflow_failed` | First failed step; native error. | Route specific cause; conclusion isn't root cause. | | `workflow_timeout` | Timed-out step, budget, terminal result. | Remove blocking work/narrow scope; justify timeout changes. | | `multiple_errors` | First distinct error versus cascades. | Fix first cause, not repeats. | | `mcp_server_failures`, `tools_not_available` | MCP startup, registration, failed calls. | Fix tool/server/config/auth; no blind permission expansion. | -| `model_unavailable` | Complete same-run matching provider inventory excludes exact model ID. | Follow [model availability](#model-availability); verify advertised candidate/API, edit pin, recompile. | -| `model_availability_unknown` | Missing, incomplete, malformed, ambiguous, or contradictory reflection. | Inspect discovery/auth; do not declare removal or switch models on missing evidence. | -| `subagent_model_not_observed` | Spawn, child usage, evidence coverage. | Missing observations do not prove nonexecution. | +| `model_unavailable` | Complete same-run provider inventory excludes exact model. | [Model availability](#model-availability); verify candidate/API, edit pin, recompile. | +| `model_availability_unknown` | Missing/incomplete/malformed/ambiguous/contradictory reflection. | Inspect discovery/auth; do not declare removal/switch models without evidence. | +| `model_resolution_failed` | Failed `model.resolution` lacking same-role startup-stop evidence. | Advisory: verify listed pins; not a proven failure cause. | +| `subagent_model_not_observed` | Spawn, child usage, evidence coverage. | Missing observations don't prove nonexecution. | | `subagent_model_unavailable` | Child model, request, provider policy/catalog. | Verify child separately; follow [model/engine guide](debug-model-engine-misconfiguration.md). | -| `subagent_failed` | Child native error; parent dispatch. | Fix child cause; parent success does not prove child success. | +| `subagent_failed` | Child native error; parent dispatch. | Fix child cause; parent success doesn't prove child success. | | `blocked_network_requests` | Destination, caller, denial, necessity. | Fix URL/tool or authorize narrow destination; retain firewall. | -| `threat_detection_job_failed` | Detector startup/runtime versus verdict. | Repair detector; failure is not a clean verdict. | +| `threat_detection_job_failed` | Detector startup/runtime versus verdict. | Repair detector; failure isn't a clean verdict. | | `threat_detected` | Private structured verdict/artifact. | Stop unsafe effects; follow [security guidance](debug-security-review.md). Never publish payloads. | | `high_token_usage`, `many_iterations` | Tokens/cache/turns, repeated calls. | [Optimize](optimize-agentic-workflow.md) scope/context/loops before raising caps. | | `agentic_resource_heavy_for_domain`, `agentic_overkill_for_agentic` | Assessment, deterministic alternatives, judgment needed. | Script mechanical work; retain necessary LLM judgment. | @@ -62,10 +63,10 @@ Inventory: [Model availability](#model-availability). | Codes | Check | Remedy | | --- | --- | --- | -| `inference_access_error`, `inference_access_denied`, `authentication_failed` | Native 401/403, credential names, permissions/policy, proxy guards. | Fix access/config; budget/policy denial does not warrant credential rotation. | +| `inference_access_error`, `inference_access_denied`, `authentication_failed` | Native 401/403, credential names, permissions/policy, proxy guards. | Fix access/config; budget/policy denial doesn't warrant credential rotation. | | `mcp_policy_error` | Blocked MCP server; organization policy. | Use approved tool/authorized policy correction; no bypass. | -| `model_not_supported_error`, `model_endpoint_mismatch_error`, `model_endpoint_mismatch` | Model, API, tool protocol, same-run inventory/routes. | Follow [model/engine guide](debug-model-engine-misconfiguration.md); endpoint change does not establish availability. | -| `http_400_response_error` | Provider diagnostic, endpoint, feature/schema. | Fix unsupported request; 400 alone does not prove model removal. | +| `model_not_supported_error`, `model_endpoint_mismatch_error`, `model_endpoint_mismatch` | Model, API, tool protocol, same-run inventory/routes. | Follow [model/engine guide](debug-model-engine-misconfiguration.md); endpoint change doesn't establish availability. | +| `http_400_response_error` | Provider diagnostic, endpoint, feature/schema. | Fix unsupported request; 400 alone doesn't prove model removal. | | `unsupported_endpoint`, `invalid_selection`, `uncorroborated_endpoint`, `uncorroborated_selection`, `harness_selection_mismatch` | Selection versus harness model/API and metadata. | Align selection/API; retain verification; never invent evidence. | | `unsupported_effort` | Effort versus engine/model support. | Use supported effort/mapping; no silent discard. | | `missing_model_pricing_error`, `unknown_model_ai_credits` | Proxy pricing table and `missing_model_pricing_model_name`. | Correct supported pricing/version/config; never disable credit accounting. | @@ -75,10 +76,10 @@ Inventory: [Model availability](#model-availability). | `effective_tokens_limit_exceeded` | Effective tokens; prompt/turn growth. | Reduce context/iterations; retain guard. | | `permission_denied_limit_exceeded` | Denied calls; allowed tools. | Stop denied calls; use allowed tools, not larger denial budget. | | `model_policy_violation`, `content_filter` | Provider/proxy reason; endpoint mismatch separately. | Correct request within policy; no credential swap/bypass. | -| `awf_api_proxy_blocking_requests` | Specific budget/policy/filter guard. | Address guard; blocking alone does not establish authentication failure. | +| `awf_api_proxy_blocking_requests` | Specific budget/policy/filter guard. | Address guard; blocking alone doesn't establish authentication failure. | | `capi_quota_exceeded_error` | Native 429; exhausted retries. | Respect quota/reset; no retry storm. | | `capi_server_error`, `engine_outage`, `infrastructure_error` | Retry-exhausted native 5xx/infrastructure timeline. | Report transient cause; bounded backoff, no unapproved redispatch. | -| `agentic_engine_timeout` | Step/SDK budget versus post-result `watchdogFired=true`. | Fix genuine stall; post-result watchdog alone does not establish timeout. | +| `agentic_engine_timeout` | Step/SDK budget versus post-result `watchdogFired=true`. | Fix genuine stall; post-result watchdog alone doesn't establish timeout. | | `sandbox_runtime_crash` | Exit/signal, resources, runtime revision. | Fix crash/resources; retain redacted diagnostic/version. | | `goal_already_active` | Unfinished goal. | Update/complete existing goal; no duplicate. | | `shell_expansion_guard_rejected` | Private denied command; allowed syntax. | Use allowed tools without dangerous expansion; no obfuscation/bypass. | diff --git a/docs/adr/70653-use-unified-session-as-primary-source-for-audit-model-resolution-findings.md b/docs/adr/70653-use-unified-session-as-primary-source-for-audit-model-resolution-findings.md new file mode 100644 index 00000000000..51143dc5345 --- /dev/null +++ b/docs/adr/70653-use-unified-session-as-primary-source-for-audit-model-resolution-findings.md @@ -0,0 +1,52 @@ +# ADR-70653: Use the Unified Session as the Primary Source for Audit Model-Resolution Findings + +**Date**: 2026-10-20 +**Status**: Draft +**Deciders**: SivaKesava1 (PR author / actor); maintainer review pending + +--- + +### Context + +`gh aw audit` only diagnosed model-availability startup failures by matching a message in Pi's `agent-stdio.log`. Other harnesses (for example Copilot and Copilot SDK sub-agent runs 39499460077, 39499464895 and 39508150750) already write fail-closed `model.resolution` records to the unified session (`usage/aw_session.jsonl`) under the T-UAS-074 contract, but audit ignored them, so these failures went undiagnosed. A failed `model.resolution` record does not say whether the failing role was a declared sub-agent or an optional repository agent, so a record alone cannot prove that it caused the run to fail. Routed runs also produce router or classifier traffic and wrapper `state: "started"` markers that can look like agent activity without the native agent ever running. + +### Decision + +We will read failed `model.resolution` events from the unified session as the primary evidence for model-resolution findings, and keep the Pi `agent-stdio.log` match only as a fallback when the session has no failed resolution events. Resolution records are accepted only with the recorded provenance (`model_resolution` / `agent` / `agent/model-resolution.json`) and fields that satisfy the T-UAS-074 contract. Only the canonical `usage/aw_session.jsonl` is read. It must be a regular file (no symlinks) of at most 256 MiB with a `session.format` v1 header and exactly one `agent.execution` record; anything else is an invalid session. A role's failure is attributed as the startup cause only when the run failed, the single `agent.execution` exited non-zero, there is no native agent activity (`session.start`, `session.init`, `assistant.*`, `tool.*` from component `agent`), and the same run's `agent-stdio.log` affirmatively links that role to the stop: a harness `Refusing to start` line naming the role, or that role's model-specific resolution error. Roles without that evidence get a single low-severity advisory (`model_resolution_failed`). Failures are grouped by engine, provider, model and cause, and each group is classified only against the same run's `awf-reflect.json`. A run therefore gets at most one `model_unavailable` (high) finding and at most one `model_availability_unknown` finding. Recorded causes other than `model_unavailable` are reported as recorded and never confirmed as unavailability. + +### Alternatives Considered + +#### Alternative 1: Extend log-message matching to each harness + +Add per-engine log patterns, like the existing Pi `agent-stdio.log` match, for Copilot and Copilot SDK output. This would follow the existing pattern, but it ties audit to free-form log text that varies by harness and version. It also ignores the structured, contract-validated records the harnesses already write. It was not chosen because the unified session is the cross-engine source of truth for this data. + +#### Alternative 2: Report every failed resolution record as a high-severity failure cause + +Treat any failed `model.resolution` record as the reason the run failed, without checking startup evidence. This is simpler, but it misattributes failures. An optional repository agent can fail resolution while the main agent continues and the run succeeds, as in the run 39508154330 negative control. It was not chosen because the record cannot tell sub-agents and repository agents apart, and the producer was deliberately left unchanged. + +#### Alternative 3: Change the producer to record the role kind (sub-agent vs repository agent) + +Add a field to `model.resolution` so audit can attribute failures directly. This would make attribution more precise. It was not chosen in this PR because it changes the T-UAS-074 producer contract and is outside the `pkg/cli` scope. Historical runs would also still lack the field. [TODO: verify whether this is planned as a follow-up] + +### Consequences + +#### Positive +- Audit diagnoses model-resolution startup failures for all harnesses that write the unified session, not only Pi. +- Attribution requires run-level startup evidence plus per-role harness stop evidence and reports a low-severity advisory in every other case, which avoids false "model unavailable" root causes on successful or continuing runs. +- Classification uses only the same run's reflection inventory, and recorded causes such as `catalog_unavailable` are preserved, so audit does not invent unavailability. +- Strict provenance, schema and file-type validation (no symlinks, bounded total size and line length, v1 header, a single `agent.execution`) limits trust in artifact contents. + +#### Negative +- Audit now depends on the T-UAS-074 record schema and the provenance constants. Producer changes must be coordinated or records will be silently ignored. +- Per-role attribution depends on harness `agent-stdio.log` wording; a harness that stops without such a line yields only the advisory. +- There are two detection paths (session-first, legacy Pi log fallback) to maintain until the legacy path can be removed. +- A new finding code (`model_resolution_failed`) adds to the audit surface; the debug playbook has a row for it, but `audit.md` still describes these findings as Pi-only. +- The audit cache schema moves to 10 so reports cached before this analysis are regenerated. + +#### Neutral +- `checkModelAvailability` was extracted from the Pi path and is now shared by both paths, with its messages unchanged. +- Sanitized replay fixtures from real runs live under `pkg/cli/testdata/model_resolution_startup/`. Derived variants are labelled as derived in the test comments. + +--- + +*ADR created by [adr-writer agent]. Review and finalize before changing status from Draft to Accepted.* diff --git a/pkg/cli/audit_cache.go b/pkg/cli/audit_cache.go index 64f76db5d2d..210a1373d8d 100644 --- a/pkg/cli/audit_cache.go +++ b/pkg/cli/audit_cache.go @@ -13,7 +13,7 @@ import ( type auditCacheSource string -const auditSchemaVersion = 9 +const auditSchemaVersion = 10 const ( auditCacheSourceFull auditCacheSource = "full" diff --git a/pkg/cli/audit_cache_test.go b/pkg/cli/audit_cache_test.go index 2c0722d406d..03c66ae75b2 100644 --- a/pkg/cli/audit_cache_test.go +++ b/pkg/cli/audit_cache_test.go @@ -78,9 +78,10 @@ func TestLoadCachedAuditDataDropsHistoricalOutcomeSnapshot(t *testing.T) { func TestLoadCachedAuditDataRejectsOlderSchema(t *testing.T) { t.Parallel() - // Earlier audits lack captured queue diagnostics and must be regenerated. - require.Equal(t, 9, auditSchemaVersion) - for _, schemaVersion := range []int{0, 5, 6, 7, 8} { + // Earlier audits lack captured queue diagnostics or unified-session model + // resolution findings and must be regenerated. + require.Equal(t, 10, auditSchemaVersion) + for _, schemaVersion := range []int{0, 5, 6, 7, 8, 9} { runDir := t.TempDir() run := WorkflowRun{DatabaseID: 42, Status: "completed", Conclusion: "success"} data, err := json.Marshal(AuditData{ diff --git a/pkg/cli/audit_finding_codes.go b/pkg/cli/audit_finding_codes.go index 3f8b64a1a19..0c1a836d014 100644 --- a/pkg/cli/audit_finding_codes.go +++ b/pkg/cli/audit_finding_codes.go @@ -8,7 +8,7 @@ const auditDebugPlaybookURL = "https://github.com/github/gh-aw/blob/main/.github func auditFindingPlaybookURL(code AuditFindingCode) string { switch code { - case AuditFindingModelUnavailable, AuditFindingModelAvailabilityUnknown: + case AuditFindingModelUnavailable, AuditFindingModelAvailabilityUnknown, AuditFindingModelResolutionFailed: return auditDebugPlaybookURL + "#model-availability" case AuditFindingSubagentModelNotObserved, AuditFindingSubagentModelUnavailable, AuditFindingSubagentFailed: return "https://github.com/github/gh-aw/blob/main/.github/aw/debug-model-engine-misconfiguration.md" @@ -27,6 +27,7 @@ const ( AuditFindingToolsNotAvailable AuditFindingCode = "tools_not_available" AuditFindingModelUnavailable AuditFindingCode = "model_unavailable" AuditFindingModelAvailabilityUnknown AuditFindingCode = "model_availability_unknown" + AuditFindingModelResolutionFailed AuditFindingCode = "model_resolution_failed" AuditFindingBlockedNetworkRequests AuditFindingCode = "blocked_network_requests" AuditFindingWorkflowSucceeded AuditFindingCode = "workflow_succeeded" AuditFindingSubagentModelNotObserved AuditFindingCode = "subagent_model_not_observed" diff --git a/pkg/cli/audit_model_availability.go b/pkg/cli/audit_model_availability.go index 2bbfbebcbe3..7343b6a6e0a 100644 --- a/pkg/cli/audit_model_availability.go +++ b/pkg/cli/audit_model_availability.go @@ -31,6 +31,24 @@ type auditModelEndpoint struct { } func generateModelAvailabilityFindings(run WorkflowRun) []AuditFinding { + if run.LogsPath == "" { + return nil + } + evidence, err := readSessionModelResolutionEvidence(run.LogsPath) + if err != nil { + auditReportLog.Printf("Could not read model resolution evidence from unified session: %v", err) + } + if evidence != nil && len(evidence.Failures) > 0 { + // Recorded resolution failures are authoritative; the legacy Pi log match must + // neither duplicate them nor override their recorded cause. + return generateSessionModelResolutionFindings(run, evidence) + } + return generateLegacyPiModelAvailabilityFindings(run) +} + +// generateLegacyPiModelAvailabilityFindings matches Pi's startup log line. It is a +// compatibility fallback for runs whose unified session has no failed model.resolution. +func generateLegacyPiModelAvailabilityFindings(run WorkflowRun) []AuditFinding { if run.LogsPath == "" || run.Conclusion != "failure" { return nil } @@ -50,42 +68,57 @@ func generateModelAvailabilityFindings(run WorkflowRun) []AuditFinding { return nil } description := fmt.Sprintf("Pi startup rejected model %q on provider %q before inference. ", model, provider) - unknown := func(reason string) []AuditFinding { + reflection, reflectionErr := readModelAvailabilityReflection(run.LogsPath) + confirmed, detail := checkModelAvailability(reflection, reflectionErr, provider, model) + if !confirmed { return []AuditFinding{{ Code: AuditFindingModelAvailabilityUnknown, Category: "configuration", Severity: "high", Title: "Model Availability Unverified", - Description: description + reason + "; model absence cannot be confirmed from this run's inventory.", + Description: description + detail + "; model absence cannot be confirmed from this run's inventory.", Impact: "Inspect model discovery and authentication evidence before changing the model or runtime.", }} } - data, err := readModelAvailabilityArtifact(run.LogsPath, "sandbox/firewall/awf-reflect.json", 8<<20) + return []AuditFinding{{ + Code: AuditFindingModelUnavailable, + Category: "configuration", + Severity: "high", + Title: "Model Unavailable", + Description: description + detail, + Impact: "The agent could not start inference. This evidence does not establish global model retirement or why provider availability changed.", + }} +} + +func readModelAvailabilityReflection(runDir string) ([]byte, error) { + data, err := readModelAvailabilityArtifact(runDir, "sandbox/firewall/awf-reflect.json", 8<<20) if err != nil { auditReportLog.Printf("Could not read same-run model reflection: %v", err) - return unknown("Same-run awf-reflect.json is unavailable or unsafe") } - endpoint, models, reason := parseModelAvailabilityInventory(data, provider) + return data, err +} + +// checkModelAvailability checks one exact model against the same run's provider inventory. +// It returns confirmed=true with a confirmation detail only when the complete, unambiguous +// inventory excludes the model; otherwise it returns the reason absence is unverified. +func checkModelAvailability(reflection []byte, reflectionErr error, provider, model string) (bool, string) { + if reflectionErr != nil { + return false, "Same-run awf-reflect.json is unavailable or unsafe" + } + endpoint, models, reason := parseModelAvailabilityInventory(reflection, provider) if reason != "" { - return unknown(reason) + return false, reason } requested := modelNameWithoutProvider(strings.ToLower(model)) requested, _, _ = strings.Cut(requested, "?") if slices.Contains(models, requested) { - return unknown("The exact model is advertised in saved reflection, contradicting the startup rejection") + return false, "The exact model is advertised in saved reflection, contradicting the startup rejection" } - description += fmt.Sprintf("Same-run awf-reflect.json confirms models_fetch_complete=true; the configured %s endpoint advertises %d model(s), but not %q.", provider, len(models), requested) + detail := fmt.Sprintf("Same-run awf-reflect.json confirms models_fetch_complete=true; the configured %s endpoint advertises %d model(s), but not %q.", provider, len(models), requested) if candidates := modelAvailabilityCandidates(requested, models, endpoint); len(candidates) > 0 { - description += " Advertised same-family candidates (verify compatibility): " + strings.Join(candidates, "; ") + "." + detail += " Advertised same-family candidates (verify compatibility): " + strings.Join(candidates, "; ") + "." } - return []AuditFinding{{ - Code: AuditFindingModelUnavailable, - Category: "configuration", - Severity: "high", - Title: "Model Unavailable", - Description: description, - Impact: "The agent could not start inference. This evidence does not establish global model retirement or why provider availability changed.", - }} + return true, detail } func parseModelAvailabilityInventory(data []byte, provider string) (*auditModelEndpoint, []string, string) { diff --git a/pkg/cli/audit_model_resolution.go b/pkg/cli/audit_model_resolution.go new file mode 100644 index 00000000000..39d4d1f5ef9 --- /dev/null +++ b/pkg/cli/audit_model_resolution.go @@ -0,0 +1,451 @@ +package cli + +import ( + "bufio" + "bytes" + "encoding/json" + "errors" + "fmt" + "io" + "os" + "path/filepath" + "regexp" + "slices" + "strings" + + "github.com/github/gh-aw/pkg/constants" +) + +// Model resolution records are runner-written before the engine starts (T-UAS-074). +// Identifiers are validated before they are quoted into findings. +var ( + modelResolutionEnginePattern = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9_-]{0,63}$`) + modelResolutionRolePattern = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9 ._:@/-]{0,127}$`) + modelResolutionModelPattern = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9._/?=&+:@-]{0,199}$`) + modelResolutionProviderPattern = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9_-]{0,63}$`) + modelResolutionCausePattern = regexp.MustCompile(`^[a-z][a-z0-9_]{0,63}$`) +) + +const modelResolutionCauseModelUnavailable = "model_unavailable" + +// sessionModelResolutionFailure is one failed model.resolution record. The record +// does not persist whether the role was a declared sub-agent or an optional +// repository agent, so it cannot by itself establish why a run failed. +type sessionModelResolutionFailure struct { + Engine string + Role string + ConfiguredModel string + Provider string + Cause string +} + +type sessionAgentExecutionSummary struct { + Categories []string `json:"categories"` + ExitCode *int `json:"exitCode"` +} + +// maxModelResolutionSessionBytes bounds the unified session read for model resolution evidence. +const maxModelResolutionSessionBytes int64 = 256 << 20 + +// sessionModelResolutionEvidence holds failed resolution records plus the startup +// and continuation evidence needed to decide whether they explain the run outcome. +type sessionModelResolutionEvidence struct { + Failures []sessionModelResolutionFailure + // NativeAgentActivity is true when the unified session contains native agent + // session.start, session.init, assistant, or tool events. Wrapper state, router + // traffic, and harness-written agent records are not native activity. + NativeAgentActivity bool + AgentExecution *sessionAgentExecutionSummary + agentExecutionSeen bool +} + +type sessionModelResolutionPayload struct { + Engine string `json:"engine"` + Role string `json:"role"` + ConfiguredModel string `json:"configured_model"` + ResolvedModel json.RawMessage `json:"resolved_model"` + Provider *json.RawMessage `json:"provider"` + Status string `json:"status"` + Cause *string `json:"cause"` +} + +func readSessionModelResolutionEvidence(runDir string) (*sessionModelResolutionEvidence, error) { + if runDir == "" { + return nil, nil + } + const relative = "usage/aw_session.jsonl" + path := filepath.Join(runDir, constants.UsageArtifactName.String(), "aw_session.jsonl") + info, err := os.Lstat(path) + if errors.Is(err, os.ErrNotExist) { + return nil, nil + } + if err != nil { + return nil, fmt.Errorf("%s: %w", relative, err) + } + if err := validateSubagentSessionSource(path); err != nil { + return nil, fmt.Errorf("%s: %w", relative, err) + } + if !info.Mode().IsRegular() { + return nil, fmt.Errorf("%s: unified session is not a regular file", relative) + } + if info.Size() > maxModelResolutionSessionBytes { + return nil, fmt.Errorf("%s: unified session exceeds the %d-byte audit input limit", relative, maxModelResolutionSessionBytes) + } + evidence, err := parseSessionModelResolutionFile(path) + if err != nil { + return nil, fmt.Errorf("%s: %w", relative, err) + } + return evidence, nil +} + +func parseSessionModelResolutionFile(path string) (evidence *sessionModelResolutionEvidence, err error) { + file, err := os.Open(path) + if err != nil { + return nil, err + } + defer func() { + if closeErr := file.Close(); closeErr != nil && err == nil { + evidence, err = nil, closeErr + } + }() + return parseSessionModelResolutionEvidence(file, maxModelResolutionSessionBytes) +} + +// parseSessionModelResolutionEvidence reads at most maxBytes of the unified session. +// A larger session is rejected as invalid rather than partially attributed. +func parseSessionModelResolutionEvidence(input io.Reader, maxBytes int64) (*sessionModelResolutionEvidence, error) { + limited := &io.LimitedReader{R: input, N: maxBytes + 1} + reader := bufio.NewReader(limited) + evidence := &sessionModelResolutionEvidence{} + seen := make(map[sessionModelResolutionFailure]struct{}) + headerSeen := false + for lineNumber := 1; ; lineNumber++ { + line, oversized, readErr := readUnifiedSessionLine(reader) + if limited.N == 0 { + return nil, fmt.Errorf("unified session exceeds the %d-byte audit input limit", maxBytes) + } + if oversized { + if !headerSeen { + return nil, fmt.Errorf("oversized unified session header on line %d", lineNumber) + } + auditReportLog.Printf("Skipping oversized unified session record on line %d", lineNumber) + } else if line = bytes.TrimSpace(line); len(line) > 0 { + if !headerSeen { + var header unifiedDetectionEvent + if err := json.Unmarshal(line, &header); err != nil { + return nil, fmt.Errorf("invalid unified session header on line %d", lineNumber) + } + if err := validateUnifiedDetectionHeader(header); err != nil { + return nil, err + } + headerSeen = true + } else if err := evidence.observe(line, lineNumber, seen); err != nil { + return nil, err + } + } + if readErr != nil { + if errors.Is(readErr, io.EOF) { + break + } + return nil, fmt.Errorf("failed to read unified session: %w", readErr) + } + } + if !headerSeen { + return nil, errors.New("unified session is missing its leading session.format header") + } + return evidence, nil +} + +func (evidence *sessionModelResolutionEvidence) observe(line []byte, lineNumber int, seen map[sessionModelResolutionFailure]struct{}) error { + var event unifiedDetectionEvent + if err := json.Unmarshal(line, &event); err != nil { + auditReportLog.Printf("Skipping malformed unified session event on line %d", lineNumber) + return nil + } + provenance := event.Provenance + switch { + case event.Type == "model.resolution": + if provenance.Component != "model_resolution" || provenance.Phase != "agent" || provenance.Path != "agent/model-resolution.json" { + return nil + } + failure, failed, valid := decodeFailedModelResolution(event.Data) + if !valid { + auditReportLog.Printf("Ignoring invalid model.resolution on line %d", lineNumber) + return nil + } + if !failed { + return nil + } + if _, duplicate := seen[failure]; duplicate { + return nil + } + seen[failure] = struct{}{} + evidence.Failures = append(evidence.Failures, failure) + case event.Type == "agent.execution" && provenance.Component == "execution" && provenance.Phase == "agent": + // The unified-session contract allows one agent.execution summary; a duplicate + // would make attribution order-dependent, so the session is rejected. + if evidence.agentExecutionSeen { + return fmt.Errorf("unified session contains multiple agent.execution records (line %d)", lineNumber) + } + evidence.agentExecutionSeen = true + var summary sessionAgentExecutionSummary + if json.Unmarshal(event.Data, &summary) == nil && summary.ExitCode != nil { + evidence.AgentExecution = &summary + } + case provenance.Component == "agent" && provenance.Phase == "agent" && + (event.Type == "session.start" || event.Type == "session.init" || + strings.HasPrefix(event.Type, "assistant.") || strings.HasPrefix(event.Type, "tool.")): + evidence.NativeAgentActivity = true + } + return nil +} + +// decodeFailedModelResolution validates a model.resolution payload against the +// T-UAS-074 contract. It reports failed=true only for status "failed" records. +func decodeFailedModelResolution(data json.RawMessage) (sessionModelResolutionFailure, bool, bool) { + var payload sessionModelResolutionPayload + if !bytes.HasPrefix(bytes.TrimSpace(data), []byte("{")) || json.Unmarshal(data, &payload) != nil { + return sessionModelResolutionFailure{}, false, false + } + if payload.Status != "failed" { + return sessionModelResolutionFailure{}, false, true + } + if !modelResolutionEnginePattern.MatchString(payload.Engine) || + !modelResolutionRolePattern.MatchString(payload.Role) || + !modelResolutionModelPattern.MatchString(payload.ConfiguredModel) || + payload.Cause == nil || !modelResolutionCausePattern.MatchString(*payload.Cause) || + !bytes.Equal(bytes.TrimSpace(payload.ResolvedModel), []byte("null")) { + return sessionModelResolutionFailure{}, false, false + } + provider := "" + if payload.Provider != nil && !bytes.Equal(bytes.TrimSpace(*payload.Provider), []byte("null")) { + if json.Unmarshal(*payload.Provider, &provider) != nil || !modelResolutionProviderPattern.MatchString(provider) { + return sessionModelResolutionFailure{}, false, false + } + } + return sessionModelResolutionFailure{ + Engine: payload.Engine, + Role: payload.Role, + ConfiguredModel: payload.ConfiguredModel, + Provider: provider, + Cause: *payload.Cause, + }, true, true +} + +// modelResolutionFailureGroup lists the roles that failed on one engine/provider/model/cause. +type modelResolutionFailureGroup struct { + Engine string + Provider string + ConfiguredModel string + Cause string + Roles []string +} + +func groupModelResolutionFailures(failures []sessionModelResolutionFailure) []modelResolutionFailureGroup { + var groups []*modelResolutionFailureGroup + for _, failure := range failures { + var group *modelResolutionFailureGroup + for _, candidate := range groups { + if candidate.Engine == failure.Engine && candidate.Provider == failure.Provider && + candidate.ConfiguredModel == failure.ConfiguredModel && candidate.Cause == failure.Cause { + group = candidate + break + } + } + if group == nil { + group = &modelResolutionFailureGroup{ + Engine: failure.Engine, Provider: failure.Provider, + ConfiguredModel: failure.ConfiguredModel, Cause: failure.Cause, + } + groups = append(groups, group) + } + if !slices.Contains(group.Roles, failure.Role) { + group.Roles = append(group.Roles, failure.Role) + } + } + result := make([]modelResolutionFailureGroup, 0, len(groups)) + for _, group := range groups { + result = append(result, *group) + } + return result +} + +func (group modelResolutionFailureGroup) describe() string { + roles := make([]string, 0, len(group.Roles)) + for _, role := range group.Roles { + roles = append(roles, fmt.Sprintf("%q", role)) + } + provider := group.Provider + if provider == "" { + provider = "(not recorded)" + } + return fmt.Sprintf("engine %s role(s) %s configured model %q (provider %s, recorded cause %s)", + group.Engine, strings.Join(roles, ", "), group.ConfiguredModel, provider, group.Cause) +} + +func describeModelResolutionGroups(groups []modelResolutionFailureGroup) string { + descriptions := make([]string, 0, len(groups)) + for _, group := range groups { + descriptions = append(descriptions, group.describe()) + } + return strings.Join(descriptions, "; ") +} + +// modelResolutionContinuationReason explains why failed records are not attributed +// as the run's failure cause. It returns "" only when the run-level startup evidence +// permits a fatal startup stop: a failed run, a failed agent execution, and no native +// agent activity. Each role still needs its own stop evidence (modelResolutionStopEvidence). +func modelResolutionContinuationReason(run WorkflowRun, evidence *sessionModelResolutionEvidence) string { + switch { + case run.Conclusion != "failure": + conclusion := run.Conclusion + if conclusion == "" { + conclusion = "unknown" + } + return fmt.Sprintf("The run conclusion is %q, so these records are not treated as a failure cause.", conclusion) + case evidence.NativeAgentActivity: + return "The unified session contains native agent session activity (session.start, session.init, assistant, or tool events), so the run continued past model resolution and these records are not treated as its failure cause." + case evidence.AgentExecution == nil: + return "The unified session has no agent execution record, so a startup stop caused by these records cannot be established." + case *evidence.AgentExecution.ExitCode == 0: + return "The agent execution record exited with code 0, so these records are not treated as the failure cause." + default: + return "" + } +} + +const modelResolutionNoStopEvidenceReason = "agent-stdio.log contains no harness refusal or model-specific execution error for these roles, so the nonzero agent exit is not treated as caused by their resolution." + +// partitionModelResolutionStopEvidence splits failures into those with affirmative +// startup-stop evidence in the same run's agent-stdio.log and those without. +func partitionModelResolutionStopEvidence(runDir string, failures []sessionModelResolutionFailure) (stopped, unproven []sessionModelResolutionFailure) { + logData, err := readModelAvailabilityArtifact(runDir, "agent-stdio.log", 32<<20) + if err != nil && !os.IsNotExist(err) { + auditReportLog.Printf("Could not read model resolution startup log: %v", err) + } + for _, failure := range failures { + if err == nil && modelResolutionStopEvidence(logData, failure) { + stopped = append(stopped, failure) + } else { + unproven = append(unproven, failure) + } + } + return stopped, unproven +} + +// modelResolutionStopEvidence reports whether the startup log affirmatively links this +// failed role to the startup stop: a harness refusal naming the role (or, for the main +// role, its configured model), or the role's model-specific resolution error. Optional +// repository agents log only a warning and suppress model-not-supported errors, so +// neither form is produced for them. +func modelResolutionStopEvidence(logData []byte, failure sessionModelResolutionFailure) bool { + subagentRefusal := "sub-agent '" + failure.Role + "'" + mainRefusal := "'" + failure.ConfiguredModel + "'" + resolutionError := fmt.Sprintf("model resolution: engine=%s role=%s configured=%s failed cause=%s:", + failure.Engine, failure.Role, failure.ConfiguredModel, failure.Cause) + for line := range bytes.Lines(logData) { + text := string(line) + if strings.Contains(strings.ToLower(text), "refusing to start") && + (strings.Contains(text, subagentRefusal) || (failure.Role == "main" && strings.Contains(text, mainRefusal))) { + return true + } + if strings.Contains(text, resolutionError) && + (failure.Role == "main" || strings.Contains(text, "The requested model is not supported")) { + return true + } + } + return false +} + +func modelResolutionStartupEvidence(evidence *sessionModelResolutionEvidence) string { + var categories []string + for _, category := range evidence.AgentExecution.Categories { + if modelResolutionCausePattern.MatchString(category) && !slices.Contains(categories, category) { + categories = append(categories, category) + } + } + text := fmt.Sprintf(" Startup evidence: agent execution exited with code %d", *evidence.AgentExecution.ExitCode) + if len(categories) > 0 { + text += " (categories: " + strings.Join(categories, ", ") + ")" + } + text += ", the unified session contains no native agent session activity, and agent-stdio.log records a harness refusal or model-specific resolution error for each listed role." + return text +} + +func modelResolutionAdvisory(groups []modelResolutionFailureGroup, reason string) AuditFinding { + return AuditFinding{ + Code: AuditFindingModelResolutionFailed, + Category: "configuration", + Severity: "low", + Title: "Model Resolution Failure Recorded", + Description: "The unified session records failed model resolution: " + describeModelResolutionGroups(groups) + ". " + reason + " A failed record can come from an optional repository agent, whose unresolved model is only a warning.", + Impact: "Advisory: verify the listed model pins. This record alone does not establish that model resolution stopped the run.", + } +} + +func generateSessionModelResolutionFindings(run WorkflowRun, evidence *sessionModelResolutionEvidence) []AuditFinding { + if reason := modelResolutionContinuationReason(run, evidence); reason != "" { + return []AuditFinding{modelResolutionAdvisory(groupModelResolutionFailures(evidence.Failures), reason)} + } + + stopped, unproven := partitionModelResolutionStopEvidence(run.LogsPath, evidence.Failures) + var findings []AuditFinding + if len(stopped) > 0 { + confirmed, unverified := classifyModelResolutionGroups(run, groupModelResolutionFailures(stopped)) + startup := modelResolutionStartupEvidence(evidence) + if len(confirmed) > 0 { + findings = append(findings, AuditFinding{ + Code: AuditFindingModelUnavailable, + Category: "configuration", + Severity: "high", + Title: "Model Unavailable", + Description: "Startup model resolution failed before the agent started inference. " + strings.Join(confirmed, " ") + startup, + Impact: "The agent could not start inference. This evidence does not establish global model retirement or why provider availability changed.", + }) + } + if len(unverified) > 0 { + findings = append(findings, AuditFinding{ + Code: AuditFindingModelAvailabilityUnknown, + Category: "configuration", + Severity: "high", + Title: "Model Availability Unverified", + Description: "Startup model resolution failed before the agent started inference. " + strings.Join(unverified, " ") + startup, + Impact: "Inspect model discovery and authentication evidence before changing the model or runtime.", + }) + } + } + if len(unproven) > 0 { + findings = append(findings, modelResolutionAdvisory(groupModelResolutionFailures(unproven), modelResolutionNoStopEvidenceReason)) + } + return findings +} + +// classifyModelResolutionGroups checks each failure group against this run's own +// inventory, keeping the recorded cause when it is not model_unavailable. +func classifyModelResolutionGroups(run WorkflowRun, groups []modelResolutionFailureGroup) (confirmed, unverified []string) { + var reflection []byte + var reflectionErr error + reflectionRead := false + for _, group := range groups { + var detail string + isConfirmed := false + switch { + case group.Cause != modelResolutionCauseModelUnavailable: + detail = fmt.Sprintf("the harness recorded cause %s, not model_unavailable, so this run's inventory is not used to confirm absence", group.Cause) + case group.Provider == "": + detail = "no provider was recorded, so no configured endpoint inventory can be matched" + default: + if !reflectionRead { + reflection, reflectionErr = readModelAvailabilityReflection(run.LogsPath) + reflectionRead = true + } + isConfirmed, detail = checkModelAvailability(reflection, reflectionErr, group.Provider, group.ConfiguredModel) + } + if isConfirmed { + confirmed = append(confirmed, group.describe()+": "+detail) + } else { + unverified = append(unverified, group.describe()+": "+detail+"; model absence cannot be confirmed from this run's inventory.") + } + } + return confirmed, unverified +} diff --git a/pkg/cli/audit_model_resolution_test.go b/pkg/cli/audit_model_resolution_test.go new file mode 100644 index 00000000000..059b6711c70 --- /dev/null +++ b/pkg/cli/audit_model_resolution_test.go @@ -0,0 +1,525 @@ +//go:build !integration + +package cli + +import ( + "context" + "encoding/json" + "fmt" + "maps" + "os" + "path/filepath" + "runtime" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// modelResolutionReplayDir holds sanitized artifacts from real runs: +// - 39499460077 (Smoke Copilot Sub Agents) and 39508150750 (the same workflow on main +// after #70389): mini-whoami/gpt-5-mini and mini54-whoami/gpt-5.4-mini fail closed. +// - 39499464895 (Smoke Copilot SDK Routed Claude): mini-whoami/gpt-5.4-mini fails; +// no main record; router classifier traffic is present. +// - 39508154330 (Smoke Copilot SDK Routed, successful): native session continuation +// events only; it has no failed resolution record of its own. +// +// Each failing run keeps its session.format header, model.resolution, agent.execution, +// execution.result, and session.collection events, its wrapper agent_execution.json and +// error-summary-only agent-session.jsonl, its awf-reflect.json reduced to provider, +// configured state, model IDs, and routing endpoints, and its agent-stdio.log reduced to +// the harness model-resolution and sub-agent lines copied from the agent job log with +// timestamps removed. Native events from 39508154330 +// keep their envelopes with data reduced to identifiers. +var modelResolutionReplayDir = filepath.Join("testdata", "model_resolution_startup") + +func readModelResolutionReplayFile(t *testing.T, run, name string) string { + t.Helper() + data, err := os.ReadFile(filepath.Join(modelResolutionReplayDir, run, name)) + require.NoError(t, err) + return string(data) +} + +// writeModelResolutionReplayRun lays out a run's artifacts as audit downloads them. +// Overrides replace files by run-relative path; an empty override omits the file. +func writeModelResolutionReplayRun(t *testing.T, runDir, run string, overrides map[string]string) string { + t.Helper() + files := map[string]string{ + "usage/aw_session.jsonl": readModelResolutionReplayFile(t, run, "aw_session.jsonl"), + "sandbox/firewall/awf-reflect.json": readModelResolutionReplayFile(t, run, "awf-reflect.json"), + "agent-session.jsonl": readModelResolutionReplayFile(t, run, "agent-session.jsonl"), + "agent_execution.json": readModelResolutionReplayFile(t, run, "agent_execution.json"), + "agent-stdio.log": readModelResolutionReplayFile(t, run, "agent-stdio.log"), + } + maps.Copy(files, overrides) + for relative, content := range files { + if content == "" { + continue + } + writeModelAvailabilityEvidence(t, runDir, relative, content) + } + return runDir +} + +func modelAvailabilityFindingsByCode(findings []AuditFinding) map[AuditFindingCode][]AuditFinding { + byCode := make(map[AuditFindingCode][]AuditFinding) + for _, finding := range findings { + switch finding.Code { + case AuditFindingModelUnavailable, AuditFindingModelAvailabilityUnknown, AuditFindingModelResolutionFailed: + byCode[finding.Code] = append(byCode[finding.Code], finding) + } + } + return byCode +} + +func assertTwoRoleConfirmedFinding(t *testing.T, findings []AuditFinding) { + t.Helper() + require.Len(t, findings, 1) + finding := findings[0] + assert.Equal(t, AuditFindingModelUnavailable, finding.Code) + assert.EqualValues(t, "high", finding.Severity) + assert.Equal(t, "Model Unavailable", finding.Title) + for _, text := range []string{ + `engine copilot role(s) "mini-whoami" configured model "gpt-5-mini" (provider copilot, recorded cause model_unavailable)`, + `engine copilot role(s) "mini54-whoami" configured model "gpt-5.4-mini" (provider copilot, recorded cause model_unavailable)`, + "models_fetch_complete=true; the configured copilot endpoint advertises 36 model(s), but not \"gpt-5-mini\"", + "advertises 36 model(s), but not \"gpt-5.4-mini\"", + "categories: model_not_supported_error", + "no native agent session activity", + "agent-stdio.log records a harness refusal or model-specific resolution error for each listed role", + } { + assert.Contains(t, finding.Description, text) + } + assert.NotContains(t, finding.Description, "haiku-whoami", "successful records must not be flagged") + assert.NotContains(t, finding.Description, `role(s) "main"`, "successful records must not be flagged") +} + +func TestAuditModelAvailabilityCopilotSDKTwoDeclaredSubagents(t *testing.T) { + runDir := writeModelResolutionReplayRun(t, t.TempDir(), "39499460077", nil) + findings := generateModelAvailabilityFindings(WorkflowRun{Conclusion: "failure", LogsPath: runDir}) + assertTwoRoleConfirmedFinding(t, findings) + + data, _ := buildLocalAuditData(ProcessedRun{Run: WorkflowRun{DatabaseID: 39499460077, Conclusion: "failure", LogsPath: runDir}}, LogMetrics{}, nil) + byCode := modelAvailabilityFindingsByCode(data.KeyFindings) + assert.Len(t, byCode[AuditFindingModelUnavailable], 1) + assert.Empty(t, byCode[AuditFindingModelAvailabilityUnknown]) + assert.Empty(t, byCode[AuditFindingModelResolutionFailed]) +} + +func TestAuditModelAvailabilityRoutedCopilotSDKWithoutMainRecord(t *testing.T) { + runDir := writeModelResolutionReplayRun(t, t.TempDir(), "39499464895", nil) + session := readModelResolutionReplayFile(t, "39499464895", "aw_session.jsonl") + require.Contains(t, session, `"purpose":"routing_classification"`, "fixture must retain router classifier traffic") + require.NotContains(t, session, `"role":"main"`) + + findings := generateModelAvailabilityFindings(WorkflowRun{Conclusion: "failure", LogsPath: runDir}) + require.Len(t, findings, 1) + assert.Equal(t, AuditFindingModelUnavailable, findings[0].Code) + assert.Contains(t, findings[0].Description, `engine copilot role(s) "mini-whoami" configured model "gpt-5.4-mini" (provider copilot, recorded cause model_unavailable)`) + assert.Contains(t, findings[0].Description, `but not "gpt-5.4-mini"`) + assert.NotContains(t, findings[0].Description, "main") +} + +func TestAuditModelAvailabilityCopilotSubagentsAfter70389(t *testing.T) { + runDir := writeModelResolutionReplayRun(t, t.TempDir(), "39508150750", nil) + var wrapper struct { + State string `json:"state"` + } + require.NoError(t, json.Unmarshal([]byte(readModelResolutionReplayFile(t, "39508150750", "agent_execution.json")), &wrapper)) + require.Equal(t, "started", wrapper.State) + require.Contains(t, readModelResolutionReplayFile(t, "39508150750", "agent-session.jsonl"), `"type":"agent.execution"`) + + require.Contains(t, readModelResolutionReplayFile(t, "39508150750", "agent-stdio.log"), "copilot sub-agent 'mini54-whoami': ") + + findings := generateModelAvailabilityFindings(WorkflowRun{Conclusion: "failure", LogsPath: runDir}) + assertTwoRoleConfirmedFinding(t, findings) +} + +// TestAuditModelAvailabilityRepositoryAgentFailureOnSuccessIsAdvisory is a deliberate +// negative-control replay: the native continuation events of successful run 39508154330 +// combined with an unchanged failed record from run 39499460077. Run 39508154330 did not +// itself record this failure. +func TestAuditModelAvailabilityRepositoryAgentFailureOnSuccessIsAdvisory(t *testing.T) { + var failedRecord string + for line := range strings.SplitSeq(readModelResolutionReplayFile(t, "39499460077", "aw_session.jsonl"), "\n") { + if strings.Contains(line, `"role":"mini-whoami"`) { + failedRecord = line + } + } + require.NotEmpty(t, failedRecord) + continuation := strings.TrimSuffix(readModelResolutionReplayFile(t, "39508154330", "aw_session.jsonl"), "\n") + require.Contains(t, continuation, `"type":"session.start"`) + session := continuation + "\n" + failedRecord + "\n" + + for _, conclusion := range []string{"success", "failure"} { + t.Run(conclusion, func(t *testing.T) { + runDir := writeModelResolutionReplayRun(t, t.TempDir(), "39499460077", map[string]string{ + "usage/aw_session.jsonl": session, + // A continued run that later fails for an unrelated reason. + "agent-stdio.log": "Error: unrelated tool failure after the agent started\n", + }) + findings := generateModelAvailabilityFindings(WorkflowRun{Conclusion: conclusion, LogsPath: runDir}) + require.Len(t, findings, 1) + assert.Equal(t, AuditFindingModelResolutionFailed, findings[0].Code) + assert.EqualValues(t, "low", findings[0].Severity) + assert.Contains(t, findings[0].Description, `role(s) "mini-whoami" configured model "gpt-5-mini"`) + assert.Contains(t, findings[0].Description, "not treated as") + assert.Contains(t, findings[0].Description, "optional repository agent") + + data, _ := buildLocalAuditData(ProcessedRun{Run: WorkflowRun{Conclusion: conclusion, Status: "completed", LogsPath: runDir}}, LogMetrics{}, nil) + byCode := modelAvailabilityFindingsByCode(data.KeyFindings) + assert.Empty(t, byCode[AuditFindingModelUnavailable]) + assert.Empty(t, byCode[AuditFindingModelAvailabilityUnknown]) + assert.Len(t, byCode[AuditFindingModelResolutionFailed], 1) + for _, recommendation := range data.Recommendations { + assert.NotContains(t, recommendation.Action, "Replace the unavailable model pin") + } + }) + } +} + +func TestAuditModelAvailabilitySessionMissingReflection(t *testing.T) { + // Derived from run 39499460077 by omitting its awf-reflect.json. + runDir := writeModelResolutionReplayRun(t, t.TempDir(), "39499460077", map[string]string{"sandbox/firewall/awf-reflect.json": ""}) + findings := generateModelAvailabilityFindings(WorkflowRun{Conclusion: "failure", LogsPath: runDir}) + require.Len(t, findings, 1) + assert.Equal(t, AuditFindingModelAvailabilityUnknown, findings[0].Code) + assert.Equal(t, "Model Availability Unverified", findings[0].Title) + assert.Contains(t, findings[0].Description, `role(s) "mini-whoami" configured model "gpt-5-mini"`) + assert.Contains(t, findings[0].Description, `role(s) "mini54-whoami" configured model "gpt-5.4-mini"`) + assert.Contains(t, findings[0].Description, "awf-reflect.json is unavailable") +} + +func TestAuditModelAvailabilitySessionIncompleteReflection(t *testing.T) { + // Derived from run 39499460077 by changing only models_fetch_complete to false. + reflection := readModelResolutionReplayFile(t, "39499460077", "awf-reflect.json") + require.Contains(t, reflection, `"models_fetch_complete": true`) + reflection = strings.Replace(reflection, `"models_fetch_complete": true`, `"models_fetch_complete": false`, 1) + runDir := writeModelResolutionReplayRun(t, t.TempDir(), "39499460077", map[string]string{"sandbox/firewall/awf-reflect.json": reflection}) + findings := generateModelAvailabilityFindings(WorkflowRun{Conclusion: "failure", LogsPath: runDir}) + require.Len(t, findings, 1) + assert.Equal(t, AuditFindingModelAvailabilityUnknown, findings[0].Code) + assert.Equal(t, "Model Availability Unverified", findings[0].Title) + assert.Contains(t, findings[0].Description, "Model discovery did not complete") +} + +func TestAuditModelAvailabilityGroupsRolesSharingModel(t *testing.T) { + // Derived from run 39499460077: the recorded mini54-whoami envelope and fields are + // kept, with its configured model changed to the model mini-whoami failed on. + session := readModelResolutionReplayFile(t, "39499460077", "aw_session.jsonl") + original := `"role":"mini54-whoami","configured_model":"gpt-5.4-mini"` + require.Contains(t, session, original) + session = strings.Replace(session, original, `"role":"mini54-whoami","configured_model":"gpt-5-mini"`, 1) + runDir := writeModelResolutionReplayRun(t, t.TempDir(), "39499460077", map[string]string{"usage/aw_session.jsonl": session}) + + findings := generateModelAvailabilityFindings(WorkflowRun{Conclusion: "failure", LogsPath: runDir}) + require.Len(t, findings, 1) + assert.Equal(t, AuditFindingModelUnavailable, findings[0].Code) + assert.Contains(t, findings[0].Description, `engine copilot role(s) "mini-whoami", "mini54-whoami" configured model "gpt-5-mini"`) + assert.Equal(t, 1, strings.Count(findings[0].Description, `configured model "gpt-5-mini"`)) + assert.NotContains(t, findings[0].Description, "gpt-5.4-mini") +} + +func TestAuditModelAvailabilityPreservesOtherResolutionCauses(t *testing.T) { + // Derived from run 39499464895: the recorded mini-whoami failure with its cause + // changed to the contract-valid catalog_unavailable. A legacy Pi startup line is + // appended to the run's startup log and must not override the recorded cause. + session := readModelResolutionReplayFile(t, "39499464895", "aw_session.jsonl") + require.Contains(t, session, `"status":"failed","cause":"model_unavailable"`) + session = strings.Replace(session, `"status":"failed","cause":"model_unavailable"`, `"status":"failed","cause":"catalog_unavailable"`, 1) + runDir := writeModelResolutionReplayRun(t, t.TempDir(), "39499464895", map[string]string{ + "usage/aw_session.jsonl": session, + "agent-stdio.log": readModelResolutionReplayFile(t, "39499464895", "agent-stdio.log") + + `[gh-aw/pi-models-json] fatal: Pi model "gpt-5.4-mini" is not advertised by the configured copilot proxy endpoint; choose a model listed by AWF /reflect` + "\n", + }) + + findings := generateModelAvailabilityFindings(WorkflowRun{Conclusion: "failure", LogsPath: runDir}) + require.Len(t, findings, 1) + assert.Equal(t, AuditFindingModelAvailabilityUnknown, findings[0].Code) + assert.Contains(t, findings[0].Description, "recorded cause catalog_unavailable") + assert.Contains(t, findings[0].Description, "not model_unavailable") + assert.NotContains(t, findings[0].Description, "Pi startup") + assert.NotContains(t, findings[0].Description, "recorded cause model_unavailable") +} + +func TestAuditModelAvailabilityLegacyFallbackRequiresNoFailedResolution(t *testing.T) { + // A session whose only resolution record succeeded leaves the Pi log as the evidence. + session := `{"type":"session.format","data":{"version":1},"provenance":{"component":"collector","phase":"conclusion","path":"usage/aw_session.jsonl","index":0}} +{"type":"model.resolution","data":{"engine":"pi","role":"main","configured_model":"claude-opus-5","resolved_model":"claude-opus-5","provider":"copilot","status":"exact","cause":null},"provenance":{"component":"model_resolution","phase":"agent","path":"agent/model-resolution.json","index":0}} +` + runDir := t.TempDir() + writeModelAvailabilityEvidence(t, runDir, "usage/aw_session.jsonl", session) + writeModelAvailabilityEvidence(t, runDir, "agent-stdio.log", modelUnavailableStartupLog) + writeModelAvailabilityEvidence(t, runDir, "sandbox/firewall/awf-reflect.json", + `{"models_fetch_complete":true,"endpoints":[{"provider":"copilot","configured":true,"models":["claude-opus-5.5"]}]}`) + findings := generateModelAvailabilityFindings(WorkflowRun{Conclusion: "failure", LogsPath: runDir}) + require.Len(t, findings, 1) + assert.Equal(t, AuditFindingModelUnavailable, findings[0].Code) + assert.Contains(t, findings[0].Description, "Pi startup rejected model") +} + +func TestAuditModelAvailabilityIgnoresInvalidResolutionSessions(t *testing.T) { + failed := `{"type":"model.resolution","data":{"engine":"copilot","role":"mini-whoami","configured_model":"gpt-5-mini","resolved_model":null,"provider":"copilot","status":"failed","cause":"model_unavailable"},"provenance":{"component":"model_resolution","phase":"agent","path":"agent/model-resolution.json","index":2}}` + for name, session := range map[string]string{ + "missing header": failed + "\n", + "unsupported format": `{"type":"session.format","data":{"version":2},"provenance":{"component":"collector"}}` + "\n" + failed + "\n", + "wrong provenance": `{"type":"session.format","data":{"version":1},"provenance":{"component":"collector"}}` + "\n" + strings.Replace(failed, `"component":"model_resolution"`, `"component":"agent"`, 1) + "\n", + "unsafe role": `{"type":"session.format","data":{"version":1},"provenance":{"component":"collector"}}` + "\n" + strings.Replace(failed, `"mini-whoami"`, `"mini\nsensitive-payload"`, 1) + "\n", + "resolved model": `{"type":"session.format","data":{"version":1},"provenance":{"component":"collector"}}` + "\n" + strings.Replace(failed, `"resolved_model":null`, `"resolved_model":"gpt-5-mini"`, 1) + "\n", + } { + t.Run(name, func(t *testing.T) { + runDir := t.TempDir() + writeModelAvailabilityEvidence(t, runDir, "usage/aw_session.jsonl", session) + assert.Empty(t, generateModelAvailabilityFindings(WorkflowRun{Conclusion: "failure", LogsPath: runDir})) + }) + } +} + +// TestAuditModelAvailabilityRepositoryAgentWarningThenUnrelatedStartupFailure is derived +// from run 39499464895: its recorded mini-whoami failure is kept, but the startup log is +// replaced with the harness's optional repository-agent warning followed by an unrelated +// provider-listener readiness failure, and the agent error categories are cleared because +// the harness suppresses model-not-supported errors for repository agents. +func TestAuditModelAvailabilityRepositoryAgentWarningThenUnrelatedStartupFailure(t *testing.T) { + session := readModelResolutionReplayFile(t, "39499464895", "aw_session.jsonl") + require.Contains(t, session, `"categories":["model_not_supported_error"]`) + session = strings.Replace(session, `"categories":["model_not_supported_error"]`, `"categories":[]`, 1) + runDir := writeModelResolutionReplayRun(t, t.TempDir(), "39499464895", map[string]string{ + "usage/aw_session.jsonl": session, + "agent-stdio.log": "[copilot-harness] warning: repository agent 'mini-whoami' (.github/agents/mini-whoami.agent.md) declares model 'gpt-5.4-mini', which could not be resolved (cause: model_unavailable); the SDK session gives it the session model.\n" + + "[copilot-harness] copilot-sdk driver mode: provider listener readiness probe failed for http://api-proxy:10002: connect ECONNREFUSED\n", + }) + + findings := generateModelAvailabilityFindings(WorkflowRun{Conclusion: "failure", LogsPath: runDir}) + require.Len(t, findings, 1) + assert.Equal(t, AuditFindingModelResolutionFailed, findings[0].Code) + assert.EqualValues(t, "low", findings[0].Severity) + assert.Contains(t, findings[0].Description, `role(s) "mini-whoami" configured model "gpt-5.4-mini"`) + assert.Contains(t, findings[0].Description, "no harness refusal or model-specific execution error") + + data, _ := buildLocalAuditData(ProcessedRun{Run: WorkflowRun{Conclusion: "failure", Status: "completed", LogsPath: runDir}}, LogMetrics{}, nil) + for _, recommendation := range data.Recommendations { + assert.NotContains(t, recommendation.Action, "Replace the unavailable model pin") + } +} + +func TestAuditModelAvailabilityOnlyRolesWithStopEvidenceAreAttributed(t *testing.T) { + // Derived from run 39499460077 by removing mini54-whoami's harness refusal line. + var lines []string + for line := range strings.SplitSeq(readModelResolutionReplayFile(t, "39499460077", "agent-stdio.log"), "\n") { + if !strings.Contains(line, "sub-agent 'mini54-whoami': ") { + lines = append(lines, line) + } + } + runDir := writeModelResolutionReplayRun(t, t.TempDir(), "39499460077", map[string]string{"agent-stdio.log": strings.Join(lines, "\n")}) + + byCode := modelAvailabilityFindingsByCode(generateModelAvailabilityFindings(WorkflowRun{Conclusion: "failure", LogsPath: runDir})) + require.Len(t, byCode[AuditFindingModelUnavailable], 1) + assert.Contains(t, byCode[AuditFindingModelUnavailable][0].Description, `role(s) "mini-whoami"`) + assert.NotContains(t, byCode[AuditFindingModelUnavailable][0].Description, "mini54-whoami") + require.Len(t, byCode[AuditFindingModelResolutionFailed], 1) + assert.Contains(t, byCode[AuditFindingModelResolutionFailed][0].Description, `role(s) "mini54-whoami"`) + assert.NotContains(t, byCode[AuditFindingModelResolutionFailed][0].Description, `"mini-whoami"`) +} + +func TestAuditModelAvailabilitySessionInitCountsAsNativeActivity(t *testing.T) { + // Derived from run 39499460077 by adding an initialization-only native trace, as + // Claude and Codex emit before failing ahead of their first assistant or tool event. + session := readModelResolutionReplayFile(t, "39499460077", "aw_session.jsonl") + + `{"type":"session.init","data":{"model":"gpt-5.3-codex"},"provenance":{"component":"agent","phase":"agent","path":"agent-session.jsonl","index":0}}` + "\n" + runDir := writeModelResolutionReplayRun(t, t.TempDir(), "39499460077", map[string]string{"usage/aw_session.jsonl": session}) + + findings := generateModelAvailabilityFindings(WorkflowRun{Conclusion: "failure", LogsPath: runDir}) + require.Len(t, findings, 1) + assert.Equal(t, AuditFindingModelResolutionFailed, findings[0].Code) + assert.Contains(t, findings[0].Description, "session.init") +} + +func TestAuditModelAvailabilityAcceptsMultiParameterModelSelectors(t *testing.T) { + // Derived from run 39499460077 with mini-whoami's configured model replaced by a + // supported multi-parameter selector in both the record and the startup log. + const model = "openai/gpt-6.1-sol?effort=high&temperature=0.2" + session := strings.Replace(readModelResolutionReplayFile(t, "39499460077", "aw_session.jsonl"), + `"role":"mini-whoami","configured_model":"gpt-5-mini"`, `"role":"mini-whoami","configured_model":"`+model+`"`, 1) + stdio := readModelResolutionReplayFile(t, "39499460077", "agent-stdio.log") + stdio = strings.ReplaceAll(stdio, "gpt-5-mini", model) + runDir := writeModelResolutionReplayRun(t, t.TempDir(), "39499460077", map[string]string{ + "usage/aw_session.jsonl": session, + "agent-stdio.log": stdio, + }) + + evidence, err := readSessionModelResolutionEvidence(runDir) + require.NoError(t, err) + require.Len(t, evidence.Failures, 2) + assert.Equal(t, model, evidence.Failures[0].ConfiguredModel) + + findings := generateModelAvailabilityFindings(WorkflowRun{Conclusion: "failure", LogsPath: runDir}) + var descriptions []string + for _, finding := range findings { + assert.NotEqual(t, AuditFindingModelResolutionFailed, finding.Code, "both roles have harness refusals") + descriptions = append(descriptions, finding.Description) + } + assert.Contains(t, strings.Join(descriptions, "\n"), `role(s) "mini-whoami" configured model "`+model+`"`) +} + +func TestAuditModelAvailabilityRejectsDuplicateAgentExecution(t *testing.T) { + // Derived from run 39499460077 by appending a replayed exit-0 agent.execution record. + session := readModelResolutionReplayFile(t, "39499460077", "aw_session.jsonl") + + `{"type":"agent.execution","data":{"categories":[],"errorCodes":[],"errorTypes":[],"exitCode":0},"provenance":{"component":"execution","phase":"agent","path":"agent-errors.jsonl","index":1}}` + "\n" + runDir := writeModelResolutionReplayRun(t, t.TempDir(), "39499460077", map[string]string{"usage/aw_session.jsonl": session}) + + _, err := readSessionModelResolutionEvidence(runDir) + require.ErrorContains(t, err, "multiple agent.execution records") + assert.Empty(t, generateModelAvailabilityFindings(WorkflowRun{Conclusion: "failure", LogsPath: runDir})) +} + +func TestAuditModelAvailabilityBoundsUnifiedSessionSize(t *testing.T) { + session := readModelResolutionReplayFile(t, "39499460077", "aw_session.jsonl") + evidence, err := parseSessionModelResolutionEvidence(strings.NewReader(session), int64(len(session))) + require.NoError(t, err) + assert.Len(t, evidence.Failures, 2) + + _, err = parseSessionModelResolutionEvidence(strings.NewReader(session), int64(len(session)-1)) + require.ErrorContains(t, err, "exceeds the") +} + +func TestAuditModelAvailabilityReadsOnlyCanonicalSessionPath(t *testing.T) { + runDir := t.TempDir() + writeModelAvailabilityEvidence(t, runDir, "aw_session.jsonl", readModelResolutionReplayFile(t, "39499460077", "aw_session.jsonl")) + evidence, err := readSessionModelResolutionEvidence(runDir) + require.NoError(t, err) + assert.Nil(t, evidence, "a root-level aw_session.jsonl is not the canonical usage artifact") +} + +func installModelResolutionAuditFakeGH(t *testing.T, runID int64) { + t.Helper() + if runtime.GOOS == "windows" { + t.Skip("shell script fake gh is not supported on Windows") + } + dir := t.TempDir() + script := fmt.Sprintf(`#!/bin/sh +case "$*" in + *actions/runs/%[1]d/artifacts*) echo '[]' ;; + *actions/runs/%[1]d*) + echo '{"databaseId":%[1]d,"workflowName":"Smoke Copilot Sub Agents","workflowPath":".github/workflows/smoke-copilot-sub-agents.lock.yml","status":"completed","conclusion":"failure"}' ;; + *actions/workflows/*/runs*) echo '[]' ;; + "run download "*) echo 'no valid artifacts found' >&2; exit 1 ;; + "run view "*) echo '{"jobs":[]}' ;; + *) echo "unexpected gh invocation: $*" >&2; exit 1 ;; +esac +`, runID) + require.NoError(t, os.WriteFile(filepath.Join(dir, "gh"), []byte(script), 0o755)) + t.Setenv("PATH", dir+string(os.PathListSeparator)+os.Getenv("PATH")) + t.Setenv("GH_TOKEN", "") + t.Setenv("GITHUB_TOKEN", "") +} + +// TestAuditModelAvailabilityRun39499460077EndToEnd runs the audit command pipeline +// against run 39499460077's replayed artifacts with a fake gh, then again from cache. +func TestAuditModelAvailabilityRun39499460077EndToEnd(t *testing.T) { + const runID int64 = 39499460077 + installModelResolutionAuditFakeGH(t, runID) + outputDir := t.TempDir() + runDir := writeModelResolutionReplayRun(t, filepath.Join(outputDir, fmt.Sprintf("run-%d", runID)), "39499460077", nil) + + audit := func() AuditData { + t.Helper() + stdout, _ := captureOutput(t, func() error { + skipped, err := AuditWorkflowRun(context.Background(), runID, AuditOptions{ + Hostname: "github.com", Owner: "github", Repo: "gh-aw", OutputDir: outputDir, JSONOutput: true, + }) + assert.False(t, skipped) + require.NoError(t, err) + return nil + }) + var data AuditData + require.NoError(t, json.Unmarshal([]byte(stdout), &data), stdout) + return data + } + + fresh := audit() + assert.Equal(t, auditCacheSourceFull, fresh.CacheSource) + assertTwoRoleConfirmedFinding(t, modelAvailabilityFindingsByCode(fresh.KeyFindings)[AuditFindingModelUnavailable]) + assert.Empty(t, modelAvailabilityFindingsByCode(fresh.KeyFindings)[AuditFindingModelAvailabilityUnknown]) + var recommended bool + for _, recommendation := range fresh.Recommendations { + recommended = recommended || strings.Contains(recommendation.Action, "Replace the unavailable model pin") + } + assert.True(t, recommended) + assert.FileExists(t, filepath.Join(runDir, runSummaryFileName)) + assert.FileExists(t, filepath.Join(runDir, "audit.json")) + + cached := audit() + assert.Equal(t, fresh.KeyFindings, cached.KeyFindings, "cached audit must preserve the finding") + + output := captureStderrForGuardPolicyReportTest(func() { renderConsole(cached, runDir) }) + assert.Contains(t, output, "Model Unavailable") + assert.Contains(t, output, "mini54-whoami") + assert.Contains(t, output, "debug-playbook.md#model-availability") +} + +// TestAuditModelAvailabilityUpgradeRegeneratesOlderCachedAudit seeds run 39499460077's +// output directory with a schema 9 audit.json written before unified-session model +// resolution analysis existed. The upgraded audit must regenerate it, while a report +// at the current schema is still reused as-is. +func TestAuditModelAvailabilityUpgradeRegeneratesOlderCachedAudit(t *testing.T) { + const runID int64 = 39499460077 + installModelResolutionAuditFakeGH(t, runID) + + for _, tc := range []struct { + name string + schemaVersion int + regenerated bool + }{ + {name: "schema 9 report", schemaVersion: 9, regenerated: true}, + {name: "current schema report", schemaVersion: auditSchemaVersion, regenerated: false}, + } { + t.Run(tc.name, func(t *testing.T) { + outputDir := t.TempDir() + runDir := writeModelResolutionReplayRun(t, filepath.Join(outputDir, fmt.Sprintf("run-%d", runID)), "39499460077", nil) + audit := func() AuditData { + t.Helper() + stdout, _ := captureOutput(t, func() error { + _, err := AuditWorkflowRun(context.Background(), runID, AuditOptions{ + Hostname: "github.com", Owner: "github", Repo: "gh-aw", OutputDir: outputDir, JSONOutput: true, + }) + require.NoError(t, err) + return nil + }) + var data AuditData + require.NoError(t, json.Unmarshal([]byte(stdout), &data), stdout) + return data + } + + // Build a matching report, then replace it with an older-shaped one that has + // no model resolution findings. + old := audit() + old.SchemaVersion = tc.schemaVersion + old.KeyFindings = []AuditFinding{{Title: "stale cached finding"}} + old.Recommendations = nil + encoded, err := json.Marshal(old) + require.NoError(t, err) + require.NoError(t, os.WriteFile(auditPath(runDir), encoded, 0o600)) + + upgraded := audit() + byCode := modelAvailabilityFindingsByCode(upgraded.KeyFindings) + if tc.regenerated { + assertTwoRoleConfirmedFinding(t, byCode[AuditFindingModelUnavailable]) + for _, finding := range upgraded.KeyFindings { + assert.NotEqual(t, "stale cached finding", finding.Title) + } + written, err := os.ReadFile(auditPath(runDir)) + require.NoError(t, err) + var cached AuditData + require.NoError(t, json.Unmarshal(written, &cached)) + assert.Equal(t, auditSchemaVersion, cached.SchemaVersion) + assert.Len(t, modelAvailabilityFindingsByCode(cached.KeyFindings)[AuditFindingModelUnavailable], 1) + } else { + assert.Empty(t, byCode[AuditFindingModelUnavailable]) + require.Len(t, upgraded.KeyFindings, 1) + assert.Equal(t, "stale cached finding", upgraded.KeyFindings[0].Title) + } + }) + } +} diff --git a/pkg/cli/audit_threat_detection_session.go b/pkg/cli/audit_threat_detection_session.go index e2c4c03115c..e1bc7efb86f 100644 --- a/pkg/cli/audit_threat_detection_session.go +++ b/pkg/cli/audit_threat_detection_session.go @@ -33,6 +33,7 @@ type unifiedDetectionEvent struct { Provenance struct { Component string `json:"component"` Phase string `json:"phase"` + Path string `json:"path"` } `json:"provenance"` } diff --git a/pkg/cli/testdata/model_resolution_startup/39499460077/agent-session.jsonl b/pkg/cli/testdata/model_resolution_startup/39499460077/agent-session.jsonl new file mode 100644 index 00000000000..130fc6c5ecb --- /dev/null +++ b/pkg/cli/testdata/model_resolution_startup/39499460077/agent-session.jsonl @@ -0,0 +1 @@ +{"type":"agent.execution","data":{"categories":["model_not_supported_error"],"errorCodes":[],"errorTypes":[],"exitCode":1}} diff --git a/pkg/cli/testdata/model_resolution_startup/39499460077/agent-stdio.log b/pkg/cli/testdata/model_resolution_startup/39499460077/agent-stdio.log new file mode 100644 index 00000000000..e4a6fb272f1 --- /dev/null +++ b/pkg/cli/testdata/model_resolution_startup/39499460077/agent-stdio.log @@ -0,0 +1,10 @@ +[copilot-harness] model resolution: engine=copilot role=main configured=gpt-5.3-codex resolved=gpt-5.3-codex provider=(unscoped) status=exact +[copilot-harness] model resolution: engine=copilot role=haiku-whoami configured=claude-haiku-4.5 resolved=claude-haiku-4.5 provider=copilot status=exact +[copilot-harness] copilot sub-agent 'haiku-whoami': model 'claude-haiku-4.5' -> 'copilot-completions/claude-haiku-4.5' +[copilot-harness] copilot sub-agent 'mini-whoami' model alias resolution: known alias 'gpt-5-mini' did not resolve against catalog (provider=copilot) +[copilot-harness] copilot sub-agent 'mini-whoami' model alias resolution: retrying awf-reflect model-catalog fetch once before failing for 'gpt-5-mini' +[copilot-harness] copilot sub-agent 'mini-whoami' model alias resolution: known alias 'gpt-5-mini' did not resolve against catalog (provider=copilot) +[copilot-harness] model resolution: engine=copilot role=mini-whoami configured=gpt-5-mini failed cause=model_unavailable: The requested model is not supported: 'gpt-5-mini' is not available in the workflow's live model catalog. For Copilot, availability also depends on the agentic-workflows integration, not just organization policy. Select a model offered to this workflow; enabling a missing model for that integration requires Copilot API support. +[copilot-harness] copilot sub-agent 'mini-whoami': The requested model is not supported: 'gpt-5-mini' is not available in the workflow's live model catalog. For Copilot, availability also depends on the agentic-workflows integration, not just organization policy. Select a model offered to this workflow; enabling a missing model for that integration requires Copilot API support. Refusing to start Copilot with a different sub-agent model. +[copilot-harness] model resolution: engine=copilot role=mini54-whoami configured=gpt-5.4-mini resolved=gpt-5.4-mini provider=copilot status=exact +[copilot-harness] copilot sub-agent 'mini54-whoami': The requested model is not supported: 'gpt-5.4-mini' is not available in the workflow's live model catalog. For Copilot, availability also depends on the agentic-workflows integration, not just organization policy. Select a model offered to this workflow; enabling a missing model for that integration requires Copilot API support. Sub-agent model 'gpt-5.4-mini' is not listed on the configured awf-reflect endpoints. Refusing to start Copilot with a different sub-agent model. diff --git a/pkg/cli/testdata/model_resolution_startup/39499460077/agent_execution.json b/pkg/cli/testdata/model_resolution_startup/39499460077/agent_execution.json new file mode 100644 index 00000000000..f53cf177df1 --- /dev/null +++ b/pkg/cli/testdata/model_resolution_startup/39499460077/agent_execution.json @@ -0,0 +1 @@ +{ "version": 1, "component": "agent", "run_id": 39499460077, "run_attempt": 1, "state": "started" } diff --git a/pkg/cli/testdata/model_resolution_startup/39499460077/aw_session.jsonl b/pkg/cli/testdata/model_resolution_startup/39499460077/aw_session.jsonl new file mode 100644 index 00000000000..3b2090dba89 --- /dev/null +++ b/pkg/cli/testdata/model_resolution_startup/39499460077/aw_session.jsonl @@ -0,0 +1,8 @@ +{"type":"session.format","data":{"version":1},"provenance":{"component":"collector","phase":"conclusion","path":"usage/aw_session.jsonl","index":0}} +{"type":"model.resolution","data":{"engine":"copilot","role":"main","configured_model":"gpt-5.3-codex","resolved_model":"gpt-5.3-codex","provider":null,"status":"exact","cause":null},"provenance":{"component":"model_resolution","phase":"agent","path":"agent/model-resolution.json","index":0}} +{"type":"model.resolution","data":{"engine":"copilot","role":"haiku-whoami","configured_model":"claude-haiku-4.5","resolved_model":"claude-haiku-4.5","provider":"copilot","status":"exact","cause":null},"provenance":{"component":"model_resolution","phase":"agent","path":"agent/model-resolution.json","index":1}} +{"type":"model.resolution","data":{"engine":"copilot","role":"mini-whoami","configured_model":"gpt-5-mini","resolved_model":null,"provider":"copilot","status":"failed","cause":"model_unavailable"},"provenance":{"component":"model_resolution","phase":"agent","path":"agent/model-resolution.json","index":2}} +{"type":"model.resolution","data":{"engine":"copilot","role":"mini54-whoami","configured_model":"gpt-5.4-mini","resolved_model":null,"provider":"copilot","status":"failed","cause":"model_unavailable"},"provenance":{"component":"model_resolution","phase":"agent","path":"agent/model-resolution.json","index":3}} +{"type":"execution.result","data":{},"provenance":{"component":"execution","phase":"agent","path":"agent_execution.json","index":0}} +{"type":"agent.execution","data":{"categories":["model_not_supported_error"],"errorCodes":[],"errorTypes":[],"exitCode":1},"provenance":{"component":"execution","phase":"agent","path":"agent-errors.jsonl","index":0}} +{"type":"session.collection","data":{"sources":[{"component":"workflow","phase":"agent","path":"runtime/aw_info.json","timestampUnit":"milliseconds","events":1},{"component":"skill","phase":"activation","path":"skill_inventory.json","events":2},{"component":"prompt","phase":"activation","path":"aw-prompts/system.txt","events":1},{"component":"prompt","phase":"activation","path":"aw-prompts/user.txt","events":1},{"component":"agent","phase":"agent","path":"agent-session.jsonl","timestampUnit":"milliseconds","events":0},{"component":"agent","phase":"agent","path":"agent-stdio.log","events":0},{"component":"firewall","phase":"agent","path":"sandbox/firewall/awf-reflect.json","events":1},{"component":"otel","phase":"agent","path":"otel.jsonl","timestampUnit":"nanoseconds","events":1},{"component":"mcp","phase":"agent","path":"mcp-logs/configuration.jsonl","timestampUnit":"milliseconds","events":1},{"component":"mcp","phase":"agent","path":"mcp-logs/rpc-messages.jsonl","timestampUnit":"milliseconds","events":6},{"component":"firewall","phase":"agent","path":"sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl","timestampUnit":"milliseconds","events":0},{"component":"firewall","phase":"agent","path":"sandbox/firewall/logs/audit.jsonl","timestampUnit":"seconds","events":4},{"component":"firewall","phase":"detection","path":"threat-detection/sandbox/firewall/logs/api-proxy-logs/otel.jsonl","timestampUnit":"milliseconds","events":2},{"component":"firewall","phase":"detection","path":"threat-detection/sandbox/firewall/logs/api-proxy-logs/token-tracker-audit.jsonl","timestampUnit":"milliseconds","events":4},{"component":"firewall","phase":"detection","path":"threat-detection/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl","timestampUnit":"milliseconds","events":2},{"component":"firewall","phase":"detection","path":"threat-detection/sandbox/firewall/logs/audit.jsonl","timestampUnit":"seconds","events":10},{"component":"model_resolution","phase":"agent","path":"agent/model-resolution.json","events":4},{"component":"safe_output","phase":"safe_outputs","path":"safe-output-items.jsonl","timestampUnit":"milliseconds","events":0},{"component":"experiment","phase":"activation","path":"experiments/state.jsonl","timestampUnit":"milliseconds","events":62},{"component":"experiment","phase":"activation","path":"experiments/assignments.json","timestampUnit":"milliseconds","events":1},{"component":"github_api","phase":"workflow","path":"github_rate_limits.jsonl","timestampUnit":"milliseconds","events":75},{"component":"execution","phase":"agent","path":"agent_execution.json","timestampUnit":"milliseconds","events":1},{"component":"execution","phase":"detection","path":"threat-detection/execution.json","timestampUnit":"milliseconds","events":1},{"component":"agent","phase":"agent","path":"agent-errors.jsonl","timestampUnit":"milliseconds","events":0},{"component":"execution","phase":"agent","path":"agent-errors.jsonl","events":1},{"component":"guardrail","phase":"activation","path":"usage/aw_session.jsonl","events":1},{"component":"workflow","phase":"conclusion","path":"usage/aw_session.jsonl","events":8},{"component":"grader","phase":"conclusion","path":"usage/graders/grader_manifest.json","events":1},{"component":"grader","phase":"conclusion","path":"usage/graders/grader_results.json","events":1}],"warnings":0,"untimedEvents":24,"absentComponents":["eval"]},"provenance":{"component":"collector","phase":"conclusion","path":"usage/aw_session.jsonl","index":1}} diff --git a/pkg/cli/testdata/model_resolution_startup/39499460077/awf-reflect.json b/pkg/cli/testdata/model_resolution_startup/39499460077/awf-reflect.json new file mode 100644 index 00000000000..17867d1126e --- /dev/null +++ b/pkg/cli/testdata/model_resolution_startup/39499460077/awf-reflect.json @@ -0,0 +1,209 @@ +{ + "models_fetch_complete": true, + "endpoints": [ + { + "provider": "openai", + "configured": false + }, + { + "provider": "anthropic", + "configured": false + }, + { + "provider": "copilot", + "configured": true, + "models": [ + "claude-haiku-4.5", + "claude-haiku-5.5", + "claude-opus-5", + "claude-opus-5.5", + "claude-sonnet-5", + "claude-sonnet-5.5", + "gpt-3.5-turbo", + "gpt-3.5-turbo-0613", + "gpt-4", + "gpt-4-0613", + "gpt-4-o-preview", + "gpt-4.1", + "gpt-4.1-2025-04-14", + "gpt-41-copilot", + "gpt-4o", + "gpt-4o-2024-05-13", + "gpt-4o-2024-08-06", + "gpt-4o-2024-11-20", + "gpt-4o-mini", + "gpt-4o-mini-2024-07-18", + "gpt-5.3-codex", + "gpt-5.6-luna", + "gpt-5.6-luna-utility", + "gpt-5.6-sol", + "gpt-5.6-terra", + "gpt-6-astra", + "gpt-6-luna", + "gpt-6-sol", + "gpt-6.1-sol", + "grok-4.6", + "grok-4.7", + "mai-code-1.1-flash", + "text-embedding-3-small", + "text-embedding-3-small-inference", + "text-embedding-ada-002", + "trajectory-compaction" + ], + "routing_models": [ + { + "model_id": "claude-haiku-4.5", + "supported_endpoints": ["/chat/completions", "/v1/messages"] + }, + { + "model_id": "claude-haiku-5.5", + "supported_endpoints": ["/v1/messages", "/chat/completions"] + }, + { + "model_id": "claude-opus-5", + "supported_endpoints": ["/v1/messages", "/chat/completions"] + }, + { + "model_id": "claude-opus-5.5", + "supported_endpoints": ["/v1/messages", "/chat/completions"] + }, + { + "model_id": "claude-sonnet-5", + "supported_endpoints": ["/v1/messages", "/chat/completions"] + }, + { + "model_id": "claude-sonnet-5.5", + "supported_endpoints": ["/v1/messages", "/chat/completions"] + }, + { + "model_id": "gpt-3.5-turbo", + "supported_endpoints": [] + }, + { + "model_id": "gpt-3.5-turbo-0613", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4-0613", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4-o-preview", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4.1", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4.1-2025-04-14", + "supported_endpoints": [] + }, + { + "model_id": "gpt-41-copilot", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4o", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4o-2024-05-13", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4o-2024-08-06", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4o-2024-11-20", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4o-mini", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4o-mini-2024-07-18", + "supported_endpoints": [] + }, + { + "model_id": "gpt-5.3-codex", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-5.6-luna", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-5.6-luna-utility", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-5.6-sol", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-5.6-terra", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-6-astra", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-6-luna", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-6-sol", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-6.1-sol", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "grok-4.6", + "supported_endpoints": ["/responses"] + }, + { + "model_id": "grok-4.7", + "supported_endpoints": ["/responses"] + }, + { + "model_id": "mai-code-1.1-flash", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "text-embedding-3-small", + "supported_endpoints": [] + }, + { + "model_id": "text-embedding-3-small-inference", + "supported_endpoints": [] + }, + { + "model_id": "text-embedding-ada-002", + "supported_endpoints": [] + }, + { + "model_id": "trajectory-compaction", + "supported_endpoints": ["/chat/completions"] + } + ] + }, + { + "provider": "gemini", + "configured": false + }, + { + "provider": "vertex", + "configured": false + } + ] +} diff --git a/pkg/cli/testdata/model_resolution_startup/39499464895/agent-session.jsonl b/pkg/cli/testdata/model_resolution_startup/39499464895/agent-session.jsonl new file mode 100644 index 00000000000..130fc6c5ecb --- /dev/null +++ b/pkg/cli/testdata/model_resolution_startup/39499464895/agent-session.jsonl @@ -0,0 +1 @@ +{"type":"agent.execution","data":{"categories":["model_not_supported_error"],"errorCodes":[],"errorTypes":[],"exitCode":1}} diff --git a/pkg/cli/testdata/model_resolution_startup/39499464895/agent-stdio.log b/pkg/cli/testdata/model_resolution_startup/39499464895/agent-stdio.log new file mode 100644 index 00000000000..3f66a0a0a78 --- /dev/null +++ b/pkg/cli/testdata/model_resolution_startup/39499464895/agent-stdio.log @@ -0,0 +1,2 @@ +[copilot-harness] model resolution: engine=copilot role=mini-whoami configured=gpt-5.4-mini resolved=gpt-5.4-mini provider=copilot status=exact +[copilot-harness] copilot sub-agent 'mini-whoami': The requested model is not supported: 'gpt-5.4-mini' is not available in the workflow's live model catalog. For Copilot, availability also depends on the agentic-workflows integration, not just organization policy. Select a model offered to this workflow; enabling a missing model for that integration requires Copilot API support. Sub-agent model 'gpt-5.4-mini' is not listed on the configured awf-reflect endpoints. Refusing to start Copilot with a different sub-agent model. diff --git a/pkg/cli/testdata/model_resolution_startup/39499464895/agent_execution.json b/pkg/cli/testdata/model_resolution_startup/39499464895/agent_execution.json new file mode 100644 index 00000000000..9108266e793 --- /dev/null +++ b/pkg/cli/testdata/model_resolution_startup/39499464895/agent_execution.json @@ -0,0 +1 @@ +{ "version": 1, "component": "agent", "run_id": 39499464895, "run_attempt": 1, "state": "started" } diff --git a/pkg/cli/testdata/model_resolution_startup/39499464895/aw_session.jsonl b/pkg/cli/testdata/model_resolution_startup/39499464895/aw_session.jsonl new file mode 100644 index 00000000000..23248720d39 --- /dev/null +++ b/pkg/cli/testdata/model_resolution_startup/39499464895/aw_session.jsonl @@ -0,0 +1,9 @@ +{"type":"session.format","data":{"version":1},"provenance":{"component":"collector","phase":"conclusion","path":"usage/aw_session.jsonl","index":0}} +{"type":"firewall.model_routing","timestamp":"2026-10-20T21:22:31.229Z","data":{"schema":"model-routing/v0.28.62","stage":"classification","purpose":"routing_classification","attempt":1,"classifierModel":"github-copilot/claude-haiku-4.5","classifierEffort":null},"provenance":{"component":"firewall","phase":"agent","path":"sandbox/firewall/logs/api-proxy-logs/model-routing.jsonl","index":0,"timestampMs":1792531351229}} +{"type":"firewall.token_usage","timestamp":"2026-10-20T21:22:32.042Z","data":{"provider":"copilot","model":"claude-haiku-4.5","purpose":"routing_classification","path":"/chat/completions","xInitiator":"agent","requestId":"0a6d2942-93ef-4039-95b3-296cb634ef6f","status":200,"aic":0.1268,"totalAic":0.1268,"durationMs":810,"usage":{"inputTokens":1103,"outputTokens":33,"cacheReadInputTokens":0,"cacheCreationInputTokens":0}},"provenance":{"component":"firewall","phase":"agent","path":"sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl","index":0,"timestampMs":1792531352042}} +{"type":"firewall.model_routing","timestamp":"2026-10-20T21:22:32.047Z","data":{"schema":"model-routing/v0.28.62","stage":"selection","classifierModel":"github-copilot/claude-haiku-4.5","classifierEffort":null,"objective":{"goal":"cost","mode":"economy"},"provider":"copilot","labels":{"task_type":"unknown","scope":"unknown","task_complexity":"unknown"},"mode":"unknown","classifierAttempts":1,"degradedClassification":false,"degradedReason":null,"selectedId":"choice-0001","selectedProvider":"copilot","selectedModel":"github-copilot/claude-haiku-4.5","selectedEffort":null,"wireModel":"claude-haiku-4.5","endpoint":"/chat/completions","router":{"name":"gh-aw-router","version":"0.1.3"},"latencyMs":832,"rankedChoices":[{"rank":1,"model":"github-copilot/claude-haiku-4.5","effort":null}],"eligibleChoices":1,"catalogueOverlap":1},"provenance":{"component":"firewall","phase":"agent","path":"sandbox/firewall/logs/api-proxy-logs/model-routing.jsonl","index":1,"timestampMs":1792531352047}} +{"type":"model.resolution","data":{"engine":"copilot","role":"mini-whoami","configured_model":"gpt-5.4-mini","resolved_model":null,"provider":"copilot","status":"failed","cause":"model_unavailable"},"provenance":{"component":"model_resolution","phase":"agent","path":"agent/model-resolution.json","index":0}} +{"type":"model_routing.outcome","data":{"status":"selected","wireModel":"claude-haiku-4.5","effectiveEndpoint":"/chat/completions","selectedEndpoint":"/chat/completions"},"provenance":{"component":"agent","phase":"agent","path":"agent/awf-routing-outcome.json","index":0}} +{"type":"execution.result","data":{},"provenance":{"component":"execution","phase":"agent","path":"agent_execution.json","index":0}} +{"type":"agent.execution","data":{"categories":["model_not_supported_error"],"errorCodes":[],"errorTypes":[],"exitCode":1},"provenance":{"component":"execution","phase":"agent","path":"agent-errors.jsonl","index":0}} +{"type":"session.collection","data":{"sources":[{"component":"workflow","phase":"agent","path":"runtime/aw_info.json","timestampUnit":"milliseconds","events":1},{"component":"skill","phase":"activation","path":"skill_inventory.json","events":2},{"component":"prompt","phase":"activation","path":"aw-prompts/system.txt","events":1},{"component":"prompt","phase":"activation","path":"aw-prompts/user.txt","events":1},{"component":"agent","phase":"agent","path":"agent-session.jsonl","timestampUnit":"milliseconds","events":0},{"component":"agent","phase":"agent","path":"agent-stdio.log","events":0},{"component":"firewall","phase":"agent","path":"sandbox/firewall/awf-reflect.json","events":1},{"component":"otel","phase":"agent","path":"otel.jsonl","timestampUnit":"nanoseconds","events":1},{"component":"mcp","phase":"agent","path":"mcp-logs/configuration.jsonl","timestampUnit":"milliseconds","events":1},{"component":"mcp","phase":"agent","path":"mcp-logs/rpc-messages.jsonl","timestampUnit":"milliseconds","events":6},{"component":"firewall","phase":"agent","path":"sandbox/firewall/logs/api-proxy-logs/model-routing.jsonl","timestampUnit":"milliseconds","events":2},{"component":"firewall","phase":"agent","path":"sandbox/firewall/logs/api-proxy-logs/otel.jsonl","timestampUnit":"milliseconds","events":1},{"component":"firewall","phase":"agent","path":"sandbox/firewall/logs/api-proxy-logs/token-tracker-audit.jsonl","timestampUnit":"milliseconds","events":2},{"component":"firewall","phase":"agent","path":"sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl","timestampUnit":"milliseconds","events":1},{"component":"firewall","phase":"agent","path":"sandbox/firewall/logs/audit.jsonl","timestampUnit":"seconds","events":8},{"component":"firewall","phase":"detection","path":"threat-detection/sandbox/firewall/logs/api-proxy-logs/otel.jsonl","timestampUnit":"milliseconds","events":3},{"component":"firewall","phase":"detection","path":"threat-detection/sandbox/firewall/logs/api-proxy-logs/token-tracker-audit.jsonl","timestampUnit":"milliseconds","events":6},{"component":"firewall","phase":"detection","path":"threat-detection/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl","timestampUnit":"milliseconds","events":2},{"component":"firewall","phase":"detection","path":"threat-detection/sandbox/firewall/logs/audit.jsonl","timestampUnit":"seconds","events":8},{"component":"model_resolution","phase":"agent","path":"agent/model-resolution.json","events":1},{"component":"agent","phase":"agent","path":"agent/awf-routing-outcome.json","events":1},{"component":"safe_output","phase":"safe_outputs","path":"safe-output-items.jsonl","timestampUnit":"milliseconds","events":0},{"component":"github_api","phase":"workflow","path":"github_rate_limits.jsonl","timestampUnit":"milliseconds","events":74},{"component":"execution","phase":"agent","path":"agent_execution.json","timestampUnit":"milliseconds","events":1},{"component":"execution","phase":"detection","path":"threat-detection/execution.json","timestampUnit":"milliseconds","events":1},{"component":"agent","phase":"agent","path":"agent-errors.jsonl","timestampUnit":"milliseconds","events":0},{"component":"execution","phase":"agent","path":"agent-errors.jsonl","events":1},{"component":"guardrail","phase":"activation","path":"usage/aw_session.jsonl","events":1},{"component":"workflow","phase":"conclusion","path":"usage/aw_session.jsonl","events":5},{"component":"grader","phase":"conclusion","path":"usage/graders/grader_manifest.json","events":1},{"component":"grader","phase":"conclusion","path":"usage/graders/grader_results.json","events":1}],"warnings":0,"untimedEvents":18,"absentComponents":["experiment","eval"]},"provenance":{"component":"collector","phase":"conclusion","path":"usage/aw_session.jsonl","index":1}} diff --git a/pkg/cli/testdata/model_resolution_startup/39499464895/awf-reflect.json b/pkg/cli/testdata/model_resolution_startup/39499464895/awf-reflect.json new file mode 100644 index 00000000000..17867d1126e --- /dev/null +++ b/pkg/cli/testdata/model_resolution_startup/39499464895/awf-reflect.json @@ -0,0 +1,209 @@ +{ + "models_fetch_complete": true, + "endpoints": [ + { + "provider": "openai", + "configured": false + }, + { + "provider": "anthropic", + "configured": false + }, + { + "provider": "copilot", + "configured": true, + "models": [ + "claude-haiku-4.5", + "claude-haiku-5.5", + "claude-opus-5", + "claude-opus-5.5", + "claude-sonnet-5", + "claude-sonnet-5.5", + "gpt-3.5-turbo", + "gpt-3.5-turbo-0613", + "gpt-4", + "gpt-4-0613", + "gpt-4-o-preview", + "gpt-4.1", + "gpt-4.1-2025-04-14", + "gpt-41-copilot", + "gpt-4o", + "gpt-4o-2024-05-13", + "gpt-4o-2024-08-06", + "gpt-4o-2024-11-20", + "gpt-4o-mini", + "gpt-4o-mini-2024-07-18", + "gpt-5.3-codex", + "gpt-5.6-luna", + "gpt-5.6-luna-utility", + "gpt-5.6-sol", + "gpt-5.6-terra", + "gpt-6-astra", + "gpt-6-luna", + "gpt-6-sol", + "gpt-6.1-sol", + "grok-4.6", + "grok-4.7", + "mai-code-1.1-flash", + "text-embedding-3-small", + "text-embedding-3-small-inference", + "text-embedding-ada-002", + "trajectory-compaction" + ], + "routing_models": [ + { + "model_id": "claude-haiku-4.5", + "supported_endpoints": ["/chat/completions", "/v1/messages"] + }, + { + "model_id": "claude-haiku-5.5", + "supported_endpoints": ["/v1/messages", "/chat/completions"] + }, + { + "model_id": "claude-opus-5", + "supported_endpoints": ["/v1/messages", "/chat/completions"] + }, + { + "model_id": "claude-opus-5.5", + "supported_endpoints": ["/v1/messages", "/chat/completions"] + }, + { + "model_id": "claude-sonnet-5", + "supported_endpoints": ["/v1/messages", "/chat/completions"] + }, + { + "model_id": "claude-sonnet-5.5", + "supported_endpoints": ["/v1/messages", "/chat/completions"] + }, + { + "model_id": "gpt-3.5-turbo", + "supported_endpoints": [] + }, + { + "model_id": "gpt-3.5-turbo-0613", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4-0613", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4-o-preview", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4.1", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4.1-2025-04-14", + "supported_endpoints": [] + }, + { + "model_id": "gpt-41-copilot", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4o", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4o-2024-05-13", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4o-2024-08-06", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4o-2024-11-20", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4o-mini", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4o-mini-2024-07-18", + "supported_endpoints": [] + }, + { + "model_id": "gpt-5.3-codex", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-5.6-luna", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-5.6-luna-utility", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-5.6-sol", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-5.6-terra", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-6-astra", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-6-luna", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-6-sol", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-6.1-sol", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "grok-4.6", + "supported_endpoints": ["/responses"] + }, + { + "model_id": "grok-4.7", + "supported_endpoints": ["/responses"] + }, + { + "model_id": "mai-code-1.1-flash", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "text-embedding-3-small", + "supported_endpoints": [] + }, + { + "model_id": "text-embedding-3-small-inference", + "supported_endpoints": [] + }, + { + "model_id": "text-embedding-ada-002", + "supported_endpoints": [] + }, + { + "model_id": "trajectory-compaction", + "supported_endpoints": ["/chat/completions"] + } + ] + }, + { + "provider": "gemini", + "configured": false + }, + { + "provider": "vertex", + "configured": false + } + ] +} diff --git a/pkg/cli/testdata/model_resolution_startup/39508150750/agent-session.jsonl b/pkg/cli/testdata/model_resolution_startup/39508150750/agent-session.jsonl new file mode 100644 index 00000000000..130fc6c5ecb --- /dev/null +++ b/pkg/cli/testdata/model_resolution_startup/39508150750/agent-session.jsonl @@ -0,0 +1 @@ +{"type":"agent.execution","data":{"categories":["model_not_supported_error"],"errorCodes":[],"errorTypes":[],"exitCode":1}} diff --git a/pkg/cli/testdata/model_resolution_startup/39508150750/agent-stdio.log b/pkg/cli/testdata/model_resolution_startup/39508150750/agent-stdio.log new file mode 100644 index 00000000000..e4a6fb272f1 --- /dev/null +++ b/pkg/cli/testdata/model_resolution_startup/39508150750/agent-stdio.log @@ -0,0 +1,10 @@ +[copilot-harness] model resolution: engine=copilot role=main configured=gpt-5.3-codex resolved=gpt-5.3-codex provider=(unscoped) status=exact +[copilot-harness] model resolution: engine=copilot role=haiku-whoami configured=claude-haiku-4.5 resolved=claude-haiku-4.5 provider=copilot status=exact +[copilot-harness] copilot sub-agent 'haiku-whoami': model 'claude-haiku-4.5' -> 'copilot-completions/claude-haiku-4.5' +[copilot-harness] copilot sub-agent 'mini-whoami' model alias resolution: known alias 'gpt-5-mini' did not resolve against catalog (provider=copilot) +[copilot-harness] copilot sub-agent 'mini-whoami' model alias resolution: retrying awf-reflect model-catalog fetch once before failing for 'gpt-5-mini' +[copilot-harness] copilot sub-agent 'mini-whoami' model alias resolution: known alias 'gpt-5-mini' did not resolve against catalog (provider=copilot) +[copilot-harness] model resolution: engine=copilot role=mini-whoami configured=gpt-5-mini failed cause=model_unavailable: The requested model is not supported: 'gpt-5-mini' is not available in the workflow's live model catalog. For Copilot, availability also depends on the agentic-workflows integration, not just organization policy. Select a model offered to this workflow; enabling a missing model for that integration requires Copilot API support. +[copilot-harness] copilot sub-agent 'mini-whoami': The requested model is not supported: 'gpt-5-mini' is not available in the workflow's live model catalog. For Copilot, availability also depends on the agentic-workflows integration, not just organization policy. Select a model offered to this workflow; enabling a missing model for that integration requires Copilot API support. Refusing to start Copilot with a different sub-agent model. +[copilot-harness] model resolution: engine=copilot role=mini54-whoami configured=gpt-5.4-mini resolved=gpt-5.4-mini provider=copilot status=exact +[copilot-harness] copilot sub-agent 'mini54-whoami': The requested model is not supported: 'gpt-5.4-mini' is not available in the workflow's live model catalog. For Copilot, availability also depends on the agentic-workflows integration, not just organization policy. Select a model offered to this workflow; enabling a missing model for that integration requires Copilot API support. Sub-agent model 'gpt-5.4-mini' is not listed on the configured awf-reflect endpoints. Refusing to start Copilot with a different sub-agent model. diff --git a/pkg/cli/testdata/model_resolution_startup/39508150750/agent_execution.json b/pkg/cli/testdata/model_resolution_startup/39508150750/agent_execution.json new file mode 100644 index 00000000000..3edab6b25b7 --- /dev/null +++ b/pkg/cli/testdata/model_resolution_startup/39508150750/agent_execution.json @@ -0,0 +1 @@ +{ "version": 1, "component": "agent", "run_id": 39508150750, "run_attempt": 1, "state": "started" } diff --git a/pkg/cli/testdata/model_resolution_startup/39508150750/aw_session.jsonl b/pkg/cli/testdata/model_resolution_startup/39508150750/aw_session.jsonl new file mode 100644 index 00000000000..83ec6f3a973 --- /dev/null +++ b/pkg/cli/testdata/model_resolution_startup/39508150750/aw_session.jsonl @@ -0,0 +1,8 @@ +{"type":"session.format","data":{"version":1},"provenance":{"component":"collector","phase":"conclusion","path":"usage/aw_session.jsonl","index":0}} +{"type":"model.resolution","data":{"engine":"copilot","role":"main","configured_model":"gpt-5.3-codex","resolved_model":"gpt-5.3-codex","provider":null,"status":"exact","cause":null},"provenance":{"component":"model_resolution","phase":"agent","path":"agent/model-resolution.json","index":0}} +{"type":"model.resolution","data":{"engine":"copilot","role":"haiku-whoami","configured_model":"claude-haiku-4.5","resolved_model":"claude-haiku-4.5","provider":"copilot","status":"exact","cause":null},"provenance":{"component":"model_resolution","phase":"agent","path":"agent/model-resolution.json","index":1}} +{"type":"model.resolution","data":{"engine":"copilot","role":"mini-whoami","configured_model":"gpt-5-mini","resolved_model":null,"provider":"copilot","status":"failed","cause":"model_unavailable"},"provenance":{"component":"model_resolution","phase":"agent","path":"agent/model-resolution.json","index":2}} +{"type":"model.resolution","data":{"engine":"copilot","role":"mini54-whoami","configured_model":"gpt-5.4-mini","resolved_model":null,"provider":"copilot","status":"failed","cause":"model_unavailable"},"provenance":{"component":"model_resolution","phase":"agent","path":"agent/model-resolution.json","index":3}} +{"type":"execution.result","data":{},"provenance":{"component":"execution","phase":"agent","path":"agent_execution.json","index":0}} +{"type":"agent.execution","data":{"categories":["model_not_supported_error"],"errorCodes":[],"errorTypes":[],"exitCode":1},"provenance":{"component":"execution","phase":"agent","path":"agent-errors.jsonl","index":0}} +{"type":"session.collection","data":{"sources":[{"component":"workflow","phase":"agent","path":"runtime/aw_info.json","timestampUnit":"milliseconds","events":1},{"component":"skill","phase":"activation","path":"skill_inventory.json","events":2},{"component":"prompt","phase":"activation","path":"aw-prompts/system.txt","events":1},{"component":"prompt","phase":"activation","path":"aw-prompts/user.txt","events":1},{"component":"agent","phase":"agent","path":"agent-session.jsonl","timestampUnit":"milliseconds","events":0},{"component":"agent","phase":"agent","path":"agent-stdio.log","events":0},{"component":"firewall","phase":"agent","path":"sandbox/firewall/awf-reflect.json","events":1},{"component":"otel","phase":"agent","path":"otel.jsonl","timestampUnit":"nanoseconds","events":1},{"component":"mcp","phase":"agent","path":"mcp-logs/configuration.jsonl","timestampUnit":"milliseconds","events":1},{"component":"mcp","phase":"agent","path":"mcp-logs/rpc-messages.jsonl","timestampUnit":"milliseconds","events":6},{"component":"firewall","phase":"agent","path":"sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl","timestampUnit":"milliseconds","events":0},{"component":"firewall","phase":"agent","path":"sandbox/firewall/logs/audit.jsonl","timestampUnit":"seconds","events":7},{"component":"firewall","phase":"detection","path":"threat-detection/sandbox/firewall/logs/api-proxy-logs/otel.jsonl","timestampUnit":"milliseconds","events":3},{"component":"firewall","phase":"detection","path":"threat-detection/sandbox/firewall/logs/api-proxy-logs/token-tracker-audit.jsonl","timestampUnit":"milliseconds","events":6},{"component":"firewall","phase":"detection","path":"threat-detection/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl","timestampUnit":"milliseconds","events":2},{"component":"firewall","phase":"detection","path":"threat-detection/sandbox/firewall/logs/audit.jsonl","timestampUnit":"seconds","events":13},{"component":"model_resolution","phase":"agent","path":"agent/model-resolution.json","events":4},{"component":"safe_output","phase":"safe_outputs","path":"safe-output-items.jsonl","timestampUnit":"milliseconds","events":0},{"component":"experiment","phase":"activation","path":"experiments/state.jsonl","timestampUnit":"milliseconds","events":63},{"component":"experiment","phase":"activation","path":"experiments/assignments.json","timestampUnit":"milliseconds","events":1},{"component":"github_api","phase":"workflow","path":"github_rate_limits.jsonl","timestampUnit":"milliseconds","events":73},{"component":"execution","phase":"agent","path":"agent_execution.json","timestampUnit":"milliseconds","events":1},{"component":"execution","phase":"detection","path":"threat-detection/execution.json","timestampUnit":"milliseconds","events":1},{"component":"agent","phase":"agent","path":"agent-errors.jsonl","timestampUnit":"milliseconds","events":0},{"component":"execution","phase":"agent","path":"agent-errors.jsonl","events":1},{"component":"guardrail","phase":"activation","path":"usage/aw_session.jsonl","events":1},{"component":"workflow","phase":"conclusion","path":"usage/aw_session.jsonl","events":8},{"component":"grader","phase":"conclusion","path":"usage/graders/grader_manifest.json","events":1},{"component":"grader","phase":"conclusion","path":"usage/graders/grader_results.json","events":1}],"warnings":0,"untimedEvents":24,"absentComponents":["eval"]},"provenance":{"component":"collector","phase":"conclusion","path":"usage/aw_session.jsonl","index":1}} diff --git a/pkg/cli/testdata/model_resolution_startup/39508150750/awf-reflect.json b/pkg/cli/testdata/model_resolution_startup/39508150750/awf-reflect.json new file mode 100644 index 00000000000..17867d1126e --- /dev/null +++ b/pkg/cli/testdata/model_resolution_startup/39508150750/awf-reflect.json @@ -0,0 +1,209 @@ +{ + "models_fetch_complete": true, + "endpoints": [ + { + "provider": "openai", + "configured": false + }, + { + "provider": "anthropic", + "configured": false + }, + { + "provider": "copilot", + "configured": true, + "models": [ + "claude-haiku-4.5", + "claude-haiku-5.5", + "claude-opus-5", + "claude-opus-5.5", + "claude-sonnet-5", + "claude-sonnet-5.5", + "gpt-3.5-turbo", + "gpt-3.5-turbo-0613", + "gpt-4", + "gpt-4-0613", + "gpt-4-o-preview", + "gpt-4.1", + "gpt-4.1-2025-04-14", + "gpt-41-copilot", + "gpt-4o", + "gpt-4o-2024-05-13", + "gpt-4o-2024-08-06", + "gpt-4o-2024-11-20", + "gpt-4o-mini", + "gpt-4o-mini-2024-07-18", + "gpt-5.3-codex", + "gpt-5.6-luna", + "gpt-5.6-luna-utility", + "gpt-5.6-sol", + "gpt-5.6-terra", + "gpt-6-astra", + "gpt-6-luna", + "gpt-6-sol", + "gpt-6.1-sol", + "grok-4.6", + "grok-4.7", + "mai-code-1.1-flash", + "text-embedding-3-small", + "text-embedding-3-small-inference", + "text-embedding-ada-002", + "trajectory-compaction" + ], + "routing_models": [ + { + "model_id": "claude-haiku-4.5", + "supported_endpoints": ["/chat/completions", "/v1/messages"] + }, + { + "model_id": "claude-haiku-5.5", + "supported_endpoints": ["/v1/messages", "/chat/completions"] + }, + { + "model_id": "claude-opus-5", + "supported_endpoints": ["/v1/messages", "/chat/completions"] + }, + { + "model_id": "claude-opus-5.5", + "supported_endpoints": ["/v1/messages", "/chat/completions"] + }, + { + "model_id": "claude-sonnet-5", + "supported_endpoints": ["/v1/messages", "/chat/completions"] + }, + { + "model_id": "claude-sonnet-5.5", + "supported_endpoints": ["/v1/messages", "/chat/completions"] + }, + { + "model_id": "gpt-3.5-turbo", + "supported_endpoints": [] + }, + { + "model_id": "gpt-3.5-turbo-0613", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4-0613", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4-o-preview", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4.1", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4.1-2025-04-14", + "supported_endpoints": [] + }, + { + "model_id": "gpt-41-copilot", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4o", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4o-2024-05-13", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4o-2024-08-06", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4o-2024-11-20", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4o-mini", + "supported_endpoints": [] + }, + { + "model_id": "gpt-4o-mini-2024-07-18", + "supported_endpoints": [] + }, + { + "model_id": "gpt-5.3-codex", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-5.6-luna", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-5.6-luna-utility", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-5.6-sol", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-5.6-terra", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-6-astra", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-6-luna", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-6-sol", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "gpt-6.1-sol", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "grok-4.6", + "supported_endpoints": ["/responses"] + }, + { + "model_id": "grok-4.7", + "supported_endpoints": ["/responses"] + }, + { + "model_id": "mai-code-1.1-flash", + "supported_endpoints": ["/responses", "ws:/responses"] + }, + { + "model_id": "text-embedding-3-small", + "supported_endpoints": [] + }, + { + "model_id": "text-embedding-3-small-inference", + "supported_endpoints": [] + }, + { + "model_id": "text-embedding-ada-002", + "supported_endpoints": [] + }, + { + "model_id": "trajectory-compaction", + "supported_endpoints": ["/chat/completions"] + } + ] + }, + { + "provider": "gemini", + "configured": false + }, + { + "provider": "vertex", + "configured": false + } + ] +} diff --git a/pkg/cli/testdata/model_resolution_startup/39508154330/aw_session.jsonl b/pkg/cli/testdata/model_resolution_startup/39508154330/aw_session.jsonl new file mode 100644 index 00000000000..1a187eb80ce --- /dev/null +++ b/pkg/cli/testdata/model_resolution_startup/39508154330/aw_session.jsonl @@ -0,0 +1,16 @@ +{"type":"session.format","data":{"version":1},"provenance":{"component":"collector","phase":"conclusion","path":"usage/aw_session.jsonl","index":0}} +{"type":"session.start","id":"ea16f71f-c8e0-42fc-a2b6-563b8185228f","parentId":null,"timestamp":"2026-10-20T22:29:53.090Z","data":{"sessionId":"3826d660-bd62-4ea5-a4f5-51b90d3b7168","model":"copilot-responses/gpt-5.6-luna"},"provenance":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":0,"timestampMs":1792535393090,"native":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":0,"persistedPath":"agent-session.jsonl"}}} +{"type":"assistant.turn_start","id":"da39cf5f-f322-4540-99d2-77a8e6addb5c","parentId":"b47dceb1-aa98-4793-af82-8ea18e9a3a99","timestamp":"2026-10-20T22:29:53.356Z","data":{"turnId":"0"},"provenance":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":4,"timestampMs":1792535393356,"native":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":4,"persistedPath":"agent-session.jsonl"}}} +{"type":"assistant.message","id":"a9db00f6-2367-46f8-ba6d-c9fdc977c120","parentId":"d12503be-ecc8-4ef6-ba37-0a1ca8bca169","timestamp":"2026-10-20T22:30:03.703Z","data":{"messageId":"ada82a6c-c38f-43f6-b1eb-3a57d84dc3c5","model":"gpt-5.6-luna"},"provenance":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":6,"timestampMs":1792535403703,"native":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":6,"persistedPath":"agent-session.jsonl"}}} +{"type":"tool.execution_start","id":"65a812a4-328c-4ef2-960c-fe8e267c743a","parentId":"a9db00f6-2367-46f8-ba6d-c9fdc977c120","timestamp":"2026-10-20T22:30:03.705Z","data":{"toolCallId":"call_5w7p9uRg2zTBcdF2n1zfHe86","toolName":"skill"},"provenance":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":7,"timestampMs":1792535403705,"native":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":7,"persistedPath":"agent-session.jsonl"}}} +{"type":"tool.execution_complete","id":"a18792c9-361f-4516-aee1-dfada52a8a75","parentId":"65a812a4-328c-4ef2-960c-fe8e267c743a","timestamp":"2026-10-20T22:30:03.708Z","data":{"toolCallId":"call_5w7p9uRg2zTBcdF2n1zfHe86","toolName":"skill","success":true},"provenance":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":8,"timestampMs":1792535403708,"native":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":8,"persistedPath":"agent-session.jsonl"}}} +{"type":"assistant.turn_end","id":"f07b04da-d759-4b9c-bfb5-828cf20b2944","parentId":"f5ad2e82-88e7-4af6-9900-cf1ff15ac6a8","timestamp":"2026-10-20T22:30:03.720Z","data":{"turnId":"0"},"provenance":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":11,"timestampMs":1792535403720,"native":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":11,"persistedPath":"agent-session.jsonl"}}} +{"type":"tool.execution_start","id":"6913d522-4023-43ea-89cc-d5f2321cf051","parentId":"1d03ac8d-2c18-441d-a877-5bf822092192","timestamp":"2026-10-20T22:30:06.055Z","data":{"toolCallId":"call_WlOtBu1T35d4hqh8gX7SpfyE","toolName":"task"},"provenance":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":15,"timestampMs":1792535406055,"native":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":15,"persistedPath":"agent-session.jsonl"}}} +{"type":"subagent.started","id":"0b9005ef-b12c-44a3-b3a2-76ea222b1837","parentId":"6913d522-4023-43ea-89cc-d5f2321cf051","timestamp":"2026-10-20T22:30:06.060Z","agentId":"a6b32315-a68b-458a-8c8b-af782d69f041","data":{"toolCallId":"call_WlOtBu1T35d4hqh8gX7SpfyE","agentName":"haiku-whoami","model":"copilot-completions/claude-haiku-4.5"},"provenance":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":16,"timestampMs":1792535406060,"native":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":16,"persistedPath":"agent-session.jsonl"}}} +{"type":"subagent.completed","id":"3e5a4dd4-89c6-4c5d-a610-abd72c79324b","parentId":"b3843285-154e-4adb-b631-ff9274c9b7ef","timestamp":"2026-10-20T22:30:07.153Z","agentId":"a6b32315-a68b-458a-8c8b-af782d69f041","data":{"toolCallId":"call_WlOtBu1T35d4hqh8gX7SpfyE","agentName":"haiku-whoami","model":"copilot-completions/claude-haiku-4.5"},"provenance":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":26,"timestampMs":1792535407153,"native":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":26,"persistedPath":"agent-session.jsonl"}}} +{"type":"tool.execution_complete","id":"de1734eb-70f9-4167-b8d9-53596f421851","parentId":"3e5a4dd4-89c6-4c5d-a610-abd72c79324b","timestamp":"2026-10-20T22:30:07.154Z","data":{"toolCallId":"call_WlOtBu1T35d4hqh8gX7SpfyE","toolName":"task","success":true},"provenance":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":27,"timestampMs":1792535407154,"native":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":27,"persistedPath":"agent-session.jsonl"}}} +{"type":"tool.execution_start","id":"8b903b8b-3279-49ed-89bd-884226878471","parentId":"76202d5a-90da-436d-abcc-527d77148810","timestamp":"2026-10-20T22:30:12.392Z","data":{"toolCallId":"call_rSxUDTs0Fu0nZUqLvwMHOi3i","toolName":"safeoutputs-create_issue"},"provenance":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":33,"timestampMs":1792535412392,"native":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":33,"persistedPath":"agent-session.jsonl"}}} +{"type":"tool.execution_complete","id":"4fec1e2b-03ff-419c-9f51-44c25330ead9","parentId":"72e8a996-daf3-4622-889b-980e30193512","timestamp":"2026-10-20T22:30:12.647Z","data":{"toolCallId":"call_rSxUDTs0Fu0nZUqLvwMHOi3i","toolName":"safeoutputs-create_issue","success":true},"provenance":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":36,"timestampMs":1792535412647,"native":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":36,"persistedPath":"agent-session.jsonl"}}} +{"type":"session.shutdown","id":"2347f78c-5718-4517-9efe-913109383948","parentId":"3a04cf7e-a69d-46f8-966b-4d57878b6ecd","timestamp":"2026-10-20T22:30:15.805Z","data":{"shutdownType":"routine","currentModel":"gpt-5.6-luna"},"provenance":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":43,"timestampMs":1792535415805,"native":{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","index":43,"persistedPath":"agent-session.jsonl"}}} +{"type":"agent.execution","data":{"categories":[],"errorCodes":[],"errorTypes":[],"exitCode":0},"provenance":{"component":"execution","phase":"agent","path":"agent-errors.jsonl","index":0}} +{"type":"session.collection","data":{"sources":[{"component":"workflow","phase":"agent","path":"runtime/aw_info.json","timestampUnit":"milliseconds","events":1},{"component":"skill","phase":"activation","path":"skill_inventory.json","events":2},{"component":"prompt","phase":"activation","path":"aw-prompts/system.txt","events":1},{"component":"prompt","phase":"activation","path":"aw-prompts/user.txt","events":1},{"component":"agent","phase":"agent","path":"sandbox/agent/logs/copilot-session-state/3826d660-bd62-4ea5-a4f5-51b90d3b7168/events.jsonl","timestampUnit":"milliseconds","events":47},{"component":"firewall","phase":"agent","path":"sandbox/firewall/awf-reflect.json","events":1},{"component":"otel","phase":"agent","path":"otel.jsonl","timestampUnit":"nanoseconds","events":1},{"component":"mcp","phase":"agent","path":"mcp-logs/configuration.jsonl","timestampUnit":"milliseconds","events":1},{"component":"mcp","phase":"agent","path":"mcp-logs/rpc-messages.jsonl","timestampUnit":"milliseconds","events":8},{"component":"firewall","phase":"agent","path":"sandbox/firewall/logs/api-proxy-logs/model-routing.jsonl","timestampUnit":"milliseconds","events":7},{"component":"firewall","phase":"agent","path":"sandbox/firewall/logs/api-proxy-logs/otel.jsonl","timestampUnit":"milliseconds","events":6},{"component":"firewall","phase":"agent","path":"sandbox/firewall/logs/api-proxy-logs/token-tracker-audit.jsonl","timestampUnit":"milliseconds","events":12},{"component":"firewall","phase":"agent","path":"sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl","timestampUnit":"milliseconds","events":6},{"component":"firewall","phase":"agent","path":"sandbox/firewall/logs/audit.jsonl","timestampUnit":"seconds","events":29},{"component":"firewall","phase":"detection","path":"threat-detection/sandbox/firewall/logs/api-proxy-logs/otel.jsonl","timestampUnit":"milliseconds","events":3},{"component":"firewall","phase":"detection","path":"threat-detection/sandbox/firewall/logs/api-proxy-logs/token-tracker-audit.jsonl","timestampUnit":"milliseconds","events":6},{"component":"firewall","phase":"detection","path":"threat-detection/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl","timestampUnit":"milliseconds","events":2},{"component":"firewall","phase":"detection","path":"threat-detection/sandbox/firewall/logs/audit.jsonl","timestampUnit":"seconds","events":12},{"component":"model_resolution","phase":"agent","path":"agent/model-resolution.json","events":1},{"component":"agent","phase":"agent","path":"agent/awf-routing-outcome.json","events":1},{"component":"safe_output","phase":"agent","path":"safeoutputs.jsonl","timestampUnit":"milliseconds","events":1},{"component":"safe_output","phase":"safe_outputs","path":"safe-output-items.jsonl","timestampUnit":"milliseconds","events":1},{"component":"github_api","phase":"workflow","path":"github_rate_limits.jsonl","timestampUnit":"milliseconds","events":67},{"component":"execution","phase":"agent","path":"agent_execution.json","timestampUnit":"milliseconds","events":1},{"component":"execution","phase":"detection","path":"threat-detection/execution.json","timestampUnit":"milliseconds","events":1},{"component":"agent","phase":"agent","path":"agent-errors.jsonl","timestampUnit":"milliseconds","events":0},{"component":"execution","phase":"agent","path":"agent-errors.jsonl","events":1},{"component":"guardrail","phase":"activation","path":"usage/aw_session.jsonl","events":1},{"component":"workflow","phase":"conclusion","path":"usage/aw_session.jsonl","events":5},{"component":"grader","phase":"conclusion","path":"usage/graders/grader_manifest.json","events":1},{"component":"grader","phase":"conclusion","path":"usage/graders/grader_results.json","events":1}],"warnings":0,"untimedEvents":19,"absentComponents":["experiment","eval"]},"provenance":{"component":"collector","phase":"conclusion","path":"usage/aw_session.jsonl","index":1}}