Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions .prettierignore
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,6 @@ schemas/logs-jsonl.schema.json

.beads/
.github/aw/

# Recorded byte-exact AWF reflection fixtures (SHA-256 pinned)
actions/setup/js/test_data/awf_reflect/*.json
17 changes: 17 additions & 0 deletions actions/setup/js/test_data/awf_reflect/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Recorded AWF `/reflect` fixtures

These files are unchanged byte-for-byte copies of `sandbox/firewall/awf-reflect.json`
from public `github/gh-aw` workflow-run artifacts. They are excluded from
prettier so that their SHA-256 digests stay verifiable. Tests derive labelled
pending-fetch and malformed variants in memory; they never edit these files.

| File | Source run | Artifact | SHA-256 | Shape |
| --- | --- | --- | --- | --- |
| `run-38995410124-awf-reflect.json` | [smoke-claude-on-copilot 38995410124](https://github.com/github/gh-aw/actions/runs/38995410124), commit `12ba3ffcbaf5760b6094f52d8f7d61938e4e951e`, 2026-10-17 | `agent` (12087091651) | `9d59511015584d359b4159737d1d44ed12ad9426a1690e62148ab53b7543648a` | Five endpoints; configured Copilot with 40 models and 40 `routing_models`; four unconfigured endpoints with null `models` and `routing_models` |
| `run-29812420778-legacy-awf-reflect.json` | [Smoke Copilot 29812420778](https://github.com/github/gh-aw/actions/runs/29812420778), commit `fed7b8398300613d6f79474d75d3ffcdc6665d18`, 2026-07-21 | `agent` (8488185681) | `da2570969e0d62bd10195f029b5472f750404b515784d8c68e9b267e0975b410` | Older AWF reflection without `routing_models`; configured Copilot with 36 models and four unconfigured endpoints with null `models` |

The legacy file is the oldest retained public reflection found. It is not the
source of a `pkg/cli/testdata/model_routing_golden/` case: those source runs are
either private or expired (for example, run 37886511291 returns HTTP 404). Every
AWF `/reflect` version since its introduction supplies endpoint `configured`, so
no recorded reflection without `configured` exists to pin.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"endpoints":[{"provider":"openai","port":10000,"base_url":"http://api-proxy:10000","configured":false,"models":null,"models_url":"http://api-proxy:10000/v1/models"},{"provider":"anthropic","port":10001,"base_url":"http://api-proxy:10001","configured":false,"models":null,"models_url":"http://api-proxy:10001/v1/models"},{"provider":"copilot","port":10002,"base_url":"http://api-proxy:10002","configured":true,"models":["claude-haiku-4.5","claude-opus-4.5","claude-opus-4.6","claude-opus-4.7","claude-opus-4.8","claude-sonnet-4.5","claude-sonnet-4.6","claude-sonnet-5","gemini-2.5-pro","gemini-3-flash-preview","gpt-3.5-turbo","gpt-3.5-turbo-0613","gpt-4","gpt-4-0613","gpt-4-o-preview","gpt-4.1","gpt-4.1-2025-04-14","gpt-4o","gpt-4o-2024-05-13","gpt-4o-2024-08-06","gpt-4o-2024-11-20","gpt-4o-mini","gpt-4o-mini-2024-07-18","gpt-5-mini","gpt-5.3-codex","gpt-5.4","gpt-5.4-mini","gpt-5.5","gpt-5.6-luna","gpt-5.6-sol","gpt-5.6-terra","mai-code-1-flash-picker","text-embedding-3-small","text-embedding-3-small-inference","text-embedding-ada-002","trajectory-compaction"],"models_url":"http://api-proxy:10002/models"},{"provider":"gemini","port":10003,"base_url":"http://api-proxy:10003","configured":false,"models":null,"models_url":"http://api-proxy:10003/v1beta/models"},{"provider":"vertex","port":10004,"base_url":"http://api-proxy:10004","configured":false,"models":null,"models_url":null}],"models_fetch_complete":true,"model_aliases":{"agent":["sonnet-6x","gpt-5.4","gpt-5.3","gemini-pro","any"],"any":["copilot/*","anthropic/*","openai/*","google/*","gemini/*"],"claude":["agent"],"codex":["agent"],"coding":["copilot/gpt-5*codex*","openai/gpt-5*codex*","gpt-5-codex","kimi"],"copilot":["agent"],"gemini":["agent"],"gemini-3-flash":["copilot/gemini-3*flash*","google/gemini-3*flash*","gemini/gemini-3*flash*"],"gemini-flash":["copilot/gemini-*flash*","google/gemini-*flash*","gemini/gemini-*flash*"],"gemini-pro":["copilot/gemini-*pro*","google/gemini-*pro*","gemini/gemini-*pro*"],"gpt-5":["copilot/gpt-5*","openai/gpt-5*"],"gpt-5-codex":["copilot/gpt-5*codex*","openai/gpt-5*codex*"],"gpt-5-mini":["copilot/gpt-5*mini*","openai/gpt-5*mini*"],"gpt-5.3":["copilot/gpt-5.3*","openai/gpt-5.3*"],"gpt-5.4":["copilot/gpt-5.4*","openai/gpt-5.4*"],"gpt-5.5":["copilot/gpt-5.5*","openai/gpt-5.5*"],"gpt-5.6":["copilot/gpt-5.6*","openai/gpt-5.6*"],"haiku":["copilot/*haiku*","anthropic/*haiku*"],"large":["fable","sonnet","gpt-5-pro","gpt-5","gemini-pro"],"mai-code":["copilot/MAI-Code*","copilot/mai-code*","openai/MAI-Code*"],"mai-code-1-flash-picker":["copilot/MAI-Code-1-Flash-picker*","copilot/mai-code-1-flash-picker*","openai/MAI-Code-1-Flash-picker*"],"mini":["haiku","gpt-5-mini","gpt-5-nano","gemini-flash-lite"],"opus":["copilot/*opus*","anthropic/*opus*"],"small":["mini"],"small-agent":["haiku","gpt-5-mini","gemini-flash"],"sonnet":["copilot/*sonnet*","anthropic/*sonnet*"],"sonnet-6x":["copilot/*sonnet-4.5*","copilot/*sonnet-4.6*","copilot/*sonnet-5*","copilot/*sonnet-4-5-*","anthropic/*sonnet-4-5-*","copilot/*sonnet-4-6*","anthropic/*sonnet-4-6*","anthropic/*sonnet-5*"],"summarization":["haiku","gpt-5-mini","gemini-flash-lite","mini"],"vision":["copilot/gemini-*image*","google/gemini-*image*","gemini/gemini-*image*","copilot/gemini-*flash*","google/gemini-*flash*","gemini/gemini-*flash*"]},"model_fallback":{"enabled":true,"strategy":"middle_power","excludeEngines":[]},"model_fallback_effective":{"openai":{"enabled":true,"strategy":"middle_power","excludeEngines":[],"suppressed":false},"anthropic":{"enabled":true,"strategy":"middle_power","excludeEngines":[],"suppressed":false},"copilot":{"enabled":false,"strategy":"middle_power","excludeEngines":[],"suppressed":true,"suppression_reason":"copilot_standard_authoritative"},"gemini":{"enabled":true,"strategy":"middle_power","excludeEngines":[],"suppressed":false},"vertex":{"enabled":true,"strategy":"middle_power","excludeEngines":[],"suppressed":false}},"ai_credits":{"total":225.64295,"by_model":{"gpt-5.4-2026-03-05":{"input_credits":188.21525,"cached_input_credits":17.0112,"cache_write_credits":0,"output_credits":20.4165,"total":225.64295}}},"runs":{"enabled":true,"max_runs":500,"invocation_count":18,"remaining_runs":482},"cache_misses":{"enabled":true,"max_cache_misses":5,"consecutive_cache_misses":0,"remaining_cache_misses":5},"permission_denied":{"enabled":false,"max_permission_denied":null,"denied_count":0},"model_api_mapping":{"available":true,"last_updated":"2026-07-16T06:03:36Z","providers":["openai","anthropic"],"models":{"openai":{"endpoints":{"responses":{"path":"/v1/responses","description":"Newer unified API for agentic, tool-using, and multimodal workflows. Required for GPT-5+ models; some older models (o1, o3, gpt-4o, gpt-4.1) support it alongside chat/completions."},"chat_completions":{"path":"/v1/chat/completions","description":"Legacy conversational API. Being deprecated for newer models by end of 2026."}},"models":[{"family":"gpt-5.6","patterns":["gpt-5.6*"],"endpoints":["chat_completions","responses"],"notes":"Supports both endpoints. Includes gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna per OpenAI SDK ChatModel."},{"family":"gpt-5.5","patterns":["gpt-5.5*"],"endpoints":["responses"],"notes":"Responses API only. Not present in OpenAI OpenAPI spec v2.3.0; reserved for future gpt-5.5 family models."},{"family":"gpt-5.4","patterns":["gpt-5.4*"],"endpoints":["chat_completions","responses"],"notes":"Supports both endpoints on OpenAI. Includes gpt-5.4-mini, gpt-5.4-nano. NOTE: gpt-5.4-mini is not accessible via /chat/completions on the Copilot API (responses-only via Copilot); the api-proxy handles this by falling back to the next alias candidate."},{"family":"gpt-5.3","patterns":["gpt-5.3*"],"endpoints":["chat_completions","responses"],"notes":"Supports both endpoints. Only gpt-5.3-chat-latest confirmed in OpenAPI spec; gpt-5.3-codex variants not present."},{"family":"gpt-5.2","patterns":["gpt-5.2*"],"endpoints":["chat_completions","responses"],"notes":"Supports both endpoints. Includes gpt-5.2-pro."},{"family":"gpt-5.1-codex-max","patterns":["gpt-5.1-codex-max"],"endpoints":["responses"],"notes":"Responses API only (ResponsesOnlyModel)."},{"family":"gpt-5.1","patterns":["gpt-5.1*"],"endpoints":["chat_completions","responses"],"notes":"Supports both endpoints for gpt-5.1, gpt-5.1-codex, gpt-5.1-mini."},{"family":"gpt-5-codex/pro","patterns":["gpt-5-codex","gpt-5-pro","gpt-5-pro-*"],"endpoints":["responses"],"notes":"Responses API only for gpt-5-codex and gpt-5-pro (ResponsesOnlyModel). Includes dated variants like gpt-5-pro-2025-10-06."},{"family":"gpt-5","patterns":["gpt-5","gpt-5-*"],"endpoints":["chat_completions","responses"],"notes":"Supports both endpoints for gpt-5, gpt-5-mini, gpt-5-nano."},{"family":"o4-mini-deep-research","patterns":["o4-mini-deep-research","o4-mini-deep-research-*"],"endpoints":["responses"],"notes":"Responses API only (ResponsesOnlyModel). Includes dated variants like o4-mini-deep-research-2025-06-26."},{"family":"o4","patterns":["o4*"],"endpoints":["chat_completions","responses"],"notes":"Reasoning model. o4-mini supports both endpoints (ModelIdsShared)."},{"family":"o3-pro/deep-research","patterns":["o3-pro","o3-pro-*","o3-deep-research","o3-deep-research-*"],"endpoints":["responses"],"notes":"Responses API only. o3-pro and o3-deep-research are in ResponsesModel but not ChatModel."},{"family":"o3","patterns":["o3*"],"endpoints":["chat_completions","responses"],"notes":"Reasoning model. o3, o3-mini support both endpoints. o3-pro/deep-research use responses only (covered by separate entry above)."},{"family":"o1-pro","patterns":["o1-pro","o1-pro-*"],"endpoints":["responses"],"notes":"Responses API only. o1-pro is in ResponsesModel but not ChatModel."},{"family":"o1","patterns":["o1*"],"endpoints":["chat_completions","responses"],"notes":"Reasoning model. o1, o1-mini, o1-preview support both endpoints. o1-pro uses responses only (covered by separate entry above)."},{"family":"computer-use-preview","patterns":["computer-use-preview*"],"endpoints":["responses"],"notes":"Responses API only. Specialized model for computer use tasks."},{"family":"codex-mini","patterns":["codex-mini*"],"endpoints":["chat_completions","responses"],"notes":"Supports both endpoints. codex-mini-latest is in ChatModel."},{"family":"chatgpt-4o-latest","patterns":["chatgpt-4o-latest"],"endpoints":["chat_completions","responses"],"notes":"Rolling alias for the latest ChatGPT-optimized GPT-4o model. Supports both endpoints (ModelIdsShared). Not matched by gpt-4o* pattern due to chatgpt- prefix."},{"family":"gpt-4o","patterns":["gpt-4o*"],"endpoints":["chat_completions","responses"],"notes":"Multimodal. Supports both endpoints. Includes audio and search preview variants."},{"family":"gpt-4.1","patterns":["gpt-4.1*"],"endpoints":["chat_completions","responses"],"notes":"Supports both endpoints. Includes gpt-4.1-mini, gpt-4.1-nano."},{"family":"gpt-4.5","patterns":["gpt-4.5*"],"endpoints":["chat_completions"],"notes":"Preview model. Chat completions only."},{"family":"gpt-4-turbo","patterns":["gpt-4-turbo*"],"endpoints":["chat_completions","responses"],"notes":"Legacy. In ModelIdsShared per OpenAI spec v2.3.0; supports both endpoints."},{"family":"gpt-4","patterns":["gpt-4","gpt-4-*"],"endpoints":["chat_completions","responses"],"notes":"Legacy. In ModelIdsShared per OpenAI spec v2.3.0; supports both endpoints."},{"family":"gpt-3.5-turbo","patterns":["gpt-3.5-turbo*"],"endpoints":["chat_completions","responses"],"notes":"Legacy. In ModelIdsShared per OpenAI spec v2.3.0; supports both endpoints. Deprecated."}]},"anthropic":{"endpoints":{"messages":{"path":"/v1/messages","description":"Unified Messages API for all Claude models. Supports tool use, streaming, extended thinking, and vision."}},"models":[{"family":"claude-fable-5","patterns":["claude-fable-5*"],"endpoints":["messages"],"notes":"Latest flagship. 1M token context."},{"family":"claude-mythos-5","patterns":["claude-mythos-5*","claude-mythos-preview*"],"endpoints":["messages"],"notes":"Invitation-only (Project Glasswing). Limited availability. 1M token context."},{"family":"claude-sonnet-5","patterns":["claude-sonnet-5*"],"endpoints":["messages"],"notes":"Best combination of speed and intelligence. 1M token context."},{"family":"claude-opus-4","patterns":["claude-opus-4*"],"endpoints":["messages"],"notes":"Includes claude-opus-4-8, claude-opus-4-7, claude-opus-4-6, claude-opus-4-5, claude-opus-4-1. 1M token context."},{"family":"claude-sonnet-4","patterns":["claude-sonnet-4*"],"endpoints":["messages"],"notes":"Includes claude-sonnet-4-6, claude-sonnet-4-5. Production workhorse. 1M token context."},{"family":"claude-haiku-4","patterns":["claude-haiku-4*"],"endpoints":["messages"],"notes":"Includes claude-haiku-4-5. Fast/cheap. 200K token context."},{"family":"claude-3.5-sonnet","patterns":["claude-3-5-sonnet*","claude-3.5-sonnet*"],"endpoints":["messages"],"notes":"Previous generation. 200K token context."},{"family":"claude-3.5-haiku","patterns":["claude-3-5-haiku*","claude-3.5-haiku*"],"endpoints":["messages"],"notes":"Previous generation fast model."},{"family":"claude-3.7-sonnet","patterns":["claude-3-7-sonnet*","claude-3.7-sonnet*"],"endpoints":["messages"],"notes":"Previous generation extended thinking model. 200K token context."},{"family":"claude-3-opus","patterns":["claude-3-opus*"],"endpoints":["messages"],"notes":"Previous generation. 200K token context."}]}},"error":null}}

Large diffs are not rendered by default.

15 changes: 14 additions & 1 deletion actions/setup/js/types/unified_session.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -228,10 +228,23 @@ export interface ModelRoutingOutcomeData {
export interface ModelInventoryData {
endpoints: {
provider: string;
models: string[];
/** AWF's endpoint configuration state; absent when the source did not supply it. */
configured?: boolean;
/** Advertised model IDs in source order; absent means unknown, not empty. */
models?: string[];
/** Routing metadata in source order; absent means unknown, not empty. */
routingModels?: {
modelId: string;
supportedEndpoints?: string[];
/** Identifier-shaped reasoning efforts; absent means unknown; a supplied empty array stays empty. */
supportedReasoningEfforts?: string[];
/**
* Positive context window size in tokens; absent means unknown.
* @asType integer
* @minimum 1
* @maximum 9007199254740991
*/
contextWindowTokens?: number;
}[];
}[];
modelsFetchComplete?: boolean;
Expand Down
4 changes: 3 additions & 1 deletion actions/setup/js/unified_session.test.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,8 @@ describe("Unified conclusion session", () => {
'{"providers":{"copilot":{"models":["a",false]}}}',
'{"provider":"copilot","models":["a",42]}',
'{"endpoints":[{"provider":"copilot","models":[],"routing_models":[{"model_id":42}]}]}',
'{"endpoints":[{"provider":"copilot","configured":true,"models":["a"],"routing_models":[{"model_id":"a","supported_reasoning_efforts":["very high"]}]}]}',
'{"endpoints":[{"provider":"copilot","configured":true,"models":["a"],"routing_models":[{"model_id":"a","context_window_tokens":0}]}]}',
])("warns once and skips malformed model inventories: %s", content => {
write("sandbox/firewall/models.json", content);
const warn = vi.fn();
Expand Down Expand Up @@ -232,7 +234,7 @@ describe("Unified conclusion session", () => {
],
[{ providers: {} }, { endpoints: [] }],
[{ provider: "copilot", models: [] }, { endpoints: [{ provider: "copilot", models: [] }] }],
[{ endpoints: [{ provider: "copilot", configured: false, models: null }] }, { endpoints: [{ provider: "copilot", models: [] }] }],
[{ endpoints: [{ provider: "copilot", configured: false, models: null }] }, { endpoints: [{ provider: "copilot", configured: false }] }],
])("keeps genuine empty or unconfigured preferred inventories instead of stale mirrors: %j", (inventory, expected) => {
write("agent/awf-reflect.json", inventory);
write("sandbox/firewall/awf-reflect.json", { endpoints: [{ provider: "stale", models: ["stale"] }] });
Expand Down
Loading
Loading