Daily Model Inventory Checker: deterministic workflow model-pin check against the live Copilot catalogue

[Copilot]

Nothing checked gh-aw's own workflow model pins against what Copilot still offers. When claude-opus-5 and claude-opus-4.8 were retired on 16 Oct, Design Decision Gate failed on every PR. daily-action-setup-security-audit was silently downgraded to Haiku and lost its threat-detection verdict, yet still showed green. The Daily Model Inventory Checker reads the live catalogue every day but never looked at pins, and its findings were agent judgement, not a check.

This PR adds a checked-in, unit-tested pin checker and wires it into that workflow.

Checker: actions/setup/js/model_pin_check.cjs

• Collection. It collects every model name configured by .github/workflows/*.md and records the workflow, source file and field for each:

  • Fields: model; engine.{model, fallback-models, provider.model, models.*, model-routing.allowed-models}; safe-outputs.threat-detection.{model, engine.*, engine-config.*}; safe-outputs.assign-to-agent.model; evals[].model, evals.model and evals.questions[].model; enclaves[].agent.model.
  • Imported .github/agents/*.md files and inline ## agent: blocks.
  • Non-reserved models: alias definitions from the workflow and its imports. permissions.models and models.providers.* are excluded.
  • Local imports are followed with the compiler's nested-path rules (determineNestedBaseDir). Remote and @builtin: imports are skipped and listed.

• Provider scope. A copilot/ (or other) prefix on the model wins. Otherwise the engine's provider applies:

  • copilot: Copilot.
  • claude, codex and pi: the main model's prefix if it has one, else Anthropic, OpenAI and Copilot respectively.
  • Explicit model-provider or provider.
  • The provider of an imported engine definition.
  • BYOK when engine.env.COPILOT_PROVIDER_BASE_URL is set.

• Classification (in this order):

  1. Any ${{ … }} → not checked, reason expression.
  2. inherited → not checked, sentinel (the parent's pin has its own row).
  3. BYOK or a non-Copilot provider → not checked, reason provider not in the Copilot catalogue.
  4. auto where the gateway does the selection (every engine except copilot, and assign-to-agent) → not checked, sentinel.
  5. Otherwise resolveConfiguredModel runs with provider: "copilot":
     • An alias that resolves → alias resolves, showing the resolved model.
     • A resolver error → unresolvable with the resolver's cause.
     • An exact result is also tested with isLiveModelId(base, buildProviderCatalog(...)):
       • live → exact
       • not live → unresolvable, model_unavailable
       • empty catalogue → unresolvable, catalog_unavailable

  resolve_model_alias.cjs and model_aliases.json are unchanged.

• Output. The checker writes three files to --out-dir:

  • model-pins-summary.md is the issue section: "Action required" first when any pin is unresolvable, then the catalogue size, counts and not-checked reasons. When the catalogue is unavailable it says so in one line instead of listing about 170 catalog_unavailable rows.
  • model-pins.md lists every pin, about 38 KB today. That is too large to sit in the issue alongside the rest of the report, so it goes in the artifact.
  • model-pins.json holds the same data with the counts.

node actions/setup/js/model_pin_check.cjs --reflect <reflect.json> --out-dir <dir> [--repo <root>]

Workflow: daily-model-inventory.md

• Step 0.5. The agent runs the checker right after the Step 0 /reflect fetch. It pastes model-pins-summary.md verbatim at the top of the issue and must not classify pins itself.
• Title rule. If counts.unresolvable > 0, the title is Model alias inventory - action required - YYYY-MM-DD, never "no changes needed". max: 1 and close-older-issues are unchanged.
• Post-step. After the agent, the same script runs again. It reads the harness's own capture (/tmp/gh-aw/agent/awf-reflect.json, then sandbox/firewall/awf-reflect.json) and falls back to Step 0's file. It appends model-pins.md to the step summary and uploads the model-pins artifact.
• Why both. /reflect is reachable only from inside the sandbox, and the agent has to put the result in the issue. The post-step makes the artifact and step summary independent of the agent.
• The existing alias and pricing analysis is unchanged.

Tests: model_pin_check.test.cjs

They use the recorded /reflect from Design Decision Gate run 38945917145 (fixtures/model_resolution_reflect.cjs) and small temporary fixture workflows. They cover the seven acceptance cases:

1. copilot/claude-opus-5 → unresolvable model_unavailable, listed under "Action required" with its file and field.
2. copilot/opus → resolves to copilot/claude-opus-5.5.
3. gpt-5-mini → exact.
4. ${{ vars.GH_AW_MODEL_AGENT_COPILOT }} → not checked, expression.
5. claude engine pinning claude-sonnet-4-6 → not checked, provider not in the Copilot catalogue.
6. A models: alias whose every candidate is gone → unresolvable.
7. No pins → empty table and no "Action required" section.

They also cover multi-field and import collection, nested imports, BYOK, inherited, copilot/ prefix precedence, an unavailable catalogue and the CLI's three output files.

Findings on main (not fixed here)

Against the recorded 40-model catalogue the checker finds 278 pins: 135 exact, 31 aliases that resolve, 4 unresolvable and 108 not checked.

Workflow	Field	Configured	Cause
daily-action-setup-security-audit.md	model	claude-opus-4.8	model_unavailable
smoke-copilot-sub-agents.md	## agent: 'nano-whoami' model	gpt-5-nano	model_unavailable
smoke-github-claude.md	engine.fallback-models[1]	copilot/claude-sonnet-4.6	model_unavailable
video-analyzer.md	model	copilot/gemini-3.7-flash	model_unavailable

The two o4-mini-aw Azure pins are not checked (BYOK). The inherited pin in dependabot-burner.md is not checked under the sentinel rule; it is still unconfirmed whether the Copilot CLI treats it as "use the parent model".

Not yet done

The end-to-end run has not been dispatched. A maintainer needs to run gh workflow run daily-model-inventory.lock.yml --ref <branch> and check the resulting issue and the model-pins artifact.

• Fixes Daily Model Inventory Checker: deterministically check workflow model pins against the live Copilot catalogue #70289

[SivaKesava1]

End-to-end verification at 5e56aa44b9: Daily Model Inventory Checker dispatched on this branch, run 39294514193 (success).

• The agent ran Step 0.5 and created [model-inventory] Model alias inventory - action required - 2026-10-19 #70312, titled Model alias inventory - action required - 2026-10-19. Its body starts with the checker's ### Action required section verbatim. close-older-issues closed [model-inventory] Model alias inventory update - 2026-10-18 #70050.
• The model-pins artifact has model-pins-summary.md, model-pins.md (38 KB) and model-pins.json. Counts: 278 pins, 135 exact, 31 alias, 4 unresolvable, 108 not checked, against a 40-model live catalogue.
• The 4 unresolvable pins are the ones listed in Daily Model Inventory Checker: deterministically check workflow model pins against the live Copilot catalogue #70289: daily-action-setup-security-audit.md claude-opus-4.8, smoke-copilot-sub-agents.md gpt-5-nano, smoke-github-claude.md engine.fallback-models[1] copilot/claude-sonnet-4.6, and video-analyzer.md copilot/gemini-3.7-flash. Both o4-mini-aw Azure pins and the inherited sub-agent pin are not checked.

One change requested:

• The post-step must not be able to fail the agent job. "Check workflow model pins" runs in the agent job with set -euo pipefail and no continue-on-error. If the checker crashes (for example on a malformed workflow file or reflect capture), the agent job fails, and safe_outputs then skips the whole inventory issue, including the alias and pricing analysis, not just the pin section. Add continue-on-error: true to that step and keep the artifact upload if: always(), then recompile.

CI is green (JS shards 1–4, typecheck, lint, stale lock files).

[Copilot]

🟡 Changes recommended

Host execution of agent-writable code and incorrect model-pin classifications remain unresolved.

10 open issues

• Post-step executes agent-modifiable checkout code on the host · New
• Imported aliases incorrectly override built-in aliases · New
• Preference-only engine incorrectly overrides imported engine ID · New
• Model precedence ignores imported defaults and execution model · New
• Detection scope ignores imported detection configuration · New
• Detection engine drops inherited BYOK configuration · New
• Detector provider uses sibling model over engine-specific model · New
• Detector provider ignores effective detection engine selection · New
• Eval model provider is not recomputed from eval override · New
• Expression-based model aliases falsely report unavailable models · New

What changed in this PR

Adds deterministic workflow model-pin validation to gh-aw’s daily inventory using the live Copilot catalogue.

Changes:

• Collect and classify workflow pins, producing JSON and Markdown reports.
• Add fixture-based coverage for collection and resolution.
• Integrate issue summaries, action-required titles, and report artifacts.

File	Description
actions/​setup/​js/​model_pin_check.test.cjs	Tests pin collection and classification.
actions/​setup/​js/​model_pin_check.cjs	Implements deterministic checking and reporting.
.github/​workflows/​daily-model-inventory.md	Integrates checks and reporting instructions.
.github/​workflows/​daily-model-inventory.lock.yml	Regenerates executable workflow steps.

_{Review effort: Max}

[Copilot]

End-to-end verification at 5e56aa44b9: Daily Model Inventory Checker dispatched on this branch, run 39294514193 (success).

• The agent ...

Done in 64f48c4: "Check workflow model pins" now has continue-on-error: true, and the model-pins upload stays if: always(). I merged main and ran make recompile; the only lock-file diff is the new key plus the frontmatter hash. The PR description now includes the E2E run, #70312 and the artifact counts.

[gh-aw-bot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
2. Review (.github/workflows/daily-model-inventory.md:357): This post-step executes the checkout's script and its required modules on the runner host after the agent exits. The checkout is writable by the agent, which reads external content and has unrestricted bash access. A prompt-injected agent could replace this code, escape the sandbox's network restrictions, and alter /tmp/gh-aw/agent_output.json after ingestion but before upload. Run ${RUNNER_TEMP}/gh-aw/actions/model_pin_check.cjs instead: Setup Scripts stages it and its dependencies before execution, and that directory is mounted read-only in the sandbox. Regenerate daily-model-inventory.lock.yml afterward. - Daily Model Inventory Checker: deterministic workflow model-pin check against the live Copilot catalogue #70295 (comment)
3. Review (actions/setup/js/model_pin_check.cjs:337): Imported aliases can overwrite built-ins here, unlike MergeImportedModelAliases in pkg/workflow/model_aliases.go:193–204, which only adds imported names that are not already defined. For example, an import defining sonnet: [copilot/claude-opus-5] makes a workflow's model: sonnet report model_unavailable, although execution uses the live built-in alias. Preserve built-in aliases when merging imports; only the main workflow may override them. Add a regression case for this precedence. - Daily Model Inventory Checker: deterministic workflow model-pin check against the live Copilot catalogue #70295 (comment)
4. Review (actions/setup/js/model_pin_check.cjs:351): A preference-only engine: {model: claude-sonnet-4-6} creates a non-null mainEngine without an ID, so this condition ignores an imported engine: claude. The next statement then defaults the engine to Copilot and falsely reports the valid Anthropic model as unavailable. Let imports supply the missing engine ID while retaining the main workflow's model preference, matching mergeImportedEngineConfig in pkg/workflow/compiler_orchestrator_engine.go. Add a fixture for this combination. - Daily Model Inventory Checker: deterministic workflow model-pin check against the live Copilot catalogue #70295 (comment)
5. Review (actions/setup/js/model_pin_check.cjs:355): This reverses the compiler's model precedence and ignores imported model defaults. resolveEngineModel prefers engine.model over top-level model, and mergeImportedEngineConfig supplies an imported model when the main workflow has none. With a Claude engine, top-level model: anthropic/claude-sonnet-4-6, engine.model: copilot/gpt-5-mini, and bare fallback claude-opus-5, this checker skips the retired fallback as Anthropic even though execution sends it to Copilot. Resolve the effective model using both precedence rules before deriving mainProvider, and cover explicit overrides and imported defaults in tests. - Daily Model Inventory Checker: deterministic workflow model-pin check against the live Copilot catalogue #70295 (comment)
6. Review (actions/setup/js/model_pin_check.cjs:358): Detection scope is derived only from the main file, but imported detection pins later reuse it. A workflow with engine: claude importing safe-outputs.threat-detection: {engine: copilot, model: claude-opus-5} therefore skips the retired pin as Anthropic, although MergeSafeOutputs adopts the imported Copilot detector. Select the explicit detection configuration from the main workflow or its imports before computing this scope, preserving main-workflow overrides, including disabled detection. Add a regression fixture. - Daily Model Inventory Checker: deterministic workflow model-pin check against the live Copilot catalogue #70295 (comment)
7. Review (actions/setup/js/model_pin_check.cjs:361): An explicit detection engine drops inherited BYOK settings even when both engines are Copilot. With engine.env.COPILOT_PROVIDER_BASE_URL set and threat-detection: {engine: copilot, model: o4-mini-aw}, describeEngine('copilot') sets byok: false, causing a false model_unavailable alert for the Azure deployment. At runtime, mergeThreatDetectionEngineEnv inherits the main environment when the effective engine IDs match, then applies detection-specific overrides. Apply that merge before deciding whether detection targets BYOK, including explicit empty endpoint overrides, and add regression coverage. - Daily Model Inventory Checker: deterministic workflow model-pin check against the live Copilot catalogue #70295 (comment)
8. Review (actions/setup/js/model_pin_check.cjs:362): The detector's engine.model overrides its sibling model in parseThreatDetectionEngineField, but this expression gives the sibling priority. For an explicit Claude detector with sibling model: anthropic/claude-sonnet-4-6, engine.model: copilot/gpt-5-mini, and a bare retired fallback, this infers Anthropic and skips the fallback even though it runs against Copilot. Prefer the engine-specific model when deriving the detector's provider, and test the conflicting-model case. - Daily Model Inventory Checker: deterministic workflow model-pin check against the live Copilot catalogue #70295 (comment)
9. Review (actions/setup/js/model_pin_check.cjs:363): Detection does not necessarily use the main engine. For a Claude workflow with main model copilot/gpt-5-mini and detection override model: claude-opus-5, getThreatDetectionEngineID selects Copilot from the main provider. This code instead infers Anthropic from the bare detection override and skips the retired pin. Custom engines can also declare a separate detection-engine (for example Cursor selects Copilot), and Pi is normalized to a supported detector. Resolve the effective detector using those rules before constructing its provider scope, with regression cases for provider-based and custom-engine selection. - Daily Model Inventory Checker: deterministic workflow model-pin check against the live Copilot catalogue #70295 (comment)
10. Review (actions/setup/js/model_pin_check.cjs:437): evals.model replaces the main model before the eval job resolves its provider (pkg/workflow/evals_steps.go:204–210), so it cannot always reuse mainScope. With engine: claude, main model: copilot/gpt-5-mini, and evals.model: claude-sonnet-4-6, execution uses Anthropic, but this checker reports the valid eval pin as unavailable in Copilot and triggers Action required. Recompute the provider from the eval override while retaining explicit provider and BYOK settings, and add a regression case. - Daily Model Inventory Checker: deterministic workflow model-pin check against the live Copilot catalogue #70295 (comment)
11. Review (actions/setup/js/model_pin_check.cjs:534): The expression check only inspects the configured name for ordinary model pins. With model: dynamic and models.dynamic: ['copilot/${{ vars.MODEL }}'], the alias-definition row is not checked, but the model row tries to resolve the literal expression and reports model_unavailable. Alias entries support runtime expressions, so this can falsely trigger Action required for a live model. Follow the applicable alias references and mark expression-dependent pins as not checked, preserving live-exact-ID precedence and ordered fallbacks. Add direct and nested expression-alias cases. - Daily Model Inventory Checker: deterministic workflow model-pin check against the live Copilot catalogue #70295 (comment)

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: 64f48c4
Sous-chef work: 0baa287c690ac7de96c331969daf1ee08dfd7b390a6b5c2743f3e2530f2b07ae 0d1bc693d61309750235f61bb066670b78d6924f3d8328e241cc0844dd3442c4 1b6bde0056c178add35c9a35147d809f91bf6dc5f5e39b07ab5f641333efaa3c 28d0216d6ef45afaaa8420dcc169d4871f1599570bfd2711e3f4f6006ab7c0db 5396834d91fecd7155871a3524ac1ec2edeb5665e901005b8ab3778b905338c8 68a9cab26964c6e06142e0f4324047af179d5b6b9c71b393316cd4e94d7abd8e a40fd78f490e97c409a76c6f9f161be881687b624dadb6f899238f7f67902bdb a7edf1844f7db6b69d7ff04a38b10344ebc3951317bf5faff9473131c9032e70 e1b367afb23bf07845836a61ba44c92944ff9d24a0efb20dada282bcb5b5e034 f1467e0101266c7e25e80583f97ec99713b9c12d5df1421c03d78743432d460a
Sous-chef state: 135e0555e2cc37bbf0a2bbecbddfac05ff94416b1ac00610c602197621e54509

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 7.25 AIC · ⌖ 13 AIC · ⊞ 1.1K · ◷
_{Comment /souschef to run again}

[Copilot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes....

Merged the latest main, fixed the requested checker and post-step issues, regenerated the workflow lock, and added regression coverage. Commit: c4e2397.

[gh-aw-bot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
2. Fix failing check JS Tests (shard 2/4) (FAILURE): https://github.com/github/gh-aw/actions/runs/39308992113/job/118258506851.

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: c4e2397
Sous-chef work: 043185828d65a534dd8b2b6b4d00e0d47bf21a2940d3f277c0ed15da7f668648
Sous-chef state: 0b08942039fd7bcbccbd08f8cf493e2b4b238606e4c52740f61753914c914233

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 3 AIC · ⌖ 6.06 AIC · ⊞ 1.1K · ◷
_{Comment /souschef to run again}

[Copilot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes....

Updated the fixture to expect mini54-whoami with gpt-5.4-mini, matching the workflow; the shard-2 test now passes locally. main was already up to date. Committed as 9f39165.

[SivaKesava1]

End-to-end verification at 27be880983: Daily Model Inventory Checker dispatched on this branch, run 39322171512, all jobs succeeded.

• It created [model-inventory] Model alias inventory - action required - 2026-10-19 #70367 (Model alias inventory - action required - 2026-10-19) with the checker's section verbatim, and closed [model-inventory] Model alias inventory - action required - 2026-10-19 #70312.
• The model-pins artifact, checked against the live 36-model catalogue, reports 278 pins: 98 exact, 47 alias, 25 unresolvable and 108 not checked.
• The results match main after today's Copilot catalogue change (gpt-5-mini, gpt-5.4, gpt-5.4-mini, gpt-5.5 and grok-4.5 removed):
  • Alias pins rose from 32 to 47: the 15 workflows that fix: use model aliases for pins Copilot no longer offers #70342 moved to gpt-5/grok now resolve.
  • The 25 unresolvable pins are exactly the ones that need exact ids: fallback models, routing allowed-models, Copilot and Pi sub-agent models, the non-alias engines (aider, crush, goose, opencode, pydantic-ai, deepseek) plus engine-conformance-pi, and video-analyzer.
  • The pins retired earlier and fixed by fix: replace retired model pins in three workflows #70328 no longer appear.
• All review threads are answered and resolved. I verified the expression-alias case locally: model: dynamic with models.dynamic: ['copilot/${{ vars.MODEL }}', …] reports both rows as not checked (expression), and model_pin_check.test.cjs passes 22/22.

Ready for review.

[github-actions]

🎉 This pull request is included in a new release.

Release: v0.92.6