Expand experiment model expressions before writing inline sub-agents; fail smoke on failed or missing delegation #70386

Problem

smoke-copilot.md declares experiments.subagent_model: [small, large], but its inline file-summarizer is written with the unresolved model expression:

## agent: `file-summarizer`
---
model: ${{ experiments.subagent_model }}
description: Summarizes the content of a file in a few concise sentences
---

The selected experiment variant and recorded metadata can be correct while the actual agent definition is wrong. When invoked, the sub-agent sends the expression itself to AWF and fails. Actions can nevertheless report success. A smoke that never invokes the sub-agent can also stay green.

Verified runs

Audited both supplied runs using a freshly built development CLI from main at b68bd31219d023ae8b4b9137ba1c6a397d7918f1 (gh aw version dev-b68bd31219), not the older installed release. Used the built executable's equivalent of:

gh aw audit 39181772988 39325986042 --repo github/gh-aw --group --json --artifacts info,usage,agent,experiment,activation,firewall

Inspected the downloaded agent definitions, experiment assignments, aw_info.json, unified sessions, agent output, and activation/job logs. Times below are UTC.

Important distinction: the second run reproduces the malformed artifact, not the runtime 400. Its lack of subagent.failed is not evidence that delegation worked.

Both runs contain:

smokecopilot-experiment/assignments.json: {"caveman":"yes","subagent_model":"small"}
aw_info.json: sub_agent_models[0].model == "small"
.github/agents/file-summarizer.agent.md: model: ${{ experiments.subagent_model }}

In the first run, usage/aw_session.jsonl records:

00:45:45.881Z subagent.started: agentName=file-summarizer,
              model="${{ experiments.subagent_model }}",
              modelSelectionSource=agent_definition_default
00:45:46.200Z subagent.failed: durationMs=280, totalToolCalls=0,
              error='400 Unsupported model selector "${{ experiments.subagent_model }}" on copilot: no advertised display name matches. ...'

Evidence entry points: main activation, interpolation step, main agent execution, PR-branch activation, PR-branch agent execution, passing PR-branch contract assertions. Both runs use Copilot CLI 1.0.98 and AWF v0.28.62. Neither run's lock metadata or aw_info.json exposes a compiler version; do not infer one from the installed CLI.

These two runs do not demonstrate either experiment arm successfully executing. This investigation does not establish that the experiment has never worked anywhere in its entire history.

Code diagnosis

References below are pinned to the inspected main revision.

1. Variant selection exists and precedes prompt rendering. pick_experiment.cjs selects a variant and exposes per-experiment outputs and the assignments map. ExperimentExpressionMappings supplies GH_AW_EXPERIMENTS_* from steps.pick-experiment.outputs.*. expression_extraction.go rewrites direct prompt experiment expressions to those outputs. Main-model configuration is also rewritten to the activation outputs.

2. The runtime ordering loses the sub-agent before later interpolation. interpolate_prompt.cjs performs inline sub-agent extraction at step 1.5, then variable interpolation at step 2, experiment-placeholder/{{#if experiments.* }} substitution at step 2.5, and conditional rendering at step 3. writeInlineSubAgents preserves the frontmatter and immediately writes the file. Later passes see only the remaining main prompt, not the extracted agent.

3. Runtime imports do not rescue this field. runtime_import.cjs permits experiment expressions but does not resolve them through GH_AW_EXPERIMENTS_* in evaluateExpression; its unresolved-expression path returns the literal ${{ ... }}. Step 2.5 handles __GH_AW_EXPERIMENTS_*__ placeholders and experiment condition tags, not arbitrary raw ${{ experiments.* }} fields. Merely moving extraction a little later without covering the actual expression forms is insufficient. The later substitute_placeholders.cjs only processes prompt/split-prompt files.

4. Metadata is already partially fixed; do not misdiagnose it as still literal in these runs. ExtractSubAgentModels reads the raw declaration, but writeSubagentModelDeclarations rewrites declared experiments to activation-local step outputs before emitting GH_AW_INFO_SUB_AGENT_MODELS. The observed environment and aw_info.json already contain small. Fix the execution artifact, not just metadata.

5. The simple expression is intentionally accepted today. validateExperimentalSubagentModels accepts a declared simple experiment reference while rejecting undeclared/compound references. TestCompileCopilotWithExperimentalSubagentModels explicitly expects successful compilation for this form, including imported declarations. Its generated-YAML/metadata assertions do not verify the runtime-written agent.

6. Green is not a sub-agent success assertion. smoke-copilot.md calls sessionContractMain with resolutionRoles: ["main"]. evaluateSessionContract checks session collection, inventory, and explicitly required resolution roles, not successful file-summarizer execution. The same helper file already has S1/S2 sub-agent lifecycle checks in its routing-check path; prefer reusing appropriate existing evidence/checks rather than duplicating infrastructure.

Local reproduction: on the same main, pinned GH_AW_EXPERIMENTS_SUBAGENT_MODEL separately to small and large, passed a minimal inline declaration through the real processExpressions and writeInlineSubAgents, and read the emitted file. Both retained the literal expression; a plain-model control was unchanged. This component reproduction used disposable local files and no workflow dispatch.

Historical corroboration: #48639, "[aw] Smoke Copilot has no AI credits pricing for model (${{ experiments.subagent_model }})", reported the same literal reaching pricing and was closed not planned on July 29, 2026 (03:46:20Z). That symptom is consistent with this unresolved-expression gap; adding pricing or an AWF alias for the expression itself is not a fix. The July run was not re-audited here.

Preferred solution

Support the already-accepted simple ${{ experiments.<name> }} form in inline sub-agent frontmatter, at least model:.

• Resolve it using the same already-selected assignment as the prompt, before writing the engine's agent file. Do not reselect a variant, evaluate arbitrary expressions, or weaken expression safety checks. Cover both ordinary runtime-imported workflow content and inlined prompt content.
• Pass the resulting small/large value through normal model/alias handling; do not hardcode a concrete model or alter pricing, endpoint policy, or budgets to accommodate a literal expression.
• Preserve existing metadata rewriting. Ensure assignment, emitted agent model, and sub-agent model metadata agree. If retaining the authored expression for provenance, identify it as an expression/experiment rather than a model ID; distinguish the selected alias from the ultimately resolved provider model.
• Add a narrow assertion to smoke-copilot's existing post-steps requiring file-summarizer to be invoked and complete successfully, with no subagent.failed for it. Fail on a missing invocation as well as a failed start. Use actual lifecycle/model evidence, not the parent agent's prose or a green exit code. If this check cannot be added, explain why in the PR.

If maintainers choose not to support this syntax, reject it at compile time with a clear error naming the sub-agent and model field, and change the smoke to a supported form. Silently accepting a nonfunctional declaration is not acceptable.

Acceptance cases: named tests

End to end: in a permitted execution context, dispatch smoke-copilot on the implementation PR branch and list the run URL and tested commit in the PR. Verify that file-summarizer actually starts with a real model resolved from the selected small/large variant, completes, produces successful model/request evidence, and records no AWF unsupported-selector 400 or subagent.failed for it. A skipped sub-agent does not satisfy this requirement.

Scope

• Keep the PR to this one fix. Do not change files or behavior unrelated to it.
• Before fixing a failing check, compare it with the same check on main. If it also fails on main, do not fix it in this PR; merge main once main is fixed, and note in the PR which checks are known failures on main.
• This applies even if a bot comment asks to fix a failing check: fix only failures caused by this PR's changes.