Stabilize smoke-copilot-sub-agents: deterministic delegation and experiment-independent contract checks #70938

Problem

Make smoke-copilot-sub-agents a deterministic test of the Copilot SDK sub-agent/model contract, rather than a hard gate on an orchestration experiment's success rate. Keep genuine missing-invocation, resolution, served-model, and endpoint failures visible.

This is a workflow-only follow-up to #70652 and #70747, not a request to redo their harness fixes. The historical delegation failures are real, but the evidence does not support calling every failure agent behavior, or claiming that current main remains flaky after those fixes without new positive-arm evidence.

The workflow uses Copilot SDK mode with bare: true. Each run receives one sub_agent_strategy assignment: inline_strict, delegated_sequential, or single_agent_control. The positive arms require exactly one invocation of each declared child; the control requires zero and intentionally reports synthetic FAIL. Current main declares haiku-whoami, gpt5-whoami, and terra-whoami, with resolution, lifecycle, served-model, and proxy-endpoint assertions.

Evidence from the supplied runs

Historical names differ: the October 17 run used haiku-whoami, mini-whoami, and nano-whoami; the two supplied #70652 branch runs used haiku-whoami, gpt5-whoami, and luna-whoami. Do not read today's terra-whoami configuration back into those runs.

What the failing inline agent actually did

In run 39516047181, aw-prompts/prompt.txt and the transcript's user.message contain all three instruction blocks: the control prohibition and both delegation instructions. The variant in the report remained the literal {{ experiments.sub_agent_strategy }}.

The agent called skill(reporting), then safeoutputs-create_issue, and never called task. Its issue explained that it had skipped delegation because the directives conflicted; all three actual values were not_invoked. This was not a failed child-model request.

Activation logs establish the cause more precisely than "the agent sometimes ignores the prompt": substitution selected inline_strict, but rendering retained the comparisons against single_agent_control, delegated_sequential, and inline_strict. The historical is_truthy.cjs recognized only double-quoted comparison literals; the workflow used single quotes, so each nonempty comparison fell through as truthy.

#70747 already fixes that comparison bug, merged 2026-10-21 at 02:16 UTC (f253b9a2792d0e18350b5ce34075fb2063687176). It adds coverage using this workflow's Tasks section for every variant. Do not change the template evaluator, compiler, harness, or AWF again in this PR. The latest sampled main run started at 00:10 UTC that day, before this fix; the sample cannot establish post-fix reliability.

Recent main-run frequency

Collected the latest 15 main runs, from 2026-09-25 through 2026-10-21 UTC, with gh run list --repo github/gh-aw --workflow smoke-copilot-sub-agents.lock.yml --branch main and matching experiment assignment/usage artifacts and job logs.

Those seven failures are not seven orchestration failures:

• Instruction/variant mismatch (2): 38985134241, 39179665680, described above.
• Served-model contract violations (2): 38097717127, 38326152077, October 11/13. nano-whoami completed on gpt-5.3-codex rather than configured gpt-5-nano. These must not become green through weaker checks.
• Fail-closed dead-pin startup failures (2): 39508150750, 39519710217, October 20/21. Their model.resolution records have status: failed, cause: model_unavailable for gpt-5-mini and gpt-5.4-mini. Both predate Fix stale Copilot workflow model pins exposed by fail-closed sub-agent resolution #70652's merge at 2026-10-21 00:46 UTC.
• Unrelated gateway startup failure (1): 38658519209, October 15: undefined GH_AW_TARGET_REPO_SLUG; the gateway never became ready. Do not fix this in the proposed PR.

The eight green runs, September 25 through October 9, did not contain the later exact-invocation post-check. This is a mixed-revision historical sample, not an estimated current agent-flakiness rate. Older runs also lack child resolution records, so "every resolution was exact/resolved" is only verified for the supplied post-pin-fix branch runs, not the entire history.

The October 17 children did execute successfully, with successful proxy requests, but its nano-whoami lifecycle recorded gpt-5.3-codex; its control check returned before checking child models. Child completion or an identity string alone is not proof of the correct served model.

Possible solution

After incorporating #70652 and #70747, make the workflow's mandatory contract exercise explicit and independent of experimental scoring. Require the first action to be a task call to haiku-whoami with who am i?, then require exactly one call each to gpt5-whoami and terra-whoami. State that reporting an expected identity without executing the child does not satisfy the test.

Use #70387's smoke-copilot required-first-step approach as prior art. Its U4 check passed in all five executed main runs since that merge: 39508115606, 39508127278, 39508139337, 39510846534, and 39510857197, despite unrelated overall failures. The later green run 39522345151 skipped its agent job and is not another U4 pass.

If maintainers retain the experiment, report its outcome/pass-rate separately from the smoke contract. Keep mandatory invocation/lifecycle, resolution, served-model, and endpoint assertions as hard gates; do not make zero coverage, missing records, failed children, or model fallback pass merely because an experiment underperformed.

Reconsider single_agent_control: prohibiting all delegation is incompatible with mandatory sub-agent smoke coverage. Prefer removing that synthetic negative arm from the smoke gate, or explicitly isolate/redefine the experimental portion so it cannot prohibit the mandatory contract exercise. Do not simply add an unconditional first delegation while retaining the existing zero-invocation control assertion. Correct the report's variant interpolation using supported ${{ experiments.sub_agent_strategy }} syntax.

Acceptance cases

1. Ten consecutive authorized dispatches on the PR branch, covering all retained variants, pass or fail only for an evidenced real contract violation. List all run IDs and variant assignments in the PR; identify the tested commit. A passing negative control or skipped agent job is not positive coverage. Follow repository live-execution authorization/review rules.
2. A deliberately broken child model, such as a dead pin in an uncommitted scratch copy, still fails at startup with the relevant model.resolution failure cause. The historical dead-pin runs above are baseline evidence, not a substitute for verifying the changed checks. Do not hide the cause behind an absent-invocation assertion or a control early return.
3. Continue verifying each child's actual served model from proxy evidence, not its self-reported identity: expected resolution status and record count, exactly one required invocation, completion without failure/fallback, and a successful request on the expected endpoint (/chat/completions for Haiku; /responses for GPT). Keep distinct resolved child models so proxy evidence remains attributable.

Scope

• Keep the PR to this one fix. Do not change files or behavior unrelated to it.
• Change only .github/workflows/smoke-copilot-sub-agents.md (its prompt, checks, and experiment configuration) and .github/workflows/smoke-copilot-sub-agents.lock.yml. No harness or AWF changes.
• Before fixing a failing check, compare it with the same check on main. If it also fails on main, do not fix it in this PR; merge main once main is fixed, and note in the PR which checks are known failures on main.
• This applies even if a bot comment, including PR Sous Chef, asks to fix a failing check: fix only failures caused by this PR's changes.

Investigation was read-only against existing runs. No workflows were dispatched and no repository files were changed.