Repository navigation
Copilot engine silently runs claude-sonnet-5 when the configured model (gemini-3.8-flash) is not available #66894
Description
Summary
A workflow configured with engine: copilot and model: gemini-3.8-flash never actually runs on Gemini. At startup the Copilot CLI cannot find the model in the CAPI /models list and silently falls back to claude-sonnet-5. The run still succeeds, and every gh-aw surface (aw_info.json, the lock-file metadata, gh aw audit) keeps reporting gemini-3.8-flash. The fallback is invisible unless you dig through the CLI process logs.
Root cause: CAPI does not offer Gemini models to the agentic-workflows integration ID, which gh-aw always sets for the Copilot engine (GITHUB_COPILOT_INTEGRATION_ID: agentic-workflows). Gemini is enabled in our org's Copilot policy and is available to the same user under other integrations.
Environment
- gh-aw v0.89.21 (lock compiled with
engine_versions.copilot: 1.0.87; session reports CLI 1.0.87 and 1.0.90) engine.id: copilot,engine.model: gemini-3.8-flash,engine.args: [--reasoning-effort, high]permissions: copilot-requests: write→COPILOT_GITHUB_TOKEN: ${{ github.token }},S2STOKENS: true,GH_AW_LLM_PROVIDER: github- Runner:
ubuntu-24.04-4core
What happens
Harness (agent-stdio.log):
[copilot-harness] auto-configuring COPILOT_PROVIDER_WIRE_API=completions for model gemini-3.8-flash
[copilot-harness] inference routing: mode=cli configuredModel="gemini-3.8-flash" endpoint=managed-by-copilot-cli
Copilot CLI process log:
[WARNING] Found COPILOT_PROVIDER_WIRE_API without COPILOT_PROVIDER_BASE_URL. Provider configuration will be ignored. Set COPILOT_PROVIDER_BASE_URL to enable BYOK mode.
[WARNING] Server-to-server auth token detected; skipping the organization MCP registry policy fetch ...
[DEBUG] [rust:capi_models] fetched models from CAPI /models {"count":31, ...}
[WARNING] [rust:model_bindings::api_resolver] Model 'gemini-3.8-flash' from session is not available. Falling back to next option.
[WARNING] [rust:model_bindings::api_resolver] Model 'gemini-3.8-flash' from COPILOT_MODEL environment variable is not available. Falling back to next option.
[INFO] [rust:model_bindings::api_resolver] Using default model: claude-sonnet-5
The session's events.jsonl records session.model_change (source: startup, gemini-3.8-flash → claude-sonnet-5), and every main-agent and subagent message runs on claude-sonnet-5.
The 31 models CAPI returned for the workflow token contain no Gemini model.
Reproduced in every run we checked (18 runs between Sep 23 and Oct 7, from the first run after we switched to gemini-3.8-flash), both before and after upgrading gh-aw. Gemini has never actually run for this workflow.
Why: the agentic-workflows integration ID has no Gemini models
The same personal token, querying GET https://api.githubcopilot.com/models with only the Copilot-Integration-Id header changed:
curl -s https://api.githubcopilot.com/models \
-H "Authorization: Bearer $(gh auth token)" \
-H "Copilot-Integration-Id: <integration>" \
| jq -r '.data[] | select(.capabilities.type == "chat") | .id'| Model | copilot-developer-cli |
agentic-workflows |
|---|---|---|
gemini-3.8-flash |
✅ policy.state: enabled |
❌ not listed |
gemini-3.7-flash |
✅ policy.state: enabled |
❌ not listed |
| Claude / GPT / other models | ✅ | ✅ (same set) |
So no Gemini model can resolve under gh-aw's Copilot engine, regardless of the org policy or the billing path (org-billed copilot-requests: write or a seat-billed COPILOT_GITHUB_TOKEN). The gemini-3.8-flash entry under github-copilot in actions/setup/js/models.json is therefore unusable for engine: copilot today. This matches the model inventory report in #66711, which lists gemini-3.5/3.6/3.7/3.8-flash as "not in Copilot SDK; present in Gemini API".
BYOK with Gemini's OpenAI-compatible endpoint is not a workaround
We tried COPILOT_PROVIDER_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai with a Gemini API key on Copilot CLI 1.0.90 and 1.0.93:
- The first turn works. Gemini returns a tool call, and the CLI executes it.
- The CLI then echoes the assistant message back verbatim, including
"refusal": null. Gemini rejects that with400 Value is not a string: null. - Removing
refusalfrom the replayed request makes it succeed, so every BYOK session fails right after its first tool call.
That part is a Copilot CLI bug, but it closes off the obvious workaround.
Expected
- A configured model that the CLI cannot resolve should fail the run, or at least surface a prominent warning in the step summary and conclusion job. It should not silently switch to a different model vendor.
aw_info.jsonandgh aw auditshould report the model that actually ran (it is available fromsession.model_changeand frommodelon assistant messages).- Gemini models should be marked unavailable for
engine: copilotin the model catalog, and the compiler should warn or fail when one is configured. Alternatively, get them enabled for theagentic-workflowsintegration.
Questions
- Is the absence of Gemini models under the
agentic-workflowsintegration intentional, or is enabling them planned?
Activity
pelikhan commentedon Oct 11, 2026
@SivaKesava1
SivaKesava1 commentedon Oct 12, 2026
Rechecked on v0.91.8 (
c16656ef90, AWF v0.28.51). A run withengine: copilotandmodel: gemini-3.8-flashno longer falls back silently in the default AWF api-proxy path. The harness refuses to start Copilot and makes no model request:Still open:
failure_cause: engine_outage). It should say the configured model isn't available to this workflow, and how to fix it, the way Select Copilot wire APIs from AWF metadata and surface model mismatches #67487 does for model/endpoint mismatches.gh aw compilegives 0 warnings for this workflow, andactions/setup/js/models.jsonstill lists Gemini models undergithub-copilot.endpoint=managed-by-copilot-cli, noCOPILOT_PROVIDER_BASE_URL). That path may still fall back silently.Model routing isn't affected: it only picks models that are live in AWF's catalog, and fails with exit 78 when none are.
@pelikhan Item 1 should be filed as its own issue. Whether Gemini should be enabled for the
agentic-workflowsintegration is a question for the CAPI side.filip-be commentedon Oct 12, 2026
+1,
gemini 3-8-flashdoesn't work with Copilot engineIit looks like #61234 doesn't address missing model inventory.
Retested on 2026-10-12 with default gh-aw and Copilot CLI version.
Then retested with latest gh-aw
v0.91.8, Copilot CLI1.0.96-2, and AWFv0.28.51.The workflow compiles, but the harness fails before starting Copilot:
Minimal workflow
SivaKesava1 commentedon Oct 12, 2026
@filip-be Thanks, that matches what we see. Your line and ours come from the same failed run:
resolve_model_alias.cjslogsknown alias 'gemini-3.8-flash' did not resolve against catalog. The catalog was fetched; Gemini just isn't in it.model-catalog retrieval prevented alias resolution ... refusing to start Copilot, which blames catalog retrieval for both cases (item 2 in my comment above).So failing closed is right. What's left on the gh-aw side is the misleading final message, the
engine_outagefailure cause, and a compile-time warning.On the missing inventory: the harness can only use models that the Copilot API offers to the workflow's org, and AWF's live catalog had no Gemini models for ours either. Check whether Gemini 3.8 Flash is enabled in your org's Copilot model policies. If it is and it's still missing, the Copilot API isn't offering it to agentic workflows, which gh-aw can't fix.
filip-be commentedon Oct 13, 2026
@SivaKesava1 Gemini 3.8 flash is enabled in my org and works perfectly fine when used locally across Copilot CLI, VS Code and GitHub Copilot App.
Could this issue be related? github/copilot-sdk#2571
SivaKesava1 commentedon Oct 13, 2026
@filip-be Thanks, that issue was the missing piece. It isn't your org's policy: the Copilot API filters models by the calling integration. With one token, I called
GET https://api.githubcopilot.com/modelsand changed only theCopilot-Integration-Idheader (13 Oct):Copilot-Integration-Idcopilot-developer-cli(standalone Copilot CLI)gemini-3.7-flash,gemini-3.8-flashcopilot-cli,vscode-chatagentic-workflows(AWF's default, so gh-aw's Copilot runs)copilot-sdkA chat request for
gemini-3.8-flashsent asagentic-workflowsreturns400 The requested model is not available for integrator "agentic-workflows". The same request ascopilot-developer-clireturns 200.So gh-aw can't use Gemini on the Copilot engine until the Copilot API enables it for the
agentic-workflowsintegration. That's why it works for you locally and fails in the workflow. github/copilot-sdk#2571 looks like the same gating for thecopilot-sdkintegration: Gemini is missing fromlistModels()in SDK mode but works in the standalone CLI.@pelikhan the fix is on the Copilot API side: enabling Gemini for the
agentic-workflowsintegration. On the gh-aw side, the remaining items are a clearer failure message ("model not available for the agentic-workflows integration") and a compile-time warning for models that integration doesn't offer.pelikhan commentedon Oct 14, 2026
@SivaKesava1 can you do the fix on the gh-aw side?
SivaKesava1 commentedon Oct 16, 2026
@pelikhan The gh-aw side is done in #68819 (thanks for picking it up). Verified on
mainf1621a88b3withengine.model: gemini-3.8-flash:gh aw compilewarns that the model may not be available to theagentic-workflowsintegration.'gemini-3.8-flash' is not available in the workflow's live model catalog. For Copilot, availability also depends on the agentic-workflows integration …" instead of blaming catalog retrieval.What's left is on the Copilot API side: enabling Gemini for the
agentic-workflowsintegration (see the integration-ID table above). Should we keep this issue open to track that, or close it and track the API side elsewhere?pelikhan commentedon Oct 16, 2026
Did you use the org token to testing? The user models are enterprise level whereas the org token is org level so the list can diverge.
SivaKesava1 commentedon Oct 16, 2026
@pelikhan Good point, our 13 Oct table used a user token. I reran it with an org token:
github.tokenwithcopilot-requests: writeon githubnext's org billing, the same token type gh-aw's Copilot runs get.Copilot-Integration-Id/modelsgemini-3.7-flashgemini-3.8-flashagentic-workflowsagentic-workflows"copilot-developer-cligemini-3.7-flashmodel_not_supportedcopilot-sdkcopilot-sdk"copilot-cli,vscode-chatSo both effects are real:
gemini-3.8-flashisn't offered to githubnext's org token under any integration, while my user token gets it undercopilot-developer-cli.gemini-3.7-flashworks ascopilot-developer-cliand is refused asagentic-workflows. So a workflow can't use Gemini on the Copilot engine until the Copilot API enables it foragentic-workflows, whatever the org's policy allows.pelikhan commentedon Oct 16, 2026
Current-state refresher (2026-10-16): Partially addressed — gh-aw safeguards landed; Gemini availability remains external.
Merged #68819 addresses the gh-aw behavior discussed above. Current main has configured-model validation, an alias-resolution diagnostic explicitly explaining the workflow live catalog and
agentic-workflowsintegration, and harness regressions covering refusal to start with an unavailable model. This supersedes the original “silently succeeds as another vendor” description for the default protected path.The latest org-token experiment in this discussion is important: token scope and integration filtering both affect inventory. It reports Gemini 3.7 rejected under
agentic-workflows, and Gemini 3.8 unavailable to that org token even under the comparison integration. Local availability therefore does not establish workflow availability.What remains is enabling the desired model for the actual workflow token/integration, not removing the fail-closed behavior. Static verification of main and existing reported experiments only; I did not issue new Copilot model/API requests or recheck every direct/BYOK path.
SivaKesava1 commentedon Oct 18, 2026
Closing: the gh-aw side landed in #68819 and was verified on
mainon 16 Oct (compile warning, harness message naming the agentic-workflows integration, and the "unavailable configured model" failure issue), so gh-aw no longer silently substitutes a model. Whether Gemini is offered to the agentic-workflows integration is decided outside gh-aw (see the org-token results above).