feat(config): send [model.headers] on every call to the model endpoint, agents' included

[earakely-scale]

Why

Some model endpoints need a header on every request. Hugging Face's Inference Providers router, for one, bills an organization only when each request carries X-HF-Bill-To: <org>.

Today a header can reach agent-env's own calls ([model.params.extra_headers]), but not an agent's. Deployed agents only receive LITELLM_BASE_URL and LITELLM_API_KEY, so each agent invents its own variable for the header, and tasks end up storing who pays.

What

[model]
base_url = "https://router.huggingface.co/v1"
api_key  = "env:HF_INFERENCE_TOKEN"

[model.headers]
"X-HF-Bill-To" = "my-org"

[model.headers] declares the endpoint's headers once. Values may be env: or secret: references, which stay raw and resolve per call, like api_key. The headers go only to the configured endpoint, since a header can carry a credential. The rule lives in one place, ModelConfig.headers_for.

Who	How they get the headers
agent-env's own calls (the direct judge)	resolve_call adds them to extra_headers; a [model.params.extra_headers] entry wins, whatever the case of its name
deployed agents (deploy_agent, the auto-deployed judge agent)	LITELLM_EXTRA_HEADERS, a JSON object, set in A2AAgent.deploy for the URL the agent will use (the passed one, else its own default_env_vars)
run_container_unit_tests_verifier	LITELLM_EXTRA_HEADERS, reserved from seed columns
install_agent	a litellm_extra_headers placeholder ({} when unset), opt-in through required_params
the protocol's reference model	sends LITELLM_EXTRA_HEADERS only to the LITELLM_BASE_URL AgentEnv set, and never lets one replace its own Authorization

Rules:

• A call or agent pointed at another URL gets no headers, and that's logged. That covers a step's litellm_base_url, user_overrides.litellm_base_url, judge_litellm_base_url, an agent's own endpoint when it isn't the configured one, and model overrides. A trailing slash on either URL doesn't count as a different endpoint, for the headers or the configured key.
• An explicit LITELLM_EXTRA_HEADERS in the task's or the agent's env wins.
• Core only writes LITELLM_EXTRA_HEADERS, never reads it, so exporting it in the shell does nothing.
• A header HTTP can't carry is refused: a name that isn't a token, or a value with non-ASCII characters, CR or LF. A literal is checked when the config loads, a reference when it resolves.
• config show and config explain mask every literal value under a headers key ([model.headers], [model.params.extra_headers]), since a name like X-Api-Key doesn't say it carries a key. References still show.
• The unit-test verifier fails when configured headers don't resolve, unless the step sets the variable itself.

Sail (sail_vm with inject_model_key, the default). The headers are added at egress, like the key, so their values never enter the Sailbox:

• Each value is a Sail secret, AGENTENV_LITELLM_H_<sha256(name, value)[:32]>, kept like the key's.
• The saved policy sets each header from its secret on requests under the endpoint's path (match.path.prefix, /v1/ for https://host/v1). Another service on the same host never gets them. The key still goes with every request to the host, in a second rule, as before.
• Inside the box, every command and file carries LITELLM_EXTRA_HEADERS with the names kept and sail-injected-header as each value. The JSON is replaced whole, in agent-env's spelling or the deploy's, so a short value can't rewrite unrelated text.
• Reconnect recognises the policy and recovers the values for scrubbing, from this process or from the configured [model.headers].
• A Sailbox refuses a command or file carrying header values Sail doesn't inject for it, as it does for a key.
• A header named authorization or x-api-key (which the key is set as) is refused, as is a value that isn't a JSON object of strings. The error doesn't quote the value.

Also fixed here: install_agent logs each command's template instead of the substituted command. That was already true of a resolved {litellm_api_key} before this PR (Greptile spotted it).

Choices to review:

• Env var, not agent config. LITELLM_EXTRA_HEADERS is a container env var, the channel the endpoint and key already use. Every agent receives it at startup, including installed CLIs and agents not built on the protocol SDK.
  • [model.params] does reach agents, but only those whose config declares model_params, and only per prompt.
  • The LITELLM_ prefix matches the existing pair. LiteLLM itself reads none of them.
• A bad header reference fails a deploy but only warns in the unit-test verifier. On deploy, an unresolvable secret: reference fails before any container starts. The verifier only warns, matching how it already treats a missing key or URL.

Tests

• tst/unit/config/test_model.py:
  • parsing, with references kept raw and non-string values rejected;
  • scoping: configured endpoint (file or env) vs a foreign URL, None, or no endpoint;
  • lazy resolution;
  • resolve_call to the configured endpoint vs a foreign override vs a model override;
  • case-insensitive [model.params.extra_headers] precedence.
• tst/unit/task_step/local_container_urls_test.py:
  • deployed agents on local and remote providers, for configured and passed URLs;
  • an agent whose own endpoint is, or isn't, the configured one;
  • an explicit value winning;
  • the unit-test verifier, pointed elsewhere, and a seed column trying to override;
  • the installed-agent placeholder.
• tst/unit/task_step/install_agent_test.py: execute with both placeholders; the values reach the command, and the log shows only the template.
• Protocol: the reference model with no headers, with endpoint headers (a lowercase authorization is dropped), and with a different endpoint.
• test_unread_keys: the expected [model] key list now includes headers.
• tst/unit/providers/sandbox_providers/sail_vm_model_key_test.py: 16 new tests, with a fake Sail SDK, covering:
  • secret naming;
  • path-scoped rules, and rules for an endpoint with no path;
  • the refused collision and refused malformed JSON (unquoted);
  • secrets stored on create and released on failure;
  • placeholders in commands and files, in agent-env's spelling and the deploy's;
  • reconnect, in this process and in another;
  • another set of headers configured, which gets refused;
  • refusal on a box that doesn't inject;
  • Sailboxes sharing headers;
  • detection as the shell reads it.
• tst/unit/config/test_describe.py: header values masked whatever the header is called.
• Mutation check: reverting each change, the review fixes included, makes its test fail.

Chaos and regression testing (Linux, real local sandboxes)

The setup:

• a c6i.2xlarge running Ubuntu 24.04 and Docker 29.9;
• a stub model recording every request, on :4000 (the configured endpoint) and :4001 (a foreign one);
• the protocol's reference agent, built from the checkout;
• the direct rubrics judge going through litellm.

At 5534c3a, all 64 checks pass:

Scenario	Checks
S1 happy path: a literal, an env: and a secret: header; the agent, the judge on litellm's OpenAI route (strict json_schema) and on its Anthropic route	11/11
S2 scoping: deploy_agent.litellm_base_url, user_overrides.litellm_base_url, an agent's own endpoint (foreign and configured), judge_litellm_base_url, explicit env_vars	27/27
S3 hostile values (quotes, $(…), backticks, ;, backslashes, JSON, 4 KB) arrive byte for byte, and no canary command ran	4/4
S4 faults: an unresolvable secret: fails before any request or container; a non-string value fails with the config error; a shell LITELLM_BASE_URL takes the headers with it; a shell LITELLM_EXTRA_HEADERS is ignored	12/12
S5 load: 32 concurrent runs, with about 20% of requests delayed and failed with 429 or 500 (67 requests, 9 failed and retried); every one to :4000 carried the headers, and nothing reached :4001	4/4
S6 the unit-test verifier's container gets both variables byte for byte, and nothing when it uses its own URL	4/4

Regression, branch vs main 230aa0f on the same box:

• Without [model.headers]: agent and judge requests carry identical header names and the same Authorization.
• Suites: identical failure sets on both sides:
  • unit: 1 failure on both, a real-sandbox context-zip test that fails as root;
  • integration-fast: 160/160 on both;
  • integration-slow: the same 2 gateway_test failures on both, and the same 491 capability skips.

Live on Sail (with a real key)

The reference agent was deployed on sail_vm with a literal and a secret: header, against an HTTPS endpoint that doesn't resolve, so no request left Sail. 13/13 checks pass.

• Injection on:
  • the saved policy sets both headers from their secrets under /v1/ and the key on every path, and holds no value;
  • Sail holds both secrets;
  • reconnect recognises the stored, normalised policy and recovers the values;
  • the agent container's LITELLM_EXTRA_HEADERS and LITELLM_API_KEY are placeholders, and no value appears anywhere in its env;
  • teardown removed the policy.
• Injection off: the container gets the real values, and there are no egress rules.
• Cleanup: no Sailboxes or policies were left, and the log holds neither the Sail key nor any value.

In transit, 19/19. This used a temporary public HTTPS stub, logging every request's headers, as the model endpoint; it's deleted now.

• Injection on:
  • the agent's container held only placeholders, yet every request the stub received carried the real header values and key, added by Sail;
  • a request from the Sailbox under /v1/ got key and headers, and one to /other/ got the key alone;
  • no placeholder ever reached the stub.
• Injection off: the agent sent the real values itself.
• Judge: the direct judge sent them through litellm in both runs.

Not in this PR

• ANTHROPIC_CUSTOM_HEADERS for Claude Code in the unit-test verifier's container. Nothing in this repo reads it, so I dropped it to keep one header encoding. A step can set it in env_vars, and it can come back when it has a consumer.
• An agent pointed at an overridden URL still receives the configured key. That's existing behaviour, unlike resolve_call, and S2 reproduced it. It's tracked separately.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; both numbered findings are fixed and no new blocking issue remains.

Summary

Adds endpoint-scoped [model.headers] to direct model calls, deployed agents, the unit-test verifier, and opt-in agent installation. Sail keeps the values outside the box and adds them as requests leave.

• Model calls send shared headers only to the configured endpoint.
• Deployed agents receive configured endpoint headers at startup.
• Sail adds configured headers outside the agent's box.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A["model.headers"] --> B["ModelConfig.headers_for"]
  B --> C{"Configured endpoint?"}
  C -->|No| D["No configured headers"]
  C -->|Yes| E["Resolve and check values"]
  E --> F["Direct call: extra_headers"]
  E --> G["Agent or verifier: LITELLM_EXTRA_HEADERS"]
  E --> H["Install: opt-in placeholder"]
  G --> I{"Sail injection enabled?"}
  I -->|No| J["Agent sends values"]
  I -->|Yes| K["Box receives placeholders"]
  K --> L["Sail adds values under endpoint path"]

Loading

_{Reviews (10) · Last reviewed commit: "fix(sail_vm): scrub the headers' JSON be..." · Reviewed by Greptile}