[release] v0.114.4

.agents/skills/agent-release-gate/SKILL.md

@@ -344,10 +344,13 @@ ever reaches the stream. An empty ledger FAILS a cell; missing evidence is not e
 
 - `resources/matrix_l1_lifecycle_routes.py` — **MANDATORY. [mechanism-blind]** the routing matrix
   itself: for each kind of mid-conversation config change, assert the route the runner took. One
-  sandbox id = applied in place, two = rebuilt. Blocks on the four unambiguous cases (no change
-  must stay warm; an instructions edit, a permissions edit and a tool-catalog edit must escalate)
-  and reports the `model` case rather than guessing at a deployment's connection shape. This is
-  the cell that would have caught the `cold1` rot described below.
+  sandbox id = applied in place, two = rebuilt. Blocks on six cases: no change must stay warm; an
+  instructions edit, a permissions edit and a tool-catalog edit must escalate; and a
+  same-connection model switch must stay warm on BOTH claude and pi_core. The pi_core model case
+  (added 2026-08-29) is the standing trap for the wire-spelling bug class: the router once keyed
+  its table on the bare "pi" literal while the wire carries "pi_core", every playground model
+  switch silently rebuilt, and the claude-only case could not see it (#6364). This is the cell
+  that would have caught the `cold1` rot described below.
 - `resources/matrix_l2_approval_across_config_change.py` — **MANDATORY. [coached]** the killer
   combination: an approval answered while a config change rides along in the SAME request. It is
   the regression test for the applied-state bug (the pool used to stamp the INCOMING fingerprint
@@ -429,6 +432,82 @@ lands on a pool miss and takes the cold decision-map path, which is exactly the
   live Gmail and GitHub Composio connections in the target project; skip it otherwise.
 - `resources/seeds/` — representative green `results.json` files kept as regression-seed references.
 
+### The incident checks — born from the free-credits 401 of 2026-08-30
+
+A free-credits user on cloud hit a 401 because a fresh Daytona sandbox's first model call raced
+the asynchronous substitution of its Daytona Secret: the provider got the raw `dtn_secret_<id>`
+placeholder. The product then blamed the user's own key, which was wrong. The same release fixed a
+family of warm-session over-evictions caused by drift between two identity views in the runner.
+These four checks make each layer's failure loud instead of silent. Run all four on every gate.
+
+- `resources/matrix_c5_first_call_race.py` — **[mechanical]** the placeholder race, and whether it
+  is reported honestly. Mints a new workflow so the sandbox is necessarily cold, sends one short
+  message so the first model call lands as early as possible, and asserts the STORED turn row came
+  back. PASSes when the turn succeeds or when the failure carries the runner's
+  `credential_delivery_failed` code with its retry copy. FAILs when the run advises adding a key
+  while the underlying refusal carries the placeholder signature (`Received=dtn_`/`dtn_secret_`) —
+  the incident itself. The assertion is deliberately body-INDEPENDENT: only the litellm proxy
+  echoes a placeholder, so on a direct provider (where BYO-key cloud users live) an echo test is
+  blind, and F6 shipped a user-blaming 401 straight through the first version of this cell. A
+  credential refusal on this cell's necessarily-fresh sandbox must never advise adding a key, echo
+  or no echo; with PR #6408 the honest classification is `credential_delivery_failed`. Against a
+  deployment predating #6408 that assertion fails by construction — pass `--pre-6408` to report it
+  as a SKIP naming the known gap instead of an unexplained failure. It also counts `Received=dtn_`
+  lines in the credits proxy and reports the
+  count as diagnostic, never as a verdict. The proxy is never guessed by name across the box: it
+  must be named with `--proxy-container`, or belong to the target stack's compose project
+  (`--compose-project`, else derived from whichever container publishes the port in
+  `AGENTA_BASE`). With no match it prints "no credits proxy in this deployment; count not
+  applicable" and carries on. Reading a foreign project's proxy invents evidence about a
+  deployment that was never under test, which is worse than reading none. A run that dies on an
+  exhausted provider key SKIPs with "environment: provider key out of credit" rather than
+  failing — but only when the stored error carries a credit or billing signature, and never
+  when a placeholder refusal is present, because that combination is the incident itself.
+- `resources/sweep_disagree.py` — **[mechanism-level invariant; run AFTER a gate session]** greps
+  the runner log for `[reconcile] shadow ... DISAGREE ...`, the line `logReconcileShadow` writes
+  when the coordinator's `configFingerprint` decision and the router's facet digests disagree.
+  That drift is the over-eviction signature and it is invisible from the wire — the turn still
+  succeeds, it just paid for a rebuild it did not need — so a log sweep is the only way to catch
+  it. `--since <iso-timestamp>` is required; `--container` defaults to autodetecting the local
+  stack's runner. Exits 0 PASS, 1 FAIL (printing the offending lines), 2 SKIP when the log is not
+  reachable. Three line shapes are excluded as known SHADOW-COMPARATOR gaps (triage 2026-08-31,
+  `f7-disagree-triage.md`): the coordinator is correct and pinned, only the shadow's model of it
+  disagrees, and the comparator fixes are a post-release follow-up — without the exceptions the
+  sweep fails on the runner's own expected behavior on every loaded window. They are never
+  silent: each excluded line is printed with its shape and the triage marker, the excluded count
+  is reported separately, each shape is anchored on both halves of the line so it cannot swallow
+  a real disagreement, and any line matching no shape still FAILS. Delete a shape when its fix
+  lands; `--no-exceptions` fails on every DISAGREE line and is how you prove one can go.
+- `resources/matrix_h1_bad_harness.py` — **[mechanical]** a malformed harness must fail closed.
+  Drives three unreadable `harness` blocks (a wrong-type value, an unknown string, a null kind) at
+  both the commit API and the live invoke, and records WHICH boundary refused (`commit_api`,
+  `invoke_http`, or `runner_stream`) rather than demanding a particular one — a refusal further
+  out is better, not worse. The invariant is that some boundary refuses attributably and no turn
+  ever runs on a defaulted harness. FAILs if a turn executes and stores output.
+- `resources/check_secrets_teardown.py` — **[mechanical]** a Daytona Secret must not outlive its
+  run. Inventories the Daytona organization's Secret NAMES (never values) before a short Daytona
+  journey, forces the teardown with a config-change eviction, then asserts every `agenta_*` Secret
+  the run created is gone within a bounded settle window. Needs a Daytona API key in the
+  environment (`DAYTONA_API_KEY` or `AGENTA_RUNNER_DAYTONA_API_KEY`) and SKIPs with the exact
+  reason without one. Run it alone: a concurrent Daytona run against the same organization looks
+  the same as a leftover. The listing walks `GET /secret/paginated` by cursor to exhaustion —
+  `/secrets` does not exist, plain `/secret` is deprecated and fails above 1500 secrets, and a
+  `page` parameter is silently ignored, so anything less than real cursor pagination is noise
+  against an organization this size. The settle loop polls `GET /secret/{secretId}` per created
+  Secret rather than re-enumerating. `test_check_secrets_teardown_pagination.py` pins the walk.
+  A journey that dies on an exhausted provider key never creates a Secret, so it SKIPs with
+  "environment: provider key out of credit" instead of failing the teardown path it never
+  exercised.
+- `qa_matrix_lib.out_of_credit(error_text, codes)` — **[shared classification; no cell of its
+  own]** the SKIP reason when a run failed ONLY because the provider key has no credit left, and
+  `None` for everything else. An exhausted key is an environment condition: a cell that renders
+  it as FAIL spends a reviewer's attention on a topped-up balance, and teaches the reader that
+  this cell's FAIL is sometimes noise, which is how a real regression gets waved through later.
+  Recognition is narrow in both directions — the `starter_credits_*` codes plus the runner's own
+  credits copy and the provider's billing refusal, and deliberately NOT a bare 401, a rate limit,
+  or the placeholder refusal. Wired into `matrix_c5_first_call_race.py` and
+  `check_secrets_teardown.py`; `test_out_of_credit_skip.py` pins the boundary from both sides.
+
 ## Contributing
 
 Before committing any resource script, run the repo-pinned ruff (`uv run --no-sync ruff format`

.agents/skills/agent-release-gate/resources/LESSONS.md

@@ -228,6 +228,100 @@ around — `CODEX_SQLITE_HOME` is split onto container-local disk) and hard link
   the API reads S3 directly, so a hit is store-side proof, and the same listing shows any 0-byte
   objects — the fingerprint of this whole bug class.
 
+## Two Daytona-era traps for the lifecycle (L*) cases — 2026-08-31
+
+**Fixture connections must be vault-backed.** A fixture built by copying the Claude defaults and
+changing only the harness keeps `llm.connection = {"mode":"self_managed","slug":null}`. A
+self_managed Pi run needs the `PI_CODING_AGENT_DIR` mount, which a gate deployment does not have,
+so turn 1 errors and the case fails before it tests anything. Set
+`llm.connection = {"mode":"agenta","slug":null}` on every non-default-harness fixture (caught on
+#6371; the Pi fixtures in `matrix_l5_live_route_observed.py` and `bench_lib.py` already do this).
+
+**A stuck-substitution rebuild is not an eviction.** Since the credential preflight (#6370), a
+fresh Daytona sandbox whose Secret wiring failed (a vendor-side per-sandbox fault, a few percent
+of creates) is convicted at ~10s and rebuilt ONCE. A warm-reuse case that counts sandbox ids can
+therefore see two ids without any lifecycle regression. Before ruling a warm case failed, grep
+the runner log for `[credential-preflight] STUCK`: if it fired inside the run, re-run the case
+instead of reporting the eviction.
+
+## On a placeholder 401, read the proxy log before you blame a key — 2026-08-31
+
+A free-credits user on cloud hit a 401 on their first message. The product told them to add the
+project's OpenAI key. That advice was wrong three ways: their key was fine, adding one would not
+have helped, and the run was retryable. The real cause was the first-call race. On a Daytona run
+the real key never enters the sandbox — it is a Daytona Secret, and the sandbox holds a
+`dtn_secret_<id>` placeholder that Daytona substitutes into egress asynchronously, with no
+confirmation signal, 10-24s after the Secret is created. A cold sandbox whose FIRST model call
+beats that propagation sends the raw placeholder, and the provider refuses it with a 401.
+
+**The rule.** A 401 from a Daytona run is not evidence about the user's key until you have read
+the litellm-proxy log. Grep it for `Received=dtn_`. If the line is there, the key was never the
+problem and no key change will fix it: the run needed a retry.
+
+**An ABSENT marker proves nothing.** The marker is one-directional evidence — present, it
+confirms a placeholder refusal; absent, it is silence, and silence has many causes. A direct
+provider never emits it at all (`api.anthropic.com` answers "Invalid bearer token" and echoes
+nothing), a remote deployment has no reachable proxy log, and incomplete log access looks
+identical to a clean window. Reading an empty grep as "so it really was the user's key" is how F6
+survived a whole release. When the marker is absent, judge on the other evidence instead: the
+stored error's CODE (`credential_delivery_failed` is the runner's own verdict and outranks any
+grep), whether the sandbox was freshly created, and whether the copy contradicts itself by
+advising a key change on a run whose key was delivered seconds earlier. The runner classifies
+this correctly as `credential_delivery_failed`
+(see `PLACEHOLDER_CREDENTIAL` in `services/runner/src/engines/sandbox_agent/errors.ts`), so a
+run that reports an add-a-key message over a placeholder refusal is a product bug, not a user
+error. The related trap already recorded above still holds: a stuck-substitution rebuild is not an
+eviction, so grep `[credential-preflight] STUCK` before calling a warm case failed.
+
+Four standing checks came out of this incident. Run all four on every gate; each one is described
+in full in the skill's resource inventory.
+
+1. `matrix_c5_first_call_race.py` — forces a cold Daytona sandbox and sends one message
+   immediately, so the first model call lands as early as it can. FAILs when a placeholder refusal
+   is reported as the user's key problem.
+2. `sweep_disagree.py` — run AFTER a gate session. Greps the runner log for
+   `[reconcile] shadow ... DISAGREE ...`, the over-eviction signature that never shows on the wire.
+3. `matrix_h1_bad_harness.py` — a malformed harness must be refused at some boundary and must
+   never run as a silent defaulted turn.
+4. `check_secrets_teardown.py` — a Daytona Secret must not outlive its run. Names only, never
+   values.
+
+## Two traps the incident checks hit on their first live run — 2026-08-31
+
+**The Daytona Secrets API is singular, paginated, and lies about `page`.** `/secrets` does not
+exist; it 404s with "Cannot GET". The real paths are `/secret`, `/secret/paginated` and
+`/secret/{secretId}` (verified against `@daytona/api-client@0.198.0` inside the runner). Plain
+`/secret` is deprecated and, per the client's own docs, "fails for organizations with more than
+1500 secrets" — and the org holds ~3510, so it is unusable. The paginated listing returns 100 per
+response and a `page` parameter is SILENTLY IGNORED: the same 100 ids come back every time, which
+makes a page-based walk loop forever on identical data while looking like progress. Follow
+`nextCursor` to exhaustion, refuse a cursor that repeats, and bound the walk. A useful side
+effect: because `/secret` and `/secret/paginated` return 403 for an under-scoped key while
+`/secrets` returns 404, you can confirm the right path without any list access at all.
+
+**Never pick a container by name match on a shared box.** `matrix_c5` originally took the first
+`docker ps` name containing "litellm" and found `starter-litellm-proxy` — a different project's
+container — while the stack under test had no proxy at all. A foreign container's log is worse
+than no log: it invents evidence about a deployment that was never under test. Resolve a
+container by its `com.docker.compose.project` label against the target stack's project (derive
+the project from whichever container publishes the port in `AGENTA_BASE`), or take it explicitly.
+When nothing matches, say so and continue.
+
+## Standing reds: expected, named, never softened
+
+A check that goes red for a filed finding stays red — softening it would hide the next real
+break behind the same shape. What it gets instead is a NAME in its failure message, so a reader
+scanning a gate report recognizes it in one line instead of chasing it as fresh breakage. This is
+how the W5 steer red is handled, and it now applies to one more:
+
+- **`matrix_h1_bad_harness.py`, the `null_kind` case — finding SF2.** A cleared harness
+  (`{"kind": null}`) is not rejected: it silently defaults to `pi_core`, so on any config whose
+  model spelling suits Pi the turn runs and the cell correctly fails. Filed for the next release,
+  not fixed in v0.114.4. The failure message says so. When SF2 is fixed the case turns green on
+  its own and the `known_finding` key stops appearing — that is the signal to delete the note.
+  A wrong-type or unknown-string harness that runs is a DIFFERENT, unfiled defect and is
+  deliberately not covered by the name.
+
 ## The checklist for the next QA run
 
 1. `docker ps` — is anything restarting? If yes, wait.