[feat] Name what evicted a warm session, and when Secrets moved - #6369
Open
mmabrouk wants to merge 3 commits into
Open
[feat] Name what evicted a warm session, and when Secrets moved#6369mmabrouk wants to merge 3 commits into
mmabrouk wants to merge 3 commits into
Conversation
Two blind spots this week's placeholder-401 investigation had to work around with production access, closed: - A 'mismatch (config)' eviction now names the changed FIELDS (fields=[mcpServers,permissions]). The applied state stores per-field digests beside the whole-request fingerprint and the facet digests, all three computed from one request by one helper so they can never describe different configurations. Names only: each side of the diff is a hash, so no config value can reach a log through this path. - Daytona Secret allocation and deletion log one line each with count, allowed hosts, and elapsed time (never ids, generated names, placeholders, or values) — the create/delete timeline that had to be reconstructed by inference from eviction lines. Documented as a deliberate, narrow exception to the delivery layer's log-nothing rule. Plus the measurement tool the Daytona questions need: scripts/probe-secret-propagation.ts measures create-to-substitution lag (and the delete-then-create eviction ordering with --delete-old) using the runner's own SDK calls against httpbin.org header echo. Not yet run; docs/design/daytona-secret-propagation/README.md carries the incident facts, the two hypotheses, and how to run it. Claude-Session: https://claude.ai/code/session_014s6jqKCsVKsNMmnrJVJkZt
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
dosubot
Bot
added
the
size:L
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. 🗂️ Base branches to auto review (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository YAML (base), Organization UI (inherited) Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
dosubot
Bot
added
dev experience
Contributor
📘 Docs preview
This comment updates in place on every push. |
Substitution is host-dependent: a heavily-used host maps a new Secret's value in under 2s, while a never-used host got no interception at all within 5 minutes. Measured from the production runner with the probe this branch ships. Claude-Session: https://claude.ai/code/session_014s6jqKCsVKsNMmnrJVJkZt
The Fable reproduction (2026-08-30) overturned two claims. First, the 'new host never substitutes' finding was a measurement artifact: Daytona scrubs responses (real values are rewritten back to placeholders before they enter the sandbox), so an echo-service instrument is blind either way; the working instrument is a provider whose error body echoes a MASKED key (api.openai.com). Second, with that instrument the fault is binary per sandbox: 15 of 20 fresh sandboxes substituted on their FIRST request (+1.5-2.9s), 5 of 20 never did (raw for 90-180s from the first request on), a twin sandbox on the same Secret worked while the stuck one stayed raw, stop+start repaired nothing, and the delete ordering is irrelevant. Not a lag — a per-sandbox registration failure with no reconciliation. The probe script now uses the masked-echo instrument and measures the stuck rate; the README carries the corrected findings and the updated Daytona question. Claude-Session: https://claude.ai/code/session_014s6jqKCsVKsNMmnrJVJkZt
mmabrouk
commented
Aug 31, 2026
Member
Author
There was a problem hiding this comment.
should not be part of the pr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Diagnosing this week's placeholder 401s (#6362) needed production access twice, for questions the runner's own logs should have answered:
mismatch (config)and nothing else. Naming the changed field took reconstructing fingerprint inputs by hand.Changes
A
mismatch (config)eviction now names the changed fields:Before:
mismatch (config) key=...; evict + coldAfter:
mismatch (config) key=... fields=[mcpServers,permissions]; evict + coldThe applied state stores per-field digests beside the whole-request fingerprint and the facet digests. One helper computes all three from one request, so the three views can never describe different configurations. The diff compares hashes, so no config value can reach a log through this path.
Secret allocation and deletion each log one line with the count, the allowed hosts, and the elapsed time:
[daytona-secrets] allocated n=2 hosts=[gateway.eu.cloud.agenta.ai] ms=840. Never an id, a generated name, a placeholder, or a value. The comment documents this as a deliberate, narrow exception to the delivery layer's log-nothing rule.Also included:
scripts/probe-secret-propagation.ts, the measurement tool for the two open Daytona questions (create-to-substitution lag, and whether deleting an old same-host Secret widens the window). It uses the runner's own SDK calls and an httpbin.org header echo. It has not been run yet;docs/design/daytona-secret-propagation/README.mdcarries the incident facts and the run instructions.Tests
Stacked on #6368.
https://claude.ai/code/session_014s6jqKCsVKsNMmnrJVJkZt