Skip to content

feat(a2a): agents the object store's grants can't reach get their objects through their own staging - #46

Merged
earakely-scale merged 73 commits into
mainfrom
edgararakelyan/staged-transfers
Oct 6, 2026
Merged

earakely-scale merged 73 commits into
mainfrom
edgararakelyan/staged-transfers

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

With the local object store, an agent on a remote sandbox (Modal, E2B, a plugin provider) can't reach the store's grants. Today agent-env therefore refuses that agent's skill bundles, snapshots and changelogs ("grants do not reach agents on the 'modal' sandbox provider"), and its trajectories come back inline. This PR makes those flows work with no setup beyond the provider's own credentials: no bucket, tunnel or inbound connection to the user's machine.

An agent that serves the staging extension (#39; every SDK agent does) gets ordinary HTTPS grants naming paths on its own /ext/staging routes. agent-env moves the bytes over connections it opens itself:

  • Reads (skill bundle files, snapshot load, changelog apply): before the call, each object is pushed into the agent's staging, streamed from the store.
  • Writes (trajectory, snapshot save): after the call, each object the agent wrote is pulled into the store. The call's staging is cleared whether the call succeeded or not.
  • Changelog capture: the namespace stays staged for the agent's life. Its increments are drained into the store every 5 seconds while a prompt runs, again when the prompt ends, and before teardown_run or a teardown_sandboxes step takes the agent down.
    • A drained increment is removed from staging only if the agent hasn't rewritten it since (If-Match).
    • If the sandbox dies mid-prompt, at most one drain interval is lost.

How it fits in:

  • Choosing the store. Every call site that builds a transfer call takes its grants from transfer_store(). That function keeps the configured store whenever the store's grants reach the agent, so hosted stores, and the local store with the local provider, are unchanged. Staging is chosen only when the grants don't reach, the agent advertises staging, and the agent's URL is HTTPS.
  • Agents without staging. An agent that doesn't advertise staging gets the inline forms, or the refusal it gets today.
  • Error hygiene.
    • Staging failures are StagingErrors that never name a staged path, because the path's id is what keeps it private.
    • Staging requests are logged by origin only.
    • If an agent can't reach its own URL, the error says so.
  • Unchanged. No request or response model changes, and no new config.

Builds on #13's grants_reach and sandbox_type plumbing, #32's per-store grant lifetimes and #39's staging extension, all now on main.

How it was tested

  • pytest tst/unit packages/agentenv-protocol/tests: 6042 passed. tst/unit/a2a_agent/staging_test.py is new. It uses an in-process SDK agent at an HTTPS URL, so the agent's own calls to its staging run too, and it covers:
    • a skill bundle pushed (3 MB file);
    • a snapshot save pulled, then loaded into the agent;
    • a trajectory pulled;
    • a changelog drained, then applied to the agent;
    • mid-prompt drains plus the final drain;
    • teardown_run draining first;
    • store selection (local agent, no staging, non-HTTPS URL);
    • a full staging failing cleanly without naming a staged path.
  • Integration tier on the local defaults (-m 'not int_test_slow'): 143 passed. The slow tier runs in CI.
  • End to end on Modal with the local object store (claude-code-cli with the staging routes): every flow and six chaos scenarios pass; results are in this comment.

🤖 Generated with Claude Code

RetriggerConfidence Score: 5/5

The PR appears safe to merge; no new issue or outstanding previous finding remains.

Fix All in CursorFindings

  1. P1 Updated objects break staged uploads ▶
  2. P2 Skill transfers repeat store lookups ▶
Fix with agent prompt
### Issue 1
src/agent_env/a2a_agent/staging.py:undefined-148
If another task replaces a skill or snapshot object while `push()` is running, `_push()` can open the new bytes but send the size read earlier as `Content-Length`. The upload then fails even though the object is available. The new preflight widens this gap by reading every size before any upload starts. Use a size tied to the bytes being sent.

### Issue 2
src/agent_env/a2a_agent/staging.py:undefined-148
`push()` now starts another metadata lookup for every object, all at once. Grant creation already reads that metadata, and a valid skill bundle can contain 1,000 files. This adds a burst of store requests before any file moves, increasing cost and risking slower transfers or request limits. Reuse the sizes already read or limit the new lookups.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Agents whose object-store grants cannot reach them can now transfer objects through their own HTTPS staging routes, with no extra storage setup. Changelog updates are copied into the store during prompts and before teardown.

  • Remote agents move object transfers through their own staging routes.
  • Remote agents keep changelog updates in the store as they work.
  • Failed agent setup now closes the sandbox it already started.
Diagram
sequenceDiagram
  participant Store as Object store
  participant Host as agent-env
  participant Stage as Agent staging
  participant Agent
  Host->>Store: Read object size
  Host->>Stage: Push objects the agent will read
  Host->>Agent: Send call with staging grants
  Agent->>Stage: Read or write objects
  Host->>Stage: Pull objects the agent wrote
  Host->>Store: Save pulled objects
  Host->>Stage: Clear the call's staging
Loading

Reviews (13) · Last reviewed commit: "fix(a2a): a staged push sends each objec..."

earakely-scale and others added 30 commits September 30, 2026 06:50
…t types

A write now lands whole or not at all. Bytes are staged in a reserved .agentenv-tmp directory inside the root and put in
place with a rename (an overwrite) or a hard link (a write-once put), so a crash or a concurrent reader never sees a
partial object, and a write-once put that loses a race raises ObjectAlreadyExistsError instead of replacing the winner.
Filesystems without hard links fall back to an exclusive create.

A write's content type is recorded in a reserved .agentenv-meta tree and returned by get_object_metadata. The default
type, which a writer that names none gets, reads back as unknown, so readers keep guessing from the name as before.
Both directories are refused as keys and left out of listings.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
LocalFilesystemObjectStore's issue_read_grant, issue_write_grant and issue_upload_policy now
return real http-get, http-put and http-post-policy grants, served by an HTTPS server inside the
process that issued them. It is started by the first grant and runs in a daemon thread: starlette
on uvicorn, one route, /v1/grants/{token}, with no access log.

- Tokens are HMAC-SHA256-signed claims (operation, key or key prefix, expiry, size bound, content
  type) under a key held only in this process's memory; the server trusts nothing else.
- GET returns the stored bytes and type. PUT requires the signed Content-Type, bounds the body to
  the grant while it streams, and replaces the object atomically. A multipart POST names its key
  before its file, must stay under the grant's prefix, and is bounded the same way.
- TLS comes from a per-user local CA under <state root>/tls, name-constrained to localhost,
  host.docker.internal and loopback or private addresses, with ca.pem and ca-bundle.pem (the
  public roots plus the CA) for containers to trust. It is never installed on the host.
- The server listens on 127.0.0.1, or the Docker bridge gateway on Linux, and grant URLs name
  host.docker.internal; grant_bind_host and grant_advertise_host override both. root is now
  optional and defaults to <state root>/object_store.

supports_transfer_grants stays false, so no agent is handed these grants yet: containers must
first trust the CA and reach the server.

starlette, uvicorn (0.47 for ssl_context_factory), python-multipart, cryptography and certifi
are declared as core dependencies; all were already installed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ble directory

ca.pem and ca-bundle.pem now live in <state root>/tls/trust/, mode 0755, apart from the CA key, so
the directory can be copied into a container whole and read by whatever user it runs as.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ocal store's grants

The local object store now offers its HTTPS transfer grants (grants = "auto", the default; "off"
hands out none), so on the all-local configuration skill bundles, trajectories, snapshot saves and
loads, and changelog capture and replay move through the object forms agents already take.

- Local containers trust the local transfer CA: LocalSandboxProvider creates the container, copies
  the CA's public trust files to /etc/agentenv/, then starts it, with SSL_CERT_FILE and
  REQUESTS_CA_BUNDLE on the bundle (public roots plus the CA) and NODE_EXTRA_CA_CERTS on the CA,
  unless the caller's env sets them. The base provider's docker run moves into protected
  _start_container / _container_args hooks, with the same script.
- Grants go only to agents they reach: ObjectStore.grants_reach(sandbox_type), true by default and
  only for "local" on the local store (a local run asks its local store). choose_transfer and every
  call builder take the agent's sandbox_type, so an agent on another provider keeps the inline
  forms it gets today.
- A legacy s3_prefix / skill_s3_url form carrying a local store's file:// URL is refused, saying
  why: grants off, an agent that takes only the S3 form, or grants that do not reach it.
- An objects-form call that fails with transfer_unavailable names this process's grant server and
  the grant_bind_host / grant_advertise_host settings.
- The validator's presigned-URI probe falls back to issue_read_grant.
- A slow integration test proves an SDK agent in a local container uploads its trajectory through
  a local grant.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…a write-once put never exposes an empty object

A content type is recorded with the identity of the file it was written for (inode, size, mtime,
which a rename keeps), so a type recorded for another write - a racing overwrite that renamed
first, or an old record an overwrite could not remove - reads back as unknown, never as the wrong
type.

On a filesystem without hard links a write-once put now checks for the key and renames into place,
instead of reserving the key with an empty file that readers could see and a crash could leave.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…c-writes' into edgararakelyan/local-transfer-grants
…ir part headers

A process now keeps the local CA it first loads, and each CA's trust files are written once into a
directory of their own under tls/trust/, so the grant server and the containers a process starts
agree on the CA even when another process renews the one on disk. A server certificate lasts as
long as its CA rather than 90 days, since a server keeps one for its whole life.

A multipart upload's part headers are bounded to 16 KiB, as its fields and file already are.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ants' into edgararakelyan/local-provider-transfer-grants
…hard links

Where no hard link can claim the key, a write-once put now claims it with a marker created
exclusively in the staging directory, held only from the existence check to the rename. A racing
writer waits for the marker to go and then finds the winner's object; a marker a crash left behind
is taken over once it is ten seconds old.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…c-writes' into edgararakelyan/local-transfer-grants

# Conflicts:
#	src/agent_env/store/object_store/local_object_store.py
…ants' into edgararakelyan/local-provider-transfer-grants

# Conflicts:
#	src/agent_env/store/object_store/local_object_store.py
…nded out, including agents placed on a local VM sandbox

The local provider now copies the CA in and sets the trust variables only when the configured
object store (behind @Local routing too) is a local store that hands out grants; otherwise it runs
the container as before, so a hosted store or grants = "off" adds no startup step that can fail.
An agent placed on a local VM sandbox, which starts through A2AAgent._run_container rather than
the provider, is given the CA the same way (and, on Linux, the host.docker.internal mapping).

The validator's image probe uses a grant only where it reaches the provider the validation agents
will deploy on.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… a copy that keeps modification times

Without hard links, a write-once put's claim marker now names its owner's pid and is taken over only
once that process has exited, not after ten seconds, so a slow live writer never loses it. On
Windows, where rename refuses an existing target, the rename itself is the claim.

A recorded content type is now paired with the file's size and modification time, and each write
stamps its staged file with a modification time no other write shares (now, to the millisecond,
plus a random remainder). Racing writes still never pair bytes with another write's type, and a copy
of the store that keeps modification times keeps its types, where the inode it no longer paired
with would not survive one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…c-writes' into edgararakelyan/local-transfer-grants
…ants' into edgararakelyan/local-provider-transfer-grants

# Conflicts:
#	src/agent_env/store/object_store/local_object_store.py
… its holder

A write-once put on a filesystem without hard links now claims its key with an exclusive flock on
a file in the staging directory rather than a marker naming a pid. The kernel drops the lock when
its holder exits, so a crashed writer never blocks the key and a reused pid can never pass for a
live one; a live holder is waited for however long it takes. The file stays in place so every
writer locks the same one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…c-writes' into edgararakelyan/local-transfer-grants
…ants' into edgararakelyan/local-provider-transfer-grants

# Conflicts:
#	src/agent_env/store/object_store/local_object_store.py
…chain led by the local provider counts as local for the validator's probe

start_trusting now removes the created container when the copy or the start fails, so a linked
agent on a local VM sandbox (whose sandbox the agent does not own, and so never tears down) leaves
its container name free for the next attempt. The validator's image-by-URI probe judges a fallback
chain by its first provider, the one that runs the agent unless it fails; ChainedSandboxProvider
gains a read-only providers property for that.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…k, and staged files of dead writes are swept

Chaos testing on exFAT, which keeps modification times to 10 ms, showed two same-size overwrites of
one key could leave one's type recorded against the other's bytes (5 in 4,129 races): the per-write
time stamp that tells writes apart is rounded away there.

A write now holds the key's lock (an exclusive flock) across placing its bytes and recording its
type, and a metadata read holds it shared, so no write lands between another's two steps and no
reader sees one write's type with another's bytes, whatever the filesystem's time resolution. An
overwrite drops the old type first, so one cut short leaves the type unknown, never wrong. Keys share
64 lock files, which also replace the per-key claim files the no-hard-link path used; Windows, which
has no flock, keeps relying on the stamp, which NTFS's 100 ns times preserve.

A store's first write also removes files staged by writes that died (untouched for an hour), which
crashes left behind in .agentenv-tmp.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ads now read the type under the key's lock

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ants' into edgararakelyan/local-provider-transfer-grants

# Conflicts:
#	src/agent_env/store/object_store/local_object_store.py
The GET handler now opens the object under the key's shared lock and reads its type against that
open file, then streams from it, so a racing overwrite can no longer pair its bytes with the type
of the object that was there when the type was read.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…es any file a live write holds

An overwrite now sets the old type record aside while its bytes land and puts it back if they do
not, so a rename that fails keeps the old object's type; one cut short after its bytes land still
reads as unknown, never as the old type.

Each staged file is held under an exclusive flock by its writer for as long as the write lives, and
the sweep removes only staged files no process holds (and none younger than a minute, which may not
be held yet), so a write paused for any length of time never loses its file. Windows, which has no
flock, is not swept.

Keys now share 256 lock files rather than 64, so fewer unrelated keys wait on one another, and the
tests that hold a lock with flock skip on Windows.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…c-writes' into edgararakelyan/local-transfer-grants
…a staged file that cannot be locked is cleaned up

Removing the set-aside type record after an overwrite's bytes land, and refreshing its time when it
is set aside, are now best effort: a failure there leaves a file the sweep removes, rather than
reporting a completed write as failed. A staged file whose lock cannot be taken is closed and
removed like any other.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…c-writes' into edgararakelyan/local-transfer-grants
…ants' into edgararakelyan/local-provider-transfer-grants

# Conflicts:
#	src/agent_env/store/object_store/local_object_store.py
…transfer-grants

# Conflicts:
#	src/agent_env/store/object_store/local_object_store.py
…ants' into edgararakelyan/local-provider-transfer-grants

# Conflicts:
#	README.md
earakely-scale and others added 2 commits October 2, 2026 09:32
…re left open

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
earakely-scale and others added 9 commits October 5, 2026 09:50
…ether in object_store/local

The local backend is now four modules, the store, its grant server, the CA
and the grant tokens, and only the local store uses the last three. They move
into one package, beside the single-module S3 and GCS backends:

  local_object_store.py   -> local/store.py
  local_grants/server.py  -> local/grant_server.py
  local_grants/tls.py     -> local/tls.py
  local_grants/tokens.py  -> local/tokens.py

A rename only: imports and docstrings follow the files, and nothing else
changes. LocalFilesystemObjectStore is still exported from
agent_env.store.object_store and agent_env.store, which is how everything
outside this repository imports it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ants' into edgararakelyan/local-provider-transfer-grants

# Conflicts:
#	tst/unit/store/local_grant_server_test.py
#	tst/unit/store/test_object_transfer_grants.py
…ansfer-grants' into edgararakelyan/store-grant-lifetime
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…grants

main now sends objects only through grants, so the reachability check moves into the
object-form refusal: a call whose grants do not reach the agent's sandbox is refused, naming
the provider. _s3_form_url and skill_add_call's agent_name, which existed for the S3 forms,
are gone; object_transfer_reach_test follows.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…etime

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Changelog enable records the staged namespace for every call, now that object calls are
the only form, and apply reads its source from agent_changelog_object_url alone.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
earakely-scale and others added 11 commits October 5, 2026 11:36
…pe without reading the object

Starlette's StreamingResponse has no HEAD case, so a HEAD streamed the whole
object only for the server to drop the body.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ants' into edgararakelyan/local-provider-transfer-grants
…ansfer-grants' into edgararakelyan/store-grant-lifetime
…provider-transfer-grants

# Conflicts:
#	src/agent_env/store/object_store/local/grant_server.py
#	src/agent_env/store/object_store/local/store.py
#	tst/unit/store/local_grant_server_test.py
#	tst/unit/store/test_object_transfer_grants.py
…ansfer-grants' into edgararakelyan/store-grant-lifetime
Main's derived-id change dropped refuse_local_derivation from the validator and added
derive_id/key_segment; keep both sides' imports.

Main's new acceptance test expected validating an @Local agent to stop at the local store
because it cannot sign URLs. With local grants the store serves the probe image to agents
on the local provider, so the test now deploys its agents on a remote provider, where the
refusal still comes before any task is written, and matches the refusal's new wording.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Main holds #13's squash, whose tree equals #13's final head (merged just before), plus the
v0.9.1269 version bump; the result is this branch's tree with that bump.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Base automatically changed from edgararakelyan/store-grant-lifetime to main October 5, 2026 23:17
earakely-scale and others added 2 commits October 5, 2026 16:18
Main holds #32's squash, whose tree equals #32's final head (merged here before), plus the
v0.9.1270 version bump; the result is this branch's tree with that bump.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…bed, without asking the store again

The quota check before a push looked every object's size up again, all at once, though the call
had read each one moments before to describe it: a skill bundle of 1,000 files cost 1,000 more
store requests. The staged store now keeps the size it read for each object and the push sends
that as the object's length, so what reaches the agent matches the size_bytes its call names.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@earakely-scale

Copy link
Copy Markdown
Collaborator Author

@greptileai review

@earakely-scale
earakely-scale merged commit 1b3959b into main Oct 6, 2026
13 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/staged-transfers branch October 6, 2026 00:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant