feat(store): the local object store issues HTTPS transfer grants

[earakely-scale]

Summary

This is the second of three PRs that let agents running in local sandboxes exchange files with the local object store over short-lived HTTPS grants, the same way they do with S3. It depends on #9 (atomic writes and kept content types) and is stacked on it. This PR adds the grants and the server that honours them. The next PR turns them on for agents.

LocalFilesystemObjectStore.issue_read_grant, issue_write_grant and issue_upload_policy now return real http-get, http-put and http-post-policy grants. So the grant payloads keep the shape agents already handle, and neither the agents nor the protocol change.

• Grant server. An HTTPS server inside the issuing process: starlette on uvicorn, started by the first grant, in a daemon thread. It has one route, /v1/grants/{token}, and follows the S3 requests the grants stand in for:
  • GET returns the stored bytes and content type, and 404 when there's no object.
  • PUT requires the signed Content-Type (403 otherwise). It rejects a body larger than the grant with a 400, checking both the declared length and while it streams, then replaces the object atomically. Retries are safe.
  • Multipart POST must name its key before its file, as S3 requires. The key has to be a plain relative path under the grant's prefix, and a violation is a 403. The size bound is enforced while streaming, fields and part headers are bounded too, and the form's Content-Type is recorded. Fields after the file are ignored.
  • A token that doesn't verify, has expired or names another method gets a 403. There are no redirects.
• Access logs are off, because they would record tokens. This is done with a quiet H11 protocol rather than access_log=False, which clears the process-wide uvicorn.access logger and would silence any other server in the process.
• Tokens. v1.<claims>.<HMAC-SHA256>, signed with a key held only in this process's memory, so a grant stops working when the process exits. The claims are the operation, the key or key prefix, the expiry, the size bound, the content type and a grant id.
• TLS from a per-user local CA. It is created under <state root>/tls/, with the key 0600, and never installed in host trust stores.
  • It is name-constrained to localhost, host.docker.internal, and loopback or private addresses, so even a leaked key can only vouch for those. Each process gets its own server certificate.
  • ca-bundle.pem is certifi's roots plus the CA, for SSL_CERT_FILE, which replaces the default trust. ca.pem is the CA alone, for trust stores that add to the defaults, such as NODE_EXTRA_CA_CERTS. Both live in a per-CA directory under tls/trust/ (0755), apart from the key, so that directory can be copied into a container whole.
  • A CA within 30 days of expiry, or one that can't be read, is replaced. A process keeps the CA it first loads, and each CA's trust files are written once into their own directory, tls/trust/<fingerprint>/. A renewal by another process therefore never changes what a running process's server or containers trust. A server certificate lasts as long as its CA.
• Where it listens. On 127.0.0.1 for macOS and Windows, where Docker reaches the host's loopback as host.docker.internal. On native Linux it uses the Docker bridge gateway, which --add-host host.docker.internal:host-gateway points at. Grant URLs name host.docker.internal.
  • grant_bind_host and grant_advertise_host override both, via [stores.object] config = { … }.
  • The advertised host must be one the CA can vouch for. root is now optional and defaults to <state root>/object_store.
• Not offered to agents yet. supports_transfer_grants stays False, so choose_transfer still picks the forms it picks today. Turning it on waits for the next PR, which makes local containers trust the CA and checks that an agent can reach the server.
• Dependencies. Core now declares starlette, uvicorn, python-multipart, cryptography and certifi. All were already installed as dependencies of mcp and httpx. uvicorn needs 0.47 or later, for ssl_context_factory.

Test plan

• pytest tst/unit packages/agentenv-protocol/tests -n auto: 5711 passed.
  • Grant cases against the real server over TLS: object_conformance.GRANT_CASES passes.
  • GET: exact bytes and type with no compression, 404 when missing, and HEAD.
  • Method checks: each grant allows only its own method (405 for DELETE); tampered, expired and closed-server grants are refused.
  • PUT: a wrong or missing type gets 403; an oversize body gets 400, both with a declared length and while chunked, and leaves nothing staged; retries overwrite.
  • POST: 403 for keys outside the prefix, traversal, //, ., a sibling, or the store's own files; 400 for an oversize upload, a key after the file, or no file; fields after the file are ignored; the content type is kept.
  • Two stores share one server and keep their own objects.
  • CA: handshakes over memory BIOs, with VERIFY_X509_STRICT, verify for localhost, host.docker.internal and 127.0.0.1, and fail for example.com and 8.8.8.8 because of the name constraints. The bundle's contents, the file modes, reuse, and replacement near expiry or when unreadable are covered.
  • Tokens: round-trip, expiry, another process's key, tampering, and invalid claims.
  • Settings: checked when the store is made; [stores.object] config keys are honoured.
  • Default bind: loopback off Linux, the bridge gateway on Linux, and loopback with a warning when docker can't say.
• Plugin API check against the base: no break.
• Manual, macOS with Rancher Desktop: from a stock agent image (Python 3.11, OpenSSL 3.5, httpx 0.28), with only SSL_CERT_FILE / NODE_EXTRA_CA_CERTS set to the mounted files:
  • httpx GET, PUT and multipart POST against a host server bound to 127.0.0.1, through host.docker.internal, all succeeded.
  • Node fetch succeeded, and public HTTPS still verified.
• pytest -m 'not int_test_slow' tst/integration --ignore=tst/integration/env/gateway/gateway_test.py: 141 passed.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; the latest change addresses the outstanding HEAD finding.

Findings

1.  HEAD reads the whole file ▶

Fix with agent prompt

### Issue 1
src/agent_env/store/object_store/local/grant_server.py:252-253
A `HEAD` request reaches `_get`, which now returns a `StreamingResponse` backed by `_chunks`. The server reads the entire object even though the request returns only headers. For large objects, size checks waste disk work. Return the size and type headers without streaming the file for `HEAD`.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The local object store now issues expiring HTTPS grants, and an in-process server uses them to read and write objects. A per-user local certificate authority and container-reachable addresses support HTTPS, while agents keep their existing transfer-method choice.

• GET grants read one object; PUT and multipart grants write objects within their limits.
• Signed tokens limit each grant by operation, scope, and expiry.
• Local HTTPS trust files and platform-specific bind defaults support access from containers.

Diagram

sequenceDiagram
    participant Client
    participant Store as Local object store
    participant Server as Grant server
    participant Files as Stored files
    Client->>Store: Request transfer grant
    Store->>Server: Register store and sign grant
    Server-->>Client: HTTPS grant URL
    Client->>Server: GET, HEAD, PUT, or POST
    Server->>Server: Check token and method
    Server->>Files: Read metadata or transfer bytes
    Server-->>Client: Response

Loading

_{Reviews (13) · Last reviewed commit: "fix(store): a HEAD through a local read ..."}

[earakely-scale]

The latest summary's two concerns are about the local store code from #9, which this PR stacks on: write-once races on filesystems without hard links, and recorded types disappearing. Both are fixed there, and this branch has merged them:

• Write-once without hard links. A write-once put claims its key with a marker that names its owner's pid. A claim is taken over only once the owner process has exited, and on Windows the non-overwriting os.rename is the claim. (1271813, 5896423)
• Recorded types. A type is paired with its file's size and a per-write unique modification time. Racing writes read back as unknown, never as the other write's type, and a copy of the store that keeps modification times keeps its types. (e91eea9, 5896423)

No inline findings remain open on this PR's own code. The four earlier threads were each fixed or answered in 38846c9.

[greptile-apps]

HEAD reads the whole file

A HEAD request reaches _get, which now returns a StreamingResponse backed by _chunks. The server reads the entire object even though the request returns only headers. For large objects, size checks waste disk work. Return the size and type headers without streaming the file for HEAD.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/agent_env/store/object_store/local_grants/server.py
Line: 252-253

Comment:
**`HEAD` reads the whole file**

A `HEAD` request reaches `_get`, which now returns a `StreamingResponse` backed by `_chunks`. The server reads the entire object even though the request returns only headers. For large objects, size checks waste disk work. Return the size and type headers without streaming the file for `HEAD`.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[earakely-scale]

Fixed in 9ba5dea: a HEAD answers from the open file's size and type and closes it without reading; Starlette's StreamingResponse has no HEAD case, so before this the server streamed the whole object only to drop it. test_a_head_reports_the_size_without_reading_the_object fails if a HEAD reads the object.