fix(store): the local object store writes atomically and keeps content types

[earakely-scale]

Summary

This is the first of three PRs that let agents running in local sandboxes exchange files with the local object store over short-lived HTTPS grants. Those grants are what the S3 store already issues. This PR makes the local store's writes safe to serve over HTTP. It doesn't serve anything yet.

• Atomic writes. put and put_file write into a reserved .agentenv-tmp directory inside the root, then move the file into place:

  • an overwrite renames over whatever is there;
  • a write-once put claims the key with a hard link, so a put that loses a race raises ObjectAlreadyExistsError and leaves the winner in place;
  • on filesystems without hard links, the key's lock (below) makes the existence check and the rename one step, so a racing writer finds the winner. On Windows, the non-overwriting os.rename is the claim.

  Readers never see a partial object, and a crash mid-write leaves nothing at the key.

• Each key has a lock. It is an exclusive flock for a write, held across placing the bytes and recording the type, and a shared one for a metadata read. So no write lands between another's two steps, and no reader sees one write's type with another's bytes, whatever the filesystem's timestamp resolution.

  • Keys share 64 lock files in the staging directory, so locking leaves no file behind per key.
  • The kernel releases a lock whose holder dies.
  • Windows has no flock and relies on the per-write timestamp below, which NTFS's 100 ns times preserve.

• Content types are kept. A write's content type is recorded under a reserved .agentenv-meta tree and returned by get_object_metadata, as S3 returns it. The record carries the identity of the file it describes: its size, and a modification time each write makes unique (now, to the millisecond, plus a random remainder), which a rename keeps and so does a copy that keeps modification times. So a type recorded for another write reads back as unknown rather than wrong. An overwrite also drops the old type before its bytes land, so one cut short leaves the type unknown, never the previous one. The default application/octet-stream is not recorded, so a write that names no type still reads back as unknown. That keeps FileArtifact.put_existing and other readers guessing the type from the file name, as they do today. Objects written before this change read back as unknown too.

• Leftover staged files are swept. A store's first write removes staged files that a dead write left behind, meaning ones untouched for an hour.

• Reserved names. .agentenv-tmp and .agentenv-meta are refused as keys, like .gitignore, and left out of list / list_at.

Test plan

• pytest tst/unit packages/agentenv-protocol/tests -n auto: 5635 passed. The new tests cover:
  • kept, default and overwritten content types, and objects written before types were kept;
  • a failed write leaving no object;
  • a write-once put losing a race;
  • the no-hard-link fallback: a failed write leaving no object, a racing put keeping the winner, and a claim that lasts exactly as long as its holder;
  • types surviving a copy that keeps modification times;
  • a type recorded for another write, and an untyped overwrite, reading back as unknown;
  • a metadata read waiting for a write holding the key, an overwrite cut short leaving the type unknown, and the sweep of dead writes' staged files;
  • reserved keys, and listings that skip the store's own files.
• pytest -m 'not int_test_slow' tst/integration --ignore=tst/integration/env/gateway/gateway_test.py: 141 passed, run with the branch's venv first on PATH for the CLI tests.
• Chaos testing, with a harness kept outside the repo, on macOS (APFS, and an exFAT disk image) and Linux (Ubuntu 22.04, kernel 6.8: ext4, tmpfs and a vfat loop mount).
  • Churn with random kills. 16 writers and 4 readers ran for 90 s. Writers did write-once puts on contested keys and overwrites, typed and untyped, while one writer was SIGKILLed every 20–200 ms: 409–772 kills per run. Across every filesystem there were zero violations: no torn object read or at rest, at most one winner per write-once key, no wrong type at rest, and no key blocked afterwards.
  • Same-key type races. Two processes overwrote one key at the same instant with equal-size payloads of different types. This targets filesystems whose timestamps are too coarse for the per-write timestamp (exFAT 10 ms, vfat 2 s). Before the key lock, exFAT mislabelled 5 of 4,129 races and vfat 87 of 4,727. With it there were 0 wrong types in 4,891 (exFAT), 6,173 (vfat), 7,018 (APFS), 6,587 (ext4) and 6,989 (tmpfs) races.
  • Disk full. A full exFAT volume made writes fail cleanly with ENOSPC, leaving no partial object.

🤖 Generated with Claude Code

Confidence Score: 4/5

The PR does not yet appear safe to merge because a failed overwrite can still lose the saved content type of the unchanged object.

Findings

1.  Old type records linger ▶

Fix with agent prompt

### Issue 1
src/agent_env/store/object_store/local_object_store.py:211-212
If removing a saved type record fails after an overwrite, the write succeeds but leaves that record in `.agentenv-tmp`. This store sweeps only on its first write, so a long-lived store will not remove it. Windows does not sweep at all. Repeated cleanup failures can leave old records behind and use disk space.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The local object store now stages writes before placing complete objects at their keys and saves non-default content types with the files they describe. It also keeps its staging and metadata paths out of object keys and listings.

• Write-once puts keep the first successful claim, including when hard links are unavailable.
• Metadata reads return a saved type only when its record matches the current file.
• Listings skip the store’s reserved paths.

_{Reviews (8) · Last reviewed commit: "fix(store): an overwrite that lands is n..."}

[greptile-apps]

Old type records linger

If removing a saved type record fails after an overwrite, the write succeeds but leaves that record in .agentenv-tmp. This store sweeps only on its first write, so a long-lived store will not remove it. Windows does not sweep at all. Repeated cleanup failures can leave old records behind and use disk space.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/agent_env/store/object_store/local_object_store.py
Line: 211-212

Comment:
**Old type records linger**

If removing a saved type record fails after an overwrite, the write succeeds but leaves that record in `.agentenv-tmp`. This store sweeps only on its first write, so a long-lived store will not remove it. Windows does not sweep at all. Repeated cleanup failures can leave old records behind and use disk space.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[earakely-scale]

Leaving this as is. A record is left behind only when deleting a file the store itself just renamed into its own staging directory fails. That means the filesystem is refusing deletes in a directory the store otherwise writes freely. The cost is one small hidden file per failure, and it never makes a type wrong. The next store instance on a POSIX filesystem sweeps it. On Windows it stays until removed by hand. Sweeping on every write, or retrying in the background, would add cost to the common path for a fault that already breaks writes elsewhere. On the summary line: an overwrite whose rename fails puts the old type back. The type is lost (read as unknown, never wrong) only if the set-aside record also vanished in the meantime, which requires another process's sweep while this commit was paused for over a minute.