perf(store): the local SQLite document store answers indexed lookups from its indexes

[earakely-scale]

What

• Indexed lookups. LocalSqliteDocumentStore now reads only the rows an index finds when a filter pins the leading fields of one of the collection's indexes by equality. The query is WHERE json_extract(doc, '$.<field>') = json_extract(?, '$') … ORDER BY rowid, which is the index's own expression, so SQLite searches the index.
  • find_one, query, count, update, update_one_and_get, replace and delete all read through it, so VersionedEntityStore.next_version() and put_bundled no longer scan.
  • evaluation.matches / sort_docs still run in Python on those rows, in rowid order as before, so results and their order don't change.
• Which filters. Pushdown takes equalities with a str or int value on top-level fields. The value is bound as JSON, so it decodes the same way as the stored field.
  • The index with the most pinned leading fields wins, and a unique index wins a tie. So id + type searches (id, version), not type.
  • Everything else still scans: None, floats, lists, In, dotted paths (which can step through arrays), and fields with no index.
• No migration. Index definitions are read from sqlite_master, so a process that reads before it ensures any index still uses them. SQLite searches (id, version) by id alone, so no new index is added and existing stores open unchanged.
• Indexes another process creates. The per-table index cache drops when PRAGMA schema_version changes, so a long-lived reader starts searching a new index without a restart.
• Lock timeouts. A write that waits out the 5 s busy timeout now raises DatabaseLockedError: <path> is locked: another process has held its write lock for over 5s, which names the database file.
  • Before, it was sqlite3.OperationalError: database is locked, with no path.
  • The first open's switch to WAL raises it too, so @local routing no longer reports a temporary lock as an unreadable store.
  • The busy timeout stays at 5 s.

Why

Every read parsed the whole collection, and the single per-user local store only grows. On main, a find_one by id takes about 70 ms at 10,000 documents and over a second at 100,000. A versioned put runs next_version(), so writes pay the same scan inside their write lock, and put_bundled slows as the store grows.

Testing

Where this was tested. Every row ran against this branch, on macOS (Apple M4 Pro, SQLite 3.50.4), with main as the control.

What	Result
Unit tier	5776 passed, 13 skipped (main: 5752 passed). 15 of the store tests fail against main's src, including all 12 query-plan cases
Query plans, in unit tests and on a 100,000-document file	SEARCH … USING INDEX on this branch, SCAN on main
Differential test (unit)	1,500 seeded operations match a store with no index, which is the old full scan
Benchmark at 1k, 10k and 100k documents	indexed lookups take 0.01–0.03 ms at any size, against 4.5 ms, 70 ms and 1.1 s on main; scans are unchanged
put_bundled of 1,000 files	2.7 s → 1.5 s into an empty store; 79 s → 1.5 s with 10,000 artifacts already present
8 writer processes	23,011 ops in 60 s, against 696 ops in 30 s on main; every write accounted for; integrity_check ok
kill -9 mid-write	20/20 rounds ok on this branch, 5/5 on main
A store written by main	44 reads give the same answers; every pre-existing row stays byte-identical; main reads it back
Lock timeout	DatabaseLockedError naming the file after about 5.4 s; main raises a bare OperationalError
CI	ubuntu: unit tier and both local integration tiers

Query plans. EXPLAIN QUERY PLAN runs on the statements each call actually executes, captured with a trace callback.

• It shows SEARCH … USING INDEX for 12 shapes: find_one by id and by (id, version), query, count, next_version, update, update_one_and_get, replace, delete, a unique index on a tie, a secondary index, and the longest pinned prefix.
• It shows SCAN for the filters that stay in Python.
• On a 100,000-document file, the plan for each call above names docs_artifacts_id_version_unique, and query(type) uses docs_artifacts_type_index. On main every call scans.

Differential test. A seeded run applies 1,500 random reads, updates, upserts, replaces and deletes to 300 random documents. It checks outcomes and order against the same store with no index.

• The documents mix missing fields, str, int, float, bool and null values, big ints, a lone surrogate, NUL, objects and arrays, including under a dotted index.
• On main, its "pushdown was exercised" check fails.

Benchmark. Median ms of 50 calls after 3 warm-up calls, on artifact-shaped documents (1.5 KB, 4 versions per id) with the indexes real stores create. Every answer was checked.

op	1k: main → branch	10k	100k
find_one(id)	4.53 → 0.029	70.7 → 0.028	1082 → 0.029
find_one(id, version)	4.63 → 0.014	70.1 → 0.013	1116 → 0.013
query(id)	5.02 → 0.025	68.2 → 0.024	1077 → 0.024
count(id)	4.75 → 0.025	68.0 → 0.024	1054 → 0.024
next_version	4.81 → 0.027	67.8 → 0.026	1056 → 0.026
update(id, version)	4.65 → 0.117	63.6 → 0.114	1037 → 0.112
count(), no filter (scan)	within noise	within noise	998 → 982

• The 100k branch figures are from a second pass over the same file.
• On the first pass the op that ran first paid for cold pages: find_one(id) took 0.33 ms and update 0.22 ms.
• Runs alternating main and this branch on the unfiltered scan overlap at every size.

Writer contention. 8 processes ran over 10,000 documents of 14 KB. The mix was 30% versioned puts onto 20 shared ids, 20% compare-and-set on 4 counters, 20% updates on 100 hot documents and 30% reads. Each process logged every successful operation, and raw sqlite3 checked the store afterwards.

• This branch, 60 s: 23,011 of 23,012 operations succeeded, at 383 ops/s. Put p50 was 31–47 ms and update p50 0.6 ms.
• main, 30 s: 696 operations, at 23 ops/s. Put p50 was 283–597 ms and update p50 182–223 ms.
• Every write accounted for, on both:
  • every logged put is in the store exactly once;
  • versions per id run 1..n with no gaps or duplicates;
  • each counter's value equals its logged count;
  • update counts match the logs;
  • integrity_check is ok.
• Lock timeouts: neither run hit one. The one failed operation on this branch is under "Not in this PR".

kill -9. A writer did continuous versioned puts and updates with the WAL active (2–4 MB) and was killed at a random point 0.2–1.5 s in. After every round, on this branch (20 rounds) and on main (5 rounds):

• integrity_check was ok;
• no logged put was missing, and no (id, version) was duplicated;
• the store reopened and accepted a write.

Existing store. A store written with main's code through the real APIs:

• Contents: entity puts, deletes, updates and replaces; unicode and quote/backslash ids; a version stored as 2.0; the index setup of the task-instance, env-artifact, run, conversation and ledger stores; compare-and-set; and an edge collection with bool, float, null, absent and array values in indexed fields.
• Reads: with this branch, 44 reads gave the same answers as main.
• Rows: sqlite_master, the schema version and all 98 pre-existing rows stayed byte-identical after this branch read and wrote.
• Back on main: it then read the store back with the same answers.

Lock timeout. Another process held the write lock, first under BEGIN IMMEDIATE with an uncommitted insert, then under BEGIN EXCLUSIVE.

• Branch: insert, update, an entity put, a new ensure_index and delete each raised DatabaseLockedError after 5.37–5.42 s, with a message naming the file: …/store.db is locked: another process has held its write lock for over 5s.
• main: they raised sqlite3.OperationalError: database is locked.
• Both: a read under the lock returned at once.

Not in this PR

• Array-valued key fields. An equality on an indexed top-level field assumes that field holds a scalar. SQLite indexes an array whole, where Mongo matches any element; the unique indexes already work this way. agent-env writes only scalars into indexed fields, and the module docstring says so. A plugin index on an array field would miss documents whose array contains the value. In a probe, Eq("red") matched 2 documents on main and 1 here.
• VersionedEntityStore.put's retry budget. Its 5 retries can run out when several processes put the same few ids hundreds of times a second, which this branch makes reachable. It raises DuplicateKeyError and writes nothing: 1 of 6,863 puts in the 8-process run.
• One id's versions are all loaded. next_version() and get(id) now read only that id's rows, but they still load and sort all its versions in Python. With 300–390 versions of 14 KB each, that is what puts cost in the contention run.
• Unindexed filters still scan: listings, count(), In, and filters on unindexed fields.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no new actionable issue was found.

Summary

This PR makes indexed equality lookups in the local SQLite store read candidate rows instead of whole collections, while keeping Python matching and sorting. It also gives write-lock timeouts a path-specific error.

• Indexed lookups use existing indexes; other filters still scan.
• Lock timeout errors now name the database file.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  F[Document filter] --> I{Index prefix matches?}
  I -- Yes --> R[Read matching rows]
  I -- No --> A[Read all rows]
  R --> P[Check filter in Python]
  A --> P

Loading

_{Reviews (4) · Last reviewed commit: "revert(cli): keep the SQLite backend's l..."}

[earakely-scale]

@greptileai review

On the lock timeout reaching CLI users as a traceback: that's a deliberate trade-off. Listing DatabaseLockedError in _USER_FACING_ERRORS made the CLI import the SQLite backend, and the CLI should depend on the store abstraction, not on one backend.

The timeout still raises DatabaseLockedError, and its message, now the traceback's last line, names the file: <path> is locked: another process has held its write lock for over 5s. Before this PR it was a bare database is locked.

@local routing still doesn't report a lock as "can't be read; move it aside". That comes from routing catching only sqlite3.DatabaseError, not from the CLI.