Skip to content

refactor(core): move the models.dev catalog cache from disk to KV - #41649

Merged
kitlangton merged 2 commits into
v2from
models-dev-kv
Aug 11, 2026
Merged

refactor(core): move the models.dev catalog cache from disk to KV#41649
kitlangton merged 2 commits into
v2from
models-dev-kv

Conversation

@kitlangton

Copy link
Copy Markdown
Contributor

What

Move the implicit models.dev provider/model catalog cache from Global.cache files into the core KV service.

The catalog still fetches on a cache miss, refreshes stale entries on the existing five-minute cadence, recovers from corrupt cached blobs, and serves an existing cached catalog when a refresh fails.

Why

Core services should not use the filesystem to acquire the server's own state. The disk cache breaks or silently becomes ineffective in filesystem-less runtimes, while the catalog cache is naturally a keyed blob.

Using the database-backed KV service makes the cache available across every core runtime, including DO-SQLite/workerd. This continues the filesystem-acquisition direction from #41618, #41622, and #41629.

How

  • Store the canonical endpoint under models-dev:catalog and custom endpoints under models-dev:catalog:<source-hash>.
  • Store { updatedAt, body }, retaining the raw JSON blob for corruption detection and replacing file mtime with an explicit freshness timestamp.
  • Decode and remove malformed KV entries before falling through to the bundled snapshot or HTTP fetch.
  • Keep stale cache entries readable; updatedAt only controls whether non-forced refresh can skip the network.
  • Replace the filesystem lock with a service-local semaphore and re-check KV after acquiring it so concurrent work in one server instance does not duplicate a miss or refresh.
  • Preserve the explicit models.file / OPENCODE_MODELS_PATH input. FSUtil remains only for that configured file source; the implicit cache no longer uses it. Global and Flock are removed from the service graph.

Testing

  • cd packages/core && bunx tsgo --noEmit
  • cd packages/core && bun run test test/models.test.ts test/plugin/models-dev.test.ts test/catalog.test.ts (28 passed)
  • cd packages/core && bun run test (1,634 passed, 16 skipped, 0 failed across 172 files)
  • Pre-push bun turbo typecheck --concurrency=3 (33 tasks successful)

Findings

  1. Existing semantics: the default cache was <Global.cache>/models.json; custom source URLs used <Global.cache>/models-<Hash.fast(source)>.json. Each file contained the raw /api.json response. get() accepted a readable cache regardless of age and memoized the normalized result indefinitely in-process. On a miss, get() fetched and atomically wrote via a temporary file plus rename. A five-minute file-mtime TTL only controlled refresh(false): refresh ran once at service startup and then five minutes after each completion, rechecked freshness under Flock, overwrote the file on success, invalidated the in-memory value, and published models-dev.refreshed. Forced refresh bypassed freshness. Refresh errors were logged and ignored, leaving the stale disk and in-memory values available. A malformed implicit cache file was removed and treated as a miss.
  2. Other readers: no CLI, runtime code, docs, or tools read either implicit cache filename. The only exact-path consumer was packages/core/test/models.test.ts, which is ported to an in-memory KV service. OPENCODE_MODELS_PATH is a separate explicit catalog file input used by the CLI/test preload and remains supported. Other models-*.json references belong to unrelated fixtures or console exports.
  3. KV design: models-dev:catalog gives the canonical catalog a stable, discoverable key; models-dev:catalog:<source-hash> preserves isolation for custom source URLs without embedding arbitrary URLs in database keys. The value remains one blob plus updatedAt, matching the cache's storage shape and preserving the old raw-response/corruption behavior while making the five-minute timestamp portable.
  4. Migration: silently abandoning the old file is safe because it is a re-fetchable cache and nothing else consumes its path. Existing installations perform one cache miss and repopulate KV. No migration/import path is needed; the obsolete temp-file write, corrupt-file cleanup, and file-lock code are removed. An already-present old cache file may remain on disk but is no longer read or modified.

@kitlangton
kitlangton marked this pull request as ready for review August 11, 2026 02:24
@kitlangton

Copy link
Copy Markdown
Contributor Author

Simplify pass complete and pushed in cf0cf928d8.

Applied

  • Composed the KV envelope and JSON body into one Schema codec. Cache reads now perform one decode and one malformed-entry cleanup branch.
  • Moved the non-forced freshness check inside the service-local semaphore. A refresh now performs one KV read/decode while queued refreshes still observe the preceding refresh's timestamp; forced refresh still bypasses freshness.
  • Extracted the canonical/custom source key derivation into cacheKey(source) with one defaultSource constant.
  • Replaced the bespoke Layer.succeed/KV.Service.of wrapper with the repository-standard Layer.mock form.
  • Corrected the single-flight test to start from an empty KV cache with fetching enabled and assert exactly one HTTP request across concurrent get() calls.

Rejected

  • Did not remove the semaphore or the miss-path re-check after acquiring it. cachedInvalidateWithTTL only single-flights get() calls; the semaphore still coordinates population with refresh, and the post-wait KV read prevents a duplicate fetch.
  • Did not store the parsed catalog directly or split timestamp metadata into another key. Keeping { updatedAt, body } as one raw blob preserves atomic cache replacement and the intended corruption/recovery boundary.
  • Did not extract a shared KV test fixture or switch these tests to the database-backed KV node. No reusable fixture exists today, and introducing one across unrelated tests would expand scope without simplifying this branch's behavior tests.
  • Did not skip KV writes in explicit models.file mode or add a database lease for cross-process locking. The former changes existing refresh behavior; the latter is not supported by the current KV interface and would add a new coordination protocol rather than simplify this change.

Verification remains green:

  • cd packages/core && bunx tsgo --noEmit
  • cd packages/core && bun run test test/models.test.ts test/plugin/models-dev.test.ts test/catalog.test.ts (28 passed)
  • Pre-push bun turbo typecheck --concurrency=3 (33 tasks successful)

@kitlangton
kitlangton merged commit 518af92 into v2 Aug 11, 2026
8 checks passed
@kitlangton
kitlangton deleted the models-dev-kv branch August 11, 2026 02:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant