src, scripts: use kv to cache directories

[flakey5]

Makes use of KV for directory listings. This aims to replace the build-r2-symlinks.mjs file with two different ones:

• build-directory-cache.mjs - This reads the entire dist-prod bucket and caches it into the KV namespace. This is pretty much only for the initial setup of the cache and for anytime we need to rebuild all of it.
• update-directory-cache.mjs - This reads only the necessary paths from the dist-prod bucket and updates them in the KV namespace. This will be the one that we will call in the workflow

A USE_KV environment variable is added to the worker that acts as a toggle between KV and S3 listings. This acts as a safeguard in case something goes wrong while we're still testing it, so we can easily switch between the two. Once we're more confident in KV, we can remove it + the entire S3Provider from the worker.

Closes #159
Closes #314

TODO:

• Pass in version argument in Node core when invoking the update redirect links action: meta: pass release version to release worker node#62777

[ovflowd]

cc @flakey5 some of the CI steps are failing

[ovflowd]

Didn't finish the review yet, but leaving initial comments here :)

[cursor]

PR Summary

Medium Risk
Introduces a new KV-backed directory listing path plus automation that writes large batches into a production KV namespace; mistakes could cause stale/incorrect directory listings or failed release automation, though a USE_KV toggle and S3 fallback reduce blast radius.

Overview
Moves directory listings toward a KV-backed cache (with a safety toggle). The worker Env gains DIRECTORY_CACHE and USE_KV, and R2Provider.readDirectory() can now serve listings from a new KvProvider (with a temporary Sentry-reported parity check against the existing cached/S3 results). S3Provider is simplified to delegate listing to a shared lib/listR2Directory.mjs helper, and contentTypeOverrides is converted from JSON to a typed TS constant.

Replaces the old symlink/listing generation script with KV cache build/update tooling and CI automation. Adds scripts/build-directory-cache.mjs (full rebuild + optional purge) and scripts/update-directory-cache.mjs (incremental per-release update), plus supporting utilities for listing R2, computing latestVersions.json, and injecting synthetic symlinks into cached listings (also updating cachedDirectories.json, docsDirectory.json, and fileSymlinks.json). GitHub Actions gains a new Build Directory Cache workflow and updates Update Redirect Links to require a version input and run the KV update script; Wrangler config and Vitest/miniflare are updated to bind the KV namespace, with new unit/e2e tests covering KV directory behavior.

^{Reviewed by Cursor Bugbot for commit 42eae4e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 3 potential issues.

Autofix Details

Bugbot Autofix prepared fixes for all 3 issues found in the latest run.

• ✅ Fixed: Version argument crashes before null check executes
  • I now validate the raw version argument before calling toLowerCase() so missing input throws the intended explicit error.
• ✅ Fixed: Subdirectories cached without trailing slashes in e2e tests
  • The e2e cache population now appends / to each subdirectory name so test data matches production directory-cache shape.
• ✅ Fixed: latest key incorrectly added as directory symlink
  • I restricted added directory symlinks to keys starting with latest-, which excludes both latest and node-latest.tar.gz.

Or push these changes by commenting:

@cursor push 29199a84ef

Preview (29199a84ef)

diff --git a/e2e-tests/util.ts b/e2e-tests/util.ts
--- a/e2e-tests/util.ts
+++ b/e2e-tests/util.ts
@@ -47,7 +47,9 @@
   });
 
   const cachedDirectory: ReadDirectoryResult = {
-    subdirectories: Object.keys(directory.subdirectories),
+    subdirectories: Object.keys(directory.subdirectories).map(
+      subdirectory => `${subdirectory}/`
+    ),
     files,
     hasIndexHtmlFile,
     lastModified: new Date(),

diff --git a/scripts/update-directory-cache.mjs b/scripts/update-directory-cache.mjs
--- a/scripts/update-directory-cache.mjs
+++ b/scripts/update-directory-cache.mjs
@@ -20,12 +20,13 @@
 import { createCloudflareClient, writeKeysToKv } from './utils/kv.mjs';
 import { createS3Client, listR2DirectoryRecursive } from './utils/r2.mjs';
 
-const VERSION = process.argv[2].toLowerCase();
-
-if (!VERSION) {
+const VERSION_ARG = process.argv[2];
+if (!VERSION_ARG) {
   throw new TypeError('version missing from args');
 }
 
+const VERSION = VERSION_ARG.toLowerCase();
+
 if (!VERSION.startsWith('v')) {
   throw new TypeError(
     'provided version not in correct format, expected vX.X.X'

diff --git a/scripts/utils/addSymlinksToDirectoryCache.mjs b/scripts/utils/addSymlinksToDirectoryCache.mjs
--- a/scripts/utils/addSymlinksToDirectoryCache.mjs
+++ b/scripts/utils/addSymlinksToDirectoryCache.mjs
@@ -55,8 +55,8 @@
 ) {
   releaseDirectory.subdirectories.push(
     ...Object.keys(latestVersionMap)
-      // We only want directories, remove the only file in the map
-      .filter(version => version !== NODE_LATEST_FILE_NAME)
+      // We only want the `latest-*` directory symlinks.
+      .filter(version => version.startsWith('latest-'))
       .map(version => `${version}/`)
   );
   releaseDirectory.subdirectories.sort();

diff --git a/scripts/utils/addSymlinksToDirectoryCache.test.ts b/scripts/utils/addSymlinksToDirectoryCache.test.ts
--- a/scripts/utils/addSymlinksToDirectoryCache.test.ts
+++ b/scripts/utils/addSymlinksToDirectoryCache.test.ts
@@ -97,6 +97,7 @@
   const latestVersionMap: LatestVersionMapping = {
     'latest-v20.x': 'v20.1.2',
     'latest-v22.x': 'v22.2.1',
+    latest: 'v22.2.1',
     'node-latest.tar.gz': 'latest/node-v22.2.1',
   };

_{This Bugbot Autofix run was free. To enable autofix for future PRs, go to the Cursor dashboard.}

[ovflowd]

As I said before, it is hard to review so many complex changes in a timely manner and one pass.

@flakey5 can you do me a favor? Can you address BugBot's reviews and maybe also ask Claude Code to scrutinize your code and do a 2nd pass in terms of code smell and what not? And then let's get this merged.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 42eae4e. Configure here.}

[cursor]

Error message displays undefined instead of file path

Low Severity

The error message in addStaticFileSymlinksToCache interpolates sourceFile (which is undefined at this point, since the !sourceFile check just passed) instead of the actual target path. The message will always read "points to invalid file 'undefined'". The variable sourceFilePath (or fileSymlinks[symlink]) holds the meaningful path that was looked up and not found.

 

^{Reviewed by Cursor Bugbot for commit 42eae4e. Configure here.}