Skip to content

feat(remote-cache): add self-hosted public cache server - #718

Draft
fengmk2 wants to merge 17 commits into
remote-cache-fetch-404from
feat/public-remote-cache
Draft

fengmk2 wants to merge 17 commits into
remote-cache-fetch-404from
feat/public-remote-cache

Conversation

@fengmk2

@fengmk2 fengmk2 commented Sep 12, 2026 •

Copy link
Copy Markdown
Member

Add packages/remote-cache for the server in #716, using Cloudflare Workers, primary D1 metadata, and a private R2 bucket. The server supports public reads and GitHub Actions OpenID Connect checks for writes. Stores use streaming uploads, atomic publication, storage limits, and automatic data expiry.

One workflow deploys related changes to a persistent staging Worker, D1 database, and R2 bucket. Internal PRs, main pushes, and manual runs share these resources. The workflow runs deployments and smoke tests in sequence, then updates PR comments with the tested revision and manual instructions. Each deployment replaces the previous revision. Closing a PR keeps staging available. Setup uses repository secrets and variables. Repository maintainers can configure staging without a GitHub environment.

The e2e plan defines automated checks and manual exercises. PR runs check public reads and rejected writes. Pushes to main also check authorized uploads, multipart storage, replacement, concurrency, and quotas. Local tests repeat smoke checks against reused storage and retain maximum-payload and scheduled-handler coverage. Real Cron execution and maximum payloads on Cloudflare require separate release exercises.

/fetch returns only { kind: "fallback", key } for fallback matches and does not read R2. Exact matches return the value and blob_id. A missing or unreadable exact value returns 503. Fetch misses and unavailable blobs return plain-text 404, as specified in the local RFC.

Setup rejects incompatible origins and repositories before it changes existing policies or saved configuration. Each Worker has separate rate-limit counters that remain stable across revisions. Workers Free CPU support remains unverified. The guide recommends Workers Paid for the full payload limits.

Motivation

Maintainers need a cache service in their own Cloudflare account. Developers and fork contributors must reuse public task results without login. Only trusted jobs on main should publish those results. Reviewers need one persistent staging environment and a clear verification result after each related change.

@socket-security

socket-security Bot commented Sep 12, 2026 •

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Sep 12, 2026 •

Copy link
Copy Markdown

fspy benchmark

linux

dynamic/launch             change  +0.98%  [ -7.12% .. +10.62%]  overhead  +271.57%
dynamic/access             change  +0.15%  [ -1.35% ..  +1.68%]  overhead   +14.09%
dynamic/access-relative    change  +0.11%  [ -0.88% ..  +0.89%]  overhead   +60.79%
dynamic/access-contended   change  -0.66%  [ -4.30% ..  +1.13%]  overhead   +15.45%
static/launch              change  -0.06%  [ -5.30% ..  +5.84%]  overhead  +739.53%
static/access              change  +0.48%  [ -0.74% ..  +2.07%]  overhead  +801.43%
static/access-relative     change  +0.02%  [ -2.29% ..  +1.54%]  overhead +1354.68%
static/access-contended    change  +0.01%  [ -0.91% ..  +1.07%]  overhead +3126.99%

macos

dynamic/launch             change  -0.17%  [ -4.00% ..  +3.57%]  overhead  +248.56%
dynamic/access             change  +0.35%  [-10.52% .. +10.21%]  overhead    +4.67%
dynamic/access-relative    change  +1.54%  [ -9.31% .. +54.69%]  overhead  +282.82%
dynamic/access-contended   change  +0.38%  [ -8.58% .. +703.53%]  overhead    -0.54%

windows

dynamic/launch             change  +2.67%  [ -6.40% ..  +9.41%]  overhead   +25.86%
dynamic/access             change  +0.19%  [ -1.48% ..  +1.13%]  overhead    +1.36%
dynamic/access-relative    change  -0.18%  [ -1.28% ..  +0.75%]  overhead    +1.12%
dynamic/access-contended   change  -0.36%  [ -6.94% ..  +0.91%]  overhead    +1.12%

@github-actions

github-actions Bot commented Sep 12, 2026 •

Copy link
Copy Markdown

Remote cache staging

Commit: c2f00bdb2afc91053b189d826e3f713a1137f5fc

Cloudflare staging deployment and smoke tests passed. You can now perform manual verification.

Endpoint: https://vp-cache-ci-staging.voidzero-docs.workers.dev/projects/manual

Download the remote-cache-e2e-36336249075-1 artifact from the workflow run. It contains manual-fetch.cbor, manual-manifest.json, and report.json.

PR checks cover public reads and rejected writes. Main-branch push checks also cover authorized HTTP stores. This staging endpoint is shared by all PRs and main. A later deployment replaces its code and manual fixture. Compare the deployment ID in the response with the artifact before manual verification. Closing this PR does not remove staging.

Manual checks and complete e2e plan.

@fengmk2
fengmk2 force-pushed the feat/public-remote-cache branch from 194d39f to e461d47 Compare September 14, 2026 15:16
@fengmk2

fengmk2 commented Sep 16, 2026

Copy link
Copy Markdown
Member Author

Deploy prompt:

Refer to the deployment instructions at https://github.com/voidzero-dev/vite-task/blob/feat/public-remote-cache/packages/remote-cache/docs/self-hosting.md to deploy a remote cache service for the current repo.

@wan9chi
wan9chi force-pushed the feat/public-remote-cache branch from df2d129 to e127e9b Compare September 27, 2026 16:04
@wan9chi
wan9chi changed the base branch from main to remote-cache-hardening September 27, 2026 16:04
@wan9chi
wan9chi added this pull request to stack #759 September 27, 2026 16:04
Base automatically changed from remote-cache-hardening to main September 27, 2026 16:32
@wan9chi
wan9chi force-pushed the feat/public-remote-cache branch from e127e9b to d4b0fea Compare September 27, 2026 16:32
fengmk2 and others added 17 commits September 28, 2026 01:07
Co-authored-by: GPT-6 Codex <codex@openai.com>
Co-authored-by: GPT-6 Codex <codex@openai.com>
Co-authored-by: GPT-6 Codex <codex@openai.com>
Co-authored-by: GPT-6 Codex <codex@openai.com>
Copy RFC #716 from c201f8e and adjust relative paths.

Co-authored-by: GPT-6 Codex <codex@openai.com>
Co-authored-by: GPT-6 Codex <codex@openai.com>
Co-authored-by: GPT-6 Codex <codex@openai.com>
Co-authored-by: GPT-6 Codex <codex@openai.com>
Co-authored-by: GPT-6 Codex <codex@openai.com>
Co-authored-by: GPT-6 Codex <codex@openai.com>
Remove the obsolete cache size study and its RFC link.

Co-authored-by: GPT-6 Codex <codex@openai.com>
Co-authored-by: GPT-6 Codex <codex@openai.com>
Co-authored-by: GPT-6 Codex <codex@openai.com>
`npm_execpath` points to pnpm's standalone executable on the Linux and
Windows runners, so running it through `node` failed. Run it directly
unless it's a JavaScript entry point.

The remote cache package also brought esbuild and workerd into the
workspace, and a root `pnpm install` without `--ignore-scripts` failed
on their unapproved build scripts. Allow them, as the standalone package
already does.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…service

`remote-cache-server` now runs the Worker from `packages/remote-cache`
in workerd through Miniflare instead of the Node test backend, so the
e2e tests exercise `vp run` against the real service.

The wrapper proxies requests to the service. It signs an upload token
for each store with a key the service fetches in place of GitHub's. It
numbers the service's random blob IDs in upload order for the request
log, and copies each blob to `remote-cache/blobs/<number>`, storing a
changed copy on the next run so tests can still corrupt an archive.

The only snapshot change is the request log for a fetch miss, which the
service answers with a 404. The cases are skipped on musl because
workerd's prebuilt binaries require glibc.

The `remote_cache_backend` fixture only tested the old backend, and the
service has its own protocol tests, so it's removed along with
`cbor-http` and its EDN formatting.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@wan9chi
wan9chi force-pushed the feat/public-remote-cache branch from 58347c9 to c2f00bd Compare September 27, 2026 17:15
@wan9chi
wan9chi removed this pull request from stack #759 September 27, 2026 17:15
@wan9chi
wan9chi changed the base branch from main to remote-cache-fetch-404 September 27, 2026 17:15
@wan9chi
wan9chi added this pull request to stack #773 September 27, 2026 17:16

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants