Skip to content

feat: add support for D1 Sessions API to mitigate stale data returned from replicas - #813

Open
kossnocorp wants to merge 12 commits into
mainfrom
fix/783-d1-bookmark
Open

kossnocorp wants to merge 12 commits into
mainfrom
fix/783-d1-bookmark

Conversation

@kossnocorp

@kossnocorp kossnocorp commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

This PR closes #783.

It adds two new primitives into D1 void/db that utilize D1 Session API:

  • getBookmark - returns the current session bookmark
  • setSession - starts a new session from a previous bookmark.

Using these allow to solve the D1 Replicas problem in the scenario read OLD -> write NEW -> read OLD and ensure sequential consistency.

Warning

Using Sessions API requires understanding of the problem, introduction of a middleware and in case with client requests, using a custom fetch wrapper.

There's no plug-in-play solution that would make it work automatigally, and to completely remove the possibility of users seeing stale data requires either explicit reading from primary or complex session coordinator code.

I suggest to consider disabling replicas by default and designing a thoughtful API on top of these primitives that would provide great DX for users that opt-into using replicas.

See guides for more information and usage examples:

TL;DR;

To utilize Sessions API the user has to create a middleware:

// middleware/01.database.ts
import { getCookie, setCookie } from 'hono/cookie';
import { defineMiddleware } from 'void';
import { getBookmark, setSession } from 'void/db';

export default defineMiddleware(async (c, next) => {
  setSession(getCookie(c, 'db-bookmark') ?? 'first-unconstrained');
  await next();

  const bookmark = getBookmark();
  if (bookmark) {
    setCookie(c, 'db-bookmark', bookmark, {
      httpOnly: true,
      secure: new URL(c.req.url).protocol === 'https:',
      sameSite: 'Lax',
      path: '/',
    });
  }
  c.header('Cache-Control', 'private, no-store');
});

To make it work with client requests, instead of the cookie (or along with the cookie transport), the devs should use a header, e.g., X-DB-Bookmark and make requests through a wrapper that sets the bookmark.

Why disabling replicas?

TL;DR; replicas are not safe and might result in support tickets, bugs and security issues when enable unknowingly.

Enabling them by default creates two problems not present in a default D1 setup and not common in other environments:

  • User might see a stale data right after they perform a write request.

  • User might see old data after seeing a new one.

  • An action triggering an account suspension might not propagate the account state and allow already suspended account to perform an action they should not be able. See Avoid stale reads with a primary-first session.

While the first two are annoyances and just a source of support requests, the last one is a security issue.

The fact that we enable replicas by default is hidden in the documentation and the user, even experienced with D1, could get caught by surprise and deploy unsafe code unknowingly.

Making it opt-in would allow devs that want/need replicas to understand cons and decide if they want it. Those who decide to opt-in will keep in mind these nuances and work around them. If they need the advantage that replicas provide, they will be happy to use said APIs regardless how awkward they are and how many docs they have to read.

For some apps it would be completely normal, but not for all.

Why not incorporate a bulletproof solution into Void?

The problem is that every query could produce fresh bookmark, even a read. So we have to track all request bookmarks and resolve the newest one.

The single cookie approach described in the guide breaks with multiple requests where a newer request could overwrite a fresh bookmark with an older one (i.e. dashboard).

So we'll have to track every query bookmark and store each in a separate cookie, so a middleware could resolve the freshest one. It might lead to hitting server/browser limits on the cookies number and break. Not to mention overhead of carrying these cookies with every request.

A built-in multi-cookie solution could work out and be a good default middleware for users that opt-in and ok with a possibility that the describe problems might appear.

But adding this overhead and still not giving 100% guarantee doesn't sound like a good default.


Reproduction

In cases where primary D1 is deployed to say WEUR but the nearest edge is SIN right after write, the SIN replica might not get the update fast enough and the user will see data before edits.

Unless you're in Europe, where the primary D1 is deployed, you can see the problem in the playground (you might not have the access, ping me): https://voidzero-void-d1-bookmarks.skoss-individual-account.workers.dev/without-bookmark?experiment=immediate&trials=20&pressure=0.5&size=0.5&probes=20

To reproduce:

  1. In "Read-only routing baseline" section click "Run probes" button. Repeat until you see the replica region closest to you (replica / APAC / SIN in my case:

    image
  2. Then go a section below "Write/read experiment" and set:

    • Read timing: Immediate server read
    • ✅ Add replication pressure
    • Pressure per trial: 512 KiB
    • Trials: 20
  3. Press "Run Trials" button. Unless you're unlucky or D1 got significant replication pressure and switched from replica to the primary D1, you'll see some of the requests getting stale data (stale data is only possible on a replica, SIN in my case):

image

Depending on your region and current state of D1 instance you might not see stale data at all. If you see that read target is primary / WEUR wait for a 1-2 minutes and start from 1. You can also increase replication pressure a bit. A high pressure makes D1 switch to primary and only hit replicas occasionally. Most likely you will encounter a network error caused by D1 being overwhelmed.


The working solution is on the "Manual bookmark" page where data is always fresh: https://voidzero-void-d1-bookmarks.skoss-individual-account.workers.dev/with-bookmark?experiment=immediate&trials=20&probes=20&pressure=0.5&size=0.5

image

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

🚀 Deploying Preview to Cloudflare 🚀

Preview URL: https://fix-783-d1-bookmark-void-cloud.voidzero-docs.workers.dev (commit a364521)

This URL reflects your latest Preview deployment

Preview Deployments by commit

Status Deployment URL Commit Updated (UTC) See this deployment's details
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://5d8f73a3-void-cloud.voidzero-docs.workers.dev a364521 2026-11-16T08:36:27.951Z Visit the dashboard ↗
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://e6d95be1-void-cloud.voidzero-docs.workers.dev 79b00d5 2026-11-16T08:30:44.179Z Visit the dashboard ↗
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://9545e410-void-cloud.voidzero-docs.workers.dev 6bd4980 2026-11-16T07:25:16.040Z Visit the dashboard ↗
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://c9005645-void-cloud.voidzero-docs.workers.dev 552cda1 2026-11-16T07:22:35.759Z Visit the dashboard ↗
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://d5572bed-void-cloud.voidzero-docs.workers.dev ca8896e 2026-11-16T06:11:58.365Z Visit the dashboard ↗
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://0c83a8a5-void-cloud.voidzero-docs.workers.dev 0f6dfa8 2026-11-16T06:03:57.634Z Visit the dashboard ↗
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://5dea4e56-void-cloud.voidzero-docs.workers.dev ff403b4 2026-11-16T01:08:47.835Z Visit the dashboard ↗
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://d9d1e272-void-cloud.voidzero-docs.workers.dev b8f635e 2026-11-13T10:07:54.715Z Visit the dashboard ↗

Comment thread packages/void/src/codegen/db-types.ts Resolved
@yangshun

Copy link
Copy Markdown
Collaborator
  • PR Better organize runtime modules #841 reorganized the runtime files into feature directories.
  • Please resolve the merge conflicts against the latest main and update imports and build/export paths that still use the old layout.
  • This PR adds runtime/db-session.ts; please place it under runtime/db/.

@kossnocorp

Copy link
Copy Markdown
Contributor Author

@yangshun thank you for the heads-up!

@kossnocorp kossnocorp changed the title feat: add support for D1 bookmarks to fight stale data returned from replicas feat: add support for D1 Sessions API to mitigate stale data returned from replicas Nov 16, 2026
@kossnocorp
kossnocorp marked this pull request as ready for review November 16, 2026 08:46
@kossnocorp
kossnocorp requested a review from a team as a code owner November 16, 2026 08:46
@cpojer
cpojer requested a review from Brooooooklyn November 16, 2026 08:47

@Brooooooklyn Brooooooklyn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, and for the careful write-up on replicas. The core runtime change is small and mostly right: outside a request, db still starts on primary for each access, auth keeps its own primary client, the new module adds no heavy imports, and the unit tests pass.

Requesting changes for:

Level Issue
medium The docs middleware's setSession() throws when a loader that used db makes a server-side fetch from void/client (same request scope)
medium The cookie and both-transports recipes turn off ISR and shared caching for every route
low The generated .void/db.d.ts doesn't match the published void/db types (getBookmark generic, DbSessionStart)
low Any property read or await on db locks setSession(); unclear error outside the request scope; constraint typos compile
low defer() note, bookmark input note, test timeouts, placeholder

Missing tests: the published module's out-of-scope path (each access uses first-primary) and the getBookmark(client) error for a client not backed by a session.

Also: the branch is behind main and needs the rebase requested earlier. The playground is a scope call for maintainers; trimming it and its browser suite would make it easier to maintain.

Comment thread packages/void/src/runtime/db/session.ts Resolved
// Framework adapters do not always enter Void's request scope. Without a
// shared session, each operation starts on primary to preserve fresh reads.
() => createPrimaryDb(requireD1()),
d1Init,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low — any property read on db counts as "first use".

The proxy get trap builds the client (and so calls withSession()) for every property, including then and symbols. So const f = db.select, logging db, await Promise.resolve(db), or return db from an async function all open the session, and a later setSession() throws. Users will read "before the first use of db" as "before the first query".

Suggestion: open the D1 session lazily (hand drizzle a small D1-shaped wrapper whose prepare/batch/exec call getSession() when they run), or at least don't build the client for then and symbol keys. The generated module in plugin-drizzle.ts has the same behaviour.

export function createD1SessionAccess(bindingName: string): DbSessionAccess {
const key = {};

function state(): DbRequestSession {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low — unclear error outside Void's request scope.

state() passes no out-of-scope callback, so setSession() and getBookmark() throw env: Request-scoped resource is unavailable outside a runtime handler.. D1 apps on Nuxt/Analog do run outside that scope (nitro.ts adds the scope plugin only for non-sqlite dialects).

Suggestion: throw a db:-specific error that points to createDb(env.DB, { session }) plus getBookmark(client), or return null from getBookmark() there.

import type { DrizzleD1Database } from 'drizzle-orm/d1';
import { getRuntimeRequestResource, requireRuntimeBinding } from '../env/runtime.ts';

export type DbSessionStart = D1SessionConstraint | (D1SessionBookmark & {});

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low — constraint typos now compile.

D1SessionConstraint | (D1SessionBookmark & {}) is just string, and createDb's session option changed from D1SessionConstraint to this type. So 'first-primay' or 'primary' type-check and are sent to D1 as a bookmark. This PR also removes the old @ts-expect-error case in db-session.typecheck.ts that caught this.

Suggestion: keep & {} for editor completion, but in setSession/createDb reject strings that start with first- and are not one of the two constraints, and add a test for it.

...(dialect === 'sqlite'
? [
'export declare function setSession(start: D1SessionConstraint | (D1SessionBookmark & {})): void;',
`export declare function getBookmark(client?: ${type}<${schemaType}>): D1SessionBookmark | null;`,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low — the generated getBookmark is narrower than the published one.

Here client is typed DrizzleD1Database<Schema> (the app's schema only), while runtime/db/index.ts is generic over TSchema. With the generated .void/db.d.ts, getBookmark(createDb(env.OTHER_DB)) fails with TS2345.

Suggestion: emit getBookmark<TSchema extends Record<string, unknown>>(client?: DrizzleD1Database<TSchema>). Please also add a check that type-checks the generated d.ts, not only the source module.

On your thread just above: in Void the sqlite dialect always means D1 (the virtual module is built on drizzle-orm/d1, and the node target blocks void/db), so I don't think a separate void/d1 is needed.

Comment thread docs/guide/database/d1.md
path: '/',
});
}
c.header('Cache-Control', 'private, no-store');

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium — this recipe turns off ISR and shared caching for the whole app.

This is global middleware with no path filter. After next() it always sets Cache-Control: private, no-store (this replaces whatever the route set) and a Set-Cookie whenever there is a bookmark. In @void/isr, hasSharedCacheOptOut treats either one as an opt-out, so every ISR or cached route stops caching. The "Supporting both transports" recipe has the same problem.

Suggestion: limit it to the paths that need read-your-writes (as the header recipe does with /api/), don't replace a Cache-Control the handler already set, and say clearly that the routes it covers are not cached.

Comment thread docs/guide/database/d1.md
Bookmarks do not affect cached responses. Disable ISR for pages that require fresh reads (`export const revalidate = 0` in the page's `.server.ts` file), and do not serve these requests from shared response caches.
:::

Use middleware to set the default `db` session before your handlers or Pages loaders run, and return its bookmark after their queries. For a browser app, a cookie carries it through redirects and navigation:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low — defer() queries miss the bookmark.

Pages defer() data streams after the response is returned, so the middleware reads getBookmark() before those queries finish. One sentence would cover it: queries inside defer() callbacks don't update the returned bookmark, so do writes that need read-your-writes before the loader returns.

Comment thread docs/guide/database/d1.md
import { getBookmark, setSession } from 'void/db';

export default defineMiddleware(async (c, next) => {
setSession(getCookie(c, 'db-bookmark') ?? 'first-unconstrained');

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low (not proven) — the bookmark comes from the client.

The cookie or header value goes straight to setSession(). Cloudflare's own examples do the same, so this isn't a blocker. But workerd keeps the bookmark that sorts highest, so a bad value that sorts above real ones never gets replaced and stays in the cookie. A short note (check the shape, fall back to 'first-unconstrained' on a bookmark error, or name the cookie per database) would help.

</p>
<p>
<strong>Related:</strong>{' '}
<a href="https://github.com/voidzero-dev/void/issues/783">Issue #783</a> | PR #XXX: TBD

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: leftover PR #XXX: TBD placeholder. Link #813 or remove it.

await route.continue({ headers });
});
await page.getByRole('button', { name: 'Run trials', exact: true }).click();
await expect

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low — likely to fail randomly. This expect.poll (and the one at line 299) uses Vitest's default 1s timeout, but it waits on a POST with a D1 write, a 303 GET, and a second intercepted POST in dev mode. Please pass { timeout: 30_000 } like the other polls in this file.

More broadly, this suite builds and runs the playground twice and mostly checks the demo UI. The README says local Miniflare can't show the replica lag, so both variants pass here. Cutting it to one mode with a few API-level checks (POST → 303, cookie/header set) would keep the void/db coverage and keep the test run fast.

@kossnocorp

Copy link
Copy Markdown
Contributor Author

@Brooooooklyn thank you for the review. I'm looking into it.

What do you think about making replicas opt-in? I can make the change disabling it in the same PR.

@Brooooooklyn

Copy link
Copy Markdown
Member

Thanks! Yes, let's make replicas opt-in here. Please add a config option, have provisioning skip read_replication unless it's set, and update the docs. Note that existing databases keep replication on.

Both provisioning paths set it today:

  • packages/void/src/cli/provision.ts (enableD1ReadReplicationViaApi, native void deploy --platform cloudflare)
  • platform/packages/api/src/provisioning/cloudflare-api.ts (create and update both send read_replication: { mode: 'auto' })

The review comments above still apply, since the session helpers are still useful once replicas are turned on.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

void/db can't carry a D1 bookmark across requests

3 participants