Skip to content

feat: add automatic collection pagination - #17

Merged
ivasilov merged 2 commits into
supabase:mainfrom
dhruvxvaishnav:feat/automatic-pagination
Sep 23, 2026
Merged

ivasilov merged 2 commits into
supabase:mainfrom
dhruvxvaishnav:feat/automatic-pagination

Conversation

@dhruvxvaishnav

@dhruvxvaishnav dhruvxvaishnav commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

What kind of change does this PR introduce?

Feature, tests, and documentation.

What is the current behavior?

Collection reads issue a single PostgREST request, so projects using the default db-max-rows cap (1,000 on Supabase) silently receive only the first 1,000 matching rows with no error.

Closes #9.

What is the new behavior?

  • Collection reads and non-aggregate queryOnce queries page past the server row cap automatically and return the complete matching set (or the caller's limit).
  • No pageSize option and no internal page size. The server cap sizes the first page; the loop never invents a limit of its own.
  • The first request sends Prefer: count=exact once. The total from Content-Range tells the loop when to stop, so a set that is an exact multiple of the cap needs no trailing empty request, and a caller offset is subtracted from the total so it does not trigger one either.
  • Later pages are fetched sequentially and advance the offset by the rows actually received, so a response shorter than requested never causes a gap. The first request's URL is unchanged, so the query key is too.
  • Filters, ordering, offsets, cursor filters, and explicit limits are preserved across pages. The keyset boundary-tie read is paged as well, so a tie class larger than the cap is not truncated.
  • limit: 0 returns [] without a request. The query's abort signal is threaded to every page. An error from any page rejects the whole load rather than returning partial data.
  • postgrestRequest now returns { data, count } and accepts count and signal options.
  • The aggregate / GROUP BY / HAVING path of queryOnce stays a single request and is documented as not paginated.

Known limitations (documented in the README)

  • Offset paging needs a total order. A read with no orderBy, or ordered on a non-unique column, can skip or repeat rows across page boundaries. Appending key columns as tie-breakers is deferred to a follow-up.
  • Offset paging is not keyset-stable under concurrent inserts or deletes mid-load; rows reconcile on the next refetch.
  • Every collection read now costs one count=exact on its first request.

Additional context

supabase/config.toml sets max_rows = 2 for the e2e stack so the multi-page path is exercised without seeding thousands of rows. All raw supabase-js reads in the e2e suite are single-row lookups, so nothing else is affected.

Validation:

  • pnpm typecheck
  • pnpm test (288 passed, 9 todo)
  • pnpm exec ultracite check (only pre-existing warnings in src/db.ts)
  • End-to-end suite against a local Supabase stack, including the new multi-page read test (run on 40b7dad; the follow-up commit 5f36d2f touches only the offset target, its unit test, and the README, and was validated with the unit suite)

@dhruvxvaishnav

Copy link
Copy Markdown
Contributor Author

Hi @ivasilov, quick follow-up: the implementation and local validation are complete, but the GitHub Actions run is still waiting for approval because this is a first-time fork contribution. When you have a chance, could you approve the workflow run? Typecheck, unit tests, build, and the full local Supabase end-to-end suite all passed.

Collection reads issued a single PostgREST request per page, which PostgREST
silently truncates at its db-max-rows cap (1000 by default). A matching set
larger than the cap loaded incomplete data with no error.

Wrap reads in an offset-paging loop (fetchAllPages) that fetches the complete
matching set, or the caller's limit. The first request asks for count=exact so
the loop knows the total up front and stops without a trailing empty probe;
later pages advance the offset by the rows actually received (self-correcting
under concurrent writes) and are capped by the observed page size and the rows
still owed.

- postgrest-request: return { data, count }; add count and signal options.
  The signal threads straight into the builder config and on to fetch.
- functions: supabaseQueryFn routes both the main read and the boundary-tie
  read through the loop and threads ctx.signal. The tie read is paged but
  unbounded by the caller's limit so every tied row returns.
- query-once: non-aggregate queryOnce is paginated for free (it loads its
  source collections through supabaseQueryFn); executeQuery (aggregate /
  groupBy / having) stays a single request and adopts the new return shape.

The query key keeps limit/offset (the loop overrides them per page on a clone),
so distinct windows of a subset still get distinct keys.

Co-Authored-By: Dhruv Vaishnav <dhruvvaishnav687@gmail.com>
@ivasilov
ivasilov force-pushed the feat/automatic-pagination branch from ee661f3 to 40b7dad Compare September 20, 2026 22:14
…mitations

`Content-Range` reports the whole matching set, so a read that starts at a
caller `offset` has only `total - offset` rows left to fetch. Using the raw
total as the target made an exact-multiple read issue one trailing empty
request. Compute the remaining rows past the starting offset instead, and add
a unit test for offset 2 over six rows at a cap of 2 (two requests, not three).

Document the paging loop's known limitations in the README: offset paging
needs a total order, it is not stable under concurrent writes, and each read
now costs one `count=exact`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@ivasilov
ivasilov merged commit 05dac5f into supabase:main Sep 23, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add pagination support beyond the PostgREST 1,000 row limit

2 participants