feat: add automatic collection pagination - #17
Merged
ivasilov merged 2 commits intoSep 23, 2026
Merged
Conversation
Contributor
Author
|
Hi @ivasilov, quick follow-up: the implementation and local validation are complete, but the GitHub Actions run is still waiting for approval because this is a first-time fork contribution. When you have a chance, could you approve the workflow run? Typecheck, unit tests, build, and the full local Supabase end-to-end suite all passed. |
ivasilov
force-pushed
the
feat/automatic-pagination
branch
from
September 12, 2026 22:27
8db2a3b to
eaf8143
Compare
ivasilov
force-pushed
the
feat/automatic-pagination
branch
from
September 16, 2026 22:30
eaf8143 to
6cb0462
Compare
Collection reads issued a single PostgREST request per page, which PostgREST
silently truncates at its db-max-rows cap (1000 by default). A matching set
larger than the cap loaded incomplete data with no error.
Wrap reads in an offset-paging loop (fetchAllPages) that fetches the complete
matching set, or the caller's limit. The first request asks for count=exact so
the loop knows the total up front and stops without a trailing empty probe;
later pages advance the offset by the rows actually received (self-correcting
under concurrent writes) and are capped by the observed page size and the rows
still owed.
- postgrest-request: return { data, count }; add count and signal options.
The signal threads straight into the builder config and on to fetch.
- functions: supabaseQueryFn routes both the main read and the boundary-tie
read through the loop and threads ctx.signal. The tie read is paged but
unbounded by the caller's limit so every tied row returns.
- query-once: non-aggregate queryOnce is paginated for free (it loads its
source collections through supabaseQueryFn); executeQuery (aggregate /
groupBy / having) stays a single request and adopts the new return shape.
The query key keeps limit/offset (the loop overrides them per page on a clone),
so distinct windows of a subset still get distinct keys.
Co-Authored-By: Dhruv Vaishnav <dhruvvaishnav687@gmail.com>
ivasilov
force-pushed
the
feat/automatic-pagination
branch
from
September 20, 2026 22:14
ee661f3 to
40b7dad
Compare
…mitations `Content-Range` reports the whole matching set, so a read that starts at a caller `offset` has only `total - offset` rows left to fetch. Using the raw total as the target made an exact-multiple read issue one trailing empty request. Compute the remaining rows past the starting offset instead, and add a unit test for offset 2 over six rows at a cap of 2 (two requests, not three). Document the paging loop's known limitations in the README: offset paging needs a total order, it is not stable under concurrent writes, and each read now costs one `count=exact`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ivasilov
approved these changes
Sep 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What kind of change does this PR introduce?
Feature, tests, and documentation.
What is the current behavior?
Collection reads issue a single PostgREST request, so projects using the default
db-max-rowscap (1,000 on Supabase) silently receive only the first 1,000 matching rows with no error.Closes #9.
What is the new behavior?
queryOncequeries page past the server row cap automatically and return the complete matching set (or the caller'slimit).pageSizeoption and no internal page size. The server cap sizes the first page; the loop never invents alimitof its own.Prefer: count=exactonce. The total fromContent-Rangetells the loop when to stop, so a set that is an exact multiple of the cap needs no trailing empty request, and a calleroffsetis subtracted from the total so it does not trigger one either.limit: 0returns[]without a request. The query's abort signal is threaded to every page. An error from any page rejects the whole load rather than returning partial data.postgrestRequestnow returns{ data, count }and acceptscountandsignaloptions.GROUP BY/HAVINGpath ofqueryOncestays a single request and is documented as not paginated.Known limitations (documented in the README)
orderBy, or ordered on a non-unique column, can skip or repeat rows across page boundaries. Appending key columns as tie-breakers is deferred to a follow-up.count=exacton its first request.Additional context
supabase/config.tomlsetsmax_rows = 2for the e2e stack so the multi-page path is exercised without seeding thousands of rows. All raw supabase-js reads in the e2e suite are single-row lookups, so nothing else is affected.Validation:
pnpm typecheckpnpm test(288 passed, 9 todo)pnpm exec ultracite check(only pre-existing warnings insrc/db.ts)40b7dad; the follow-up commit5f36d2ftouches only the offset target, its unit test, and the README, and was validated with the unit suite)