ts-sdk: order client-cache range bounds by the wrapped integer - #5970
Open
captain-mirage wants to merge 1 commit into
Open
captain-mirage wants to merge 1 commit into
captain-mirage wants to merge 1 commit into
Conversation
`Identity`, `ConnectionId`, `Timestamp`, `TimeDuration` and `Uuid` are all
indexable column types, so `index.filter(new Range(...))` over a column of
one of those types type-checks and is the documented way to scan a btree
index in the client cache.
Each of those is a SATS one-element product wrapping a single integer, and
each is an ordinary object in JS. `scalarCompare` in `src/sdk/table_cache.ts`
compares range bounds with bare `===` and `<`, so:
- `===` is reference equality, so a bound built from a fresh instance never
compares equal to the instance stored in the cache. An `excluded` lower
bound therefore keeps the boundary row and an `included` upper bound drops
it.
- `<` coerces via `toString`. `ConnectionId`, `Timestamp` and `TimeDuration`
have no `toString` override, so every pair stringifies to `[object Object]`
and the comparison is meaningless: a `Timestamp` lower bound matches every
row, and a `ConnectionId` range matches none. `Identity` and `Uuid` happen
to stringify to fixed-width hex, so their ordering is accidentally right
but their boundaries are still off by one.
Unwrap the well-known wrapper field before comparing. The host orders these
columns by the integer they wrap, because `AlgebraicValue`'s derived `Ord`
descends through the one-element product, so comparing the wrapped `bigint`
reproduces the host's ordering exactly -- including negative `Timestamp` and
`TimeDuration` values, which a hex-string normalisation would get wrong.
The `x === y` fast path stays first, so identical primitives still short
circuit, and everything else costs one `typeof` per operand on a path that
already allocates a key array per row.
Equality terms and unique-index `find` were never affected: both go through
`deepEqual`, which already compares these wrappers structurally.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of Changes
Identity,ConnectionId,Timestamp,TimeDurationandUuidall implementIndexable, sot.timestamp().index('btree')is a supported column declaration andindex.filter(new Range(...))over such a column type-checks. In the client cache thoserange scans return the wrong rows.
scalarCompareinsrc/sdk/table_cache.ts(line 41 on1906706) is the comparator usedfor the lower/upper bound of a
Range:Each of those five types is a SATS one-element product wrapping a single integer, and each
is an ordinary object in JS, so both operators do the wrong thing:
===is reference equality. A bound built from a fresh instance never compares equal tothe instance stored in the cache, so
scalarComparecan never return0. Anexcludedlower bound therefore keeps the boundary row and an
includedupper bound drops it.<coerces viavalueOf/toString.ConnectionId,TimestampandTimeDurationhave no
toStringoverride, so every pair stringifies to[object Object]and thecomparison is meaningless.
IdentityandUuiddo overridetoStringto fixed-widthhex, so their ordering happens to come out right, but their boundaries are still off
by one from the
===problem above.Concretely, on a table with
at: t.timestamp().index('btree')and rows at 10µs, 20µs and30µs:
and with
connection: t.connectionId().index('btree'):There is no error — the scan just silently returns the wrong set, and which way it is
wrong depends on the column type.
Root cause and fix
src/sdk/table_cache.ts:41(base190670675f8a26849d15670faac598764cd468eb), reachedfrom
matchRangeat lines 165 and 172.The fix unwraps the well-known wrapper field before comparing:
Comparing the wrapped
bigintis not an approximation — it is exactly the host'sordering.
AlgebraicValue's derivedOrddescends through the one-element product, andthe wrapped scalars are
u256(Identity),u128(ConnectionId,Uuid) andi64(
Timestamp,TimeDuration). BigInt comparison reproduces all five, including negativeTimestamp/TimeDurationvalues, which any hex-string normalisation would order wrongly(a pre-epoch timestamp would sort above the epoch).
Detecting the wrapper by field name rather than
instanceofmatches howProductType.intoMapKey(src/lib/algebraic_type.ts:591) already identifies these types,and keeps working when a consumer ends up with two copies of the package in its module
graph.
What is not affected
Equality terms and unique-index
findwere never broken: both go throughdeepEqual(
src/lib/util.ts:9), which already compares these wrappers structurally. The bug isconfined to
Rangebounds. The change adds a test for the equality path anyway so theboundary between the two is explicit.
This is the client-cache mirror of the server-side range logic fixed in #5479 — the same
"prefix columns are equality, the bound applies only to the last term" rule, implemented
separately in
TableCacheImpl.matchRange.Alternative considered
Give the five classes a
valueOf(): bigintand rewritescalarCompareas a plainthree-way compare. That is a smaller diff in
table_cache.ts, and it would makeidentity < otherwork everywhere. I did not do it becausevalueOfis a public,implicit coercion hook: adding it changes
==,+,<and sorting semantics forIdentity/Timestamp/… in all consumer code, silently, andvalueOfreturning abigintmakesNumber(timestamp)throw. That is a public API change to five classes tofix one comparator. Happy to switch if you would prefer it.
A third option is to hoist the decision out of the comparator entirely:
#makeReadonlyIndexknows each indexed column's
AlgebraicType, so it could pick a specialised comparator percolumn at index-construction time and pay nothing per row. That is strictly better on
paper but a larger diff in the index-construction path, and the per-row cost of the
current shape is already negligible (see below). Also happy to do that instead.
I also considered exporting the wrapper field list from
src/lib/algebraic_type.ts, wherespecialProductDeserializersalready enumerates exactly these five names, so there is onedefinition site. That adds a public export, so I kept the list local to
table_cache.ts;say the word if you would rather have the shared constant.
Performance
scalarCompareis not on the cache-maintenance path —applyOperationsdoes not call it.It is called at most twice per row per
Rangescan, frommatchRange, which alreadyallocates a key array per row (
columns.map(...)) and linearly scans the whole table(
// TODO: this just scans the whole table). Thex === yfast path stays first, soidentical primitives still short-circuit; a non-identical primitive pair now costs two
typeofchecks. That is far below the noise floor of the surroundingmapallocation.Bundle impact, measured with the repo's own
size-limitbudgets:esm min (brotli)sdk esm min (brotli)Out of scope
Enum (sum-typed) columns are also
Indexable, and their JS representation is a{ tag, value }object, soRangebounds over an enum column are wrong too. Ordering asum needs the variant's declaration index, not a wrapped scalar, so it is a different fix
and I left it alone.
Uint8Array/array columns are notIndexablein the typed builderAPI, so they cannot reach this path.
API and ABI breaking changes
None. No exported type or signature changes;
scalarCompareand the new helper aremodule-private. The only observable difference is that
Rangefilters overIdentity/ConnectionId/Timestamp/TimeDuration/Uuidcolumns now return the rowsthe host would return. Code that had worked around the old behaviour (for example by
filtering on
__timestamp_micros_since_unix_epoch__by hand) is unaffected, since thatpath never went through this comparator.
Rollback safety impact
n/a
Expected complexity level and risk
from
matchRange's two bound checks. No wire format, no serialization, no serverinteraction. The one thing worth a reviewer's eye is the claim that BigInt ordering equals
the host's ordering for all five types —
crates/lib/src/identity.rs(u256, derivedOrd),crates/lib/src/connection_id.rs(u128),crates/sats/src/{timestamp,time_duration}.rs(
i64) andcrates/sats/src/uuid.rs(u128) are the references.Testing
crates/bindings-typescript/tests/table_cache_range_bounds.test.ts(8 cases, inline
table()+tablesToSchema+TableCacheImplfixtures, in thestyle of
tests/table_cache_resolved_indexes.test.ts): aRangebound over each ofIdentity,ConnectionId,Uuid,TimestampandTimeDurationreturns exactlythe rows in range with the right boundary inclusivity; a pre-epoch
Timestampsortsbelow
Timestamp.UNIX_EPOCH; bare-value (equality) terms and primitive-columnranges are unchanged.
masterfor the right reason — 6 of the 8fail, with a
Timestamplower bound matching every row, aConnectionIdrangematching none, and
Identity/Uuidranges dropping the row that sits exactly on anincludedupper bound. The two that pass onmasterare the equality-term andprimitive-column controls.
pnpm test— 31 files, 324 tests, 0 failures.pnpm lintclean.pnpm build(
tsup+tsc -p tsconfig.build.json) clean.pnpm size— all 13 budgets green.Rangescan overa
Timestampcolumn now agrees betweenctx.db.<table>.<index>.filter(...)on theclient cache and the same scan on the server would be worth having; I only have unit
coverage for the client side.