Skip to content

ts-sdk: order client-cache range bounds by the wrapped integer - #5970

Open
captain-mirage wants to merge 1 commit into
clockworklabs:masterfrom
captain-mirage:fix/ts-table-cache-scalar-compare
Open

captain-mirage wants to merge 1 commit into
clockworklabs:masterfrom
captain-mirage:fix/ts-table-cache-scalar-compare

Conversation

@captain-mirage

Copy link
Copy Markdown

Description of Changes

Identity, ConnectionId, Timestamp, TimeDuration and Uuid all implement
Indexable, so t.timestamp().index('btree') is a supported column declaration and
index.filter(new Range(...)) over such a column type-checks. In the client cache those
range scans return the wrong rows.

scalarCompare in src/sdk/table_cache.ts (line 41 on 1906706) is the comparator used
for the lower/upper bound of a Range:

// Strict scalar compare for index term values.
const scalarCompare = (x: any, y: any): number => {
  if (x === y) return 0;
  // Compare booleans/numbers/bigints/strings with JS ordering.
  return x < y ? -1 : 1;
};

Each of those five types is a SATS one-element product wrapping a single integer, and each
is an ordinary object in JS, so both operators do the wrong thing:

  • === is reference equality. A bound built from a fresh instance never compares equal to
    the instance stored in the cache, so scalarCompare can never return 0. An excluded
    lower bound therefore keeps the boundary row and an included upper bound drops it.
  • < coerces via valueOf/toString. ConnectionId, Timestamp and TimeDuration
    have no toString override, so every pair stringifies to [object Object] and the
    comparison is meaningless. Identity and Uuid do override toString to fixed-width
    hex, so their ordering happens to come out right, but their boundaries are still off
    by one from the === problem above.

Concretely, on a table with at: t.timestamp().index('btree') and rows at 10µs, 20µs and
30µs:

const inRange = [...ctx.db.events.at.filter(
  new Range({ tag: 'included', value: new Timestamp(20n) }, { tag: 'unbounded' })
)];
// master: all three rows       expected: the 20µs and 30µs rows

and with connection: t.connectionId().index('btree'):

const inRange = [...ctx.db.events.connection.filter(
  new Range({ tag: 'excluded', value: new ConnectionId(10n) },
            { tag: 'excluded', value: new ConnectionId(30n) })
)];
// master: []                   expected: the 20 row

There is no error — the scan just silently returns the wrong set, and which way it is
wrong depends on the column type.

Root cause and fix

src/sdk/table_cache.ts:41 (base 190670675f8a26849d15670faac598764cd468eb), reached
from matchRange at lines 165 and 172.

The fix unwraps the well-known wrapper field before comparing:

const specialProductFields = [
  '__identity__',
  '__connection_id__',
  '__timestamp_micros_since_unix_epoch__',
  '__time_duration_micros__',
  '__uuid__',
] as const;

const comparableTerm = (value: any): any => {
  if (value === null || typeof value !== 'object') return value;
  for (const field of specialProductFields) {
    const inner = value[field];
    if (typeof inner === 'bigint') return inner;
  }
  return value;
};

Comparing the wrapped bigint is not an approximation — it is exactly the host's
ordering. AlgebraicValue's derived Ord descends through the one-element product, and
the wrapped scalars are u256 (Identity), u128 (ConnectionId, Uuid) and i64
(Timestamp, TimeDuration). BigInt comparison reproduces all five, including negative
Timestamp/TimeDuration values, which any hex-string normalisation would order wrongly
(a pre-epoch timestamp would sort above the epoch).

Detecting the wrapper by field name rather than instanceof matches how
ProductType.intoMapKey (src/lib/algebraic_type.ts:591) already identifies these types,
and keeps working when a consumer ends up with two copies of the package in its module
graph.

What is not affected

Equality terms and unique-index find were never broken: both go through deepEqual
(src/lib/util.ts:9), which already compares these wrappers structurally. The bug is
confined to Range bounds. The change adds a test for the equality path anyway so the
boundary between the two is explicit.

This is the client-cache mirror of the server-side range logic fixed in #5479 — the same
"prefix columns are equality, the bound applies only to the last term" rule, implemented
separately in TableCacheImpl.matchRange.

Alternative considered

Give the five classes a valueOf(): bigint and rewrite scalarCompare as a plain
three-way compare. That is a smaller diff in table_cache.ts, and it would make
identity < other work everywhere. I did not do it because valueOf is a public,
implicit coercion hook: adding it changes ==, +, < and sorting semantics for
Identity/Timestamp/… in all consumer code, silently, and valueOf returning a
bigint makes Number(timestamp) throw. That is a public API change to five classes to
fix one comparator. Happy to switch if you would prefer it.

A third option is to hoist the decision out of the comparator entirely: #makeReadonlyIndex
knows each indexed column's AlgebraicType, so it could pick a specialised comparator per
column at index-construction time and pay nothing per row. That is strictly better on
paper but a larger diff in the index-construction path, and the per-row cost of the
current shape is already negligible (see below). Also happy to do that instead.

I also considered exporting the wrapper field list from src/lib/algebraic_type.ts, where
specialProductDeserializers already enumerates exactly these five names, so there is one
definition site. That adds a public export, so I kept the list local to table_cache.ts;
say the word if you would rather have the shared constant.

Performance

scalarCompare is not on the cache-maintenance path — applyOperations does not call it.
It is called at most twice per row per Range scan, from matchRange, which already
allocates a key array per row (columns.map(...)) and linearly scans the whole table
(// TODO: this just scans the whole table). The x === y fast path stays first, so
identical primitives still short-circuit; a non-identical primitive pair now costs two
typeof checks. That is far below the noise floor of the surrounding map allocation.

Bundle impact, measured with the repo's own size-limit budgets:

entry before after limit
esm min (brotli) 21.26 kB 21.39 kB 30 kB
sdk esm min (brotli) 20.48 kB 20.56 kB 30 kB

Out of scope

Enum (sum-typed) columns are also Indexable, and their JS representation is a
{ tag, value } object, so Range bounds over an enum column are wrong too. Ordering a
sum needs the variant's declaration index, not a wrapped scalar, so it is a different fix
and I left it alone. Uint8Array/array columns are not Indexable in the typed builder
API, so they cannot reach this path.

API and ABI breaking changes

None. No exported type or signature changes; scalarCompare and the new helper are
module-private. The only observable difference is that Range filters over
Identity/ConnectionId/Timestamp/TimeDuration/Uuid columns now return the rows
the host would return. Code that had worked around the old behaviour (for example by
filtering on __timestamp_micros_since_unix_epoch__ by hand) is unaffected, since that
path never went through this comparator.

Rollback safety impact

n/a

Expected complexity level and risk

  1. One module-private comparator in one file, behind an unchanged fast path, reached only
    from matchRange's two bound checks. No wire format, no serialization, no server
    interaction. The one thing worth a reviewer's eye is the claim that BigInt ordering equals
    the host's ordering for all five types — crates/lib/src/identity.rs (u256, derived
    Ord), crates/lib/src/connection_id.rs (u128), crates/sats/src/{timestamp,time_duration}.rs
    (i64) and crates/sats/src/uuid.rs (u128) are the references.

Testing

  • Added crates/bindings-typescript/tests/table_cache_range_bounds.test.ts
    (8 cases, inline table() + tablesToSchema + TableCacheImpl fixtures, in the
    style of tests/table_cache_resolved_indexes.test.ts): a Range bound over each of
    Identity, ConnectionId, Uuid, Timestamp and TimeDuration returns exactly
    the rows in range with the right boundary inclusivity; a pre-epoch Timestamp sorts
    below Timestamp.UNIX_EPOCH; bare-value (equality) terms and primitive-column
    ranges are unchanged.
  • Verified the new tests fail on unmodified master for the right reason — 6 of the 8
    fail, with a Timestamp lower bound matching every row, a ConnectionId range
    matching none, and Identity/Uuid ranges dropping the row that sits exactly on an
    included upper bound. The two that pass on master are the equality-term and
    primitive-column controls.
  • pnpm test — 31 files, 324 tests, 0 failures. pnpm lint clean. pnpm build
    (tsup + tsc -p tsconfig.build.json) clean. pnpm size — all 13 budgets green.
  • If you have an integration environment handy, a sanity check that a Range scan over
    a Timestamp column now agrees between ctx.db.<table>.<index>.filter(...) on the
    client cache and the same scan on the server would be worth having; I only have unit
    coverage for the client side.

`Identity`, `ConnectionId`, `Timestamp`, `TimeDuration` and `Uuid` are all
indexable column types, so `index.filter(new Range(...))` over a column of
one of those types type-checks and is the documented way to scan a btree
index in the client cache.

Each of those is a SATS one-element product wrapping a single integer, and
each is an ordinary object in JS. `scalarCompare` in `src/sdk/table_cache.ts`
compares range bounds with bare `===` and `<`, so:

  - `===` is reference equality, so a bound built from a fresh instance never
    compares equal to the instance stored in the cache. An `excluded` lower
    bound therefore keeps the boundary row and an `included` upper bound drops
    it.
  - `<` coerces via `toString`. `ConnectionId`, `Timestamp` and `TimeDuration`
    have no `toString` override, so every pair stringifies to `[object Object]`
    and the comparison is meaningless: a `Timestamp` lower bound matches every
    row, and a `ConnectionId` range matches none. `Identity` and `Uuid` happen
    to stringify to fixed-width hex, so their ordering is accidentally right
    but their boundaries are still off by one.

Unwrap the well-known wrapper field before comparing. The host orders these
columns by the integer they wrap, because `AlgebraicValue`'s derived `Ord`
descends through the one-element product, so comparing the wrapped `bigint`
reproduces the host's ordering exactly -- including negative `Timestamp` and
`TimeDuration` values, which a hex-string normalisation would get wrong.

The `x === y` fast path stays first, so identical primitives still short
circuit, and everything else costs one `typeof` per operand on a path that
already allocates a key array per row.

Equality terms and unique-index `find` were never affected: both go through
`deepEqual`, which already compares these wrappers structurally.
@CLAassistant

CLAassistant commented Sep 22, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants