Skip to content

Fix substring keyspace match in shard discovery - #63

Merged
rpatel27 merged 1 commit into
mainfrom
cursor/fix-sibling-keyspace-shards-e737
Sep 16, 2026
Merged

rpatel27 merged 1 commit into
mainfrom
cursor/fix-sibling-keyspace-shards-e737

Conversation

@rpatel27

@rpatel27 rpatel27 commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Fixes a confirmed bug reported by Trengo (PlanetScale ticket #37380). This ports the same three fixes already landed on airbyte-source in planetscale/airbyte-source#158.

Every Singer sync against the trengo database logged one ERROR per stream:

shard provided in VGTID, trengo_etl/-, not found in the trengo keyspace

The real trengo/- data still synced, and the job succeeded, because cursor-peek failures wrapped a nil err.

Root cause

  1. GetVitessShards ran show vitess_shards like "%trengo%" and TrimPrefix(name, "trengo/"). The sibling keyspace trengo_etl matches the LIKE; because the row does not start with trengo/, TrimPrefix left trengo_etl/- in every stream's saved state.
  2. In Read, when cursor peek failed (lcErr), the code did errors.Wrap(err, ...) while err was still nil, so the ERROR was logged but the shard counted as synced.
  3. Fixing discovery alone would leave the stale trengo_etl/- key in saved state. Once the nil-error bug is fixed, those streams would start failing.

This matches the existing fivetran-source #90 fix and airbyte-source #158 fix (GetVitessShards + shardsForKeyspace).

Changes

  • Discovery uses show vitess_shards like "<keyspace>/%" and parses with an exact keyspace cut (shardsForKeyspace), so sibling keyspaces that share a prefix cannot leak in.
  • Cursor-peek failures return a non-nil error (errors.Wrap(lcErr, ...) / errors.New for an empty stop position).
  • When loading/merging sync state, drop shard keys that current discovery no longer returns and seed empty cursors for newly discovered shards, keeping valid cursors for real shards.

Test plan

  • TestShardsForKeyspace_SiblingKeyspacePrefix — trengo vs trengo_etl, plus a sharded test / test_sharded case
  • TestSync_PrunesStaleSiblingKeyspaceShards — drops trengo_etl/-, keeps the real trengo/- cursor, seeds empty cursors for newly discovered shards
  • TestRead_ReturnsErrorWhenCursorPeekFails — peek Recv error is returned instead of a successful nil wrap
  • TestRead_ReturnsErrorWhenCursorPeekStopPositionEmpty — empty stop position returns a non-nil error
  • go test ./... passes

References: planetscale/airbyte-source#158, Trengo / PlanetScale ticket #37380

Open in Web Open in Cursor 

Trengo (ticket #37380) saw one ERROR per stream because GetVitessShards
matched sibling keyspace trengo_etl via LIKE "%trengo%" and TrimPrefix left
trengo_etl/- in every stream's saved state. Cursor peek then failed, but
Read wrapped a nil err so the job still succeeded.

Match shards with exact "<keyspace>/%", return the real peek error, and
prune saved shard keys that discovery no longer returns.

Port of planetscale/airbyte-source#158.

Co-authored-by: Rupesh Patel <rpatel27@users.noreply.github.com>
@rpatel27
rpatel27 marked this pull request as ready for review September 15, 2026 08:45
@cursor cursor Bot changed the title Fix sibling keyspace shard leakage in Vitess discovery Fixing substring keyspace match issue. Sep 15, 2026
@cursor cursor Bot changed the title Fixing substring keyspace match issue. Fix substring keyspace match in shard discovery Sep 15, 2026
@rpatel27
rpatel27 merged commit cb23afe into main Sep 16, 2026
1 check passed
@rpatel27
rpatel27 deleted the cursor/fix-sibling-keyspace-shards-e737 branch September 16, 2026 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants