Skip to content

Fix substring keyspace match in shard discovery - #158

Merged
rpatel27 merged 1 commit into
mainfrom
cursor/fix-sibling-keyspace-shards-d1d5
Sep 16, 2026
Merged

rpatel27 merged 1 commit into
mainfrom
cursor/fix-sibling-keyspace-shards-d1d5

Conversation

@rpatel27

@rpatel27 rpatel27 commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Fixes a confirmed bug reported by Trengo (PlanetScale ticket #37380).

Every Airbyte sync against the trengo database logged one ERROR per stream:

[trengo:primary:<table> shard : trengo_etl/-] Error fetching latest cursor position: ... shard provided in VGTID, trengo_etl/-, not found in the trengo keyspace

The real trengo/- data still synced, and the job succeeded, because cursor-peek failures wrapped a nil err.

Root cause

  1. GetVitessShards ran show vitess_shards like "%trengo%" and TrimPrefix(name, "trengo/"). The sibling keyspace trengo_etl matches the LIKE; because the row does not start with trengo/, TrimPrefix left trengo_etl/- in every stream's saved state.
  2. In Read, when cursor peek failed (lcErr), the code did errors.Wrap(err, ...) while err was still nil, so the ERROR was logged but the shard counted as synced.
  3. Fixing discovery alone would leave the stale trengo_etl/- key in saved state. Once the nil-error bug is fixed, those streams would start failing.

This matches the existing fivetran-source #90 fix (GetVitessShards + shardsForKeyspace).

Changes

  • Discovery uses show vitess_shards like "<keyspace>/%" and parses with an exact keyspace cut (shardsForKeyspace), so sibling keyspaces that share a prefix cannot leak in.
  • Cursor-peek failures return a non-nil error (errors.Wrap(lcErr, ...) / errors.New for an empty stop position).
  • readState drops shard keys that current discovery no longer returns and seeds empty cursors for newly discovered shards, keeping valid cursors for real shards.

Test plan

  • TestShardsForKeyspace_SiblingKeyspacePrefix — trengo vs trengo_etl, plus a sharded test / test_sharded case
  • TestReadState_PrunesStaleSiblingKeyspaceShards — drops trengo_etl/-, keeps the real trengo/- cursor, seeds empty cursors for newly discovered shards
  • TestRead_ReturnsErrorWhenCursorPeekFails — peek Recv error is returned instead of a successful nil wrap
  • TestRead_ReturnsErrorWhenCursorPeekStopPositionEmpty — empty stop position returns a non-nil error
  • go test ./... passes locally
Open in Web Open in Cursor 

Trengo (ticket #37380) saw one ERROR per stream because GetVitessShards
matched sibling keyspace trengo_etl via LIKE "%trengo%" and TrimPrefix left
trengo_etl/- in every stream's saved state. Cursor peek then failed, but
Read wrapped a nil err so the job still succeeded.

Match shards with exact "<keyspace>/%", return the real peek error, and
prune saved shard keys that discovery no longer returns.

Co-authored-by: Rupesh Patel <rpatel27@users.noreply.github.com>
@rpatel27
rpatel27 marked this pull request as ready for review September 15, 2026 08:35
@cursor cursor Bot changed the title Fix sibling keyspace shard leakage in Vitess discovery Fixing substring keyspace match issue. Sep 15, 2026
@cursor cursor Bot changed the title Fixing substring keyspace match issue. Fix substring keyspace match in shard discovery Sep 15, 2026
rpatel27 added a commit to planetscale/singer-tap that referenced this pull request Sep 16, 2026
Trengo (ticket #37380) saw one ERROR per stream because GetVitessShards
matched sibling keyspace trengo_etl via LIKE "%trengo%" and TrimPrefix left
trengo_etl/- in every stream's saved state. Cursor peek then failed, but
Read wrapped a nil err so the job still succeeded.

Match shards with exact "<keyspace>/%", return the real peek error, and
prune saved shard keys that discovery no longer returns.

Port of planetscale/airbyte-source#158.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Rupesh Patel <rpatel27@users.noreply.github.com>
@rpatel27
rpatel27 merged commit fd38146 into main Sep 16, 2026
8 checks passed
@rpatel27
rpatel27 deleted the cursor/fix-sibling-keyspace-shards-d1d5 branch September 16, 2026 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants