Repository navigation
Fix substring keyspace match in shard discovery - #158
Merged
Merged
Conversation
Trengo (ticket #37380) saw one ERROR per stream because GetVitessShards matched sibling keyspace trengo_etl via LIKE "%trengo%" and TrimPrefix left trengo_etl/- in every stream's saved state. Cursor peek then failed, but Read wrapped a nil err so the job still succeeded. Match shards with exact "<keyspace>/%", return the real peek error, and prune saved shard keys that discovery no longer returns. Co-authored-by: Rupesh Patel <rpatel27@users.noreply.github.com>
rpatel27
marked this pull request as ready for review
September 15, 2026 08:35
5 tasks done
maxenglander
approved these changes
Sep 16, 2026
rpatel27
added a commit
to planetscale/singer-tap
that referenced
this pull request
Sep 16, 2026
Trengo (ticket #37380) saw one ERROR per stream because GetVitessShards matched sibling keyspace trengo_etl via LIKE "%trengo%" and TrimPrefix left trengo_etl/- in every stream's saved state. Cursor peek then failed, but Read wrapped a nil err so the job still succeeded. Match shards with exact "<keyspace>/%", return the real peek error, and prune saved shard keys that discovery no longer returns. Port of planetscale/airbyte-source#158. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rupesh Patel <rpatel27@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes a confirmed bug reported by Trengo (PlanetScale ticket #37380).
Every Airbyte sync against the
trengodatabase logged one ERROR per stream:The real
trengo/-data still synced, and the job succeeded, because cursor-peek failures wrapped a nilerr.Root cause
GetVitessShardsranshow vitess_shards like "%trengo%"andTrimPrefix(name, "trengo/"). The sibling keyspacetrengo_etlmatches the LIKE; because the row does not start withtrengo/, TrimPrefix lefttrengo_etl/-in every stream's saved state.Read, when cursor peek failed (lcErr), the code diderrors.Wrap(err, ...)whileerrwas still nil, so the ERROR was logged but the shard counted as synced.trengo_etl/-key in saved state. Once the nil-error bug is fixed, those streams would start failing.This matches the existing fivetran-source #90 fix (
GetVitessShards+shardsForKeyspace).Changes
show vitess_shards like "<keyspace>/%"and parses with an exact keyspace cut (shardsForKeyspace), so sibling keyspaces that share a prefix cannot leak in.errors.Wrap(lcErr, ...)/errors.Newfor an empty stop position).readStatedrops shard keys that current discovery no longer returns and seeds empty cursors for newly discovered shards, keeping valid cursors for real shards.Test plan
TestShardsForKeyspace_SiblingKeyspacePrefix—trengovstrengo_etl, plus a shardedtest/test_shardedcaseTestReadState_PrunesStaleSiblingKeyspaceShards— dropstrengo_etl/-, keeps the realtrengo/-cursor, seeds empty cursors for newly discovered shardsTestRead_ReturnsErrorWhenCursorPeekFails— peek Recv error is returned instead of a successful nil wrapTestRead_ReturnsErrorWhenCursorPeekStopPositionEmpty— empty stop position returns a non-nil errorgo test ./...passes locally