Skip to content

Retry inconsistent region metadata while waiting for migrations - #19163

Merged
CRZbulabula merged 1 commit into
apache:masterfrom
d-wang-commit:fix-region-migration-metadata-retry
Nov 6, 2026
Merged

CRZbulabula merged 1 commit into
apache:masterfrom
d-wang-commit:fix-region-migration-metadata-retry

Conversation

@d-wang-commit

Copy link
Copy Markdown
Contributor

Description

Parallel SHOW REGIONS queries can observe different stages of an asynchronous region migration. For example, one query can return replicas {1,3} while another returns {1,2,3}. The cluster test result set then throws InconsistentDataException while comparing rows, aborting the migration wait and triggering cluster teardown before its deadline.

Retry this exception in four existing migration waits: Stream consensus-directory preservation, IoTConsensus empty-deletion and compression-ratio checks, and the replica-membership helper shared by tree/table multi-data-directory deletion tests. An inconsistent observation is treated as an unmet condition. The original one-minute or ten-minute deadlines and all final assertions remain unchanged.

Tests

  • UT: no new unit tests; existing unit tests were not run because this change only adjusts integration-test polling.
  • IT: updated the existing waits and their shared helper; no new test cases. All 7 cases passed on the final commit (0 failures, 0 errors, 0 skipped) with JDK 17 on Windows.
    • IoTDBRegionMigrateNormalForIoTV2StreamIT (3 cases)
    • IoTDBRegionMigrateWithLastEmptyDeletionIT (1 case)
    • IoTDBRegionMigrateWithCompressionRatioIT (1 case)
    • IoTDBRegionMigrateWithDeletionMultiDataDirIT (1 case)
    • IoTDBRegionMigrateWithDeletionMultiDataDirTableIT (1 case)
  • Validation uses with-integration-tests,ClusterIT with explicit ClusterIT, TableClusterIT, and DailyIT category selection. Reactor verification for integration-test and its dependencies, including Spotless checks, passed.

Side effects and risks

A persistent metadata inconsistency now fails at the existing polling deadline instead of aborting immediately. Retrying is limited to InconsistentDataException in these migration waits; other exceptions still propagate, and the shared result-set consistency checks remain enabled. Production behavior is unchanged.


This PR has:

  • been self-reviewed.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR
  • IoTDBRegionMigrateNormalForIoTV2StreamIT: retries transient metadata inconsistencies before checking consensus directories.
  • IoTDBRegionMigrateWithLastEmptyDeletionIT: retries inconsistent membership observations during migration.
  • IoTDBRegionMigrateWithCompressionRatioIT: retries inconsistent region observations in the post-migration compression-ratio wait.
  • RegionMigrateFileAssertions: applies the same retry policy to the shared replica-membership wait.

Treat transient cross-node SHOW REGIONS mismatches as unmet conditions
in migration waits while preserving their deadlines and final assertions.
@CRZbulabula
CRZbulabula merged commit 3ab38d1 into apache:master Nov 6, 2026
39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants