Repository navigation
[Bug] Recognize IoTConsensusV2 class name aliases on ConfigNode - #18989
Open
luoluoyuyu wants to merge 5 commits into
Open
luoluoyuyu wants to merge 5 commits into
luoluoyuyu wants to merge 5 commits into
Conversation
Caideyipi
reviewed
Oct 23, 2026
Caideyipi
left a comment
Collaborator
There was a problem hiding this comment.
Review
The normalization is applied when ConfigNode reads its setting and when it compares a joining node's parameters, so both startup and mixed-version registration use the canonical class name. The pipe creation and repair checks follow the same alias handling, and the new tests cover both legacy names. I found no blocking issue in the changed paths.
ConfigNode compared the raw value of data_region_consensus_protocol_class with ConsensusFactory.IOT_CONSENSUS_V2, while DataNode normalizes the configured value through ConsensusFactory.normalizeConsensusProtocolClass before using it. When the configuration used the legacy org.apache.iotdb.consensus.pipe.PipeConsensus name or the implementation class name org.apache.iotdb.consensus.pipe.IoTConsensusV2, ConfigNode concluded that IoTConsensusV2 was not configured and therefore: - createInitialConsensusPipes() never created the __consensus.* pipes on database creation; - checkAndRepairConsensusPipes() returned immediately, so PipeMetaSyncer could never repair missing pipes; - RouteBalancer disabled the automatic data region leader balance. Since the IoTConsensusV2 write path only writes the local state machine, the missing pipes silently degraded the data region replication factor to one: no error was reported and syncLag stayed at zero. Normalize the value once at the configuration boundary so CONF never holds an alias, log the normalization when the configuration file uses an alias, keep the comparison sites and the RouteBalancer cache defensive, and normalize the value sent by a joining ConfigNode so a mixed-version cluster is not rejected.
…ass at startup ConfigNode decides whether it has to create consensus pipes by comparing data_region_consensus_protocol_class with the canonical IoTConsensusV2 class name. An unrecognized value therefore silently disables IoTConsensusV2 replication: data_replication_factor=2 degrades to a single replica without any error, and the pipe based syncLag metric keeps reporting 0. Fail the startup check with a ConfigurationException that lists the accepted classes instead of starting a cluster that cannot replicate.
…gions without pipes PipeMetaSyncer#sync() returned early whenever canSkipNextSync() was true, which is exactly the state of a cluster whose consensus pipes were never created: the pipe task info is empty and its version never changes. The reconciliation therefore only ran inside the unskipped path and could never repair the very cluster it exists for. Move the interval check in front of the early return, and add a region level ERROR when an expected DataRegion has no consensus pipe at all. The existing per pipe warning is only emitted once the operator already knows which pipes are expected, so the degradation of a whole region had no single, obvious signal before a replica actually failed.
…atches The reconciliation ran on a fixed round cadence, so a region that lost its consensus pipes could stay unreplicated for up to five sync intervals even after the starvation fix made the check reachable again. Probe the pipe count on every sync round instead: the number of ordered replica pairs over the DataRegion replica sets is exactly the number of consensus pipes IoTConsensusV2 requires, and comparing it with the number of existing consensus pipes neither builds a pipe name nor copies a replica set (the new PartitionManager#getReplicaPairCount iterates the partition tables in place). A mismatch means at least one region has lost its replication channel, so the full reconciliation runs immediately and keeps running every round until the counts match again. Equal counts still trigger the full check periodically, because a same-size but different pipe set cannot be detected by counting. The probe is skipped entirely when IoTConsensusV2 is not configured, and a single-replica region expects no pipe at all, so single-replica clusters never report a mismatch.
luoluoyuyu
force-pushed
the
fix/cn-normalize-consensus-protocol-alias
branch
from
November 6, 2026 03:24
0c32041 to
4a85526
Compare
Keep production comments to what the code does not already say: the long explanations belonged in the commit messages, not next to the statements. Give every new test a short comment stating the scenario it covers and the behaviour it asserts.
luoluoyuyu
force-pushed
the
fix/cn-normalize-consensus-protocol-alias
branch
from
November 6, 2026 04:12
4a85526 to
9a75acc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The problem in one paragraph
data_region_consensus_protocol_classaccepts three spellings of IoTConsensusV2, becauseDataNodenormalizes the value through
ConsensusFactory.normalizeConsensusProtocolClass()before comparingit.
ConfigNodecompared the raw string withConsensusFactory.IOT_CONSENSUS_V2only, so with analiased value every DataNode believed it was running IoTConsensusV2 while ConfigNode believed it was
not. IoTConsensusV2 in BATCH mode only writes the local state machine and relies on
__consensus.*pipes to ship the data to the followers, and those pipes are created by ConfigNode. ConfigNode
therefore never created them, and
data_replication_factor=2silently degraded to one replica: noerror, no slow query,
syncLagstill reported0.The three accepted spellings
org.apache.iotdb.consensus.iot.IoTConsensusV2org.apache.iotdb.consensus.pipe.IoTConsensusV2org.apache.iotdb.consensus.pipe.PipeConsensusAll three are legal inputs. Only the first one was recognized by ConfigNode.
Symptom
SHOW PIPESlists no__consensus.*pipe at all, on any DataNode.data_replication_factor=2/3is honoured on paper only: the peers are in the replica set, butthey never receive a byte, because the leader never opens the replication channel.
iot_consensus_v2{type="syncLag"}is0because a followerthat has never received anything has nothing to lag behind. There is no error, no warning, and no
"degraded" region.
since the region was created is gone.
RouteBalancercachesthe same value and reaches the same conclusion, so a hot leader stays hot.
Root causes
The bug has four independent layers. Fixing only the first one leaves the cluster degraded, which is
why this PR addresses all four.
1. ConfigNode does not recognize the aliases
RegionMaintainHandlercomparedCONF.getDataRegionConsensusProtocolClass()with a raw stringliteral in four places:
createInitialConsensusPipes()— so no pipe was created when the database was created;checkAndRepairConsensusPipes()— so the periodic self-healing path returned immediately;isIoTConsensusV2DataRegion()— so the region was treated as a non-V2 region;MultiLeaderbranch ofcreatePeer— so the replica sets were built as if IoTConsensusV2were absent.
RouteBalancercached the same raw value and disabled the data region leader balance for the samereason.
2. The reconciliation is starved exactly when it is needed
Even with a canonical value (
CONFis rewritten to the canonical form on restart, see below), themissing pipes were still not created.
PipeMetaSyncer#sync()skips an entire round whenand
canSkipNextSync()requiresisLastSyncedPipeTaskInfoEmpty && pipeMetaKeeper.isEmpty() && lastSyncedVersion == latestVersion, i.e. the skip is enabled precisely when the cluster has nopipe at all.
checkConsensusPipesIfDue()used to run after that early return and counted rounds:CONSENSUS_PIPE_CHECK_INTERVAL_ROUND = 5rounds are needed before the reconciliation runs. A clusterthat lost all of its pipes skips from round 2 on, so the counter never advances past 1 and the
reconciliation never runs.
The result is an inversion of the health signal:
This also explains why restarting does not reliably recover an affected cluster. The persisted value
in
system.propertiesis canonical (SystemPropertiesUtilsnormalizes both on write and on read, andConfigNodeStartupCheck.checkSystemProperties()overwritesCONFwith it), so after a restartConfigNode does recognize IoTConsensusV2 — but the reconciliation that would recreate the pipes is
still starved while the cluster has no pipe, so a cluster that only ever used consensus pipes stays
degraded indefinitely.
3. The reconciliation, when it runs, is slow and reactive
PipeMetaSyncersyncs everypipe_meta_syncer_sync_interval_minutes(default 3 minutes) andreconciles every 5 syncs, i.e. every 15 minutes, and only on the ConfigNode leader. A replication
channel lost at runtime (a peer replaced, a pipe dropped by hand, a partially applied configuration
change) stayed broken for up to that long before anything noticed.
4. Nothing reports the degradation
There is no error and no warning: a region whose pipes were never created looks exactly like a
healthy one, because
syncLagis0in both cases (a follower that never received a byte hasnothing to lag behind). The first symptom is data loss after a leader failover.
The fix
1. Normalize at the configuration boundary and reject what cannot be acted on
ConfigNodeConfig#setDataRegionConsensusProtocolClass()now normalizes throughConsensusFactory.normalizeConsensusProtocolClass(), soCONFnever holds an alias regardless ofthe source (properties file, system properties, programmatic setup).
ConfigNodeDescriptornormalizes on read and logs a WARN when a legacy name is found, so thedegradation is visible in the ConfigNode log instead of being silent.
ConfigManager#checkConfigNodeGlobalConfig()normalizes the value sent by a joining ConfigNode, soa mixed-version cluster is not rejected during a rolling upgrade.
RouteBalancernormalizes its cached value.RegionMaintainHandler#isIoTConsensusV2Configured()is now the single place that answers "is thiscluster on IoTConsensusV2?", used by all four call sites above.
ConfigNodeStartupCheck#checkDataRegionConsensusProtocolClass()rejects any value that is not oneof the four supported classes, after normalization, with a message that lists the accepted values.
An unrecognized class name is not a cosmetic typo: it silently disables replication, so failing at
startup is strictly better than starting a cluster that cannot replicate.
2. Never starve the reconciliation
checkConsensusPipesIfDue()moved before the early return ofPipeMetaSyncer#sync(), so the counteradvances on every attempted round, whether or not the round was skipped. A cluster with no pipes
now reconciles every 5 rounds (~15 minutes by default) like any other cluster, instead of never.
3. Reconcile as soon as the pipe count mismatches
Relying on the 15-minute cadence alone still leaves a 15-minute window, and the counter can be reset
by unrelated activity. Every round now runs a cheap probe first:
A mismatch means at least one region lost its replication channel, so the full reconciliation runs
immediately and keeps running every round until the counts match again.
The probe is designed to be affordable on every 3-minute round:
PartitionManager#getReplicaPairCount()is a new zero-copy chain(
PartitionInfo→DatabasePartitionTable→RegionGroup#getReplicaNum); it does not callgetAllReplicaSetsMap(), which deep-copies every replica set. It also skips pre-deleted databases,mirroring the filter of
checkAndRepairConsensusPipes().PipeTaskInfo#getConsensusPipeCount()walks the pipe meta under the read lock and countsPipeType.CONSENSUS; it does not build a single__consensus.*name, unlikegetConsensusPipeStatusMap().falsewhen IoTConsensusV2 is not configured, so other protocols do not pay for it.0pipes, so anRF=1cluster never reports a mismatch.Counting is a trigger, not a proof of correctness: a pipe of a replaced peer can keep the total
unchanged while pointing at the wrong endpoint. The 15-minute full reconciliation is therefore kept.
4. Report every region that has no pipe
The reconciliation now walks the replica sets and logs one WARN per DataRegion that has no consensus
pipe at all, before the per-pipe messages. The per-pipe warning only fires once the operator already
knows which pipes are expected; this one names the region and says that its followers receive no data.
syncLagis deliberately left untouched: it keeps its current semantics instead of gaining a specialvalue, so existing dashboards, alert rules and aggregations keep working unchanged.
Behaviour changes
pipe.IoTConsensusV2/pipe.PipeConsensus)Recovering a cluster that is already degraded
Upgrading the ConfigNode leader is enough: the probe sees the mismatch on the next sync round and
recreates the missing
__consensus.*pipes, no restart and no manualCREATE PIPEneeded.The pipes only carry data written after they exist. Data written while the channel was missing
exists only in the leader's state machine; whether it is back-filled depends on the pipe/snapshot
start semantics, so operators of an affected cluster should verify the replicas (backup, or a replica
consistency check) instead of assuming the history was repaired.
Tests
23 new tests, all green:
ConfigNodeConfigTest(2) — both aliases are stored in canonical form; canonical and Ratis valuesare preserved.
ConfigNodeStartupCheckTest(3) — the four supported classes pass; both aliases pass and arenormalized; an unrecognized class fails with a message that names the accepted values.
PipeMetaSyncerTest(2) — the reconciliation runs on its own cadence even whensync()is skippedevery round, and a count mismatch reconciles on the next round and keeps reconciling until the
counts match.
RegionMaintainHandlerConsensusPipeTest(16) — pipe creation under both aliases, the per-regionWARN for a region without any pipe, the count probe (mismatch / match / single replica / non-V2),
plus the existing create-drop-restart cases.
Also re-run for regression:
PipeConfigNodeMetricsRestartTestandIoTConsensusV2ServerMetricsTest(6 tests) — 29 tests in total, 0 failures. Verified with
mvn test-compile(full reactor) andmvn test-compile -P with-zh-locale;spotlessapplied.Alternatives considered
syncLag = -1for a region without a pipe. Rejected, andsyncLagis left untouched bythis PR. A negative lag would poison every aggregation (
sum,avg,abs) and every existingdashboard, would make
RF=1clusters permanently "abnormal", would flap during replica-set changes,and would break under version skew (a new DataNode reporting
-1to an old ConfigNode). Thedegradation is reported by the per-region WARN instead.
would have to be maintained by every code path that touches replica sets or pipes. The
reconciliation derives it from the partition table and the pipe meta every time, so it cannot
disagree with reality for long.
data is missing on a follower is still readable and writable, and dropping a replica destroys the
evidence and the chance to back-fill.
the existing create-procedure queue already serializes the work. It can be added if a very large
cluster turns out to need it.
This PR has:
Key changed/added classes (or packages if there are too many classes) in this PR
org.apache.iotdb.confignode.conf.ConfigNodeConfigorg.apache.iotdb.confignode.conf.ConfigNodeDescriptororg.apache.iotdb.confignode.conf.ConfigNodeStartupCheckorg.apache.iotdb.confignode.procedure.env.RegionMaintainHandlerorg.apache.iotdb.confignode.manager.pipe.coordinator.runtime.PipeMetaSyncerorg.apache.iotdb.confignode.manager.pipe.coordinator.task.PipeTaskCoordinatororg.apache.iotdb.confignode.persistence.pipe.PipeTaskInfoorg.apache.iotdb.confignode.manager.partition.PartitionManagerorg.apache.iotdb.confignode.persistence.partition.PartitionInfoorg.apache.iotdb.confignode.persistence.partition.DatabasePartitionTableorg.apache.iotdb.confignode.manager.load.balancer.RouteBalancerorg.apache.iotdb.confignode.manager.ConfigManagerorg.apache.iotdb.confignode.persistence.partition.RegionGroupConfigNodeMessages/ManagerMessages/ProcedureMessages(en/zh)