Skip to content

[Bug] Recognize IoTConsensusV2 class name aliases on ConfigNode - #18989

Open
luoluoyuyu wants to merge 5 commits into
apache:masterfrom
luoluoyuyu:fix/cn-normalize-consensus-protocol-alias
Open

luoluoyuyu wants to merge 5 commits into
apache:masterfrom
luoluoyuyu:fix/cn-normalize-consensus-protocol-alias

Conversation

@luoluoyuyu

@luoluoyuyu luoluoyuyu commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

Description

The problem in one paragraph

data_region_consensus_protocol_class accepts three spellings of IoTConsensusV2, because DataNode
normalizes the value through ConsensusFactory.normalizeConsensusProtocolClass() before comparing
it. ConfigNode compared the raw string with ConsensusFactory.IOT_CONSENSUS_V2 only, so with an
aliased value every DataNode believed it was running IoTConsensusV2 while ConfigNode believed it was
not. IoTConsensusV2 in BATCH mode only writes the local state machine and relies on __consensus.*
pipes to ship the data to the followers, and those pipes are created by ConfigNode. ConfigNode
therefore never created them, and data_replication_factor=2 silently degraded to one replica: no
error, no slow query, syncLag still reported 0.

The three accepted spellings

Value Status
org.apache.iotdb.consensus.iot.IoTConsensusV2 canonical
org.apache.iotdb.consensus.pipe.IoTConsensusV2 implementation class name, valid before the package rename
org.apache.iotdb.consensus.pipe.PipeConsensus legacy name kept for compatibility

All three are legal inputs. Only the first one was recognized by ConfigNode.

Symptom

  • SHOW PIPES lists no __consensus.* pipe at all, on any DataNode.
  • data_replication_factor=2 / 3 is honoured on paper only: the peers are in the replica set, but
    they never receive a byte, because the leader never opens the replication channel.
  • Every monitoring signal looks healthy. iot_consensus_v2{type="syncLag"} is 0 because a follower
    that has never received anything has nothing to lag behind. There is no error, no warning, and no
    "degraded" region.
  • The failure surfaces only when the leader is lost: the follower is elected, and the data written
    since the region was created is gone.
  • The automatic leader balance of data regions is disabled at the same time: RouteBalancer caches
    the same value and reaches the same conclusion, so a hot leader stays hot.

Root causes

The bug has four independent layers. Fixing only the first one leaves the cluster degraded, which is
why this PR addresses all four.

1. ConfigNode does not recognize the aliases

RegionMaintainHandler compared CONF.getDataRegionConsensusProtocolClass() with a raw string
literal in four places:

  • createInitialConsensusPipes() — so no pipe was created when the database was created;
  • checkAndRepairConsensusPipes() — so the periodic self-healing path returned immediately;
  • isIoTConsensusV2DataRegion() — so the region was treated as a non-V2 region;
  • the MultiLeader branch of createPeer — so the replica sets were built as if IoTConsensusV2
    were absent.

RouteBalancer cached the same raw value and disabled the data region leader balance for the same
reason.

2. The reconciliation is starved exactly when it is needed

Even with a canonical value (CONF is rewritten to the canonical form on restart, see below), the
missing pipes were still not created. PipeMetaSyncer#sync() skips an entire round when

isLastPipeSyncSuccessful && canSkipNextSync()

and canSkipNextSync() requires isLastSyncedPipeTaskInfoEmpty && pipeMetaKeeper.isEmpty() && lastSyncedVersion == latestVersion, i.e. the skip is enabled precisely when the cluster has no
pipe at all
. checkConsensusPipesIfDue() used to run after that early return and counted rounds:
CONSENSUS_PIPE_CHECK_INTERVAL_ROUND = 5 rounds are needed before the reconciliation runs. A cluster
that lost all of its pipes skips from round 2 on, so the counter never advances past 1 and the
reconciliation never runs.

The result is an inversion of the health signal:

Cluster state Rounds that reach the counter Reconciliation
at least one user pipe exists (sync never skipped) every round runs every 5 rounds, ~15 min
no pipe at all (the degraded state) only the first one never runs

This also explains why restarting does not reliably recover an affected cluster. The persisted value
in system.properties is canonical (SystemPropertiesUtils normalizes both on write and on read, and
ConfigNodeStartupCheck.checkSystemProperties() overwrites CONF with it), so after a restart
ConfigNode does recognize IoTConsensusV2 — but the reconciliation that would recreate the pipes is
still starved while the cluster has no pipe, so a cluster that only ever used consensus pipes stays
degraded indefinitely.

3. The reconciliation, when it runs, is slow and reactive

PipeMetaSyncer syncs every pipe_meta_syncer_sync_interval_minutes (default 3 minutes) and
reconciles every 5 syncs, i.e. every 15 minutes, and only on the ConfigNode leader. A replication
channel lost at runtime (a peer replaced, a pipe dropped by hand, a partially applied configuration
change) stayed broken for up to that long before anything noticed.

4. Nothing reports the degradation

There is no error and no warning: a region whose pipes were never created looks exactly like a
healthy one, because syncLag is 0 in both cases (a follower that never received a byte has
nothing to lag behind). The first symptom is data loss after a leader failover.

The fix

1. Normalize at the configuration boundary and reject what cannot be acted on

  • ConfigNodeConfig#setDataRegionConsensusProtocolClass() now normalizes through
    ConsensusFactory.normalizeConsensusProtocolClass(), so CONF never holds an alias regardless of
    the source (properties file, system properties, programmatic setup).
  • ConfigNodeDescriptor normalizes on read and logs a WARN when a legacy name is found, so the
    degradation is visible in the ConfigNode log instead of being silent.
  • ConfigManager#checkConfigNodeGlobalConfig() normalizes the value sent by a joining ConfigNode, so
    a mixed-version cluster is not rejected during a rolling upgrade.
  • RouteBalancer normalizes its cached value.
  • RegionMaintainHandler#isIoTConsensusV2Configured() is now the single place that answers "is this
    cluster on IoTConsensusV2?", used by all four call sites above.
  • ConfigNodeStartupCheck#checkDataRegionConsensusProtocolClass() rejects any value that is not one
    of the four supported classes, after normalization, with a message that lists the accepted values.
    An unrecognized class name is not a cosmetic typo: it silently disables replication, so failing at
    startup is strictly better than starting a cluster that cannot replicate.

2. Never starve the reconciliation

checkConsensusPipesIfDue() moved before the early return of PipeMetaSyncer#sync(), so the counter
advances on every attempted round, whether or not the round was skipped. A cluster with no pipes
now reconciles every 5 rounds (~15 minutes by default) like any other cluster, instead of never.

3. Reconcile as soon as the pipe count mismatches

Relying on the 15-minute cadence alone still leaves a 15-minute window, and the counter can be reset
by unrelated activity. Every round now runs a cheap probe first:

hasConsensusPipeCountMismatch()
  = Σ over DataRegions of N * (N - 1)      // pipes the replica sets require
  ≠ number of existing CONSENSUS pipes     // pipes that exist

A mismatch means at least one region lost its replication channel, so the full reconciliation runs
immediately and keeps running every round until the counts match again.

The probe is designed to be affordable on every 3-minute round:

  • PartitionManager#getReplicaPairCount() is a new zero-copy chain
    (PartitionInfo → DatabasePartitionTable → RegionGroup#getReplicaNum); it does not call
    getAllReplicaSetsMap(), which deep-copies every replica set. It also skips pre-deleted databases,
    mirroring the filter of checkAndRepairConsensusPipes().
  • PipeTaskInfo#getConsensusPipeCount() walks the pipe meta under the read lock and counts
    PipeType.CONSENSUS; it does not build a single __consensus.* name, unlike
    getConsensusPipeStatusMap().
  • The probe returns false when IoTConsensusV2 is not configured, so other protocols do not pay for it.
  • Single-replica regions expect 0 pipes, so an RF=1 cluster never reports a mismatch.
  • Any exception inside the probe is caught and logged; the periodic full check still runs.

Counting is a trigger, not a proof of correctness: a pipe of a replaced peer can keep the total
unchanged while pointing at the wrong endpoint. The 15-minute full reconciliation is therefore kept.

4. Report every region that has no pipe

The reconciliation now walks the replica sets and logs one WARN per DataRegion that has no consensus
pipe at all, before the per-pipe messages. The per-pipe warning only fires once the operator already
knows which pipes are expected; this one names the region and says that its followers receive no data.

syncLag is deliberately left untouched: it keeps its current semantics instead of gaining a special
value, so existing dashboards, alert rules and aggregations keep working unchanged.

Behaviour changes

Situation Before After
alias configured (pipe.IoTConsensusV2 / pipe.PipeConsensus) accepted, silently treated as "not V2" normalized, treated as V2
unknown class name configured accepted, silently treated as "not V2" ConfigNode refuses to start, message lists the accepted values
cluster with pipes, a pipe lost at runtime repaired within ~15 min repaired within one sync round (~3 min)
cluster with no pipe at all never repaired repaired on the next sync round (~3 min), without a restart
region without any pipe invisible WARN per region in the ConfigNode log

Recovering a cluster that is already degraded

Upgrading the ConfigNode leader is enough: the probe sees the mismatch on the next sync round and
recreates the missing __consensus.* pipes, no restart and no manual CREATE PIPE needed.

The pipes only carry data written after they exist. Data written while the channel was missing
exists only in the leader's state machine; whether it is back-filled depends on the pipe/snapshot
start semantics, so operators of an affected cluster should verify the replicas (backup, or a replica
consistency check) instead of assuming the history was repaired.

Tests

23 new tests, all green:

  • ConfigNodeConfigTest (2) — both aliases are stored in canonical form; canonical and Ratis values
    are preserved.
  • ConfigNodeStartupCheckTest (3) — the four supported classes pass; both aliases pass and are
    normalized; an unrecognized class fails with a message that names the accepted values.
  • PipeMetaSyncerTest (2) — the reconciliation runs on its own cadence even when sync() is skipped
    every round, and a count mismatch reconciles on the next round and keeps reconciling until the
    counts match.
  • RegionMaintainHandlerConsensusPipeTest (16) — pipe creation under both aliases, the per-region
    WARN for a region without any pipe, the count probe (mismatch / match / single replica / non-V2),
    plus the existing create-drop-restart cases.

Also re-run for regression: PipeConfigNodeMetricsRestartTest and IoTConsensusV2ServerMetricsTest
(6 tests) — 29 tests in total, 0 failures. Verified with mvn test-compile (full reactor) and
mvn test-compile -P with-zh-locale; spotless applied.

Alternatives considered

  • Report syncLag = -1 for a region without a pipe. Rejected, and syncLag is left untouched by
    this PR. A negative lag would poison every aggregation (sum, avg, abs) and every existing
    dashboard, would make RF=1 clusters permanently "abnormal", would flap during replica-set changes,
    and would break under version skew (a new DataNode reporting -1 to an old ConfigNode). The
    degradation is reported by the per-region WARN instead.
  • Store a "degraded" flag on the region. Rejected: it is derived state that can go stale, and it
    would have to be maintained by every code path that touches replica sets or pipes. The
    reconciliation derives it from the partition table and the pipe meta every time, so it cannot
    disagree with reality for long.
  • Mark the region as degraded / remove the under-replicated replicas. Rejected: a region whose
    data is missing on a follower is still readable and writable, and dropping a replica destroys the
    evidence and the chance to back-fill.
  • Rate-limit the pipe creation. Not done here: a burst is bounded by the number of regions, and
    the existing create-procedure queue already serializes the work. It can be added if a very large
    cluster turns out to need it.

This PR has:

  • been self-reviewed.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added documentation for new or modified features or behaviors.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR
  • org.apache.iotdb.confignode.conf.ConfigNodeConfig
  • org.apache.iotdb.confignode.conf.ConfigNodeDescriptor
  • org.apache.iotdb.confignode.conf.ConfigNodeStartupCheck
  • org.apache.iotdb.confignode.procedure.env.RegionMaintainHandler
  • org.apache.iotdb.confignode.manager.pipe.coordinator.runtime.PipeMetaSyncer
  • org.apache.iotdb.confignode.manager.pipe.coordinator.task.PipeTaskCoordinator
  • org.apache.iotdb.confignode.persistence.pipe.PipeTaskInfo
  • org.apache.iotdb.confignode.manager.partition.PartitionManager
  • org.apache.iotdb.confignode.persistence.partition.PartitionInfo
  • org.apache.iotdb.confignode.persistence.partition.DatabasePartitionTable
  • org.apache.iotdb.confignode.manager.load.balancer.RouteBalancer
  • org.apache.iotdb.confignode.manager.ConfigManager
  • org.apache.iotdb.confignode.persistence.partition.RegionGroup
  • i18n: ConfigNodeMessages / ManagerMessages / ProcedureMessages (en / zh)

@Caideyipi Caideyipi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review

The normalization is applied when ConfigNode reads its setting and when it compares a joining node's parameters, so both startup and mixed-version registration use the canonical class name. The pipe creation and repair checks follow the same alias handling, and the new tests cover both legacy names. I found no blocking issue in the changed paths.

ConfigNode compared the raw value of data_region_consensus_protocol_class with
ConsensusFactory.IOT_CONSENSUS_V2, while DataNode normalizes the configured value
through ConsensusFactory.normalizeConsensusProtocolClass before using it. When the
configuration used the legacy org.apache.iotdb.consensus.pipe.PipeConsensus name
or the implementation class name
org.apache.iotdb.consensus.pipe.IoTConsensusV2, ConfigNode concluded that
IoTConsensusV2 was not configured and therefore:

- createInitialConsensusPipes() never created the __consensus.* pipes on database
  creation;
- checkAndRepairConsensusPipes() returned immediately, so PipeMetaSyncer could
  never repair missing pipes;
- RouteBalancer disabled the automatic data region leader balance.

Since the IoTConsensusV2 write path only writes the local state machine, the
missing pipes silently degraded the data region replication factor to one: no
error was reported and syncLag stayed at zero.

Normalize the value once at the configuration boundary so CONF never holds an
alias, log the normalization when the configuration file uses an alias, keep the
comparison sites and the RouteBalancer cache defensive, and normalize the value
sent by a joining ConfigNode so a mixed-version cluster is not rejected.
…ass at startup

ConfigNode decides whether it has to create consensus pipes by comparing
data_region_consensus_protocol_class with the canonical IoTConsensusV2 class
name. An unrecognized value therefore silently disables IoTConsensusV2
replication: data_replication_factor=2 degrades to a single replica without
any error, and the pipe based syncLag metric keeps reporting 0.

Fail the startup check with a ConfigurationException that lists the accepted
classes instead of starting a cluster that cannot replicate.
…gions without pipes

PipeMetaSyncer#sync() returned early whenever canSkipNextSync() was true,
which is exactly the state of a cluster whose consensus pipes were never
created: the pipe task info is empty and its version never changes. The
reconciliation therefore only ran inside the unskipped path and could never
repair the very cluster it exists for.

Move the interval check in front of the early return, and add a region level
ERROR when an expected DataRegion has no consensus pipe at all. The existing
per pipe warning is only emitted once the operator already knows which pipes
are expected, so the degradation of a whole region had no single, obvious
signal before a replica actually failed.
…atches

The reconciliation ran on a fixed round cadence, so a region that lost its
consensus pipes could stay unreplicated for up to five sync intervals even
after the starvation fix made the check reachable again.

Probe the pipe count on every sync round instead: the number of ordered
replica pairs over the DataRegion replica sets is exactly the number of
consensus pipes IoTConsensusV2 requires, and comparing it with the number of
existing consensus pipes neither builds a pipe name nor copies a replica set
(the new PartitionManager#getReplicaPairCount iterates the partition tables
in place). A mismatch means at least one region has lost its replication
channel, so the full reconciliation runs immediately and keeps running every
round until the counts match again. Equal counts still trigger the full check
periodically, because a same-size but different pipe set cannot be detected
by counting.

The probe is skipped entirely when IoTConsensusV2 is not configured, and a
single-replica region expects no pipe at all, so single-replica clusters never
report a mismatch.
@luoluoyuyu
luoluoyuyu force-pushed the fix/cn-normalize-consensus-protocol-alias branch from 0c32041 to 4a85526 Compare November 6, 2026 03:24
Keep production comments to what the code does not already say: the long
explanations belonged in the commit messages, not next to the statements.
Give every new test a short comment stating the scenario it covers and the
behaviour it asserts.
@luoluoyuyu
luoluoyuyu force-pushed the fix/cn-normalize-consensus-protocol-alias branch from 4a85526 to 9a75acc Compare November 6, 2026 04:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants