Skip to content

Fix tier folder recovery and DiskFull status after configuration reload - #19161

Open
hongzhi-gao wants to merge 5 commits into
apache:masterfrom
hongzhi-gao:fix/tier-manager-full-disk-recovery
Open

hongzhi-gao wants to merge 5 commits into
apache:masterfrom
hongzhi-gao:fix/tier-manager-full-disk-recovery

Conversation

@hongzhi-gao

@hongzhi-gao hongzhi-gao commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem

When every directory managed by a FolderManager is full during DataNode startup or configuration reload, construction fails. TierManager previously skipped that manager in its tier lists. Freeing disk space could then restore the DataNode to Running, but a later TsFile allocation could still fail with IndexOutOfBoundsException or select the wrong tier because the list no longer matched the configured tier levels.

A successful configuration reload could also clear ReadOnly(DiskFull) even when the same directories remained full or another disk-full source had not recovered.

Changes

  • Keep an entry for each configured tier and directory type before constructing its FolderManager. Retry a failed construction on the next access and reuse successfully constructed managers.
  • Register recovery checks against the current tier configuration. A directory reload or a new disk-full event invalidates recovery results obtained before that change.
  • Remove the unconditional transition to Running after a successful reload. Heartbeat recovery now requires both the aggregate disk-space check and every registered disk-full recovery check to pass.

Verification

  • Focused DataNode and node-commons unit/component tests passed, including initialization while full, retry after recovery, reload with unchanged/full directories, an unrelated disk-full source, and concurrent recovery checks.
  • python3 .github/scripts/check-it-selection.py passed.
  • mvn spotless:apply -o -pl integration-test -P with-integration-tests -Ddevelocity.off=true passed.
  • mvn verify -o -DskipUTs '-Dit.test=IoTDBDiskFullFolderRecoveryIT' -DfailIfNoTests=false -Dfailsafe.failIfNoSpecifiedTests=false -pl integration-test -am -P with-integration-tests -Ddevelocity.off=true passed: 1 test, 0 failures, 0 errors, 0 skipped. The IT verifies ReadOnly(DiskFull) during a full-directory reload, continued read-only status after an unchanged reload, recovery to Running, and successful TsFile creation and query.

The IT raises the disk-space warning threshold to exercise the real DataNode allocation and recovery paths without physically filling the host filesystem.

@hongzhi-gao hongzhi-gao closed this Nov 6, 2026
@hongzhi-gao hongzhi-gao changed the title Fix tier folder allocation after disk-full initialization Fix tier folder recovery and DiskFull status after configuration reload Nov 6, 2026
@hongzhi-gao hongzhi-gao reopened this Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant