Skip to content

Support asynchronous timeseries and table column deletion - #19186

Open
Caideyipi wants to merge 5 commits into
apache:masterfrom
Caideyipi:feature/async-schema-deletion
Open

Caideyipi wants to merge 5 commits into
apache:masterfrom
Caideyipi:feature/async-schema-deletion

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Tree-model DELETE/DROP TIMESERIES and table-model ALTER TABLE ... DROP COLUMN now return after durable admission of a background procedure. Callers can inspect tree deletion with SHOW DELETE TIMESERIES PROGRESS and column deletion with DESCRIBE ... DETAILS, then wait for completion before dependent DDL.

  • Add a bounded, configurable tree-deletion scheduler that controls queue length, the minimum interval between execution starts, and task concurrency.
  • Preserve query-ID idempotency, expose task IDs and progress, and keep terminal progress separate from durable finalization. Return clear admission messages for asynchronous table and column deletion.
  • Persist admission and completion transitions, isolate stored procedure state, and retain same-database DDL reservations through database deletion/recreation and recovery. Keep tree/table database namespaces separate and protect database configuration from being overwritten during recreation.
  • Update integration-test helpers and dependent tests to wait for durable deletion completion, including JDBC and Session paths.

Tree-model asynchronous deletion configuration

Configure these ConfigNode settings in iotdb-system.properties; changes take effect after a ConfigNode restart.

Configuration Default Meaning
timeseries_deletion_queue_capacity 1024 Maximum number of tree-model timeseries deletion tasks waiting in the queue.
timeseries_deletion_execution_interval_ms 1000 (1 second) Minimum interval between starting/dequeuing tree-model timeseries deletion tasks.
timeseries_deletion_max_concurrent_tasks 1 Maximum number of tree-model timeseries deletion tasks executing concurrently.

The execution interval applies when tasks leave the queue to start execution. Enqueue requests are not throttled by this interval. With the defaults, up to 1024 tasks can wait, one task executes at a time, and successive task starts are at least 1 second apart.

Example: submit deletions and track completion

CREATE DATABASE root.async_delete_demo;
CREATE TIMESERIES root.async_delete_demo.d1.s1 WITH DATATYPE=DOUBLE, ENCODING=GORILLA;
CREATE TIMESERIES root.async_delete_demo.d1.s2 WITH DATATYPE=DOUBLE, ENCODING=GORILLA;

DELETE TIMESERIES root.async_delete_demo.d1.s1;
DELETE TIMESERIES root.async_delete_demo.d1.s2;

SHOW DELETE TIMESERIES PROGRESS;

Each DELETE TIMESERIES returns after its task has been durably admitted, with a task/procedure ID in the response message. It does not wait for deletion to finish. The two requests can be submitted back-to-back; with the default scheduler settings, their tasks execute one at a time and start at least 1 second apart.

SHOW DELETE TIMESERIES PROGRESS returns ProcedureId, QueryId, Paths, State, Progress, and ErrorMessage. Match each returned procedure ID and poll until its State is SUCCESS before recreating the corresponding timeseries or issuing dependent DDL. FINALIZING still means completion is being persisted; ROLLEDBACK indicates failure and should be checked with ErrorMessage.

For example, after the task for root.async_delete_demo.d1.s1 reports SUCCESS, the series can be recreated:

CREATE TIMESERIES root.async_delete_demo.d1.s1 WITH DATATYPE=DOUBLE, ENCODING=GORILLA;

Validation

  • Current submitted source: Spotless, git diff --check, and a clean English integration-test dependency-closure build passed (clean verify -pl integration-test -am -P with-integration-tests,SimpleIT). The selected audit IT was skipped by its existing class-level @Ignore (1 selected, 1 skipped); its test body did not run.
  • Immediately preceding snapshot, with identical production sources: 56 targeted unit tests and 8 targeted IT executions passed. The IT runs covered 1C1D and 1C3D, including Memory and PBTree schema modes for tree-model cases. The subsequent change only tightened the audit IT's procedure-ID completion wait; those runtime results are historical rather than reruns of the final snapshot.
  • Verification reused third-party dependency caches, disabled Maven extensions/build caching, and skipped Enforcer. Required IoTDB dependencies were built from clean source; downloaded IoTDB SNAPSHOT binaries were not used.
  • Not run: full-reactor English/Chinese builds, complete distribution packaging, the remaining affected IT matrix (including the Session-path test update), the audit IT body, and real-cluster leader-switch race scenarios.

This PR has:

  • been self-reviewed.
  • added unit tests or modified existing tests to cover new code paths.
  • added integration tests.
  • been tested in a test IoTDB cluster, with the snapshot and scope limits described above.

Key changed classes

ProcedureManager, ProcedureExecutor, DeleteTimeSeriesProcedure, DeleteTimeSeriesProcedureScheduler, ConfigProcedureStore, SQL parsing/progress RPC classes, and integration-test completion helpers.

@Caideyipi
Caideyipi marked this pull request as draft November 9, 2026 08:11
@Caideyipi
Caideyipi marked this pull request as ready for review November 9, 2026 08:17
@Caideyipi

Copy link
Copy Markdown
Collaborator Author

CI reported an error in IoTDBTableDDLHAIT.testHAWithOneDataNodeIsDown while the test waited for asynchronous timeseries deletion. The progress query used the cluster-wide result wrapper, which also queried the DataNode intentionally stopped by this test. The test now polls deletion progress through a single-node statement on its existing live connection.

Validation passed on Windows 11 with JDK 17: mvn -T 2 -Ddevelocity.off=true clean verify -DskipUTs -Dit.test=IoTDBTableDDLHAIT#testHAWithOneDataNodeIsDown -DfailIfNoTests=false -Dfailsafe.failIfNoSpecifiedTests=false -pl integration-test -am -PTableClusterIT -P with-integration-tests (1 test, 0 failures, 0 errors, 0 skipped). mvn spotless:apply -pl integration-test -P with-integration-tests and git diff --check also passed.

Updated from source head 3e66ecaf42e763ae42fc653dd27cf548fcbb9352; final commit: 971feafe81b9dec11a115f36cedfb8b715aa3aa2. Both the source branch and PR head match the final commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant