fix(core): drain tasks before graph deletion

[imbajin]

A cancelled task can still be executing or saving its completion callback. Graph deletion previously treated logical cancellation or a failed scheduler close as sufficient to remove data and registrations. This change waits for the actual local owners to exit first.

• Fence task admission in both schedulers and retain execution/cancellation owners through callback persistence and transaction cleanup. Closing a scheduler alone leaves unstarted persistent tasks in storage.
• Drain before clearing graph data or PD metadata. On a drain timeout, retain the graph and its registration for an explicit retry; remove configuration only after successful local closure.
• Keep schedulers registered after uncertain cleanup and close the shared executor only after its schedulers are gone. Keep long-running cron queries outside the admission monitor so shutdown can reach its timed barrier. Use initialized/actual-start/drain fixtures while preserving the precise missing-graph, task-deletion and ephemeral-absence assertions.

flowchart LR
    A[Stop task admission] --> B[Drain execution, callbacks and transaction cleanup]
    B -->|Drained| C[Delete graph data]
    B -->|Timeout| D[Retain graph; retry after owners exit]

Loading

Scope is intentionally limited to local task shutdown and safe deletion. Stop writes and tasks on every Server before maintenance, and do not recreate the same graph name until deletion completes. Backend/metadata deletion is nontransactional; errors after destructive cleanup require operator inspection. Owner-cleanup failure does not count as a successful drain.

Hot reload recovery, cross-Server notification reconciliation, same-name replacement and task-record deletion policies are deferred to separate work. Use a restart for configuration changes. Their full source is preserved outside this PR; no global lifecycle coordinator or fallback executor is added.

Validation: Java 17 formatting, clean reactor compilation, focused scheduler/deletion/permission regressions and the full TaskCoreTest class passed. The blocked-cron regression fails before the lock correction and verifies the real timeout barrier, admission rejection and successful retry afterward. TTL regressions use a current input date and verify exact expiry while preserving all visibility checks; related Memory and real RocksDB cases passed. Latest-head ASF Linux suites (including HStore), macOS Intel/ARM compatibility and Commons/RPC passed, with complete coverage uploaded. The paired org CI also passed. A final review more than an hour after the last fix found no new actionable feedback or unresolved threads.

See docs/server-runtime.md for the maintenance contract. Paired source: org #285 / ASF #3308. The broader website documentation #519 is held as a draft pending scope reconciliation and is not a merge prerequisite.

[codecov]

Codecov Report

❌ Patch coverage is 62.58065% with 116 lines in your changes missing coverage. Please review.
✅ Project coverage is 52.72%. Comparing base (68fb045) to head (bb3ebc8).

Files with missing lines	Patch %	Lines
...pache/hugegraph/task/DistributedTaskScheduler.java	58.15%	56 Missing and 21 partials ⚠️
...n/java/org/apache/hugegraph/core/GraphManager.java	46.66%	15 Missing and 1 partial ⚠️
...g/apache/hugegraph/task/StandardTaskScheduler.java	81.81%	7 Missing and 7 partials ⚠️
...n/java/org/apache/hugegraph/StandardHugeGraph.java	46.15%	6 Missing and 1 partial ⚠️
...in/java/org/apache/hugegraph/task/TaskManager.java	66.66%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3308      +/-   ##
============================================
+ Coverage     52.63%   52.72%   +0.09%     
- Complexity     2435     6120    +3685     
============================================
  Files           770      770              
  Lines         67694    67933     +239     
  Branches       9172     9210      +38     
============================================
+ Hits          35631    35821     +190     
- Misses        28520    28555      +35     
- Partials       3543     3557      +14     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bitflicker64]

Blocking: yes. Summary: The deletion ordering and the retained registrations work as described, but three paths behave differently from master for callers that are not retrying a failed deletion: a drop with the local scheduler no longer waits task.wait_timeout for short tasks, a plain close of the distributed scheduler cancels dispatched tasks that never started, and reload() stops halfway when one graph cannot close. Evidence: read the full diff 87a06a9..9918ff2 (14 files, +3653/-169) and the surrounding code in GraphManager, StandardHugeGraph, TaskManager, HugeTask and both schedulers; built both trees with JDK 17 (mvn -pl hugegraph-server/hugegraph-test -am install -DskipTests) and ran one JUnit probe on each: dropping a memory graph while a 1.5 s task runs returns after 1720 ms on 87a06a9 and throws after 0 ms on 9918ff2; the scheduler-close and reload() points are from reading the code and from the assertions in DistributedTaskAdmissionTest, not from a run. All check runs at 9918ff2 are green or skipped.

[imbajin]

The scope has been narrowed to local task admission, physical execution/callback drain, and safe graph deletion. Global lifecycle coordination, hot reload restoration, cross-Server notification reconciliation, task-record deletion recovery, and fallback executors have been removed from this PR and recorded for separate work.

Previous descriptions claiming those broader guarantees are superseded. Configuration changes require a restart while reload recovery is deferred; all Servers must stop writes and tasks before deletion. The broader website documentation PRs are now drafts pending reconciliation.

Core regression tests and clean compilation passed. The missing-graph fixture now uses normal GraphManager construction and still requires the original IllegalArgumentException. Current-head Linux CI and complete coverage are pending.