feat(server): coordinate loading owners

[imbajin]

Purpose of the PR

When concurrent Loaders finish independently, one can restore NONE while another still imports. Keep LOADING until the last owner releases it on the same live graph instance, and provide controlled recovery when an import loses its handle.

Source PR: hugegraph#304.

Main Changes

• Bind handles to instance, epoch, token and authenticated principal; make acquire/release idempotent and keep cancelled or released handles terminal. Pass the request principal explicitly even when a graph has no authentication proxy.
• Require administrator authorization, explicit stop/drain confirmation and exact instance/epoch/revision for recovery. A new owner invalidates an earlier recovery snapshot.
• Return HTTP 409 for expected mode/owner conflicts. Audit successful mode/owner transitions under the state lock using mode, epoch and owner counts; omit tokens and principal values.
• Keep RPC and resource cleanup in separate PRs.

Requires the paired Loader integration for concurrent ordinary imports. Older Servers keep historical behavior; restore uses the enum mode path. Protection covers one stable Server/graph instance.

Verifying these changes

• Need tests and can be verified as follows:
  • Core/auth/API regressions, formatting and Java 17 reactor compilation pass.
  • A real HugeFactory memory graph without an auth proxy rejects release by a different caller. Audit regressions cover actual, repeated, rejected and recovery operations without credential values.
  • Real standalone Server and independent Loader processes verify concurrent ownership, CSV readback, restore-to-ordinary recovery, stale revision/epoch conflicts and terminal late handles.
  • Three independent reviewers checked both the source change and the bounded Server/Loader runtime evidence.

New-head CI, non-admin HTTP, abrupt-crash and distributed acceptance remain open. The capacity-rotation first-acquire availability race is tracked separately; automatic capability/token refresh is not added.

Does this PR potentially affect the following parts?

• Dependencies
• Modify configurations
• The public API
• Other affects
• Nope

Documentation Status

• Doc - TODO
• Doc - Done: docs/server-runtime.md.
• Doc - No Need

Conceptual overview

同一稳定 Server / graph 实例的并发导入 owner 与 LOADING 时序；最后释放恢复 NONE，不提供跨节点 fencing 或自动过期。

Conceptual illustration; not test or release-acceptance evidence.

[codecov]

Codecov Report

❌ Patch coverage is 82.11382% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.97%. Comparing base (92af2f8) to head (7762b90).

Files with missing lines	Patch %	Lines
...rc/main/java/org/apache/hugegraph/LoadingMode.java	87.20%	0 Missing and 11 partials ⚠️
...va/org/apache/hugegraph/api/profile/GraphsAPI.java	68.42%	3 Missing and 3 partials ⚠️
.../src/main/java/org/apache/hugegraph/HugeGraph.java	0.00%	3 Missing ⚠️
.../org/apache/hugegraph/auth/HugeGraphAuthProxy.java	87.50%	0 Missing and 1 partial ⚠️
...n/java/org/apache/hugegraph/StandardHugeGraph.java	85.71%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3331      +/-   ##
============================================
+ Coverage     54.31%   56.97%   +2.66%     
+ Complexity     2250      130    -2120     
============================================
  Files           770      515     -255     
  Lines         67952    49851   -18101     
  Branches       9213     7229    -1984     
============================================
- Hits          36905    28404    -8501     
+ Misses        27470    18536    -8934     
+ Partials       3577     2911     -666     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[imbajin]

Independent cross-review of the new Server loading-owner delta: 8.5/10, source-only. No confirmed P1/P2 defect found in same-instance acquire/release serialization, active-principal binding, idempotence, cancellation tombstones or epoch/instance fencing. This is not end-to-end Loader acceptance: the paired client must retain the fixed handle through all writers and cancellation cleanup, reject unsupported Servers before import, and retry uncertain acquisition with the same tuple. Before closing the shared GraphMode gate, preserve a real authenticated REST + two-file staggered-completion check, including lost-response/late-acquire cancellation and retained ownership after backend failure. Current tests mainly exercise the state machine and mocked auth delegation. Cross-Server/LB coordination, expiring leases and automatic crash recovery are explicitly outside scope.

[bitflicker64]

Blocking: yes. Summary: The owner state machine and the instance and epoch fencing behave as described, but an owner leaked by a killed Loader cannot be cleared without the original tuple and user, which leaves the graph in LOADING until the graph instance is reopened. One minor note on the HTTP status of the rejections. Evidence: full diff of 0b7c0de..4a42a77 read with ExceptionFilter, HugeGraphAuthProxy, GraphManager.createGraph and GraphTransaction; LoadingMode and LoadingModeTest from this head compiled standalone on JDK 17 (6 tests pass), plus a probe for the cross-principal release and the enum setter. Server Compatibility CI passed on this head, but HugeGraph-Server CI (the memory job that runs the unit suite, server-rocksdb, cluster-test, pd_store) was still queued, so HugeGraphAuthProxyTest and the permission path rest on reading the code.

[bitflicker64]

Blocking: yes. Score: 5.5/10. Summary: The ten PR files are identical to 4a42a77 (git diff 4a42a77 ba1dbb9 over them is empty), so the two open threads on LoadingMode.java still apply at this head. One new finding on epoch rotation, inline. Evidence: LoadingMode from this head compiled on JDK 17 with a probe that reproduces the rotation case; receipt handling read in apache/hugegraph-toolchain#815 at 63cfd76. All non-skipped CI checks pass.

[bitflicker64]

Important: after an epoch rotation, a Loader cannot tell a retired handle from a cancelled one, so it aborts an import it could have started.

The 1025th distinct release in an epoch, including an unknown release with a fresh token, runs this.epoch++ (line 98). If that lands between a Loader's GET .../mode/owners and its PUT acquire, this line returns released for a token it never saw. Probe on this head: B reads {epoch=0}, a rotation follows, and B's acquire gets state=released, mode=NONE. In #815, LoadingModeOwner.verify throws InvalidResponseException for that receipt, and LoadContext keeps the same owner, so a retry fails too. LoadingModeTest.java:75 and the "permanently retires" sentence in docs/server-runtime.md treat this as intended, but they give the client no way to recover.

Please return a distinct state (for example stale) for any older-epoch tuple that is not active, so the client can read the capability again and acquire with a new token. Update the assertion at LoadingModeTest.java:75 to match.

[imbajin]

The capacity-rotation first-acquire race is confirmed on the frozen Server and Client classes: a never-acquired old handle and an already-cancelled old handle both receive the same terminal state, while an active old handle remains usable until release. We are keeping this availability thread open separately from privileged recovery. Merely returning stale and automatically minting a new token would lose cancellation fencing; safe refresh needs to account for the context lifecycle, unknown acquire outcomes, and the distinction between automatic rotation and administrator recovery. The current increment preserves permanent terminal old tuples and does not add that automatic retry.

[bitflicker64]

Blocking: no. Summary: No new blocking finding at this head, two minor notes inline; the recover action and the 409 mapping behave as documented and answer the two earlier threads. The epoch-rotation thread stays open and still applies: the code under it (LoadingMode.java line 100) is unchanged, and it is not repeated here. Evidence: full diff of 92af2f8..623be2f read with AuthenticationFilter, HugeAuthenticator, HugeGraphAuthProxy, ExceptionFilter, GraphManager.createGraph and HugeFactory.open; LoadingMode and LoadingModeTest from this head compiled standalone on JDK 17 (9 tests pass) with a probe for the empty-principal release, recovery of a manual mode and a stale revision; HugeGraph-Server CI run 38472600754 on this head ran UnitTestSuite (1137 tests, 0 failures), CoreTestSuite and ApiTestSuite in the build-server / server_memory (Java 17) job. GraphSpaceAPITest covers the recover and 409 branches with mocks; the acquire and release branch has no test at the API layer and nothing in ApiTestSuite calls mode/owners.