fix(loader): guard concurrent graph loading

[imbajin]

Purpose of the PR

When two Loader processes import one graph on one live Server instance, the first to finish can reset LOADING while the other still owns the import. Use the paired Server ownership API so each ordinary import releases only its own handle.

Source PR: hugegraph#68.

Main Changes

• Keep one fixed instance/epoch/token handle across acquire and release. Permission, network and malformed-response failures never fall back to enum writes.
• Preserve old Server ordinary/restore behavior. On an owner-capable Server, ordinary import requires NONE or owner-held LOADING; restore retains RESTORING until an operator safely resets it.
• Keep tokens out of diagnostics. Document administrator recovery after stopping and draining all imports/writes, using current instance/epoch/revision when the original handle is lost.
• Explicitly run LegacyServerCompatibilityTest in the release-1.7 CI leg.

Release-failure diagnostics now direct users to stop and drain related imports, then follow the Server administrator recovery procedure in the Loader README.

Verifying these changes

• Need tests and can be verified as follows:
  • Client/Loader unit tests, formatting and Checkstyle pass; capability responses containing revision preserve the original acquire/release tuple.
  • Ordinary and restore imports pass against the official ASF Server 1.7 distribution after signature and checksum verification.
  • Two independent Loader JVMs import CSV into the paired standalone Server: concurrent ownership preserves LOADING after the first shutdown and returns NONE after the last; independent reads verify the rows.
  • Controlled administrator recovery permits ordinary import after restore, rejects stale revision/epoch recovery, and keeps late old handles terminal.

New-head CI and broader Hubble/distributed/crash scenarios remain separate gates. Capacity-rotation refresh is tracked in the Server review thread; no automatic new-token retry is added.

Does this PR potentially affect the following parts?

• Dependencies
• Modify configurations
• The public API
• Other affects
• Nope

Documentation Status

• Doc - TODO
• Doc - Done: Loader README.
• Doc - No Need

Conceptual overview

Protocol-1 imports retain one fixed owner handle and release it after workers stop. Only a recognized missing endpoint uses legacy mode; same-Server ownership and administrator recovery require paired Server #3331.

Conceptual illustration; not test or release-acceptance evidence.

[codecov]

Codecov Report

❌ Patch coverage is 61.44578% with 32 lines in your changes missing coverage. Please review.
✅ Project coverage is 47.11%. Comparing base (8e6ed84) to head (73ad996).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
...g/apache/hugegraph/structure/LoadingModeOwner.java	63.33%	3 Missing and 8 partials ⚠️
...ava/org/apache/hugegraph/api/graphs/GraphsAPI.java	0.00%	10 Missing ⚠️
...ava/org/apache/hugegraph/driver/GraphsManager.java	0.00%	5 Missing ⚠️
.../apache/hugegraph/loader/executor/LoadContext.java	81.48%	1 Missing and 4 partials ⚠️
...a/org/apache/hugegraph/loader/HugeGraphLoader.java	90.90%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master     #815      +/-   ##
============================================
+ Coverage     47.04%   47.11%   +0.07%     
- Complexity     4517     4547      +30     
============================================
  Files           602      603       +1     
  Lines         29448    29534      +86     
  Branches       3454     3482      +28     
============================================
+ Hits          13853    13915      +62     
- Misses        14312    14324      +12     
- Partials       1283     1295      +12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[imbajin]

[P2] Preserve the ownership recovery handle when both loading and release fail

Reviewed commit: 1b3bd93.

LoadContext.unsetLoadingMode() includes the owner tuple in its release-failure exception, but HugeGraphLoader.stopThenShutdown() calls context.close() in a plain finally block. close() can rethrow the earlier error stored by failLoading(), replacing the tuple-bearing release exception rather than preserving it as a suppressed exception. Logger/progress cleanup failures can also mask it.

For example, acquisition may be applied server-side while its response is lost, followed by a release request that cannot reach the Server. The non-expiring owner can remain active, yet the surfaced error contains only the original network failure and omits the generated tuple needed for the documented manual recovery. close() then marks the context closed, so another shutdown() returns without retrying.

Please preserve both failures and reliably expose the original recovery tuple. Add a regression test through load() plus shutdown() where acquisition/loading and release both fail; a direct unsetLoadingMode() test does not cover the masking finally path.

Relevant locations: HugeGraphLoader.java lines 220–234 and 925–932; executor/LoadContext.java lines 269–277 and 298–338. This is a source-level finding independently cross-checked by two reviewers, not a reproduced network-failure test.

Source:

hugegraph-toolchain/hugegraph-loader/src/main/java/org/apache/hugegraph/loader/HugeGraphLoader.java

Lines 925 to 932 in 1b3bd93

	try {
	// Keep dependencies alive if cancellation could not terminate the workers.
	if (tasksStopped) {
	try {
	this.context.unsetLoadingMode();
	} finally {
	this.context.close();
	}

[imbajin]

Addressed the release-error review. When release and context cleanup both fail, shutdown now preserves the original release failure and its instance/epoch/token recovery handle, with the cleanup failure attached as suppressed. Resources still close, and a cleanup-only failure keeps its original exception.

Regression checks cover lost replies and real logger cleanup. The published two-file update matches the source checked by all three independent reviewers. Recovery remains an explicit REST release with the same handle and authenticated identity. Current-head CI and full matching Server/Loader/Hubble verification remain open.

[bitflicker64]

Blocking: yes. Summary: The owner handle is kept fixed through acquire and release, the 404 fallback and the release-failure reporting in stopThenShutdown() read correctly, and the earlier P2 about the masked release exception is addressed at this head. Three points remain: an ordinary import now fails on a graph that a previous restore import left in RESTORING, the handle is never recorded unless a release fails, and LegacyServerCompatibilityTest is not run by any CI job. Evidence: full diff 8e6ed84..63cfd76 read with LoadContext, HugeGraphLoader, GraphsAPI, ServerException, the loader pom.xml profiles and loader-ci.yml at this head, and LoadingMode, StandardHugeGraph and the Server GraphsAPI at apache/hugegraph#3331 head 4a42a77; no tests were run for this review and every check on this head was still queued at review time.

[bitflicker64]

Blocking: no. Score: 9/10 (10 - 0.5 - 0.5 for two Minor). Summary: the owner tuple stays fixed across acquire and release, the 404 fallback only triggers on the Jersey not-found reply, and cleanup keeps the release failure with the close failure suppressed. The earlier points about the masked release error, the RESTORING prerequisite, the lost-handle recovery and the CI run of LegacyServerCompatibilityTest are addressed at this head. Evidence: read the full diff with LoadContext, HugeGraphLoader, GraphsAPI and ServerException, and LoadingMode/GraphsAPI at apache/hugegraph#3331 head 7762b90. LoadingModeOwnerTest (4 tests) and LoadContextTest (35 tests) pass locally. The release-1.7 loader leg ran LegacyServerCompatibilityTest (2 tests, 0 failures), and all checks pass at 324206d. Open: apache/hugegraph#3331 is not merged yet, so the protocol-1 path has only been checked against stub servers in this build. I did not run the two-Loader scenario against the paired Server.

[bitflicker64]

Minor: Hubble never releases this handle. LoadTask.run() calls only loader.load(), and stop() and reconnect() drop the loader without shutdown(), so unsetLoadingMode() runs only from the JVM shutdown hook.

On legacy Servers Hubble already left LOADING behind, but an enum PUT to NONE cleared it. On a #3331 Server each Hubble task and each resume adds an active owner. The Server then rejects every enum mode change, including restore, until Hubble exits or an administrator runs recover.

Could Hubble call loader.shutdown() at the end of LoadTask.run() and in stop(), or could the README call out this Hubble limitation next to the recovery steps?

[bitflicker64]

Blocking: no. Summary: The only change since 324206d is the release-failure message, which now points at the Server administrator recovery procedure instead of a retry, so that earlier Minor thread is addressed; the owner handle, the narrow 404 fallback and the cleanup ordering in stopThenShutdown() are unchanged and still read correctly. One new Minor note on where the README places the ownership section, inline. Evidence: full diff 8e6ed84..73ad996 and the delta 324206d..73ad996 read with LoadContext, HugeGraphLoader.setGraphMode/load/stopThenShutdown, client GraphsAPI and LoadingModeOwner, and LoadingMode.update and the Server GraphsAPI at apache/hugegraph#3331 head 7762b90; walked a rejected acquire (for example a leftover RESTORING) through shutdown: the release of the unregistered token returns released with the current mode, which verify accepts, so the original rejection is not masked. LoadContextTest still asserts only the unchanged Failed to release LOADING owner prefix. No tests were run for this review. At review time the Module validation client-ci and loader-ci matrix jobs for this head (including the release-1.7 leg that runs LegacyServerCompatibilityTest) had been queued for about three and a half hours, so no CI result covers this head yet. The Hubble release gap (open Minor thread on LoadContext.java line 238) is handled by the paired #814 and is not repeated here.

[bitflicker64]

Minor: this section lands under ## 5. License, after the license sentence and the Flink CDC link, so the README now shows the ownership protocol and the administrator recovery steps as a subsection of License. Someone reading #### Graph mode during startup and cleanup in 3.1 (line 136), which now opens with "On Servers without the loading-owner endpoint", has no pointer to the protocol-1 behaviour, and the release-failure message in LoadContext.unsetLoadingMode() sends users to a recovery procedure they will not find under 3.1 or 4. Requested change: move this block into 3.1 as a #### subsection right after "Graph mode during startup and cleanup" (or link to it from line 138), and leave ## 5. License with only the license text.