fix(store): size the RocksDB memory budget from available memory

[byteayan]

Purpose of the PR

• close [Bug] Store ships rocksdb.total_memory_size of 32 GB, ignores the container memory limit, and Docker users cannot override it #3254

Both shipped application-pd.yml files set rocksdb.total_memory_size: 32000000000, and the pd profile is always included, so the Runtime.maxMemory() fallback in AppConfig.init() never ran. RaftRocksdbOptions splits the value into the WriteBufferManager cache (66%) and the block cache. Both are native LRUCaches, so neither -Xmx nor the container limit bounds them, and a separate 1 GiB raft log cache sits on top. In the run in the issue all three Stores were OOM-killed at a 4 GiB limit once each held about 1 GB of RocksDB data.

Docker users had no clean way to change it. docker-entrypoint.sh builds SPRING_APPLICATION_JSON from five HG_STORE_* keys, and none of them is a rocksdb.* key. Bind-mounting a config file or passing -Drocksdb.total_memory_size=<bytes> in JAVA_OPTS both work, but neither is documented.

As the issue points out, dropping the line from the yml alone would hand RocksDB a budget equal to the heap, which is a second heap-sized allocation on top of the first.

Main Changes

• New RocksdbMemoryBudget in hg-store-core/.../options. An unset, blank or 0 value means auto: half of (physical memory − max heap − MaxDirectMemorySize − 1.5 GiB reserve), clamped to 256 MiB to 32 GB. The reserve covers the raft log cache plus stacks, metaspace and other native memory. Physical memory comes from OperatingSystemMXBean.getTotalMemorySize(), which on JDK 17 reports the cgroup v1/v2 limit inside a container. Older JDK 17 updates may not see a limit set only on a parent cgroup, so the budget also reads the cgroup itself (v2: memory.max from the process's cgroup up to the mount root; v1: hierarchical_memory_limit) and uses the lower value. The membership path is rebased onto the mount's root from /proc/self/mountinfo, so a cgroupfs mounted from a subtree such as /kubepods is walked from the right directory. Every mount that contains the process's cgroup is checked and the lowest limit wins, so mount order doesn't matter. Inside a private cgroup namespace a limit above the container can't be seen, and the deployment guide says to pin the value there. MaxDirectMemorySize is counted as the JVM reports it, or as the heap size when unset, since that is what the JVM then allows. bin/start-hugegraph-store.sh now passes -XX:MaxDirectMemorySize equal to the heap it picks, unless -j sets -Xmx or the cap itself, in which case the JVM default (that heap) or the given cap applies. Either way the budget counts a cap the JVM enforces. If the Store couldn't locate its own cgroup in /proc/self/mountinfo (an inherited mount in a cgroup namespace), the limit at the visible mount root still counts and auto-sizing logs a warning to set the budget explicitly. If physical memory can't be read, the previous heap-size default applies.
  • The issue's setup (4 GiB limit, -Xmx1g, -XX:MaxDirectMemorySize=512m) gets 512 MiB, and an 8 GiB limit gets 2.5 GiB. I checked the 4 GiB case with resolve() inside eclipse-temurin:17-jre-jammy under a 4 GiB Docker limit: 536870912 bytes. With -Xmx2g and no direct-memory cap it lands on the 256 MiB floor. The 32 GB cap means no host gets a bigger budget than before. On bare metal the default launcher (2 GiB heap, 2 GiB direct cap) on a 16 GiB host gets 5.25 GiB. A raised heap should come with a smaller explicit cap, for example -j "-Xmx36g -XX:MaxDirectMemorySize=2g" (12.25 GiB on 64 GiB); without one, direct memory counts as a second heap and the floor applies. A large node that relied on the fixed 32 GB should pin the value after upgrading.
  • A positive value is used unchanged, with one exception: values under 1 MiB are refused. The write buffer manager gets 2/3 of the budget, so 1 would give a 0-byte manager, which RocksDB treats as no memtable limit at all. 4GB and other non-numeric values now fail at startup with "must be a number of bytes", where before they failed with a bare NumberFormatException. Nothing shipped (configs, compose files, Helm, CI) sets a value under 1 MiB.
  • Startup logs the chosen value and its inputs. It warns when the 256 MiB floor applies (heap plus direct memory already fill the limit), when an explicit value is under 256 MiB, and when an explicit value exceeds the memory left after the heap, direct memory and the reserve. That last case is what an upgraded install with an old mounted application-pd.yml will hit; the Store also warns when it sees exactly 32000000000, the old shipped value, and the deployment guide has an upgrade step (set it to 0, remove it, or set HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE).
• AppConfig.init() uses it. While there I fixed the raft disruptor sizing for budgets under 1 GB: log(0) rounded to a buffer size of 0. It now gives 32. The formula moved to RocksdbMemoryBudget.disruptorBufferSize() so it has a test. The shipped application.yml pins disruptorBufferSize: 1024, so this only matters when that key is absent.
• Both application-pd.yml files ship total_memory_size: 0, which also removes the [Bug] Store ships rocksdb.total_memory_size of 32 GB, ignores the container memory limit, and Docker users cannot override it #3254 TODO from chore: add TODO breadcrumbs for the Helm chart's server workarounds #3260. The test resources keep their explicit values.
• docker-entrypoint.sh accepts HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE. Values must be digits only; anything else exits with code 2. It is added to SPRING_APPLICATION_JSON only when set, so an unset variable leaves the config file or auto-sizing in charge. When set it outranks both the config file and -Drocksdb.total_memory_size (checked against Spring Boot 2.5's property order).
• helm/hugegraph/values-cluster.yaml: the store comment said container limits don't configure the RocksDB budget. It now says how the budget is derived (2.5 GiB for that preset) and how to pin it with store.extraEnv.
• CI fixture, needed for this PR's HStore job: start-server.sh now writes usePD, schema.sync_deletion and the HBase timeouts idempotently. Since fix(api): allow anonymous cypher queries #3366, run-api-test.sh starts the same directory twice, and the duplicated usePD failed the second init-store with '[true, true]'. The new test-start-server.sh covers two starts on one directory.

In a small container the write buffer manager now flushes memtables much earlier than before, which means more small L0 files and lower write throughput in exchange for staying inside the limit. Both caps are soft, so RocksDB can still go over briefly under heavy writes. I have not measured either effect. Setting the value explicitly restores the old behaviour.

Verifying these changes

• Need tests and can be verified as follows:
  • New RocksdbMemoryBudgetTest (23 tests, added to CoreSuiteTest, which the store-core-test profile runs). It covers:
    • explicit values kept unchanged, and values under 1 MiB refused;
    • unset, blank, 0 and 00 going to auto;
    • refusal of 4GB, -1, 1.5e9 and a value past Long.MAX_VALUE;
    • the 4 GiB and 8 GiB examples above, both clamp bounds, and the unknown-physical fallback;
    • the disruptor size for 512 MiB, 1 GB and 32 GB;
    • the derived write_buffer_size staying at or above RocksDB's 1 MiB minimum, checked through the same HugeConfig validation as startup;
    • unset direct memory counted as the heap, and bare-metal sizes with and without an explicit direct cap;
    • an inherited namespace mount (root /..) that can't be matched being flagged, with the visible root's limit still applied;
    • cgroup limits read from fake v1 and v2 trees: a limit set only on the pod, the container's own lower limit, the namespace-root view, v1 and v2 subtree mounts rebased through mountinfo (including an escaped mount point), several mounts in different orders, unlimited values, and an unreadable /proc/self/cgroup.
  • New test-store-docker-entrypoint.sh (9 cases), added to the store job in pd-store-ci.yml. It checks that the rocksdb key is absent when the variable is unset or empty, that bytes and 0 pass through, that app.data-path is still set, and that 4GB, -1, 1.5e9, 1 2 and 1"} are refused. It clears the variable from the caller's environment first, so "unset" really is unset.
  • mvn test -pl hugegraph-store/hg-store-test -P store-core-test on JDK 17: 128 tests, 0 failures. test-start-hugegraph-store.sh --gc-only (run in the store CI job) covers the launcher with a stub java: no -j gets the computed cap, a heap-only -j (lower or raised) leaves the cap to the JVM, and an explicit -j cap applies. python3 -m unittest discover -s .github/scripts -p 'test_*.py' passes. helm unittest helm/hugegraph: 294 tests pass. mvn editorconfig:check and git diff --check are clean.
  • Not tested: a full Store started in a memory-limited container, and the load from the issue.
  • test-store-docker-entrypoint.sh is mapped to the store suite in .github/scripts/ci-policy.py, so a PR that changes only that script still runs it; the selector test covers the mapping.

Does this PR potentially affect the following parts?

• Dependencies (add/update license info & regenerate_known_dependencies.sh)
• Modify configurations
• The public API
• Other affects (typed here)
• Nope

Documentation Status

• Doc - TODO: required documentation is pending; complete it before merging.
• Doc - Done: documentation is included here or linked below.
• Doc - No Need: no user-visible documentation is affected.

Documentation files in this PR or paired hugegraph-doc PR:

• hugegraph-store/docs/deployment-guide.md: the auto-sizing rule and its inputs, the 1 MiB minimum, the new env var and its precedence in the compose and Kubernetes examples, and the RocksDB block (total_memory_size: 0; the nonexistent block_cache_size key replaced with write_buffer_ratio).
• hugegraph-store/README.md: the env var in the docker run section.
• Paired docs(store): document the auto-sized RocksDB memory budget hugegraph-doc#535 (merge together with this PR): content/{en,cn}/docs/quickstart/hugegraph/hugegraph-hstore.md (the default, the auto-sizing rule and the env var) and content/{en,cn}/docs/guides/hugegraph-docker-cluster.md (the new HG_STORE_* row). The hstore page's "checked against Server master commit" note should move to this PR's merge commit once both land.

Conceptual overview

Concept illustration: derive the RocksDB budget from memory remaining after heap, direct memory and reserve, with clamp bounds and a 1 MiB derived write-buffer floor. Cache budgets remain soft.

[codecov]

Codecov Report

❌ Patch coverage is 81.56425% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.02%. Comparing base (92af2f8) to head (9e6dc50).
⚠️ Report is 28 commits behind head on master.

Files with missing lines	Patch %	Lines
...e/hugegraph/store/options/RocksdbMemoryBudget.java	81.56%	15 Missing and 18 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3355      +/-   ##
============================================
+ Coverage     54.31%   56.02%   +1.71%     
- Complexity     2250     6486    +4236     
============================================
  Files           770      775       +5     
  Lines         67952    68679     +727     
  Branches       9213     9631     +418     
============================================
+ Hits          36905    38477    +1572     
+ Misses        27470    26624     -846     
- Partials       3577     3578       +1     

Flag	Coverage Δ
commons	96.37% <ø> (?)
hstore	35.82% <ø> (?)
pd	49.27% <ø> (?)
server_hbase	41.88% <ø> (?)
server_memory	54.39% <ø> (?)
server_rocksdb	42.35% <ø> (?)
store	32.60% <81.56%> (?)

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bitflicker64]

Blocking: yes. Summary: Score 7/10. The container case from #3254 is handled, and the earlier write_buffer_size thread is fixed by writeBufferSize() at 1927fbe. One blocking problem: auto-sizing counts an unset -XX:MaxDirectMemorySize as a second full heap, and the tarball launcher never sets it, so bare-metal installs with a large heap drop from the shipped 32000000000 bytes to the 256 MiB floor on upgrade. Evidence: the head's RocksdbMemoryBudget.java compiled standalone and resolve(null) run on a 16 GiB host with JDK 17.0.20 returned 268435456 for -Xmx8g and 3221225472 for -Xmx8g -XX:MaxDirectMemorySize=512m; start-hugegraph-store.sh, util.sh calc_xmx, the Store Dockerfile, RaftRocksdbOptions and the Helm store template were read at e691489. Only HugeGraph-Server CI and check-license are listed for this head, both queued, so the new Store tests were not seen passing in CI.

[bitflicker64]

Blocking: no. Summary: Score 9/10. The earlier threads are fixed at this head: the derived write_buffer_size is clamped to 1 MiB, and an unset -XX:MaxDirectMemorySize now counts as at most 1 GiB, so a large-heap bare-metal Store no longer drops to the 256 MiB floor. The auto budget fits the #3254 container setup, and an explicit value still overrides it. Evidence: read RocksdbMemoryBudget, AppConfig.init(), RaftRocksdbOptions, RocksDBOptions, the Store launcher and calc_xmx, the Dockerfile, the entrypoint and both Helm values files at 84e8e27; worked through the arithmetic for 4 GiB/-Xmx1g/512m (512 MiB), the Helm cluster preset (2.5 GiB) and 16 GiB/-Xmx8g (2.75 GiB); ran test-store-docker-entrypoint.sh locally (9 passed); CI at this head ran CoreSuiteTest with the new tests (52 tests, 0 failures) and the entrypoint step, and cluster, docker and helm jobs pass. Only codecov/project fails, because the base is 19 commits behind.

[imbajin]

Blocking: yes. Summary: Auto-sizing can still use host memory on supported older JDK 17 builds that miss parent-only cgroup limits. Evidence: exact-head source and cgroup behavior review; see inline comments for the mounted-config upgrade and Store test-selection gaps.

[imbajin]

[P2] Consider compatible readable cgroup mount views instead of returning the first match.

The original mount-root translation finding is fixed at this revision: independent Java 21 synthetic-file checks using the exact resolver, with only Lombok logging stubbed, now find the 4 GiB limit and produce the 512 MiB budget. Fourteen bounded checks pass, but three multi-mount discrepancies remain, confirmed by two independent reviews.

The resolver returns the first matching cgroup mount without considering later readable views. With membership /kubepods/pod1/ctr, an unrelated first mount or a leaf-only first mount followed by a full readable hierarchy can hide a visible 4 GiB ancestor. With a simulated 64 GiB host, 1 GiB heap and 512 MiB direct cap, the budget is then 32,000,000,000 bytes instead of 536,870,912. Reversing mount order succeeds; this reproduces for both v2 and v1.

Please consider compatible readable mount views and their ancestor limits, with order-invariance regressions. Selecting only the most-specific mount can still omit visible ancestors. Multiple mounts are a plausible layout: see the OpenJDK duplicate-mount fix and runc issue #2404.

Evidence boundary: synthetic file fixtures, not real mounts, full Store startup, performance or load measurement. A correct JVM container limit can mask the overbudget path; this is conditional, not a claim that every container is affected. No new numerical score is assigned.

[byteayan]

@imbajin thanks, fixed in 535c2d0. The resolver now reads every cgroup2 mount, or every v1 mount carrying memory, from mountinfo instead of the first one. It skips mounts whose root doesn't contain the process's cgroup, walks each remaining one from its rebased path, and keeps the lowest limit. A leaf-only or unrelated mount listed first can no longer hide a fuller view listed later. New tests run three mount orders for v2 (unrelated, leaf-only and full views) and two for v1 (with a cpu-only mount that's ignored). Each finds the 4 GiB pod limit and, with a 64 GiB host, 1 GiB heap and 512 MiB direct cap, gives 512 MiB. Store core tests: 126 passed.

[bitflicker64]

Blocking: no. Score: 10/10.

Since 33e53ee, 535c2d0 reads every cgroup2 mount, or every v1 mount carrying memory, from mountinfo. It skips mounts whose root does not contain the process cgroup and keeps the lowest limit across the rest. I reran my earlier synthetic case against this head. A leaf-only cgroup2 mount listed before the full hierarchy now gives 4294967296 instead of -1, and the hybrid v2 plus v1 layout still resolves to the v1 limit. I also tried a private namespace with 0::/, a 0::/../sib path, a /.. mount root, the v1 9223372036854771712 sentinel, a nested child cgroup and an escaped mount point. Each gave the expected limit or fell back to the mount root.

Evidence: on a cgroup v2 host (JDK 17.0.20) I ran resolve() inside a transient slice with MemoryMax=4G and an unlimited service, using -Xmx1g -XX:MaxDirectMemorySize=512m. The JVM reported 16473444352, resolve(null) and resolve("0") returned 536870912, and resolve("32000000000") was kept. Locally at 535c2d0, RocksdbMemoryBudgetTest passed 21 tests and CoreSuiteTest passed 62 with 0 failures. CI at this head is still running.

Earlier threads:

• @imbajin, cgroup mount first match: fixed in 535c2d0.
• @imbajin, subtree mount root not rebased: fixed in 33e53ee, still holds.
• @imbajin, entrypoint test not picked by CI policy: fixed in 3fb0da0.
• @imbajin, parent-only limit missed by older JDK 17, and the old application-pd.yml keeping 32 GB: fixed and documented in 03a4d43, with a WARN on 32000000000.
• @imbajin, derived write_buffer_size below 1 MiB: fixed in 1927fbe.
• Mine, unset MaxDirectMemorySize counted as a full heap: fixed in 84e8e27, still holds.

Not covered: a full Store under write load in a memory-limited container.

[imbajin]

Rechecked the multi-mount finding against the updated source, reviewed 2026-10-16. The prior finding is fixed: all applicable mount views are considered, unrelated roots are excluded, and the minimum visible limit wins rather than the first mount. Two independent source reviews found no new blocker in this repair.

I independently replayed the three earlier counterexamples and added reordered v1/v2 mount and containment controls: 40 bounded Java 21 synthetic-file assertions passed. The executable copy retained the exact resolver/sizing code and replaced only Lombok logging with a no-op logger. This was not a full Store/container load test or a performance measurement.

Provisional assessment: 8.5/10, based on the earlier full review plus this verified repair, not a fresh full Store acceptance. Current-head CI and the pending paired documentation still need completion; the author's 126 Store tests remain author-reported rather than my own execution.

[byteayan]

@imbajin thanks for the thorough rounds on this one. Everything you raised is now addressed at 535c2d0: the ancestor and subtree cgroup limits, the multi-mount order, the upgrade step for an old application-pd.yml, and the CONTRACT_SUITES mapping. All 19 checks are green on that commit, and the paired doc PR is apache/hugegraph-doc#535, linked above with Doc - Done. Could you take a final look when you have a moment?

[imbajin]

Blocking: yes. Summary: Automatic sizing can miss the process cgroup limit in a supported cgroup-namespace mount layout and can reserve too little for uncapped direct memory. Either condition can leave the Store with a RocksDB budget above its available container memory. Evidence: exact-head source review of RocksdbMemoryBudget, the default Store launcher and gRPC configuration; see inline comments for the two cases.

[imbajin]

Updated independent cross-review: 8.0/10, pending focused handling of the namespace mapping case noted in #3355 (comment). I independently reproduced the missing-child-limit case using synthetic proc/mountinfo/cgroup files and the exact resolver source, compiled with only Lombok logging replaced by a no-op logger. Eight new bounded Java 21 assertions passed, and all 40 earlier controls still pass. An inherited mount rooted at /.. with process membership / falls back to the parent mount root and misses the limited child. If the JVM also reports host memory, auto-sizing can overestimate the budget; an accurate JVM-reported limit masks the failure. This is a conditional gap in the new protection, not a demonstrated production OOM or a claim that it regresses the previous fixed default. Please distinguish ambiguous mapping from an unlimited hierarchy and provide narrow safe handling or an explicit-budget diagnostic with a regression fixture. Normalizing /.. alone cannot recover the hidden child path.

The direct-memory finding #3355 (comment) also identifies a real accounting assumption: an independent Java 21 probe with -Xmx2g and unset MaxDirectMemorySize reported a 2 GiB allowance, while this code reserves 1 GiB. This heuristic is explicit in the source. No buffers or Store workload were allocated, so no direct-buffer workload failure or OOM was demonstrated. Counting the default heap-sized allowance is the conservative option; imposing a new runtime direct-memory cap would need separate validation.

Evidence boundary: two independent source reviews, synthetic resolver tests and a lightweight JVM allowance probe; no live namespace, JDK 17, full Store, OOM or performance acceptance. Current review source:

hugegraph/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RocksdbMemoryBudget.java

Lines 220 to 237 in 535c2d0

	}
	boolean matched = false;
	for (String[] mount : v2 ? v2Mounts : v1Mounts) {
	if (!contains(mount[0], parts[2])) {
	// A mount of another subtree says nothing about this cgroup
	continue;
	}
	matched = true;
	Path point = Paths.get(mount[1]);
	String relative = rebase(parts[2], mount[0]);
	limit = lowest(limit, v2 ? cgroupV2Limit(point, relative) :
	cgroupV1Limit(point, relative));
	}
	if (!matched) {
	String relative = rebase(parts[2], "/");
	limit = lowest(limit, v2 ? cgroupV2Limit(defaultRoot, relative) :
	cgroupV1Limit(defaultRoot.resolve("memory"), relative));
	}

. Prior 8.5 assessment is superseded by this new evidence on the same revision.

CI status at publication (2026-10-17 01:36 UTC): 19 exact-head checks passed, 2 were skipped, and 0 remain running/queued. Both Codecov checks passed. CI source: https://github.com/apache/hugegraph/actions/runs/38938886440. These reported CI results do not establish coverage of the synthetic namespace-mapping case and are separate from our bounded independent controls.

[imbajin]

Updated independent cross/adversarial review of the current revision: 8.5/10; waiting for current-head CI and a narrow launcher/documentation clarification.

The unmatched-cgroup flag, visible mount-root fallback, explicit-budget warning and regression fixtures satisfy the diagnostic alternative requested in our prior review. This clears that review hold; it does not mean the hidden child limit is recovered. An accurate JVM-reported bound still protects sizing, while an overreporting JVM plus a hidden tighter child remains a disclosed limitation. No new fail-fast requirement is being introduced.

The direct-memory accounting correction is sound: count the configured allowance, or the full heap-sized JVM allowance when unset. The previous 1 GiB heuristic is gone.

One new compatibility/documentation point: the launcher injects MaxDirectMemorySize from computed XMX before USER_OPTION. A heap-only -j override therefore leaves that generated direct cap unchanged. For example, computed 2 GiB followed by -j -Xmx512m still permits 2 GiB direct memory; raising heap alone retains the smaller computed cap. This changes the previous implicit heap-sized direct allowance, and the deployment guide's statement that raising heap without an explicit cap counts a second heap does not describe this launcher path. Tiny JVM argument-precedence probes confirm the allowance behavior; no workload failure was demonstrated.

The smallest compatibility-preserving option is to remove the newly generated direct-memory flag and keep the corrected full-heap accounting. If the fixed computed cap is intentional, document it explicitly and add lowered/raised heap-only -j regression cases alongside the explicit-cap override case. This is a narrow compatibility clarification, not evidence of a demonstrated OOM or a request for a broader resolver redesign.

Evidence: two independent source reviews, bounded Java 21 synthetic cgroup/accounting checks and lightweight JVM allowance probes without buffer allocations. These are not live namespace, JDK 17, Store workload, OOM, performance or same-RC acceptance tests. Current-head CI, checked at 2026-10-17 06:47 UTC, has 5 successful checks, 2 skipped checks and no failed checks. 11 checks remain running/queued. Older-head results are not inherited.

[byteayan]

@imbajin thanks, good catch on the heap-only -j. Fixed in 119f819: the launcher now adds -XX:MaxDirectMemorySize=<computed heap> only when -j sets neither -Xmx nor -XX:MaxDirectMemorySize. With a heap-only -j the JVM defaults the cap to that heap, and with an explicit cap the cap applies; the budget counts whichever the JVM reports. test-start-hugegraph-store.sh (already run in the store CI job) gains four cases: no -j gives the computed 1024m cap, -j -Xmx512m and -j -Xmx4g leave the cap out, and -j -XX:MaxDirectMemorySize=256m ends with 256m. Against the previous launcher both heap-only cases fail with the stale 1024m cap. The deployment guide now describes this path.

[imbajin]

Follow-up independent cross/adversarial review of the updated revision: 8.5/10; the standard heap-only -j compatibility concern from our previous review is addressed. Await current-head CI.

The launcher now omits its generated direct-memory cap when -j supplies -Xmx or an explicit direct cap; the documentation matches those ordinary invocation paths. Independent execution of the exact shipped --gc-only launcher fixture passed all 15 cases, including default, lowered heap, raised heap and explicit direct-cap cases. This uses stub Java for argument capture, not a real Store launch. Additional tiny JVM 21 probes confirmed the effective heap/direct allowance behavior without allocating buffers.

A non-blocking edge remains: the detection pattern expects a literal space before -Xmx, so tab/newline-separated options or the -XX:MaxHeapSize alias can retain the computed cap. Those cases were reproduced with lightweight argument probes; no Store workload failure was demonstrated. Optional simplification is still to omit the generated flag entirely and retain the corrected heap-sized accounting, or align detection with actual option tokenization. This does not reopen the ordinary -j fix or require a broader redesign.

Namespace diagnostic handling and direct-memory accounting are unchanged from the previous assessment, with the same disclosed evidence boundaries. Current-head CI, checked at 2026-10-17 07:00 UTC, has 11 successful checks, 2 skipped checks and no failed checks. 7 checks remain running/queued. Cancelled results from the previous revision are not treated as failures or inherited passes.

[byteayan]

@imbajin thanks, closed that edge too in 643db96. The launcher now splits -j the way it reaches the JVM (unquoted, so on spaces, tabs and newlines) and matches each option against -Xmx*, -XX:MaxHeapSize=* and -XX:MaxDirectMemorySize=*, rather than looking for a literal " -Xmx". test-start-hugegraph-store.sh --gc-only gains a tab-separated -Xmx4g case and a -XX:MaxHeapSize=4g case; all 17 pass, and against 119f819 exactly those two fail with the computed 1024m cap.

[imbajin]

Independent update for the current revision: 8.5/10 for the reviewed implementation; merge remains blocked on the current CI failure below.

The last launcher delta addresses the optional tab/newline and MaxHeapSize alias cases from our previous review. The exact shipped stub-launcher suite passes 17/17, both changed scripts pass bash -n, and lightweight JVM 21 probes confirm option precedence. These are bounded independent checks, not live Store tests.

The current HStore job fails during the second init-store invocation with Invalid value for key 'usePD': '[true, true]'. Anonymous Cypher tests passed 2/2 first. Independent attribution against the actual CI merge revision shows this is a shared test-fixture interaction exposed by merged Server PR #3366, not a demonstrated regression in this PR's RocksDB budgeting.

At the actual tested merge, run-api-test.sh starts the same directory twice, but start-server.sh appends usePD on each HStore start. The start script is identical in the PR head and base, and the two-start runner comes from the base.

A bounded independent twice-invoked exact-script fixture, with service commands stubbed, reproduced duplicate usePD, schema.sync_deletion, and the HBase timeout properties. Only the usePD Java failure is observed in CI; no Java failure is claimed for the other duplicate keys. Please make these appends idempotent, for example portable delete-before-append for each active key (escaping literal dots), and add a twice-start regression. The existing write_property helper alone neither inserts absent keys nor removes duplicates. A shared fixture fix should let current-head HStore CI reach the authenticated API phase; a blind rerun does not address the deterministic duplicate configuration.

No production OOM or runtime performance failure was observed. Current CI is separate from same-RC acceptance.

Remaining exact-head checks at publication (2026-10-17 07:38 UTC): 18 passed, 2 were skipped, and only HStore failed. No checks remain running or queued. Older-head results are not inherited.

[byteayan]

@imbajin thanks for tracing that. Master is merged in (a5942ff), and 9e6dc50 makes the start-server.sh writes idempotent. A set_property helper deletes every existing line for the key, dots escaped, before appending it once. It now handles usePD, schema.sync_deletion and both HBase timeouts. The new test-start-server.sh runs the real script twice on one fixture, with init-store and the launcher stubbed, for hstore and hbase, and checks each key appears exactly once. All 7 checks pass, and against master's start-server.sh 5 of them fail on the duplicated keys. The test runs in the Server rocksdb job's startup-contracts step and is mapped in CONTRACT_SUITES, and the selector tests cover the mapping.

[imbajin]

Independent repair review of the current revision: 8.5/10 overall; the duplicate-usePD HStore failure reported in our prior review is resolved on this revision. Current-head CI has completed successfully.

The startup fixture now replaces active usePD, schema.sync_deletion and HBase timeout assignments idempotently. Independent execution of the exact new twice-start fixture passes 7/7; running it against the old start script fails 5 checks, so it detects the original problem. All 20 CI-policy tests pass, and the new directly invoked fixture is executable and wired to the actual startup-contract step. Additional bounded duplicate/colon/lookalike/comment controls pass. These local checks use stub service commands, not a live Store. The master merge boundary was inspected separately; no unrelated imported master changes are claimed as newly reviewed.

Real current-head HStore CI now passes the formerly failing path: anonymous Cypher 2/2; second init-store completes; authenticated API suite 190 tests with 0 failures/errors and 53 skips; coverage and shutdown checks succeed. The log explicitly checks out the merge of this head into the tested base. Raft and TinkerPop phases were skipped, and this is not same-RC acceptance.

Latest workflow status at 2026-10-17 08:10 UTC: 20 checks passed, 1 were skipped, and no checks failed. The workflow completed successfully; no checks remain running or queued.

[bitflicker64]

Blocking: no. Summary: Score 9/10. At 9e6dc50 an unset or 0 rocksdb.total_memory_size is sized from the cgroup or physical limit minus heap, the direct memory the JVM actually allows and a 1.5 GiB reserve, and an explicit value of at least 1 MiB still wins. The earlier threads (derived write_buffer_size floor, parent-only cgroup limits, subtree and multi-mount rebasing, the inherited namespace mount, direct-memory accounting, the upgrade step for an old mounted application-pd.yml, CONTRACT_SUITES mapping, duplicate usePD) are handled at this head, and I found nothing new to raise. Remaining limits are disclosed in the PR description and the deployment guide: a limit set only above a private cgroup namespace stays invisible, a hidden child limit behind an inherited mount only gets a WARN, a raised heap without a direct cap counts direct memory as a second heap and can drop to the 256 MiB floor, and no full Store has been run under a container limit. Evidence: read the full exact-head diff plus RaftRocksdbOptions, RocksDBOptions, AppConfig.init(), start-hugegraph-store.sh, calc_xmx, the Store Dockerfile, the entrypoint and both Helm values files; compiled RocksdbMemoryBudget.java from this head standalone (Lombok logger swapped for a print stub) on OpenJDK 17.0.20.1 and got 536870912 for 4 GiB/1 GiB/512 MiB, 2684354560 for the 8 GiB Helm cluster preset, 5637144576 for the 16 GiB default launcher, a disruptor size of 32 for 512 MiB and the 1 MiB write-buffer floor; ran test-store-docker-entrypoint.sh (9 passed) and test-start-server.sh (7 passed) locally; every check on this head passed (riscv64 skipped), including pd_store / store, which runs CoreSuiteTest with the new RocksdbMemoryBudgetTest.

[imbajin]

Blocking: no. Summary: One portability issue remains in the startup property helper on systems using stock BSD sed. Evidence: the exact substitution returns unbalanced brackets with the local Darwin system sed; the current-head CI run has no failed checks and skips riscv64.

[imbajin]

⚠️ Important: On Darwin systems using the stock BSD sed, the / delimiter in this substitution is parsed before the bracket expression closes, so the command returns unbalanced brackets. With set -e, the unconditional set_property "$CONF" "schema.sync_deletion" call exits before server startup. Please escape / or use another delimiter and cover the BSD sed case. Evidence: the exact expression exits 1 on the local Darwin system; the current-head macOS Actions job passed, so this is specific to sed implementations with this parsing behavior.