Repository navigation
Conversation
The Store fix for apache/hugegraph#3254 ships rocksdb.total_memory_size as 0 and sizes the budget from the memory left after the heap, direct memory and a 1.5 GiB reserve, starting from the cgroup limit in a container. The HStore page still listed the old 32000000000 value and said 0 meant the JVM max heap. Update the rocksdb.total_memory_size row with the new rule, note the minimum of 1 GB in the raft.disruptorBufferSize derivation, and add HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE to the Store env var tables in the HStore quick start and the Docker cluster guide, in English and Chinese.
…db-memory-budget-rebased
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: The new rocksdb.total_memory_size text, the raft.disruptorBufferSize rounding note and the HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE rows match apache/hugegraph#3355 at 535c2d01b. One minor gap: the paired change also puts a 1 MiB floor on the derived rocksdb.write_buffer_size, and that row still gives the old formula. Evidence: read the full diff at a516891 (4 files, en and cn) and checked it against RocksdbMemoryBudget, AppConfig, RaftRocksdbOptions, RocksDBOptions, both application-pd.yml files and the Store docker-entrypoint.sh at the #3355 head. Note that #3355 is still open, so the 0 distribution value is correct only once it merges, as the PR description says.
apache/hugegraph#3355 fills an absent or 0 rocksdb.write_buffer_size with total_memory_size / 1000, but never below the 1 MiB that RocksDBOptions accepts. Any budget under about 1 GB, including the 256 MiB auto-sizing floor, therefore gets 1048576. Say so in the EN and CN rows.
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: The rocksdb.total_memory_size, write_buffer_size, raft.disruptorBufferSize and HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE text matches apache/hugegraph#3355 at 535c2d01b, and EN and CN agree. Two gaps remain: the Docker cluster guide says an unset variable sizes the budget from the container memory limit, but the Compose files it documents set no limit, so each Store sizes from the whole Docker host or VM; and the HStore page still says its tables were checked at master commit 2f827d6e, which predates the values this PR now lists. Evidence: read the full diff at 120c614 (4 files, EN and CN) and checked it against RocksdbMemoryBudget, AppConfig, RaftRocksdbOptions, RocksDBOptions, both application-pd.yml files, the Store docker-entrypoint.sh and start-hugegraph-store.sh at the #3355 head, the docker/*.yml Compose files (no mem_limit or deploy.resources), and application-pd.yml/AppConfig at 2f827d6e; links resolve to existing pages, git diff --check is clean and the site build is green. The earlier write_buffer_size floor comment is fixed in 120c614.
The Docker guide said an unset HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE sizes the budget from the container memory limit, but the Compose files it documents set no limit, so each Store sizes from the whole Docker host. Say so, and tell readers to set a limit or pin the variable when several Stores share a host. The HStore row now says the cgroup limit applies when one is set, otherwise the host's memory. The HStore page also said its tables were checked at a master commit from before apache/hugegraph#3355; note that the three RocksDB and disruptor rows describe that PR until its merge commit replaces it.
apache/hugegraph#3355 now budgets the JVM's real direct-memory allowance: an unset -XX:MaxDirectMemorySize counts as the heap, and bin/start-hugegraph-store.sh sets the flag to the heap it picks unless -j passes one. Drop the "at most 1 GiB" from the EN and CN rows.
apache/hugegraph#3355 now leaves -XX:MaxDirectMemorySize out when -j sets -Xmx or the cap itself, so a heap-only -j gets the JVM default. Update the EN and CN rows.
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: The new rocksdb.total_memory_size, write_buffer_size, raft.disruptorBufferSize and HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE text matches apache/hugegraph#3355 and the EN and CN pages agree; one minor gap remains because #3355 changed the launcher's direct-memory rule after this head. Evidence: read the full diff at b9e2b34 (4 files, EN and CN) and checked it against RocksdbMemoryBudget, AppConfig, RaftRocksdbOptions, the Store RocksDBOptions, both application-pd.yml files, the Store docker-entrypoint.sh and start-hugegraph-store.sh at the #3355 head 9e6dc506, plus the docker/*.yml Compose files (no memory limits); the site build checks on this head pass.
Purpose of the PR
apache/hugegraph#3355 changes the Store's shipped
rocksdb.total_memory_sizeto0. With0or no value, the Store sizes the RocksDB budget as half of what is left after the JVM max heap,MaxDirectMemorySizeand a 1.5 GiB reserve, clamped to 256 MiB to 32 GB. An unsetMaxDirectMemorySizecounts as the heap size, at most 1 GiB. Inside a container it starts from the cgroup memory limit, including a limit on a parent cgroup the Store can see. The Docker entrypoint also gainsHG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE. The site still documented the old32000000000value and said that0meant the JVM max heap.Changes, in English and Chinese:
quickstart/hugegraph/hugegraph-hstore.md:rocksdb.total_memory_sizerow: the new distribution value0, the auto-sizing rule, the off-heap caches plus the separate 1 GiB Raft log cache, the 1 MiB minimum for an explicit value, and plain bytes only.application-pd.ymlkept from an earlier release still says32000000000and has to be set to0to auto-size.raft.disruptorBufferSizerow, which the code fix now guarantees.guides/hugegraph-docker-cluster.md: added the variable to the Store table, with one line pointing to the configuration reference.bash dist/validate-links.shpasses andgit diff --checkis clean. I did not build the site locally because Hugo is not installed on this machine.