diff --git a/content/cn/docs/guides/hugegraph-docker-cluster.md b/content/cn/docs/guides/hugegraph-docker-cluster.md index 190868908..f6dda9d09 100644 --- a/content/cn/docs/guides/hugegraph-docker-cluster.md +++ b/content/cn/docs/guides/hugegraph-docker-cluster.md @@ -196,10 +196,17 @@ curl -o /dev/null -w '%{http_code}\n' -u "admin:${HUGEGRAPH_ADMIN_PASSWORD}" \ | `HG_STORE_GRPC_PORT` | 否 | `8500` | `grpc.port` | | `HG_STORE_REST_PORT` | 否 | `8520` | `server.port` | | `HG_STORE_DATA_PATH` | 否 | `/hugegraph-store/storage` | `app.data-path` | +| `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` | 否 | 无 | `rocksdb.total_memory_size`,单位字节 | {.td-table--wide} > **已弃用的别名**:`PD_ADDRESS` → `HG_STORE_PD_ADDRESS`、`GRPC_HOST` → `HG_STORE_GRPC_HOST`、`RAFT_ADDRESS` → `HG_STORE_RAFT_ADDRESS` +RocksDB 缓存位于 Java 堆外。不设置 `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` 时,将按容器内存限制自动计算;若容器未设置限制,则按宿主机(或 Docker Desktop 虚拟机)内存计算。 +上面的 Compose 文件默认未设置内存限制,每个 Store 均会按整台宿主机估算预算;多 Store 共用单机时,建议为服务配置内存限制,或将该变量显式设为字节数。 +需要注意,容器限制仅约束 RocksDB 预算:Store 启动脚本仍会按宿主机空闲内存自动分配 `-Xmx`(最多 2 GiB),并将直接内存设为相同大小。 +因此请同步通过 `JAVA_OPTS`(镜像会以 `-j` 传给启动脚本)显式约束 JVM 内存,例如容器限制为 4 GiB 时可设置 `JAVA_OPTS="-Xmx1g -XX:MaxDirectMemorySize=512m"`; +若直接使用默认值,堆内存、直接内存、1.5 GiB 预留加上 256 MiB 最低预算总和已超过 2 GiB。详见 [HStore 配置参考](/cn/docs/quickstart/hugegraph/hugegraph-hstore/)中的 `rocksdb.total_memory_size`。 + ### Server 变量 与 PD、Store 不同,Server 入口脚本没有必填变量:只有实际设置了的变量才会被写入配置文件。但分布式部署至少需要 `HG_SERVER_BACKEND` 和 `HG_SERVER_PD_PEERS`。 diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-hstore.md b/content/cn/docs/quickstart/hugegraph/hugegraph-hstore.md index b3649bbf5..a7e5e9b5f 100644 --- a/content/cn/docs/quickstart/hugegraph/hugegraph-hstore.md +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-hstore.md @@ -131,6 +131,7 @@ docker run -d \ | `HG_STORE_GRPC_PORT` | 否 | `8500` | `grpc.port` | gRPC 服务端口。 | | `HG_STORE_REST_PORT` | 否 | `8520` | `server.port` | REST API 端口。 | | `HG_STORE_DATA_PATH` | 否 | `/hugegraph-store/storage` | `app.data-path` | 数据存储路径。 | +| `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` | 否 | 无 | `rocksdb.total_memory_size` | RocksDB 内存预算,单位字节,只能是数字;其他值会使入口脚本以退出码 2 退出。未设置时使用 `application-pd.yml` 中的值或自动计算。 | {.td-table--wide} 当前主线 entrypoint 将这些变量写入 `SPRING_APPLICATION_JSON`,覆盖镜像内 `conf/application.yml`;其余配置仍从该文件和 `application-pd.yml` 读取,然后以前台方式启动 Java。旧变量 `PD_ADDRESS`、`GRPC_HOST`、`RAFT_ADDRESS` 仍可用,但会输出弃用警告。 @@ -146,7 +147,7 @@ Store 从 `conf/` 读取两个配置文件。1.7.0 发布标签的源文件见 [ #### 4.1 application.yml -以下配置参考表核对的是 server 主线提交 `2f827d6e8c9c62ae858f2fc122b3a192d015e2f4` 的发行目录值和 Java 默认注入值;1.7.0 发布包请使用其版本标签对应的配置文件,不要跨版本覆盖。 +以下配置参考表核对的是 server 主线提交 `2f827d6e8c9c62ae858f2fc122b3a192d015e2f4` 的发行目录值和 Java 默认注入值,其中 `rocksdb.total_memory_size`、`rocksdb.write_buffer_size` 和 `raft.disruptorBufferSize` 三行描述的是 apache/hugegraph#3355 的行为;1.7.0 发布包请使用其版本标签对应的配置文件,不要跨版本覆盖。 #### 4.2 application-pd.yml @@ -178,7 +179,7 @@ Store 从 `conf/` 读取两个配置文件。1.7.0 发布标签的源文件见 [ | 配置项 | 当前主线目录值 | 代码默认值 | 说明 | |--------|--------|------------|------| | `raft.address` | `127.0.0.1:8510` | 必填 | 本节点的 Raft 服务地址,格式为 `host:port`,必须能被其他 Store 节点访问。这里不需要配置 peer 列表:分区 Raft 组的成员由 PD 下发。 | -| `raft.disruptorBufferSize` | `1024` | `0` | Raft 任务队列大小。设为 `0` 时按 `rocksdb.total_memory_size` 推导:将该内存量的 GB 数取最接近的 2 的幂,再乘以 32。 | +| `raft.disruptorBufferSize` | `1024` | `0` | Raft 任务队列大小。设为 `0` 时按 `rocksdb.total_memory_size` 推导:将该内存量的 GB 数(至少为 1)取最接近的 2 的幂,再乘以 32。 | | `raft.max-log-file-size` | `600000000000` | `50000000000` | Raft 日志的最大字节数。 | | `raft.snapshotInterval` | `1800` | `300` | 生成 Raft 快照的时间间隔,单位秒。 | | `raft.snapshotLogIndexMargin` | 未设置 | `0` | 距上次快照的最小 applied index 差值,达到后才真正生成快照。设为 `0` 关闭该判断。 | @@ -207,8 +208,8 @@ Store 从 `conf/` 读取两个配置文件。1.7.0 发布标签的源文件见 [ | 配置项 | 当前主线目录值 | 代码默认值 | 说明 | |--------|--------|------------|------| -| `rocksdb.total_memory_size` | `32000000000` | `51539607552` | 本节点所有 RocksDB 实例共享的内存预算。缺失或为 `0` 时使用 JVM 最大堆内存。 | -| `rocksdb.write_buffer_size` | `32000000` | `33554432` | memtable 大小,单位字节。缺失或为 `0` 时取 `total_memory_size / 1000`。 | +| `rocksdb.total_memory_size` | `0` | 自动计算,见说明 | 本节点所有 RocksDB 实例共享的内存预算,单位字节,按 `write_buffer_ratio` 分为写缓存和 block cache。两者都在堆外,不受 `-Xmx` 限制,另有固定 1 GiB 的 Raft 日志缓存不计入其中。缺失或为 `0` 时,取物理内存减去 JVM 最大堆、`MaxDirectMemorySize`(未设置时按堆大小计算;`bin/start-hugegraph-store.sh` 默认把它设为所选的堆大小;`-j` 自行设置了 `-Xmx`、`-XX:MaxHeapSize` 或该上限时不再添加)和 1.5 GiB 预留后剩余部分的一半,并限制在 256 MiB 到 32 GB 之间;容器内设置了内存限制时以 cgroup 内存限制为准,Store 能看到的父 cgroup 上的限制也会计入;否则以宿主机内存为准。JVM 和 cgroup 文件都拿不到可用的内存大小时,改用 JVM 最大堆,同样限制在 256 MiB 到 32 GB 之间。沿用旧版本的 `application-pd.yml` 仍是 `32000000000`,需改为 `0` 才会自动计算。正数按原值使用,且必须是纯字节数,最小为 `1048576`(1 MiB);更小的值或 `4GB` 之类的值会导致启动失败。 | +| `rocksdb.write_buffer_size` | `32000000` | `33554432` | memtable 大小,单位字节。缺失或为 `0` 时取 `total_memory_size / 1000`,最小为 `1048576`(1 MiB)。 | | `rocksdb.min_write_buffer_number_to_merge` | `16` | `16` | 落盘前合并的 memtable 数量。 | | `rocksdb.write_buffer_ratio` | 未设置 | `0.66` | `total_memory_size` 中分配给写缓存的比例,其余作为 block cache。 | {.td-table--wide} @@ -325,7 +326,10 @@ master 构建接受 `g1`、`G1`、`zgc` 和 `ZGC`。 - `-d true`(默认):以后台守护进程方式运行,脚本立即返回,并把 Java 进程号写入 `bin/pid`。 - `-d false`:以前台模式运行,脚本通过 `exec` 替换为 Java 进程,容器或进程管理器的进程即为 Java 本身。在 Docker 或进程管理器(systemd、supervisord)下运行时请使用此参数,以便在崩溃时自动检测并重启服务。 -JVM 内存方面,如果没有自行设置 `JAVA_OPTIONS`:`-Xms512m`,`-Xmx` 取空闲内存的一半并限制在 512 MB 到 2048 MB 之间。脚本还会加上 `-XX:MetaspaceSize=256M`、OOM 时把堆转储写入 `logs/`,以及滚动的 GC 日志 `logs/gc.log`。生产节点通常需要大得多的堆,请显式指定,例如 `-j "-Xmx32g -Xms32g"`。 +JVM 内存方面,如果没有自行设置 `JAVA_OPTIONS`:`-Xms512m`,`-Xmx` 取空闲内存的一半并限制在 512 MB 到 2048 MB 之间。 +脚本还会加上与该堆大小相同的 `-XX:MaxDirectMemorySize`、`-XX:MetaspaceSize=256M`、OOM 时把堆转储写入 `logs/`,以及滚动的 GC 日志 `logs/gc.log`。 +生产节点通常需要大得多的堆,请显式指定,并同时设置直接内存上限,例如 `-j "-Xmx32g -Xms32g -XX:MaxDirectMemorySize=2g"`。 +不设上限时,JVM 允许的直接内存与 32 GB 的堆相同,RocksDB 预算会把它计算在内,在内存不足约 66 GB 的主机上会降到 256 MiB 的下限。 当 `ulimit -n` 或 `ulimit -u` 低于 1024 时脚本会拒绝启动;在 x86_64 和 arm64 上,如果能成功下载并校验对应的动态库,脚本会预加载 jemalloc。 diff --git a/content/en/docs/guides/hugegraph-docker-cluster.md b/content/en/docs/guides/hugegraph-docker-cluster.md index 0deb2ffe1..2eff9aeac 100644 --- a/content/en/docs/guides/hugegraph-docker-cluster.md +++ b/content/en/docs/guides/hugegraph-docker-cluster.md @@ -196,10 +196,19 @@ The following tables describe current master entrypoints and Compose files, not | `HG_STORE_GRPC_PORT` | No | `8500` | `grpc.port` | | `HG_STORE_REST_PORT` | No | `8520` | `server.port` | | `HG_STORE_DATA_PATH` | No | `/hugegraph-store/storage` | `app.data-path` | +| `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` | No | None | `rocksdb.total_memory_size`, in bytes | {.td-table--wide} > **Deprecated aliases**: `PD_ADDRESS` → `HG_STORE_PD_ADDRESS`, `GRPC_HOST` → `HG_STORE_GRPC_HOST`, `RAFT_ADDRESS` → `HG_STORE_RAFT_ADDRESS`. +RocksDB caches live outside the Java heap. Leave `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` unset to size them from the container memory limit, or from the +memory of the Docker host (such as the Docker Desktop VM) when the container has no limit. The Compose files above set no limit, so each Store sizes its budget +from the whole host; set a memory limit on Store services or set this variable to a plain number of bytes when several Stores share one host. +A container limit bounds only the RocksDB budget: the Store launcher still picks `-Xmx` from the host's free memory (up to 2 GiB) and sets direct memory to the +same size. Size those for the limit too through `JAVA_OPTS` (passed to the launcher as `-j`), for example `JAVA_OPTS="-Xmx1g -XX:MaxDirectMemorySize=512m"` for +a 4 GiB container limit; with defaults, heap plus direct memory, the 1.5 GiB reserve, and the 256 MiB minimum budget already exceed a 2 GiB limit. +See `rocksdb.total_memory_size` in the [HStore configuration reference](/docs/quickstart/hugegraph/hugegraph-hstore/). + ### Server Variables Unlike PD and Store, Server has no mandatory entrypoint variables: only set values are written to configuration. Distributed deployment nevertheless requires at least `HG_SERVER_BACKEND` and `HG_SERVER_PD_PEERS`. diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-hstore.md b/content/en/docs/quickstart/hugegraph/hugegraph-hstore.md index ad62bf4e3..2d12c7fd7 100644 --- a/content/en/docs/quickstart/hugegraph/hugegraph-hstore.md +++ b/content/en/docs/quickstart/hugegraph/hugegraph-hstore.md @@ -132,6 +132,7 @@ docker run -d \ | `HG_STORE_GRPC_PORT` | No | `8500` | `grpc.port` | gRPC service port. | | `HG_STORE_REST_PORT` | No | `8520` | `server.port` | REST API port. | | `HG_STORE_DATA_PATH` | No | `/hugegraph-store/storage` | `app.data-path` | Data path. | +| `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` | No | None | `rocksdb.total_memory_size` | RocksDB memory budget in bytes, digits only; other values stop the entrypoint with exit code 2. When unset, the value in `application-pd.yml` or auto-sizing applies. | {.td-table--wide} The master entrypoint writes these values into `SPRING_APPLICATION_JSON`, overriding image `conf/application.yml`. Other options still come from that file and `application-pd.yml`; Java runs in the foreground. Deprecated `PD_ADDRESS`, `GRPC_HOST`, and `RAFT_ADDRESS` remain supported with warnings. @@ -147,7 +148,7 @@ Store reads two files under `conf/`. Release 1.7.0 sources: [application.yml](ht #### 4.1 application.yml -The following tables were checked against distribution values and Java defaults at Server master commit `2f827d6e8c9c62ae858f2fc122b3a192d015e2f4`. Use tagged configuration for release 1.7.0 without replacing it with another version's files. +The following tables were checked against distribution values and Java defaults at Server master commit `2f827d6e8c9c62ae858f2fc122b3a192d015e2f4`, except the `rocksdb.total_memory_size`, `rocksdb.write_buffer_size` and `raft.disruptorBufferSize` rows, which describe apache/hugegraph#3355. Use tagged configuration for release 1.7.0 without replacing it with another version's files. #### 4.2 application-pd.yml @@ -180,7 +181,7 @@ This file contains only `rocksdb` and Actuator exposure settings. | Key | Master distribution value | Code default | Meaning | |-----|---------|--------------|---------| | `raft.address` | `127.0.0.1:8510` | required | Raft service address of this node, `host:port`. Must be reachable from every other Store node. There is no peer list to configure: PD tells each node which peers belong to a partition's Raft group. | -| `raft.disruptorBufferSize` | `1024` | `0` | Raft task queue size. `0` derives it from `rocksdb.total_memory_size`, by rounding that size in GB to the nearest power of two and multiplying by 32. | +| `raft.disruptorBufferSize` | `1024` | `0` | Raft task queue size. `0` derives it from `rocksdb.total_memory_size`, by rounding that size in GB (at least 1) to the nearest power of two and multiplying by 32. | | `raft.max-log-file-size` | `600000000000` | `50000000000` | Maximum byte size of Raft logs. | | `raft.snapshotInterval` | `1800` | `300` | Seconds between Raft snapshots. | | `raft.snapshotLogIndexMargin` | not set | `0` | Minimum applied-index distance since the last snapshot before a snapshot is actually written. `0` disables the distance check. | @@ -209,8 +210,8 @@ This file contains only `rocksdb` and Actuator exposure settings. | Key | Master distribution value | Code default | Meaning | |-----|---------|--------------|---------| -| `rocksdb.total_memory_size` | `32000000000` | `51539607552` | Memory budget shared by all RocksDB instances on this node. When absent or `0`, the node uses the JVM max heap instead. | -| `rocksdb.write_buffer_size` | `32000000` | `33554432` | Memtable size in bytes. When absent or `0`, the node uses `total_memory_size / 1000`. | +| `rocksdb.total_memory_size` | `0` | Auto-sized, see Meaning | Memory budget in bytes shared by all RocksDB instances on this node, split into the write cache and the block cache by `write_buffer_ratio`. Both caches are off-heap, so `-Xmx` does not limit them, and a fixed 1 GiB Raft log cache comes on top. When absent or `0`, the node uses half of the memory left after the JVM max heap, `MaxDirectMemorySize` (counted as the heap size when unset; `bin/start-hugegraph-store.sh` sets it to the heap it picks unless `-j` sets `-Xmx`, `-XX:MaxHeapSize` or the cap itself) and a 1.5 GiB reserve, clamped to 256 MiB to 32 GB. In a container the starting point is the cgroup memory limit when one is set, including a limit set only on a parent cgroup the Store can see; otherwise it is the host's memory. If neither the JVM nor the cgroup files report a usable memory size, the budget falls back to the JVM max heap, clamped to the same 256 MiB to 32 GB. An `application-pd.yml` kept from an earlier release still sets `32000000000`; set it to `0` to auto-size. A positive value is used as given and must be plain bytes, at least `1048576` (1 MiB); a smaller value or one such as `4GB` stops startup. | +| `rocksdb.write_buffer_size` | `32000000` | `33554432` | Memtable size in bytes. When absent or `0`, the node uses `total_memory_size / 1000`, at least `1048576` (1 MiB). | | `rocksdb.min_write_buffer_number_to_merge` | `16` | `16` | Number of memtables merged together before a flush. | | `rocksdb.write_buffer_ratio` | not set | `0.66` | Share of `total_memory_size` given to the write cache. The rest becomes the block cache. | {.td-table--wide} @@ -330,7 +331,11 @@ Daemon mode: - `-d true` (default): run as a background daemon. The script returns immediately and writes the Java pid to `bin/pid`. - `-d false`: run in the foreground. The script `exec`s Java, so the container or supervisor process is Java itself. Use this under Docker or a process supervisor (systemd, supervisord) so crashes are detected and the service is restarted automatically. -JVM memory, unless you set `JAVA_OPTIONS` yourself: `-Xms512m`, and `-Xmx` set to half the free memory, clamped to the 512 MB to 2048 MB range. The script also adds `-XX:MetaspaceSize=256M`, a heap dump on out-of-memory into `logs/`, and a rolling GC log at `logs/gc.log`. Production nodes normally need a much larger heap, so pass one explicitly, for example `-j "-Xmx32g -Xms32g"`. +JVM memory, unless you set `JAVA_OPTIONS` yourself: `-Xms512m`, and `-Xmx` set to half the free memory, clamped to the 512 MB to 2048 MB range. +The script also adds `-XX:MaxDirectMemorySize` equal to that heap, `-XX:MetaspaceSize=256M`, a heap dump on out-of-memory into `logs/`, and a rolling GC log at +`logs/gc.log`. Production nodes normally need a much larger heap, so pass one explicitly together with a direct-memory cap, for example +`-j "-Xmx32g -Xms32g -XX:MaxDirectMemorySize=2g"`. Without the cap the JVM allows as much direct memory as the 32 GB heap, the RocksDB budget counts it, +and on a host with less than about 66 GB it falls to the 256 MiB floor. The script refuses to start if `ulimit -n` or `ulimit -u` is below 1024, and it preloads jemalloc on x86_64 and arm64 when the shared object can be downloaded and verified.