diff --git a/content/cn/docs/guides/hugegraph-docker-cluster.md b/content/cn/docs/guides/hugegraph-docker-cluster.md index 15df1a8afe..49722ff265 100644 --- a/content/cn/docs/guides/hugegraph-docker-cluster.md +++ b/content/cn/docs/guides/hugegraph-docker-cluster.md @@ -293,7 +293,25 @@ curl -o /dev/null -w '%{http_code}\n' -u "admin:${HUGEGRAPH_ADMIN_PASSWORD}" \ 5. **数据在意料之外地保留了下来**:`docker compose down` 会保留命名卷。要同时删除该拓扑的数据,请使用 `docker compose down -v` -**查看运行时日志**:使用 `docker logs `(如 `docker logs hg-pd0`)可直接查看日志,无需进入容器。单机镜像 `hugegraph/hugegraph` 设置了 `STDOUT_MODE=true`,其服务日志会输出到容器 stdout。`hugegraph/server`(HStore)镜像没有设置该变量,因此对 HStore 拓扑的 Server 执行 `docker logs` 只能看到入口脚本的输出,其余内容需在容器内查看 `logs/hugegraph-server.log`。 +**查看运行时日志**:使用 `docker logs `(如 `docker logs hg-pd0`)可直接查看日志,无需进入容器。基于当前 master 构建的 PD、Store 以及两个 Server 镜像(`hugegraph/hugegraph` 和 HStore 的 +`hugegraph/server`)都设置了 `STDOUT_MODE=true`,服务日志会输出到容器 stdout。对 Server 而言,Hadoop、ZooKeeper、SOFA、Netty 和 Commons 的 WARN 及以上日志也会输出到 stdout;Hadoop、Netty 和 Commons 的 +INFO 日志以及审计日志、慢查询日志仍只写入文件,ZooKeeper 和 SOFA 不输出 WARN 以下的日志。`/hugegraph-server/logs` 中包含 `hugegraph-server.log`、JVM +崩溃日志(`hs_err_pid*.log`)和内存溢出时的堆转储(`heapdump_*/java_pid*.hprof`,每次启动一个目录,Server 及其启动的 JVM 分别写入各自的文件,除非某个 JVM 复用了同一次启动中先前的 PID;如果无法创建该目录,转储会直接写入 +`/hugegraph-server/logs`)。两个镜像都声明了 `VOLUME /hugegraph-server`,因此 `docker restart` 和 Compose 重新创建容器都会保留这些文件;`docker compose down` 会留下这个匿名卷,下一次 `up` 不会再挂载它,之后执行 +`down -v` 也不会删除它(请在 `down` 之前用 `docker inspect` 记下卷名,再用 `docker volume rm ` 删除;`docker volume prune` 会同时删除 Docker +主机上所有其他未使用的匿名卷),所以请使用具名卷或绑定挂载来保留这些文件。Kubernetes 每次重启都会新建容器,因此需要在 `/hugegraph-server/logs` 挂载 `emptyDir` 或 PersistentVolumeClaim。请为每个 Pod 使用单独的卷,或在共享 PVC 上为每个 +Pod 使用单独的目录(`subPathExpr: $(POD_NAME)`,并通过 Downward API 从 `metadata.name` 传入 `POD_NAME`;Helm Chart 两者都没有设置):崩溃文件的名字各不相同,但日志文件本身的名字是固定的。只有当存储支持多个节点同时读写挂载(通常是 +`ReadWriteMany`)时,不同节点上的 Pod 才能共享同一个 PVC;`ReadWriteOnce` 的 PVC 只能被一个节点读写挂载,否则请为每个 Pod 使用单独的 PVC。Helm Chart 目前还不会挂载日志卷。堆转储可能和 JVM 堆一样大,而且每次启动都会使用新目录,因此反复内存溢出的 +Server 会把卷写满;请按需要保留的转储数量规划卷大小,并确保不超过 `ephemeral-storage` 限制或 `emptyDir` 的 `sizeLimit`(被驱逐时 `emptyDir` 会连同转储一起删除),也不要使用 `medium: Memory`。存活探针失败后,容器最早可能在转储开始后约 +`(failureThreshold - 1) × periodSeconds + timeoutSeconds` 被重启,kubelet 最多再等待 `terminationGracePeriodSeconds` +就会终止它;请让这段时间长于写入一次堆转储所需的时间,否则在写转储的过程中重启会留下不完整的文件。启动脚本从不删除转储或 `heapdump_*` 目录,因为 computer 任务的 JVM 可能在 Server 退出后仍在使用它;每次启动到 Java 阶段都会留下一个目录,除非发生内存溢出否则为空。在该次启动的 +HugeGraph JVM 都已退出后可以删除旧目录,但不要动正在运行的 Server 最新的那个目录。要对所有 JVM 关闭堆转储,请在容器的环境变量中设置 `JAVA_TOOL_OPTIONS=-XX:-HeapDumpOnOutOfMemoryError`(可以使用 `docker run -e`,或在 +Compose 中 Server 服务的 `environment:` 里添加,自带的 Compose 文件没有设置它,或在 Kubernetes 中使用 `env`;只在宿主机 shell 中导出不会生效),这样会保留镜像默认的 `JAVA_OPTS`;把该参数加到 `JAVA_OPTS` 中只对 Server +自身的 JVM 生效,而且会替换镜像默认值(`-XX:+UseContainerSupport -XX:MaxRAMPercentage=50 ...`),需要把这些参数一并写上。 + +> **版本范围**:上文描述的是基于当前 master 构建的镜像(PD、Store 和单机 Server 见 [apache/hugegraph#2980](https://github.com/apache/hugegraph/pull/2980),HStore Server 见 +> [apache/hugegraph#3258](https://github.com/apache/hugegraph/pull/3258))。1.7.0 及更早的镜像都没有设置 `STDOUT_MODE`,包括上文使用的单机镜像 `hugegraph/hugegraph:1.7.0`:`docker logs` +> 只能看到入口脚本的输出,服务日志仍在容器内的 `logs/` 中(Server 为 `logs/hugegraph-server.log`)。 ## 容器监控与健康检查 diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-server.md b/content/cn/docs/quickstart/hugegraph/hugegraph-server.md index 481d2108c5..a8575f63d1 100644 --- a/content/cn/docs/quickstart/hugegraph/hugegraph-server.md +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-server.md @@ -536,6 +536,19 @@ Connecting to HugeGraphServer (http://127.0.0.1:8080/graphs)......OK `bin/stop-hugegraph.sh` 支持 `-m true|false`(默认 `true`),用于控制停止服务时是否同时移除 crontab 监控任务。 +`bin/hugegraph-server.sh` 会把 `-XX:+HeapDumpOnOutOfMemoryError`、`-XX:HeapDumpPath` 和 `-XX:ErrorFile` 放在 `JAVA_TOOL_OPTIONS` 的最前面。因此无论是否设置了 +`JAVA_OPTIONS`(下文所述的例外情况除外),内存溢出时的堆转储(`heapdump_<主机名>_<启动时间>/java_pid.hprof`)和 JVM 崩溃日志(`hs_err_pid_<主机名>_<启动时间>.log`)都会写入 `logs/`,其中 `<启动时间>` +为本地时间,格式是 `YYYYMMDD-HHMMSS`。每次启动都有自己的 `heapdump_*` 目录,因此服务及其启动的 JVM(例如 computer 任务)会分别写入各自的转储(HotSpot 按 JVM 的 PID 命名转储文件和崩溃日志,因此同一次启动中复用了先前 PID 的 JVM +无法写入它的转储,它的崩溃日志也会覆盖先前那个 JVM 的)。主机名可避免共用同一日志目录的多个服务选到相同的名字;如果同一主机在同一秒内已经用过该名字,会再加上计数(`<启动时间>-1`、`-2`……)。如果无法创建该目录(例如磁盘已满),服务仍会启动,堆转储改为尽力写入 `logs/` +本身;磁盘已满时可能也没有空间写入转储。堆转储可能和堆一样大。启动脚本从不删除转储或 `heapdump_*` 目录:每次启动到 Java 阶段都会留下一个目录,除非发生内存溢出否则为空;启动脚本保留它,是因为 computer 任务的 JVM 可能在服务退出后仍在使用它。在该次启动的 HugeGraph JVM +都已退出后,可以移走或删除旧目录,但不要动正在运行的服务最新的那个目录;没有它,下一次转储会被写成一个同名文件。JVM 会在这些默认值之后读取你自己的 `JAVA_TOOL_OPTIONS`、`JDK_JAVA_OPTIONS`(包括 `@argfile`)、命令行(`JAVA_OPTIONS`;未设置或为空时为 +`-j`)和 `_JAVA_OPTIONS`,因此在其中任意一处为上述参数指定的值都会覆盖默认值。只有环境变量会传给服务启动的 JVM:要对所有 JVM 关闭堆转储,请设置 `JAVA_TOOL_OPTIONS=-XX:-HeapDumpOnOutOfMemoryError`;通过 `-j` 或 +`JAVA_OPTIONS` 关闭只对服务自身的 JVM 生效。无论哪种方式,每次启动到 Java 阶段仍会创建空的 `heapdump_*` 目录,除非创建失败而改为写入 `logs/`。如果日志路径中包含双引号或 `%`,启动脚本会给出警告,堆转储直接写入 +`logs/`,不使用每次启动的目录,也不设置崩溃日志的默认值;通过 `JAVA_TOOL_OPTIONS` 关闭堆转储的方式仍然有效。同时包含两种引号的路径则完全不设置堆转储默认值。由于设置了这些默认值,JVM 启动时会在 stderr 输出 `Picked up JAVA_TOOL_OPTIONS: ...`。 + +> **版本范围**:以上行为适用于当前 master([apache/hugegraph#3258](https://github.com/apache/hugegraph/pull/3258))。1.7.0 只在未设置 `JAVA_OPTIONS` 时添加堆转储参数,堆转储直接写入 +> `logs/java_pid.hprof`,JVM 崩溃日志写入安装目录。 + ### 5.2 使用 Docker 在 [3.1 使用 Docker 容器](#31-使用-docker-容器-便于测试) 中,我们已经介绍了如何使用 `docker` 部署 Server 服务。还可以通过切换后端存储或设置参数,在 Server 启动时加载样例图。 diff --git a/content/en/docs/guides/hugegraph-docker-cluster.md b/content/en/docs/guides/hugegraph-docker-cluster.md index 098aa946bc..0360a7c634 100644 --- a/content/en/docs/guides/hugegraph-docker-cluster.md +++ b/content/en/docs/guides/hugegraph-docker-cluster.md @@ -293,7 +293,35 @@ Standalone publishes only `8080` and `8088`; minimal HStore publishes `8620` (PD 5. **Unexpected retained data**: `docker compose down` keeps named volumes. To delete topology data too, use `docker compose down -v`. -**Runtime logs**: `docker logs ` (for example, `docker logs hg-pd0`) shows logs without entering containers. Standalone `hugegraph/hugegraph` sets `STDOUT_MODE=true` and sends service logs to stdout. The HStore `hugegraph/server` image does not set this variable: `docker logs` shows only entrypoint output; inspect `logs/hugegraph-server.log` inside the container for service logs. +**Runtime logs**: `docker logs ` (for example, `docker logs hg-pd0`) shows logs without entering containers. In images built from current +master, the PD, Store and both Server images (`hugegraph/hugegraph` and the HStore `hugegraph/server`) set `STDOUT_MODE=true` and send service logs to stdout. +For the Server, WARN and above from Hadoop, ZooKeeper, SOFA, Netty and Commons also reaches stdout; INFO from Hadoop, Netty and Commons and the audit and +slow-query logs stay in files, and ZooKeeper and SOFA log nothing below WARN. `/hugegraph-server/logs` holds `hugegraph-server.log`, JVM crash logs +(`hs_err_pid*.log`) and out-of-memory heap dumps (`heapdump_*/java_pid*.hprof`, one directory per start, so the Server and the JVMs it starts get separate files +unless a JVM reuses an earlier PID from that start; if the directory cannot be created, dumps go to `/hugegraph-server/logs` itself). Both images declare +`VOLUME /hugegraph-server`, so `docker restart` and a Compose recreate keep them; `docker compose down` leaves that anonymous volume behind unattached, and a +later `down -v` does not remove it (note the volume name from `docker inspect` before `down` and delete it with `docker volume rm `; `docker volume prune` +would also delete every other unused anonymous volume on the Docker host), so use a named volume or bind mount to keep them reachable. Kubernetes starts a new +container on every restart, so mount an `emptyDir` or a PersistentVolumeClaim at `/hugegraph-server/logs`. Give each pod its own volume, or a per-pod directory +on a shared PVC (`subPathExpr: $(POD_NAME)`, with `POD_NAME` passed in from `metadata.name` through the Downward API; the Helm chart sets neither): crash files +get distinct names, but the log files themselves have fixed names. Pods on different nodes can share one PVC only if its storage supports read-write mounts from +several nodes (typically `ReadWriteMany`); a `ReadWriteOnce` claim is writable from one node only, so otherwise give each pod its own PVC. The Helm chart does +not mount a logs volume yet. A heap dump can be as large as the JVM heap, and every start uses a new directory, so a Server that keeps running out of memory +fills the volume; size it for the dumps you want to keep, keep it within any `ephemeral-storage` limit or `emptyDir` `sizeLimit` (eviction deletes the +`emptyDir` with the dump), and do not use `medium: Memory`. A failing liveness probe can restart the container as soon as about +`(failureThreshold - 1) × periodSeconds + timeoutSeconds` after a dump starts, and the kubelet kills it at most `terminationGracePeriodSeconds` after that; make +that time longer than writing a heap dump takes, or a restart in the middle of a dump leaves a truncated file. The launcher never deletes dumps or `heapdump_*` +directories, because a computer-job JVM can go on using one after its Server exits; each start that reaches Java leaves one, empty unless something ran out of +memory. Delete old ones once no HugeGraph JVM from that launch is running, never the newest one of a running Server. To turn dumps off for every JVM, set +`JAVA_TOOL_OPTIONS=-XX:-HeapDumpOnOutOfMemoryError` in the container's environment (`docker run -e`, an `environment:` entry on the Server service in Compose, +which the shipped Compose files do not set, or `env` in Kubernetes; exporting it in the host shell has no effect), which keeps the image's default `JAVA_OPTS`; +putting the flag in `JAVA_OPTS` affects the Server JVM only and replaces that default (`-XX:+UseContainerSupport -XX:MaxRAMPercentage=50 ...`), so repeat those +flags. + +> **Version scope**: the paragraph above describes current master images ([apache/hugegraph#2980](https://github.com/apache/hugegraph/pull/2980) for PD, Store +> and the standalone Server, [apache/hugegraph#3258](https://github.com/apache/hugegraph/pull/3258) for the HStore Server). No 1.7.0 or older image sets +> `STDOUT_MODE`, including the standalone `hugegraph/hugegraph:1.7.0` used above: `docker logs` shows only entrypoint output, and service logs stay in `logs/` +> inside the container (`logs/hugegraph-server.log` for the Server). ## Container Monitoring and Health Checks diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-server.md b/content/en/docs/quickstart/hugegraph/hugegraph-server.md index fc62026772..06b9ce471b 100644 --- a/content/en/docs/quickstart/hugegraph/hugegraph-server.md +++ b/content/en/docs/quickstart/hugegraph/hugegraph-server.md @@ -535,6 +535,27 @@ This indicates the successful creation of the sample graph. `bin/stop-hugegraph.sh` accepts `-m true|false` (default `true`), which controls whether the cron monitor task is removed along with the service. +`bin/hugegraph-server.sh` puts `-XX:+HeapDumpOnOutOfMemoryError`, `-XX:HeapDumpPath` and `-XX:ErrorFile` first in `JAVA_TOOL_OPTIONS`, so, whether or not +`JAVA_OPTIONS` is set and with the exceptions below, an out-of-memory heap dump (`heapdump__/java_pid.hprof`) and a JVM crash log +(`hs_err_pid__.log`) land in `logs/`, where `` is the local time as `YYYYMMDD-HHMMSS`. Each start gets its own `heapdump_*` +directory, so the server and the JVMs it starts, such as computer jobs, write separate dumps (HotSpot names each dump and crash log after the JVM's PID, so a +JVM that reuses an earlier PID from the same start cannot write its dump, and its crash log replaces the earlier one's). The host name keeps servers that share +one log directory from picking the same name, and a counter (`-1`, `-2`, ...) is added when the same host already used that second. If the +directory cannot be created, for example on a full disk, the server still starts and dumps go to `logs/` itself, as a best effort: a full disk may have no room +for the dump either. A heap dump can be as large as the heap. The launcher never deletes dumps or `heapdump_*` directories: each start that reaches Java leaves +one, empty unless something ran out of memory, and the launcher keeps it because a computer-job JVM can go on using it after the server exits. Move or delete +old ones once no HugeGraph JVM from that launch is running, but never the newest one of a running server; without it, the next dump is written as a file with +that name. The JVM reads your own `JAVA_TOOL_OPTIONS`, `JDK_JAVA_OPTIONS` (including `@argfiles`), the command line (`JAVA_OPTIONS`, or `-j` when `JAVA_OPTIONS` +is unset or empty) and `_JAVA_OPTIONS` after these defaults, so a value for one of these flags in any of them overrides the default. Only the environment +variables reach the JVMs the server starts: to turn dumps off for all of them, set `JAVA_TOOL_OPTIONS=-XX:-HeapDumpOnOutOfMemoryError`, while `-j` or +`JAVA_OPTIONS` turns them off for the server JVM only. Either way each start that reaches Java still creates its empty `heapdump_*` directory, unless creating +it failed and dumps fell back to `logs/`. If the logs path contains a double quote or `%`, the launcher warns and heap dumps go to `logs/` itself, with no +per-launch directory and no crash log default; the `JAVA_TOOL_OPTIONS` opt-out still applies. A path with both quote characters gets no heap dump default at +all. Because of the defaults, the JVM prints `Picked up JAVA_TOOL_OPTIONS: ...` on stderr at startup. + +> **Version scope**: this applies to current master ([apache/hugegraph#3258](https://github.com/apache/hugegraph/pull/3258)). In 1.7.0 the heap dump flags are +> added only when `JAVA_OPTIONS` is unset, heap dumps go directly to `logs/java_pid.hprof`, and JVM crash logs go to the install directory. + ### 5.2 Use Docker to startup In [3.1 Use Docker container](#31-use-docker-container-convenient-for-testdev), we introduced how to deploy `hugegraph-server` with Docker. You can also switch storage backends or preload a sample graph by setting the corresponding parameters.