From 22df33c2c47dc590b39eb65dbd8c9ff07a830376 Mon Sep 17 00:00:00 2001 From: Shuby Mishra Date: Thu, 19 Nov 2026 19:42:32 +0000 Subject: [PATCH] docs: add multi-GPU guide and surface Harvey LAB showcase Signed-off-by: Shuby Mishra --- README.md | 4 ++ docs/configuration.md | 6 ++ docs/getting-started.md | 4 ++ docs/multi-gpu.md | 97 +++++++++++++++++++++++++++++++++ docs/setup/gke-setup.md | 25 +++++++++ docs/setup/harvey-lab-on-gke.md | 9 +-- 6 files changed, 141 insertions(+), 4 deletions(-) create mode 100644 docs/multi-gpu.md diff --git a/README.md b/README.md index 402e6b17..5841ba8e 100644 --- a/README.md +++ b/README.md @@ -130,11 +130,15 @@ Detailed guides and runnable examples are structured under `docs/` and `examples - [Text-to-SQL SFT Notebook](examples/sft/text-to-sql/texttosql_sft_notebook.ipynb) - Reinforcement Learning: - [Text-to-SQL RL Recipe](examples/text-to-sql/README.md) + - [Harvey LAB RL Recipe](examples/harvey_labs/README.md) (8-GPU long-context agentic RL) - **Technical Documentation**: - [Architecture](docs/architecture.md) + - [Multi-GPU Training and Sampling](docs/multi-gpu.md) + - [Configuration](docs/configuration.md) - [Tinker Client Compatibility](docs/tinker-client-compatibility.md) - **Deployment**: - [Kubernetes Deployment Guide (GKE)](docs/setup/gke-setup.md) + - [Harvey LAB RL on GKE](docs/setup/harvey-lab-on-gke.md) ## Roadmap diff --git a/docs/configuration.md b/docs/configuration.md index e239e3d6..465fa423 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -69,6 +69,12 @@ REDIS_URL=redis://127.0.0.1:6379/0 VLLM_ARCHITECTURE_OVERRIDE=Gemma4ForCausalLM | `OPEN_RL_ACCEL_TIMESLICER_HOST` | unset | Node-local accelerator time-slicer host for Kubernetes workers. When set, the worker uses TCP instead of the Unix socket; Kubernetes sets this from `status.hostIP`. | | `OPEN_RL_ACCEL_TIMESLICER_PORT` | `9753` | Node-local accelerator time-slicer TCP port for Kubernetes workers. | | `OPEN_RL_LORA_JOBS_PER_RUNTIME` | unset | The most LoRA jobs one shared trainer and sampler serve. The next job on the same base model starts another runtime, so step time stops growing with the job count. The scheduler places that runtime like any LoRA runtime: it waits for GPUs up to `OPEN_RL_PLACEMENT_TIMEOUT` and is failed past it, and it may share a GPU with the first runtime unless the scheduler limits the seats per claim. `1` gives every job a runtime of its own. Unset or `0` puts every LoRA job on a base model on one runtime; the server refuses to start on any other value. | +| `OPEN_RL_AUTOMODEL_IMAGE` | `ghcr.io/openrl-dev/openrl/automodel:latest` | Worker image used when a client requests `openrl.trainer_backend=automodel`. | + +For per-job multi-GPU settings (`openrl.trainer_backend`, +`openrl.trainer_device_count`, `openrl.trainer_cp`, `openrl.sampler_replicas`, +and `openrl.exclusive`) and their restrictions, see +[Multi-GPU Training and Sampling](multi-gpu.md). For local FFT subprocess mode, start `python -m accel_timeslicer.serve` before the workers run. The local launcher tags each worker with a time-slice job id and diff --git a/docs/getting-started.md b/docs/getting-started.md index 08225550..691ae947 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -319,6 +319,10 @@ Any Tinker SDK client works against OpenRL, so the larger examples carry over: - [Text-to-SQL RL recipe](../examples/text-to-sql/README.md): SFT then RL with SQL execution rewards on Gemma 4. +- [Harvey LAB RL recipe](../examples/harvey_labs/README.md): 8-GPU long-context + agentic RL with Automodel CP4 training and 4 vLLM sampler replicas. +- [Multi-GPU training and sampling](multi-gpu.md): `TINKER_TAGS` settings, + layouts, and exclusivity rules for scaling past 2 GPUs. - [Pig Latin](../examples/sft/pig-latin/piglatin_sft_notebook.ipynb) and [Text-to-SQL](../examples/sft/text-to-sql/texttosql_sft_notebook.ipynb) SFT notebooks. diff --git a/docs/multi-gpu.md b/docs/multi-gpu.md new file mode 100644 index 00000000..f55bddb4 --- /dev/null +++ b/docs/multi-gpu.md @@ -0,0 +1,97 @@ +# Multi-GPU Training and Sampling + +OpenRL scales across GPUs on Kubernetes with no changes to your training loop. +Set `openrl.*` tags in `TINKER_TAGS` (or `user_metadata` on `create_model`): + +```bash +export TINKER_TAGS=openrl.trainer_backend=automodel,openrl.trainer_device_count=4,openrl.trainer_cp=4,openrl.sampler_replicas=4,openrl.recipe=rl +``` + +That single line runs a 4-GPU FSDP2 + CP4 Automodel trainer alongside 4 +prefix-routed vLLM sampler replicas — the exact 8-GPU layout used by the +[Harvey LAB RL recipe](../examples/harvey_labs/README.md). + +## Settings + +| Setting | Values | Default | Effect | +| --- | --- | --- | --- | +| `openrl.trainer_backend` | `pytorch`, `automodel`, or image ref | `pytorch` | `automodel` runs NeMo Automodel (FSDP2 + context parallelism) for multi-GPU LoRA. | +| `openrl.trainer_device_count` | `1`–`8` | `1` | GPUs claimed on one node for a single-node `torchrun` trainer (`openrl.trainer_gpus` is still accepted as a deprecated alias). | +| `openrl.trainer_cp` | `1`–`8` | `1` | GPUs sharing one sequence (context parallelism); the remaining `trainer_device_count / trainer_cp` GPUs run data parallel. | +| `openrl.sampler_replicas` | `1`–`8` | `1` | Single-GPU `vllm serve` Workloads behind a CPU dispatcher + prefix-aware router (Envoy + llm-d). | +| `openrl.exclusive` | `true` / `false` | `false` | Dedicated trainer and sampler with no runtime sharing or GPU time-slicing. | +| `openrl.recipe` | `rl` / `sft` | unset | `rl` starts sampler replicas together with the trainer instead of waiting for the first sample call. | + +## Common layouts + +| Goal | `TINKER_TAGS` | GPUs | +| --- | --- | --- | +| Default LoRA (shared workers) | *(none)* | 2 (1 trainer + 1 sampler) | +| Sampling-bound RL | `openrl.sampler_replicas=4,openrl.recipe=rl` | 5 (1 trainer + 4 samplers) | +| Sharded SFT trainer (e.g. 27B on 4×L4) | `openrl.trainer_backend=automodel,openrl.trainer_device_count=4` | 4 (4 trainer) | +| Long-context RL (Harvey LAB on 8×H100 / 8×B200) | `openrl.trainer_backend=automodel,openrl.trainer_device_count=4,openrl.trainer_cp=4,openrl.sampler_replicas=4,openrl.recipe=rl` | 8 (4 CP4 trainer + 4 samplers) | + +## Restrictions and exclusivity + +- **Multi-GPU trainers are always exclusive.** `openrl.trainer_device_count > 1` + gives the job its own trainer and sampling pool — no sharing with other LoRA + jobs and no GPU time-slicing. +- **Automodel + LoRA only.** `openrl.trainer_device_count > 1` requires + `openrl.trainer_backend=automodel` and `openrl.fine_tuning_type=lora`. Full + fine-tuning (`openrl.fine_tuning_type=full`) is 1 GPU only. +- **Single-node trainer (2–8 GPUs).** All trainer GPUs land on one node, and + `openrl.trainer_cp` must divide `openrl.trainer_device_count`. +- **1 GPU per sampler replica.** Samplers scale horizontally (`1`–`8` replicas), + not via tensor parallelism, so the model and its KV cache must fit one GPU. +- **Sampler exclusivity:** + - **FFT:** `openrl.sampler_replicas > 1` requires `openrl.exclusive=true`. + - **LoRA (`trainer_device_count > 1`):** already exclusive; all replicas serve + only that job. + - **LoRA (`trainer_device_count=1`, non-exclusive):** jobs on the same base + model can share a multi-replica sampling pool (sized to the max + `sampler_replicas` requested); each replica still holds its GPU exclusively. +- **Kubernetes scheduler only.** `trainer_device_count > 1` and + `sampler_replicas > 1` require `OPEN_RL_WORKER_MANAGER=scheduler` (not + supported locally). +- **Automodel image.** `openrl.trainer_backend=automodel` pulls + `OPEN_RL_AUTOMODEL_IMAGE` (default + `ghcr.io/openrl-dev/openrl/automodel:latest`), separate from the default + worker image. + +## Example: Harvey LAB on 8 GPUs + +The [Harvey LAB RL recipe](../examples/harvey_labs/README.md) trains +`Qwen/Qwen3.5-9B` with GRPO over a 262,144-token context window (4 CP4 trainer +GPUs + 4 prefix-routed vLLM samplers), lifting the held-out criterion pass rate +from 0.54 to 0.78 in 35 steps. + +1. Create an 8-GPU DRA node pool (see the 8×H100 snippet in + [GKE setup](setup/gke-setup.md)). +2. Raise the API server's context window limits before creating workers: + ```bash + kubectl -n openrl-system set env deployment/open-rl-api-server \ + VLLM_MAX_MODEL_LEN=262144 OPEN_RL_TRAIN_TOKEN_BUDGET=262144 + ``` +3. Run the recipe with the 8-GPU tags: + ```bash + export TINKER_API_KEY=tml-dummy-key + export TINKER_TAGS=openrl.trainer_backend=automodel,openrl.trainer_device_count=4,openrl.trainer_cp=4,openrl.sampler_replicas=4,openrl.recipe=rl + uv --project examples run harvey-train base_url=http://127.0.0.1:9003 log_path=artifacts/harvey-labs/my-run + ``` + +For the full in-cluster deployment (driver Job, Agent Sandbox warm pool, and +judge), see [Harvey LAB RL on GKE](setup/harvey-lab-on-gke.md). + +## Verifying placement and utilization + +```bash +# Cluster view: multi-GPU DRA claims and sampler replicas +kubectl -n openrl-system get workloads,resourceclaims,pods -o wide + +# Client view (no kubectl needed; is printed after create_model) +curl -s http://127.0.0.1:9003/api/v1/openrl/models/ | jq .summary +``` + +For per-GPU assignments and DCGM utilization, use the +[inspection API](configuration.md#inspection-api) (`/api/v1/inspect/activity` +and `/api/v1/inspect/query`). diff --git a/docs/setup/gke-setup.md b/docs/setup/gke-setup.md index 1c574cb8..bec23722 100644 --- a/docs/setup/gke-setup.md +++ b/docs/setup/gke-setup.md @@ -85,6 +85,31 @@ gcloud container node-pools create open-rl-l4 \ --disk-size=200 ``` +
+Multi-GPU node pool (8 × H100 for Harvey LAB and large models) + +For multi-GPU training and sampling (for example, the 4-GPU Automodel CP4 trainer +plus 4 vLLM samplers used by [Harvey LAB RL](harvey-lab-on-gke.md)), create an +8-GPU node pool with the same DRA labels and `gpu-driver-version=disabled`: + +```bash +gcloud container node-pools create open-rl-h100 \ + --cluster="${CLUSTER}" \ + --location="${REGION}" \ + --node-locations="${ZONE}" \ + --machine-type=a3-highgpu-8g \ + --accelerator=type=nvidia-h100-80gb,count=8,gpu-driver-version=disabled \ + --node-labels=openrl.io/enabled=true,gke-no-default-nvidia-gpu-device-plugin=true,nvidia.com/gpu.present=true \ + --image-type=COS_CONTAINERD \ + --num-nodes=1 \ + --disk-size=200 +``` + +See [Multi-GPU Training and Sampling](../multi-gpu.md) for the `TINKER_TAGS` +settings, layouts, and exclusivity rules. + +
+ Connect `kubectl`: ```bash diff --git a/docs/setup/harvey-lab-on-gke.md b/docs/setup/harvey-lab-on-gke.md index 89636d45..ce376c61 100644 --- a/docs/setup/harvey-lab-on-gke.md +++ b/docs/setup/harvey-lab-on-gke.md @@ -43,10 +43,11 @@ If Workloads for the run already exist, delete them after updating the API server so OpenRL recreates the worker pods with the new limits. The Job requests its trainer and samplers through `TINKER_TAGS` with -`openrl.trainer_backend=automodel,openrl.trainer_device_count=4,openrl.trainer_cp=4,openrl.sampler_replicas=4`. -Without context parallelism, the trainer runs out of GPU memory. One 80 GB H100 -fits about 80k tokens of Qwen3.5-9B, and data parallelism does not help because -each GPU still holds a full sequence. +`openrl.trainer_backend=automodel,openrl.trainer_device_count=4,openrl.trainer_cp=4,openrl.sampler_replicas=4` +(see [Multi-GPU Training and Sampling](../multi-gpu.md) for all multi-GPU +options and exclusivity rules). Without context parallelism, the trainer runs +out of GPU memory. One 80 GB H100 fits about 80k tokens of Qwen3.5-9B, and data +parallelism does not help because each GPU still holds a full sequence. ## 2. Build the driver image