Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,11 +130,15 @@ Detailed guides and runnable examples are structured under `docs/` and `examples
- [Text-to-SQL SFT Notebook](examples/sft/text-to-sql/texttosql_sft_notebook.ipynb)
- Reinforcement Learning:
- [Text-to-SQL RL Recipe](examples/text-to-sql/README.md)
- [Harvey LAB RL Recipe](examples/harvey_labs/README.md) (8-GPU long-context agentic RL)
- **Technical Documentation**:
- [Architecture](docs/architecture.md)
- [Multi-GPU Training and Sampling](docs/multi-gpu.md)
- [Configuration](docs/configuration.md)
- [Tinker Client Compatibility](docs/tinker-client-compatibility.md)
- **Deployment**:
- [Kubernetes Deployment Guide (GKE)](docs/setup/gke-setup.md)
- [Harvey LAB RL on GKE](docs/setup/harvey-lab-on-gke.md)

## Roadmap

Expand Down
6 changes: 6 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,12 @@ REDIS_URL=redis://127.0.0.1:6379/0 VLLM_ARCHITECTURE_OVERRIDE=Gemma4ForCausalLM
| `OPEN_RL_ACCEL_TIMESLICER_HOST` | unset | Node-local accelerator time-slicer host for Kubernetes workers. When set, the worker uses TCP instead of the Unix socket; Kubernetes sets this from `status.hostIP`. |
| `OPEN_RL_ACCEL_TIMESLICER_PORT` | `9753` | Node-local accelerator time-slicer TCP port for Kubernetes workers. |
| `OPEN_RL_LORA_JOBS_PER_RUNTIME` | unset | The most LoRA jobs one shared trainer and sampler serve. The next job on the same base model starts another runtime, so step time stops growing with the job count. The scheduler places that runtime like any LoRA runtime: it waits for GPUs up to `OPEN_RL_PLACEMENT_TIMEOUT` and is failed past it, and it may share a GPU with the first runtime unless the scheduler limits the seats per claim. `1` gives every job a runtime of its own. Unset or `0` puts every LoRA job on a base model on one runtime; the server refuses to start on any other value. |
| `OPEN_RL_AUTOMODEL_IMAGE` | `ghcr.io/openrl-dev/openrl/automodel:latest` | Worker image used when a client requests `openrl.trainer_backend=automodel`. |

For per-job multi-GPU settings (`openrl.trainer_backend`,
`openrl.trainer_device_count`, `openrl.trainer_cp`, `openrl.sampler_replicas`,
and `openrl.exclusive`) and their restrictions, see
[Multi-GPU Training and Sampling](multi-gpu.md).

For local FFT subprocess mode, start `python -m accel_timeslicer.serve` before the
workers run. The local launcher tags each worker with a time-slice job id and
Expand Down
4 changes: 4 additions & 0 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -319,6 +319,10 @@ Any Tinker SDK client works against OpenRL, so the larger examples carry over:

- [Text-to-SQL RL recipe](../examples/text-to-sql/README.md): SFT then RL with
SQL execution rewards on Gemma 4.
- [Harvey LAB RL recipe](../examples/harvey_labs/README.md): 8-GPU long-context
agentic RL with Automodel CP4 training and 4 vLLM sampler replicas.
- [Multi-GPU training and sampling](multi-gpu.md): `TINKER_TAGS` settings,
layouts, and exclusivity rules for scaling past 2 GPUs.
- [Pig Latin](../examples/sft/pig-latin/piglatin_sft_notebook.ipynb) and
[Text-to-SQL](../examples/sft/text-to-sql/texttosql_sft_notebook.ipynb) SFT
notebooks.
Expand Down
97 changes: 97 additions & 0 deletions docs/multi-gpu.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# Multi-GPU Training and Sampling

OpenRL scales across GPUs on Kubernetes with no changes to your training loop.
Set `openrl.*` tags in `TINKER_TAGS` (or `user_metadata` on `create_model`):

```bash
export TINKER_TAGS=openrl.trainer_backend=automodel,openrl.trainer_device_count=4,openrl.trainer_cp=4,openrl.sampler_replicas=4,openrl.recipe=rl
```

That single line runs a 4-GPU FSDP2 + CP4 Automodel trainer alongside 4
prefix-routed vLLM sampler replicas — the exact 8-GPU layout used by the
[Harvey LAB RL recipe](../examples/harvey_labs/README.md).

## Settings

| Setting | Values | Default | Effect |
| --- | --- | --- | --- |
| `openrl.trainer_backend` | `pytorch`, `automodel`, or image ref | `pytorch` | `automodel` runs NeMo Automodel (FSDP2 + context parallelism) for multi-GPU LoRA. |
| `openrl.trainer_device_count` | `1`–`8` | `1` | GPUs claimed on one node for a single-node `torchrun` trainer (`openrl.trainer_gpus` is still accepted as a deprecated alias). |
| `openrl.trainer_cp` | `1`–`8` | `1` | GPUs sharing one sequence (context parallelism); the remaining `trainer_device_count / trainer_cp` GPUs run data parallel. |
| `openrl.sampler_replicas` | `1`–`8` | `1` | Single-GPU `vllm serve` Workloads behind a CPU dispatcher + prefix-aware router (Envoy + llm-d). |
| `openrl.exclusive` | `true` / `false` | `false` | Dedicated trainer and sampler with no runtime sharing or GPU time-slicing. |
| `openrl.recipe` | `rl` / `sft` | unset | `rl` starts sampler replicas together with the trainer instead of waiting for the first sample call. |

## Common layouts

| Goal | `TINKER_TAGS` | GPUs |
| --- | --- | --- |
| Default LoRA (shared workers) | *(none)* | 2 (1 trainer + 1 sampler) |
| Sampling-bound RL | `openrl.sampler_replicas=4,openrl.recipe=rl` | 5 (1 trainer + 4 samplers) |
| Sharded SFT trainer (e.g. 27B on 4×L4) | `openrl.trainer_backend=automodel,openrl.trainer_device_count=4` | 4 (4 trainer) |
| Long-context RL (Harvey LAB on 8×H100 / 8×B200) | `openrl.trainer_backend=automodel,openrl.trainer_device_count=4,openrl.trainer_cp=4,openrl.sampler_replicas=4,openrl.recipe=rl` | 8 (4 CP4 trainer + 4 samplers) |

## Restrictions and exclusivity

- **Multi-GPU trainers are always exclusive.** `openrl.trainer_device_count > 1`
gives the job its own trainer and sampling pool — no sharing with other LoRA
jobs and no GPU time-slicing.
- **Automodel + LoRA only.** `openrl.trainer_device_count > 1` requires
`openrl.trainer_backend=automodel` and `openrl.fine_tuning_type=lora`. Full
fine-tuning (`openrl.fine_tuning_type=full`) is 1 GPU only.
- **Single-node trainer (2–8 GPUs).** All trainer GPUs land on one node, and
`openrl.trainer_cp` must divide `openrl.trainer_device_count`.
- **1 GPU per sampler replica.** Samplers scale horizontally (`1`–`8` replicas),
not via tensor parallelism, so the model and its KV cache must fit one GPU.
- **Sampler exclusivity:**
- **FFT:** `openrl.sampler_replicas > 1` requires `openrl.exclusive=true`.
- **LoRA (`trainer_device_count > 1`):** already exclusive; all replicas serve
only that job.
- **LoRA (`trainer_device_count=1`, non-exclusive):** jobs on the same base
model can share a multi-replica sampling pool (sized to the max
`sampler_replicas` requested); each replica still holds its GPU exclusively.
- **Kubernetes scheduler only.** `trainer_device_count > 1` and
`sampler_replicas > 1` require `OPEN_RL_WORKER_MANAGER=scheduler` (not
supported locally).
- **Automodel image.** `openrl.trainer_backend=automodel` pulls
`OPEN_RL_AUTOMODEL_IMAGE` (default
`ghcr.io/openrl-dev/openrl/automodel:latest`), separate from the default
worker image.

## Example: Harvey LAB on 8 GPUs

The [Harvey LAB RL recipe](../examples/harvey_labs/README.md) trains
`Qwen/Qwen3.5-9B` with GRPO over a 262,144-token context window (4 CP4 trainer
GPUs + 4 prefix-routed vLLM samplers), lifting the held-out criterion pass rate
from 0.54 to 0.78 in 35 steps.

1. Create an 8-GPU DRA node pool (see the 8×H100 snippet in
[GKE setup](setup/gke-setup.md)).
2. Raise the API server's context window limits before creating workers:
```bash
kubectl -n openrl-system set env deployment/open-rl-api-server \
VLLM_MAX_MODEL_LEN=262144 OPEN_RL_TRAIN_TOKEN_BUDGET=262144
```
3. Run the recipe with the 8-GPU tags:
```bash
export TINKER_API_KEY=tml-dummy-key
export TINKER_TAGS=openrl.trainer_backend=automodel,openrl.trainer_device_count=4,openrl.trainer_cp=4,openrl.sampler_replicas=4,openrl.recipe=rl
uv --project examples run harvey-train base_url=http://127.0.0.1:9003 log_path=artifacts/harvey-labs/my-run
```

For the full in-cluster deployment (driver Job, Agent Sandbox warm pool, and
judge), see [Harvey LAB RL on GKE](setup/harvey-lab-on-gke.md).

## Verifying placement and utilization

```bash
# Cluster view: multi-GPU DRA claims and sampler replicas
kubectl -n openrl-system get workloads,resourceclaims,pods -o wide

# Client view (no kubectl needed; <model_id> is printed after create_model)
curl -s http://127.0.0.1:9003/api/v1/openrl/models/<model_id> | jq .summary
```

For per-GPU assignments and DCGM utilization, use the
[inspection API](configuration.md#inspection-api) (`/api/v1/inspect/activity`
and `/api/v1/inspect/query`).
25 changes: 25 additions & 0 deletions docs/setup/gke-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,31 @@ gcloud container node-pools create open-rl-l4 \
--disk-size=200
```

<details>
<summary><b>Multi-GPU node pool (8 × H100 for Harvey LAB and large models)</b></summary>

For multi-GPU training and sampling (for example, the 4-GPU Automodel CP4 trainer
plus 4 vLLM samplers used by [Harvey LAB RL](harvey-lab-on-gke.md)), create an
8-GPU node pool with the same DRA labels and `gpu-driver-version=disabled`:

```bash
gcloud container node-pools create open-rl-h100 \
--cluster="${CLUSTER}" \
--location="${REGION}" \
--node-locations="${ZONE}" \
--machine-type=a3-highgpu-8g \
--accelerator=type=nvidia-h100-80gb,count=8,gpu-driver-version=disabled \
--node-labels=openrl.io/enabled=true,gke-no-default-nvidia-gpu-device-plugin=true,nvidia.com/gpu.present=true \
--image-type=COS_CONTAINERD \
--num-nodes=1 \
--disk-size=200
```

See [Multi-GPU Training and Sampling](../multi-gpu.md) for the `TINKER_TAGS`
settings, layouts, and exclusivity rules.

</details>

Connect `kubectl`:

```bash
Expand Down
9 changes: 5 additions & 4 deletions docs/setup/harvey-lab-on-gke.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,11 @@ If Workloads for the run already exist, delete them after updating the API
server so OpenRL recreates the worker pods with the new limits.

The Job requests its trainer and samplers through `TINKER_TAGS` with
`openrl.trainer_backend=automodel,openrl.trainer_device_count=4,openrl.trainer_cp=4,openrl.sampler_replicas=4`.
Without context parallelism, the trainer runs out of GPU memory. One 80 GB H100
fits about 80k tokens of Qwen3.5-9B, and data parallelism does not help because
each GPU still holds a full sequence.
`openrl.trainer_backend=automodel,openrl.trainer_device_count=4,openrl.trainer_cp=4,openrl.sampler_replicas=4`
(see [Multi-GPU Training and Sampling](../multi-gpu.md) for all multi-GPU
options and exclusivity rules). Without context parallelism, the trainer runs
out of GPU memory. One 80 GB H100 fits about 80k tokens of Qwen3.5-9B, and data
parallelism does not help because each GPU still holds a full sequence.

## 2. Build the driver image

Expand Down
Loading