You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Right now debugging a slow or stuck job means jumping between Redis, kubectl, Prometheus, and Cloud Logging by hand. We want to put a read-only inspection API in front of all of that on the API server, with a lightweight dashboard and agent-friendly summary endpoints on top.
Design
Where the data comes from
Redis: OpenRL's own state for each model (metadata, live sessions, per-step op records, worker heartbeats, and placements).
Kubernetes: pods, nodes, Workloads, and the GPUs in each node's ResourceSlices.
DCGM + Prometheus: GPU activity from a DCGM exporter running on every GPU node, stored alongside vLLM metrics in Prometheus (Managed Prometheus on GKE).
Cloud Logging: worker logs, labeled by model.
What we want to expose
Everything lives as read-only routes under /api/v1/inspect on the API server:
Sessions: live sessions and the models created under each one.
Models: lifecycle state (active, waiting, placed, failed, ended), config/settings, and step count.
Steps: the ops in each step with their timing and metrics, plus any client-reported metrics.
Workers & placements: which pod and GPU served a model, and when.
GPUs & nodes: cluster capacity, which GPUs are currently held, and how busy each one is.
Logs: worker and client logs for a model.
Summaries (for agents): a compact per-model and per-session rollup covering pace, failures, lost or stuck requests, and where to pull logs.
Open question: the API already says run today, while the server codebase calls it a model. We should pick one name and stick to it before landing the rest of these routes.
2. Deploy setup: split out separate service accounts for the API server and scheduler, add a DCGM exporter to every GPU bundle, and write up the IAM setup docs.
3. Cluster view and logs: wire the Kubernetes data into /inventory and add /logs.
4. Placements: record placements from Workload status.
5. GPU metrics and activity: add /query, /activity, and per-GPU duty.
6. Recorder and sampler telemetry: buffered writes, worker heartbeats, and per-batch sampler records. Gated on no regression in sampler throughput.
7. Trainer telemetry: op records, queue wait time, step counter, and /steps. Gated on trainer step time.
8. Agent summaries: model and session summary endpoints.
9. UI: ship the whole dashboard in one PR (static assets, no build step).
Design update from review of #392, plus two open items.
Logs are opt-in and go through a log store. The API server reads logs only when OPEN_RL_LOG_STORE is set. cloud-logging is the first store, behind a small adapter registry. Worker logs are found by namespace and pod name, which any log shipper records, so the store needs no OpenRL-specific setup.
Client logs come from the same store. The client sets the session tag openrl.client_log_id=<id> and ships its logs labelled openrl_client=<id>, from a pod or from anywhere else. The server no longer reads client files from the shared volume.
Open items
A log store adapter outside GCP, such as Loki. Until one exists, clusters without Cloud Logging cannot read logs through OpenRL. It is one adapter file and one registry entry.
Decide where client metrics come from. The server reads no client metrics now, so the dashboard has no client metric charts. Options: read them from a location the client declares, or have clients push them to the server.
Status: phase 2 is done (#391). Phase 3 is #392 and phase 4 is #393.
Activity
ShubyM commentedon Oct 29, 2026
Design update from review of #392, plus two open items.
Logs are opt-in and go through a log store. The API server reads logs only when
OPEN_RL_LOG_STOREis set.cloud-loggingis the first store, behind a small adapter registry. Worker logs are found by namespace and pod name, which any log shipper records, so the store needs no OpenRL-specific setup.Client logs come from the same store. The client sets the session tag
openrl.client_log_id=<id>and ships its logs labelledopenrl_client=<id>, from a pod or from anywhere else. The server no longer reads client files from the shared volume.Open items
Status: phase 2 is done (#391). Phase 3 is #392 and phase 4 is #393.