Skip to content

feat(workers): read HF_TOKEN from an optional hf-token Secret - #464

Open
NopeCharon wants to merge 1 commit into
openrl-dev:mainfrom
NopeCharon:feat/worker-hf-token-secret
Open

NopeCharon wants to merge 1 commit into
openrl-dev:mainfrom
NopeCharon:feat/worker-hf-token-secret

Conversation

@NopeCharon

Copy link
Copy Markdown
Contributor

Covers the HF_TOKEN item of #249 ("The bundle has no HF_TOKEN hook, so gated models (Gemma) cannot load") and item 5 of TODOs.md.

Worker pods that the scheduler starts get no Hugging Face token. A job on a gated model (Gemma 3, Llama) can't download its weights on a cluster, and every worker downloads public models anonymously, under the Hub's lower rate limits. The old static worker Deployments read HF_TOKEN from an optional hf-token Secret. That reference was dropped along with those Deployments in #277, and the pod template the API server renders never picked it up.

pod_env() now gives every trainer and sampler container HF_TOKEN from the token key of the hf-token Secret in the workers' namespace, the same Secret and key the old manifests used:

- name: HF_TOKEN
  valueFrom:
    secretKeyRef: {name: hf-token, key: token, optional: true}
  • The reference is optional, so a cluster without the Secret starts its workers exactly as before.
  • The token stays in the Secret. The API server's own HF_TOKEN is never copied into the Workload, where anyone who can read Workloads would see it.
  • The one-off e2e client and vLLM eval Jobs in k8s/eval read the same Secret, because they load the same models' tokenizers and weights.

Docs: a new "Use gated models" section in docs/setup/lora-dra.md with the kubectl create secret command. The GKE guide and docs/configuration.md link to it, and TODOs.md item 5 is marked completed.

Tests: added test_workers_read_the_hugging_face_token_from_its_secret. It covers a trainer and a sampler, with HF_TOKEN set on the API server. It fails without the change. All 412 tests in make test pass, and ruff check/format are clean. I also applied the rendered pod_env() of a trainer and a sampler to a k3s v1.31 cluster. Without the Secret both pods started with HF_TOKEN unset. After kubectl create secret generic hf-token --from-literal=token=... they got the token. Both edited k8s/eval Jobs pass kubectl apply --dry-run=server.

Workers that the scheduler starts had no Hugging Face token, so a job on
a gated model (Gemma 3, Llama) could not download its weights on a
cluster, and every worker downloaded public models anonymously, under
the Hub's lower rate limits. The static worker Deployments used to read
HF_TOKEN from an optional `hf-token` Secret; that went away with them
when the bundles moved to the scheduler (openrl-dev#277), and the pod template the
API server renders never picked it up.

pod_env() now gives every trainer and sampler container
HF_TOKEN from the `token` key of the `hf-token` Secret in the workers'
namespace, the same Secret and key the old manifests used. The
reference is optional, so a cluster without the Secret starts its
workers exactly as before. The token stays in the Secret: the API
server's own HF_TOKEN is never copied into the Workload, where anyone
who can read Workloads would see it.

The one-off e2e client and vLLM eval Jobs under k8s/eval read the same
Secret, since they load the same models' tokenizers and weights.

docs/setup/lora-dra.md gets a "Use gated models" section with the
kubectl command that creates the Secret, the GKE guide and
docs/configuration.md point at it, and item 5 of TODOs.md is marked
completed.

Signed-off-by: NopeCharon <115794254+NopeCharon@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant