Repository navigation
feat(workers): read HF_TOKEN from an optional hf-token Secret - #464
Open
NopeCharon wants to merge 1 commit into
Open
NopeCharon wants to merge 1 commit into
NopeCharon wants to merge 1 commit into
Conversation
Workers that the scheduler starts had no Hugging Face token, so a job on a gated model (Gemma 3, Llama) could not download its weights on a cluster, and every worker downloaded public models anonymously, under the Hub's lower rate limits. The static worker Deployments used to read HF_TOKEN from an optional `hf-token` Secret; that went away with them when the bundles moved to the scheduler (openrl-dev#277), and the pod template the API server renders never picked it up. pod_env() now gives every trainer and sampler container HF_TOKEN from the `token` key of the `hf-token` Secret in the workers' namespace, the same Secret and key the old manifests used. The reference is optional, so a cluster without the Secret starts its workers exactly as before. The token stays in the Secret: the API server's own HF_TOKEN is never copied into the Workload, where anyone who can read Workloads would see it. The one-off e2e client and vLLM eval Jobs under k8s/eval read the same Secret, since they load the same models' tokenizers and weights. docs/setup/lora-dra.md gets a "Use gated models" section with the kubectl command that creates the Secret, the GKE guide and docs/configuration.md point at it, and item 5 of TODOs.md is marked completed. Signed-off-by: NopeCharon <115794254+NopeCharon@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Covers the HF_TOKEN item of #249 ("The bundle has no HF_TOKEN hook, so gated models (Gemma) cannot load") and item 5 of
TODOs.md.Worker pods that the scheduler starts get no Hugging Face token. A job on a gated model (Gemma 3, Llama) can't download its weights on a cluster, and every worker downloads public models anonymously, under the Hub's lower rate limits. The old static worker Deployments read
HF_TOKENfrom an optionalhf-tokenSecret. That reference was dropped along with those Deployments in #277, and the pod template the API server renders never picked it up.pod_env()now gives every trainer and sampler containerHF_TOKENfrom thetokenkey of thehf-tokenSecret in the workers' namespace, the same Secret and key the old manifests used:HF_TOKENis never copied into the Workload, where anyone who can read Workloads would see it.k8s/evalread the same Secret, because they load the same models' tokenizers and weights.Docs: a new "Use gated models" section in
docs/setup/lora-dra.mdwith thekubectl create secretcommand. The GKE guide anddocs/configuration.mdlink to it, andTODOs.mditem 5 is marked completed.Tests: added
test_workers_read_the_hugging_face_token_from_its_secret. It covers a trainer and a sampler, withHF_TOKENset on the API server. It fails without the change. All 412 tests inmake testpass, and ruff check/format are clean. I also applied the renderedpod_env()of a trainer and a sampler to a k3s v1.31 cluster. Without the Secret both pods started withHF_TOKENunset. Afterkubectl create secret generic hf-token --from-literal=token=...they got the token. Both editedk8s/evalJobs passkubectl apply --dry-run=server.