Add bucket+mount transport for Jobs script upload

[davanstrien]

Use bucket transport for Jobs script upload

When hf jobs uv run is given local files (scripts, configs, etc.), we now upload them to a {namespace}/jobs-artifacts bucket and mount it into the job container at /data instead of base64-encoding them into an environment variable.

This replaces the old bash -c + xargs + base64 -d pipeline which was fragile and required manual shell quoting. The bucket approach is simpler, easier to debug, and importantly lets jobs write output artifacts back to /data/ since the mount is read-write.

How it works

• Local files are uploaded to {namespace}/jobs-artifacts under a {YYYYMMDDTHHMMSS}-{6 hex digits} subfolder (makes it easy to find artifacts for a specific job by timestamp)
• A Volume scoped to that subfolder is mounted at /data, so the job container sees its files directly at the mount root
• The uv run command references /data/<filename> — no more shell wrappers
• Failures propagate directly (no silent fallback)
• If a user already has a volume mounted at /data, we raise a clear ValueError

What changed vs the original base64 approach

The base64 transport path is fully removed — there's no opt-in flag, no fallback. Bucket transport is the only mode now. This was the simplest thing to do since base64 was experimental/hacky and maintaining two transport modes wasn't worth it.

Constants

Two new constants in constants.py:

• HF_JOBS_ARTIFACTS_BUCKET_NAME = "jobs-artifacts"
• HF_JOBS_ARTIFACTS_MOUNT_PATH = "/data"

Files changed

• src/huggingface_hub/constants.py — new bucket/mount constants
• src/huggingface_hub/hf_api.py — _create_uv_command_env_and_secrets rewritten to use bucket transport, new _upload_scripts_to_bucket helper
• tests/test_cli.py — new TestBucketTransport class replacing the old shell-quoting tests (happy path, failure propagation, mount collision, multi-file upload)

---

Note

Medium Risk
Changes the transport mechanism for local files in Jobs from env-var base64 injection to Hub bucket uploads, which affects job execution and storage semantics. Risk is mainly around bucket permissions/mount conflicts and ensuring volumes are composed correctly for existing users.

Overview
Jobs uv run local file transport is rewritten to use a Hub bucket mount instead of base64-in-env. When run_uv_job/create_scheduled_uv_job is given local scripts/configs, files are uploaded to a per-job subfolder in {namespace}/jobs-artifacts and mounted read-write at /data, and the generated command now runs as plain uv run referencing /data/... paths.

Adds HF_JOBS_ARTIFACTS_BUCKET_NAME and HF_JOBS_ARTIFACTS_MOUNT_PATH constants, introduces _upload_scripts_to_bucket, and propagates an extra_volumes return value so callers automatically append the artifacts Volume. The old bash -c + xargs + base64 transport is removed, failures now propagate directly, and the code explicitly errors if a user-provided volume already uses the reserved /data mount; tests are updated to cover the new bucket transport behavior and edge cases.

^{Reviewed by Cursor Bugbot for commit 44af196. Bugbot is set up for automated code reviews on this repo. Configure here.}

[davanstrien]

cc @Wauplin @lhoestq

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[lhoestq]

Awesome :)

[Wauplin]

Thanks for working on that @davanstrien ! Pretty excited by what it will unlock so let's double-down on this direction 😃

[Wauplin]

(converted to draft, @davanstrien could you reset to "ready for review" when ready? -just trying to filter my notifications here 😄 )

[davanstrien]

(converted to draft, @davanstrien could you reset to "ready for review" when ready? -just trying to filter my notifications here 😄 )

yes sorry for the noise!

[davanstrien]

Thanks both! Pushed follow-up commits working through the feedback. TL;DR: took @Wauplin's stronger suggestion and dropped the base64 path entirely, which enabled some other simplifications.

@lhoestq — the scripts/ rename landed earlier in 93e00708, and your "default to True" suggestion got subsumed by the base64 removal below (the flag is gone entirely).

@Wauplin:

• import uuid at top level — 5808552e. Then actually removed from the top-level imports in cf7e0193 once uuid was replaced (see next item).

• Move bucket constants to constants.py — 1a78fe92. Added HF_JOBS_ARTIFACTS_BUCKET_NAME and HF_JOBS_ARTIFACTS_MOUNT_PATH.

• YYYYMMDDTHHMMSS-{6 hex} subfolder naming — cf7e0193. Uses datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%S') + secrets.token_hex(3).

• Mount with path=scripts_prefix — 9c85c1ea. The bucket volume is now scoped to the per-job subfolder via Volume.path, so the container sees only its own files at the mount root. _upload_scripts_to_bucket no longer returns a prefix.

• Mount at /data — 554f5f64. Bucket name stays jobs-artifacts.

• Drop base64 entirely — 91219b22. The three failure modes that previously fell back to base64 now raise:

  • /data mount collision with a user volume → ValueError
  • hf_xet missing → ImportError with install hint
  • bucket create/upload failure → propagates

  HF_JOBS_USE_BUCKET_TRANSPORT is gone; ~50 lines of shell-quoting + xargs plumbing and 2 tests removed.

• Read-write mount — df347c72. Already RW by default, so this just makes the intent explicit at the call site (read_only=False), documents it in the docstring, and pins the invariant with a test assertion. Double-checked on the Hub side that omitted readOnly and explicit-false normalise to the same downstream payload — zero wire-level change.

Smoke-tested end-to-end against production. Ran hf jobs uv run smoke.py where the script prints its argv, lists /data, and writes /data/smoke_output.txt.

Job logs:

sys.argv = ['/data/hf_jobs_smoke.py']
/data contents: ['hf_jobs_smoke.py']    # only its own file
Wrote /data/smoke_output.txt - size 19 bytes

Bucket state after:

bucket://davanstrien/jobs-artifacts/scripts/20260413T133239-8bd453/
├── hf_jobs_smoke.py     ← uploaded by client
└── smoke_output.txt     ← written by job at runtime

[Wauplin]

Thanks! Some minor comments but logic looks good to me. Haven't checked the tests much yet

[lhoestq]

lgtm !

[Wauplin]

Approved once comments are addressed 😃

Tested it locally on a dummy script and it worked fine https://huggingface.co/buckets/Wauplin/jobs-artifacts/tree/20260415T092902-5ca267

I also took the liberty to update the PR description since the PR scope changed a bit since you created it

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 0fb46cc. Configure here.}