diff --git a/.github/workflows/check-run-on-jobs.yml b/.github/workflows/check-run-on-jobs.yml new file mode 100644 index 0000000..4b646b2 --- /dev/null +++ b/.github/workflows/check-run-on-jobs.yml @@ -0,0 +1,18 @@ +name: Check run-on-jobs catalogue +# Fails when a recipe changes without rebuilding the Space, or when a declared model leaves the Hub. +on: + pull_request: + schedule: + - cron: '0 7 * * 1' + workflow_dispatch: + +jobs: + check: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: '3.12' + - name: Space files match the recipes, and declared models exist + run: python3 tools/build-catalog.py --check --verify-models diff --git a/.github/workflows/sync-run-on-jobs.yml b/.github/workflows/sync-run-on-jobs.yml new file mode 100644 index 0000000..5f3db9c --- /dev/null +++ b/.github/workflows/sync-run-on-jobs.yml @@ -0,0 +1,15 @@ +name: Sync run-on-jobs → HF +on: + push: + branches: [main] + paths: ['run-on-jobs/**', '.github/workflows/sync-run-on-jobs.yml'] + +jobs: + sync: + uses: ./.github/workflows/_sync-folder.yml + with: + folder: run-on-jobs + repo_id: uv-scripts/run-on-jobs + repo_type: space + secrets: + HF_TOKEN: ${{ secrets.HF_TOKEN }} diff --git a/run-on-jobs/AGENTS.md b/run-on-jobs/AGENTS.md new file mode 100644 index 0000000..5d73744 --- /dev/null +++ b/run-on-jobs/AGENTS.md @@ -0,0 +1,37 @@ +# Run a model on Hugging Face Jobs + +Each recipe is one script, run from its URL with `hf jobs uv run`. There is nothing to clone or install except the `hf` CLI. + +## Set up + +- Check `hf version` (needs 1.32+). If it's missing: `curl -LsSf https://hf.co/cli/install.sh | bash`, then `hf auth login`. +- Jobs need pay-as-you-go credit, and each job costs money. Confirm with the user before starting one. +- The CLI is the current reference: `hf --help`, `hf jobs uv run --help`, `hf jobs hardware`. To install the Hugging Face CLI skill, run `hf skills add`. + +## Find a recipe + +- `hf download uv-scripts/run-on-jobs recipes.json --repo-type space`, then look for the model in `recipes[].models`. +- `kind`: `run` applies a model to data; `train` fine-tunes one. +- `source`: `header` carries its launch settings; `documented` is an example from the script's docs; `flags-needed` has no command, so read the script first. +- Read a script's docstring and options: `curl -s | head -100`. Running it locally may fail on GPU-only dependencies. +- Each task's README explains its recipes, for example https://huggingface.co/datasets/uv-scripts/ocr. OCR also has `models.json`, with language coverage and support levels. +- Fuller guides: [uv-recipes](https://github.com/davanstrien/uv-scripts/tree/main/skills/uv-recipes) and, for few-shot fine-tuning, [setfit-on-jobs](https://github.com/davanstrien/uv-scripts/tree/main/skills/setfit-on-jobs). + +## Run it + +Start from the recipe's `command` field. Most recipes go from dataset to dataset: + + hf jobs uv run --max-samples 10 + +Recipes that read files use mounts. A local folder also works as a source: + + hf jobs uv run -v hf://buckets//:/input:ro -v hf://buckets//:/output /input /output + +- Preview for free with `--dry-run`, and check the flavor: without a header, a GPU recipe can resolve to `cpu-basic`. +- Flags before the URL configure the job and override the header, such as `--flavor` and `--timeout`. Flags after it go to the script. If a script flag has the same name as a job flag, put `--` before the URL. +- Start with a small `--max-samples` if the script has it, and add `--private` if the output shouldn't be public. + +## Check it + +- Use `--detach` for long runs, then `hf jobs logs -f `, `hf jobs wait ` or `hf jobs inspect `. `hf jobs cancel ` stops one. +- Look at the output rows. A completed job doesn't prove they're correct. diff --git a/run-on-jobs/README.md b/run-on-jobs/README.md new file mode 100644 index 0000000..60b42d9 --- /dev/null +++ b/run-on-jobs/README.md @@ -0,0 +1,42 @@ +--- +title: Run on Jobs +emoji: ⚡ +colorFrom: blue +colorTo: gray +sdk: static +pinned: false +short_description: Batch-run this model on your own data with HF Jobs +models: + - allenai/olmOCR-2-7B-1025-FP8 + - ATH-MaaS/OvisOCR2 + - baidu/Qianfan-OCR + - baidu/Unlimited-OCR + - baidu/Unlimited-OCR-2 + - datalab-to/lift + - datalab-to/surya-ocr-2 + - deepseek-ai/DeepSeek-OCR + - deepseek-ai/DeepSeek-OCR-2 + - facebook/sam3 + - fastino/gliner2.5-multi-v1 + - HuggingFaceTB/SmolLM3-3B + - lightonai/LightOnOCR-2-1B + - lightonai/LightOnOCR-3-4B + - NemoStation/Marlin-2B + - numind/NuExtract3 + - nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B + - PaddlePaddle/PaddleOCR-VL-1.6 + - Qwen/Qwen2-VL-7B-Instruct + - Qwen/Qwen3.6-35B-A3B + - Qwen/Qwen3.8-27B + - rednote-hilab/dots.mocr + - rednote-hilab/dots.ocr + - sentence-transformers/all-MiniLM-L6-v2 + - tencent/HunyuanOCR + - zai-org/GLM-OCR +--- + +# Run on Jobs + +A catalogue of [uv-scripts](https://huggingface.co/uv-scripts) recipes: one command runs a model over your own data on [Hugging Face Jobs](https://huggingface.co/docs/hub/jobs). + +This Space is generated from [davanstrien/uv-scripts](https://github.com/davanstrien/uv-scripts). To change it, edit the recipes or `tools/build-catalog.py` there. Agents should read [AGENTS.md](./AGENTS.md). diff --git a/run-on-jobs/assets/nasa-food-list.jpg b/run-on-jobs/assets/nasa-food-list.jpg new file mode 100644 index 0000000..6448539 Binary files /dev/null and b/run-on-jobs/assets/nasa-food-list.jpg differ diff --git a/run-on-jobs/index.html b/run-on-jobs/index.html new file mode 100644 index 0000000..2c955ea --- /dev/null +++ b/run-on-jobs/index.html @@ -0,0 +1,448 @@ + + + + + +Run on Jobs + + + + + +
+

Use Hub models on your own data

+

Run a model over your whole dataset, or fine-tune it on your own examples. Each command starts a job on Hugging Face Jobs and saves the results to the Hub. There is nothing to clone.

+
+
+

Read seven scanned NASA pages with GLM-OCR and save the text to a new public dataset on your account. Add --private to keep it private.

hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/glm-ocr.py \
+    uv-scripts/ocr-demo your-username/ocr-demo-results
The scan
Scanned page listing Space Shuttle foods
What the Job wrote
Space Shuttle Food and Beverage List
+
+Foods*
+
+Applesauce (T)
+Apricots, dried (IM)
+Asparagus (R)
+Bananas (FD)
+Beef almondine (R)
+Beef, corned (I) (T)
+Beef and gravy (T)
+Beef, ground w/pickle sauce (T)
+Beef jerky (IM)
+Beef patty (R)
+Beef, slices w/barbeque sauce (T)
+Beef steak (I) (T)
+Beef stroganoff w/noodles (R)
+Bread, seedless rye (I) (NF)
+Broccoli au gratin (R)
+Breakfast roll (I) (NF)
+Candy, Life Savers*, assorted flavors (NF)
+Cauliflower w/cheese (R)
+Cereal, bran flakes (R)
+Cereal, cornflakes (R)
+Cereal, granola (R)
+Cereal, granola w/blueberries (R)
+Cereal, granola w/raisins (R)
+Cheddar cheese spread (T)
+Chicken a la king (T)
+
+Chicken and noodles (R)
+Chicken and rice (R)
+Chili mac w/beef (R)
+Cookies, pecan (NF)
+Cookies, shortbread (NF)
+Crackers, graham (NF)
+Eggs, scrambled (R)
+Food bar, almond crunch (NF)
+Food bar, chocolate chip (NF)
+Food bar, granola (NF)
+Food bar, granola/raisin (NF)
+Food bar, peanut butter/granola (NF)
+Frankfurters (Vienna sausage) (T)
+Fruitcake (NF)
+Fruit cocktail (T)
+Green beans, french w/mushrooms (R)
+Green beans and broccoli (R)
+Ham (I) (T)
+Jam/Jelly (T)
+Macaroni and cheese (R)
+Meatballs w/barbeque sauce (T)
+Nuts, almonds (NF)
+Nuts, cashews (NF)
+Nuts, peanuts (NF)
+
+Peach ambrosia (R)
+Peaches, dried (IM)
+Peaches, (T)
+Peanut butter
+Pears (FD)
+Pears (T)
+Peas w/butter sauce (R)
+Pineapple, crushed (T)
+Pudding, butterscotch (T)
+Pudding, chocolate (R) (T)
+Pudding, lemon (T)
+Pudding, vanilla (R) (T)
+Rice pilaf (R)
+Salmon (T)
+Sausage patty (R)
+Shrimp creole (R)
+Shrimp cocktail (R)
+Soup, cream of mushroom (R)
+Spaghetti w/meatless sauce (R)
+Strawberries (R)
+Tomatoes, stewed (T)
+Tuna (T)
+Turkey and gravy (T)
+Turkey, smoked/sliced (I) (T)
+Turkey tetrazzini (R)
+Vegetables, mixed Italian (R)
+
+Beverages
+
+Apple drink
+Cocoa
+Coffee, black
+Coffee w/cream
+Coffee w/cream and sugar
+Coffee w/sugar
+Grape drink
+Grapefruit drink
+Instant breakfast, chocolate
+Instant breakfast, strawberry
+
+Instant breakfast, vanilla
+Lemonade
+Orange drink
+Orange-grapefruit drink
+Orange-pineapple drink
+Strawberry drink
+Tea
+Tea w/lemon and sugar
+Tea w/sugar
+Tropical punch
+
+Condiments
+
+Barbeque sauce
+Catsup
+Mustard
+Pepper
+Salt
+Hot pepper sauce
+Mayonnaise
+
+*Abbreviations in parentheses indicate type of food T = thermostabilized, I = irradiated, IM = intermediate moisture, FD = freeze dried, R = rehydratable, and NF = natural form.
+
+Shuttle Menu
+
+The Space Shuttle menu currently features more than 70 food items and 20 beverages. Shuttle travelers will have a varied menu every day for six days rather than the personal preference meals used in previous space missions. Each day, three meals are allowed, with a repeat of menus after six days. The pantry also provides plenty of foods for snacks and between meal beverages and for individual menu changes. The pantry also stores additional contingency food to last 96 hours.
+
+Food System Constraints
+
+The primary objective in designing a food system for the Space Shuttle is to provide food that is safe and nutritious, light in weight and compact, and is packaged in a convenient form that allows easy manipulation in the weightless environment of an orbiting spacecraft. To achieve this objective requires a careful consideration of three important factors: biological, operational, and engineering. Under each factor are many constraints that affect the final choice of food and how it is

Our run: davanstrien/ocr-demo-results. 7 pages, 2.4 minutes of processing on a10g-small, after a few minutes of setup.

+

Transcribe four minutes of a NASA podcast, with speaker tags and word timestamps, into your own Bucket.

hf buckets create your-username/transcripts --private
+hf jobs uv run --timeout 45m \
+    -v hf://buckets/uv-scripts/transcription-demo/podcast:/input:ro \
+    -v hf://buckets/your-username/transcripts/podcast:/output \
+    https://huggingface.co/datasets/uv-scripts/transcription/raw/main/nemotron-transcribe.py \
+    /input /output --timestamps
<spk:0> Jim, thank you so much for coming on the podcast today. ... <spk:3> It is, uh, the mission control room, of course, is now open for business ...

Our run: about 7.5 minutes on rtx-pro-6000, about $0.34. Most of that was model setup.

+

Embed the first 100 IMDb reviews and save them as a private dataset.

hf jobs uv run --flavor l4x1 -s HF_TOKEN \
+    https://huggingface.co/datasets/uv-scripts/embeddings/raw/main/generate-embeddings.py \
+    stanfordnlp/imdb your-username/imdb-embeddings \
+    --column text --max-samples 100 --private

One L4 GPU embeds about 900 rows per second with all-MiniLM-L6-v2.

+

Train a news topic classifier from 16 examples, with no GPU, and save the model to your account.

hf jobs uv run --flavor cpu-basic --secrets HF_TOKEN \
+    https://huggingface.co/datasets/uv-scripts/classification/raw/main/train-setfit.py \
+    fancyzhx/ag_news your-username/ag-news-setfit \
+    --num-samples 4 --max-train-pool 2000 --max-eval-samples 100 \
+    --max-seq-length 128 --sampling-strategy undersampling --max-minutes 5

Our run: 0.77 accuracy on 100 test articles, against 0.32 for always guessing the most common topic. Training took 20 seconds on cpu-basic.

+
+

You need the hf CLI (1.32 or newer), a login and pay-as-you-go credit. Add --dry-run to see the GPU and settings without starting a job.

+

Find your model

+

Replace the capitalised placeholders and your-username. Check the script if you're unsure what a value should be.

+ + + +

OCR

  • allenai/olmOCR-2-7B-1025-FP8command ready · a10g-small

    olmocr2-vllm.py command ready · a10g-small

    Convert document images to markdown using olmOCR-2 with vLLM.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/olmocr2-vllm.py \
+    INPUT_DATASET OUTPUT_DATASET
  • ATH-MaaS/OvisOCR2command ready · a10g-small

    ovis-ocr2.py command ready · a10g-small

    Convert document images to markdown using OvisOCR2 with vLLM.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/ovis-ocr2.py \
+    INPUT_DATASET OUTPUT_DATASET --batch-size 16
    1 more recipe

    ovis-ocr2-saturate.py command ready · a10g-small

    Convert document images to markdown using OvisOCR2 via saturate.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/ovis-ocr2-saturate.py \
+    <INPUT_DATASET> <OUTPUT_DATASET>
  • baidu/Qianfan-OCRcommand ready · a10g-small

    qianfan-ocr.py command ready · a10g-small

    Convert document images to markdown using Qianfan-OCR with vLLM.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/qianfan-ocr.py \
+    INPUT_DATASET OUTPUT_DATASET --max-samples 10
  • baidu/Unlimited-OCRcommand ready · a10g-small

    unlimited-ocr-vllm.py command ready · a10g-small

    Convert document images to markdown using Baidu Unlimited-OCR with vLLM.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/unlimited-ocr-vllm.py \
+    your-input-dataset your-output-dataset --max-samples 10
  • baidu/Unlimited-OCR-2command ready · rtx-pro-6000

    unlimited-ocr2.py command ready · rtx-pro-6000

    OCR whole documents (book volumes, folders of scans) with Baidu Unlimited-OCR-2.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/unlimited-ocr2.py \
+    "hf://buckets/biglam/britannica/source/pages/1771EncyclopediaBritannicaNLS/*.jp2" \
+    <user>/britannica-1771
    2 more recipes

    unlimited-ocr2-live.py command ready, experimental · rtx-pro-6000

    unlimited-ocr2.py with a live view you can watch (and screen-record) while the Job runs.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/unlimited-ocr2-live.py \
+    "hf://buckets/biglam/britannica/source/pages/1771EncyclopediaBritannicaNLS/*.jp2" \
+    <user>/britannica-1771 --live-port 7860

    unlimited-ocr2-saturate.py command ready, experimental · rtx-pro-6000

    OCR whole documents (book volumes, PDFs) with Baidu Unlimited-OCR-2 via saturate.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/unlimited-ocr2-saturate.py \
+    "hf://buckets/biglam/britannica/source/pages/1771EncyclopediaBritannicaNLS/*.jp2" \
+    <user>/britannica-1771-uocr2
  • datalab-to/liftcommand ready · a100-large

    lift-extract.py command ready · a100-large

    Extract structured JSON from document images OR multi-page PDFs using Datalab's lift model (datalab-to/lift, 9B, Qwen3.5-based).

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/lift-extract.py \
+    INPUT_DATASET OUTPUT_DATASET \
+    --schema '{"type":"object","properties":{"title":{"type":"string"}}}' \
+    --max-samples 5 --shuffle --seed 42
  • datalab-to/surya-ocr-2command ready · a10g-small

    surya-ocr.py command ready · a10g-small

    Document intelligence on images OR multi-page PDFs with Datalab's Surya OCR 2 (datalab-to/surya-ocr-2, 650M, Qwen3.5-style).

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/surya-ocr.py \
+    INPUT_DATASET OUTPUT_DATASET \
+    --max-samples 5 --shuffle --seed 42
  • deepseek-ai/DeepSeek-OCRcommand ready · a10g-small

    deepseek-ocr-vllm.py command ready · a10g-small

    Convert document images to markdown using DeepSeek-OCR with vLLM.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/deepseek-ocr-vllm.py \
+    my-dataset my-output --max-samples 10
  • deepseek-ai/DeepSeek-OCR-2command ready · a10g-small

    deepseek-ocr2-vllm.py command ready · a10g-small

    Convert document images to markdown using DeepSeek-OCR-2 with vLLM.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/deepseek-ocr2-vllm.py \
+    my-dataset my-output --max-samples 10
  • lightonai/LightOnOCR-2-1Bcommand ready · a10g-small

    lighton-ocr2.py command ready · a10g-small

    Convert document images to markdown using LightOnOCR-2 with vLLM.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/lighton-ocr2.py \
+    <INPUT_DATASET> <OUTPUT_DATASET>
    1 more recipe

    lighton-ocr2-saturate.py command ready · a10g-small

    Convert document images to markdown using LightOnOCR-2 via saturate.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/lighton-ocr2-saturate.py \
+    <INPUT_DATASET> <OUTPUT_DATASET>
  • lightonai/LightOnOCR-3-4Bcommand ready · a10g-small

    lighton-ocr3-bucket.py command ready · a10g-small

    OCR images and PDFs from an HF Bucket (or a directory) using LightOnOCR-3, writing markdown files.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/lighton-ocr3-bucket.py \
+    hf://buckets/biglam/britannica/source/pages/00070476bsb \
+    hf://buckets/your-username/ocr-output/britannica --mode grounding --max-samples 20
    1 more recipe

    lighton-ocr3.py command ready · a10g-small

    Convert document images to markdown (or layout blocks) using LightOnOCR-3 with vLLM.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/lighton-ocr3.py \
+    INPUT_DATASET OUTPUT_DATASET --max-samples 10
  • numind/NuExtract3command ready · a10g-small

    nuextract3.py command ready · a10g-small

    Convert document images to markdown OR extract structured JSON using NuExtract3 with vLLM.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/nuextract3.py \
+    INPUT_DATASET OUTPUT_DATASET --max-samples 5 --shuffle --seed 42
  • openbmb/MiniCPM-OCR-1.0command ready · a10g-large

    minicpm-ocr.py command ready, experimental · a10g-large

    Convert document images to markdown with OpenBMB's MiniCPM-OCR 1.0 pipeline.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/minicpm-ocr.py \
+    INPUT_DATASET OUTPUT_DATASET --max-samples 20
  • PaddlePaddle/PaddleOCR-VL-1.6command ready · a10g-small

    paddleocr-vl-1.6.py command ready · a10g-small

    Convert document images to text/tables/formulas using PaddleOCR-VL-1.6 with vLLM.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/paddleocr-vl-1.6.py \
+    INPUT_DATASET OUTPUT_DATASET
  • rednote-hilab/dots.mocrcommand ready · a10g-small

    dots-mocr.py command ready · a10g-small

    Convert document images to markdown using dots.mocr with vLLM.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/dots-mocr.py \
+    INPUT_DATASET OUTPUT_DATASET
  • rednote-hilab/dots.ocrcommand ready · a10g-small

    dots-ocr.py command ready · a10g-small

    Convert document images to markdown using DoTS.ocr with vLLM.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/dots-ocr.py \
+    INPUT_DATASET OUTPUT_DATASET
  • tencent/HunyuanOCRcommand ready · a10g-small

    hunyuan-ocr-1.5.py command ready · a10g-small

    Convert document images to markdown using HunyuanOCR-1.5 with vLLM.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/hunyuan-ocr-1.5.py \
+    INPUT_DATASET OUTPUT_DATASET
  • zai-org/GLM-OCRcommand ready · a10g-small

    glm-ocr.py command ready · a10g-small

    Convert document images to markdown using GLM-OCR with vLLM.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/glm-ocr.py \
+    {source_dataset} \
+    <OUTPUT_DATASET> \
+    --image-column {image_column} \
+    --batch-size {batch_size} \
+    --task {task}

Transcription

  • CohereLabs/cohere-transcribe-03-2026example command

    cohere-transcribe.py example command

    Transcribe audio files from a directory using Cohere Transcribe via transformers.

    hf jobs uv run --flavor l4x1 \
+    -s HF_TOKEN \
+    -v hf://buckets/your-username/audio-input:/input:ro \
+    -v hf://buckets/your-username/transcripts:/output \
+    https://huggingface.co/datasets/uv-scripts/transcription/raw/main/cohere-transcribe.py \
+    /input /output --language en --compile
    2 more recipes

    easytranscriber-transcribe.py example command

    Transcribe audio with word-level timestamps using kb-labb/easytranscriber.

    hf jobs uv run --flavor l4x1 -s HF_TOKEN \
+    -e UV_TORCH_BACKEND=cu128 \
+    -v hf://buckets/your-username/audio-files:/input:ro \
+    -v hf://buckets/your-username/transcripts-aligned:/output \
+    https://huggingface.co/datasets/uv-scripts/transcription/raw/main/easytranscriber-transcribe.py \
+    /input /output \
+    --language en --emit-txt --emit-srt

    cohere-transcribe-vllm.py read the script first

    Transcribe audio files from a directory using Cohere Transcribe via vLLM.

    Read the script for its options. There is no ready-made command yet.

  • distil-whisper/distil-large-v3.5example command

    easytranscriber-transcribe.py example command

    Transcribe audio with word-level timestamps using kb-labb/easytranscriber.

    hf jobs uv run --flavor l4x1 -s HF_TOKEN \
+    -e UV_TORCH_BACKEND=cu128 \
+    -v hf://buckets/your-username/audio-files:/input:ro \
+    -v hf://buckets/your-username/transcripts-aligned:/output \
+    https://huggingface.co/datasets/uv-scripts/transcription/raw/main/easytranscriber-transcribe.py \
+    /input /output \
+    --language en --emit-txt --emit-srt
  • facebook/mms-1b-allexample command

    easytranscriber-transcribe.py example command

    Transcribe audio with word-level timestamps using kb-labb/easytranscriber.

    hf jobs uv run --flavor l4x1 -s HF_TOKEN \
+    -e UV_TORCH_BACKEND=cu128 \
+    -v hf://buckets/your-username/audio-files:/input:ro \
+    -v hf://buckets/your-username/transcripts-aligned:/output \
+    https://huggingface.co/datasets/uv-scripts/transcription/raw/main/easytranscriber-transcribe.py \
+    /input /output \
+    --language en --emit-txt --emit-srt
  • ibm-granite/granite-speech-5.0-470m-turboctcexample command

    granite-turboctc-transcribe.py example command

    Transcribe English audio files with IBM Granite Speech 5.0 TurboCTC (470M).

    hf jobs uv run --flavor l4x1 -s HF_TOKEN \
+    -e UV_TORCH_BACKEND=cu128 \
+    -v hf://buckets/your-username/audio-input:/input:ro \
+    -v hf://buckets/your-username/transcripts:/output \
+    https://huggingface.co/datasets/uv-scripts/transcription/raw/main/granite-turboctc-transcribe.py \
+    /input /output
  • nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4Bcommand ready · rtx-pro-6000

    nemotron-transcribe.py command ready · rtx-pro-6000

    Transcribe audio with NVIDIA Nemotron 3.5 Transcribe via an in-job vLLM server.

    hf jobs uv run \
+    -v hf://buckets/your-username/audio-files:/input:ro \
+    -v hf://buckets/your-username/transcripts:/output \
+    https://huggingface.co/datasets/uv-scripts/transcription/raw/main/nemotron-transcribe.py \
+    /input /output
  • OpenMOSS-Team/MOSS-Transcribe-Diarizeexample command

    moss-transcribe-diarize.py example command

    Transcribe + diarize audio files using MOSS-Transcribe-Diarize (0.9B).

    hf jobs uv run --flavor l4x1 -s HF_TOKEN \
+    -e UV_TORCH_BACKEND=cu128 \
+    -v hf://buckets/your-username/audio-files:/input:ro \
+    -v hf://buckets/your-username/transcripts:/output \
+    https://huggingface.co/datasets/uv-scripts/transcription/raw/main/moss-transcribe-diarize.py \
+    /input /output --emit-txt --emit-srt
    1 more recipe

    moss-transcribe-diarize-server.py read the script first

    High-throughput transcription + diarization via an in-job sgl-omni server.

    Read the script for its options. There is no ready-made command yet.

Segmentation

  • facebook/sam3example command

    detect-objects.py example command

    Detect objects in images using Meta's SAM3 (Segment Anything Model 3).

    hf jobs uv run --flavor a100-large \
+    -s HF_TOKEN \
+    https://huggingface.co/datasets/uv-scripts/sam3/raw/main/detect-objects.py \
+    INPUT_DATASET OUTPUT_DATASET \
+    --class-name photograph \
+    --confidence-threshold 0.5
    1 more recipe

    segment-objects.py example command

    Segment objects in images using Meta's SAM3 (Segment Anything Model 3).

    hf jobs uv run --flavor a100-large \
+    -s HF_TOKEN \
+    https://huggingface.co/datasets/uv-scripts/sam3/raw/main/segment-objects.py \
+    INPUT_DATASET OUTPUT_DATASET \
+    --class-name photograph

Object detection

  • Qwen/Qwen3.6-35B-A3Bcommand ready · a100-large

    qwen3vl-detect.py command ready · a100-large

    Instruction-oriented object detection with Qwen3-VL via vLLM.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/vlm-object-detection/raw/main/qwen3vl-detect.py \
+    INPUT_DATASET OUTPUT_DATASET
    1 more recipe

    qwen3vl-detect-fewshot.py command ready · a100-large

    Instruction-oriented object detection with Qwen3-VL via vLLM.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/vlm-object-detection/raw/main/qwen3vl-detect-fewshot.py \
+    INPUT_DATASET OUTPUT_DATASET

Video

  • NemoStation/Marlin-2Bcommand ready · a10g-small

    marlin-caption.py command ready · a10g-small

    Caption videos with timestamped events using Marlin-2B, writing a resumable dataset.

    hf jobs uv run \
+    -v hf://buckets/your-username/my-videos:/input:ro \
+    https://huggingface.co/datasets/uv-scripts/video/raw/main/marlin-caption.py \
+    /input hf://buckets/your-username/my-videos/captions
  • Qwen/Qwen3.8-27Bcommand ready · a100-large

    qwen38-caption.py command ready · a100-large

    Caption whole videos — no chunking — with Qwen3.8-27B, writing a resumable dataset.

    hf jobs uv run \
+    -v hf://buckets/your-username/my-videos:/input:ro \
+    https://huggingface.co/datasets/uv-scripts/video/raw/main/qwen38-caption.py \
+    /input hf://buckets/your-username/my-videos/captions

Classification

  • fastino/gliner2.5-multi-v1command ready · t4-small

    classify-gliner2.py command ready · t4-small

    Classify a text column of a Hub dataset with GLiNER2 — zero-shot, or with your fine-tuned model.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/classification/raw/main/classify-gliner2.py \
+    fancyzhx/ag_news your-username/ag-news-gliner2 \
+    --labels World Sports Business "Science and technology" --max-samples 1000
  • HuggingFaceTB/SmolLM3-3Bcommand ready · l4x1

    classify-dataset-sglang.py command ready · l4x1

    Classify text columns in Hugging Face datasets using SGLang with reasoning-aware models.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/classification/raw/main/classify-dataset-sglang.py \
+    --input-dataset your-username/emails \
+    --column content \
+    --labels "spam,ham" \
+    --output-dataset your-username/emails-classified \
+    --reasoning
    1 more recipe

    classify-dataset.py command ready · l4x1

    Classify text columns in Hugging Face datasets using vLLM with structured outputs.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/classification/raw/main/classify-dataset.py \
+    --input-dataset your-username/emails \
+    --column content \
+    --labels "spam,ham" \
+    --output-dataset your-username/emails-classified \
+    --enable-reasoning

Embeddings

  • BAAI/bge-base-en-v1.5example command

    embed-to-lance.py example command

    Embed a Hugging Face dataset and push it back as a Lance vector index — a Hub dataset that IS a searchable vector database.

    hf jobs uv run --flavor l4x1 -s HF_TOKEN \
+    https://huggingface.co/datasets/uv-scripts/embeddings/raw/main/embed-to-lance.py \
+    stanfordnlp/imdb your-username/imdb-vecdb --column text --model BAAI/bge-base-en-v1.5 --private
  • Qwen/Qwen3-Embedding-0.6Bexample command

    generate-embeddings-vllm.py example command

    High-throughput embedding generation with vLLM pooling mode — the "scale" variant of generate-embeddings.py, for large decoder embedding models (e.g.

    hf jobs uv run --flavor l4x1 -s HF_TOKEN \
+    https://huggingface.co/datasets/uv-scripts/embeddings/raw/main/generate-embeddings-vllm.py \
+    stanfordnlp/imdb your-username/imdb-embeddings --column text --model Qwen/Qwen3-Embedding-0.6B --private
  • sentence-transformers/all-MiniLM-L6-v2example command

    generate-embeddings.py example command

    Generate embeddings for a Hugging Face dataset (text OR images) with sentence-transformers, and push the result back to the Hub as a new dataset with…

    hf jobs uv run --flavor l4x1 -s HF_TOKEN \
+    https://huggingface.co/datasets/uv-scripts/embeddings/raw/main/generate-embeddings.py \
+    stanfordnlp/imdb  your-username/imdb-embeddings \
+    --column text --model sentence-transformers/all-MiniLM-L6-v2
    1 more recipe

    launch-embedding-fleet.py read the script first

    Fan one embedding run out across N Hugging Face Jobs, then consolidate.

    Read the script for its options. There is no ready-made command yet.

Text generation

  • Qwen/Qwen2-VL-7B-Instructcommand ready · l40sx1

    vlm-classify.py command ready · l40sx1

    Classify images using Vision Language Models with vLLM.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/vllm/raw/main/vlm-classify.py \
+    your-username/INPUT_DATASET \
+    your-username/OUTPUT_DATASET \
+    --classes "title-page,content,index,other"
  • Qwen/Qwen3-30B-A3B-Instruct-2507command ready · a100-large

    generate-responses.py command ready · a100-large

    Generate responses for prompts in a dataset using vLLM for efficient GPU inference.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/vllm/raw/main/generate-responses.py \
+    input output --max-model-len 32768

Fine-tuning

  • fastino/gliner2.5-multi-v1command ready · t4-small

    train-gliner2.py command ready · t4-small

    Fine-tune GLiNER2 into a text classifier — a small model (74M to 287M parameters, depending on the base checkpoint) that already works zero-shot.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/classification/raw/main/train-gliner2.py \
+    biglam/blbooksgenre your-username/gliner2-blbooks-genre \
+    --dataset-config title_genre_classifiction --text-column title
  • LiquidAI/LFM2.5-Encoder-350Mcommand ready · l4x1

    train-classifier.py command ready · l4x1

    Fine-tune a text-classification encoder on a Hub dataset and push the trained model to the Hub.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/classification/raw/main/train-classifier.py \
+    fancyzhx/ag_news your-username/my-news-classifier \
+    --max-samples 2000 --epochs 1
  • sentence-transformers/all-MiniLM-L6-v2example command

    train-setfit.py example command

    Few-shot text classification with SetFit — train on 8-64 labelled examples per class, on CPU or GPU.

    hf jobs uv run --flavor cpu-basic --secrets HF_TOKEN \
+    https://huggingface.co/datasets/uv-scripts/classification/raw/main/train-setfit.py \
+    fancyzhx/ag_news your-username/ag-news-setfit \
+    --num-samples 8
+
+ +

For agents

+

Read AGENTS.md and recipes.json.

+
+
+ + + diff --git a/run-on-jobs/recipes.json b/run-on-jobs/recipes.json new file mode 100644 index 0000000..deaa39d --- /dev/null +++ b/run-on-jobs/recipes.json @@ -0,0 +1,796 @@ +{ + "recipes": [ + { + "kind": "run", + "task": "Classification", + "script": "classification/classify-dataset-sglang.py", + "url": "https://huggingface.co/datasets/uv-scripts/classification/raw/main/classify-dataset-sglang.py", + "models": [ + "HuggingFaceTB/SmolLM3-3B" + ], + "support": null, + "source": "header", + "flavor": "l4x1", + "timeout": "1h", + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/classification/raw/main/classify-dataset-sglang.py \\\n --input-dataset your-username/emails \\\n --column content \\\n --labels \"spam,ham\" \\\n --output-dataset your-username/emails-classified \\\n --reasoning", + "summary": "Classify text columns in Hugging Face datasets using SGLang with reasoning-aware models.", + "declared": true + }, + { + "kind": "run", + "task": "Classification", + "script": "classification/classify-dataset.py", + "url": "https://huggingface.co/datasets/uv-scripts/classification/raw/main/classify-dataset.py", + "models": [ + "HuggingFaceTB/SmolLM3-3B" + ], + "support": null, + "source": "header", + "flavor": "l4x1", + "timeout": "1h", + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/classification/raw/main/classify-dataset.py \\\n --input-dataset your-username/emails \\\n --column content \\\n --labels \"spam,ham\" \\\n --output-dataset your-username/emails-classified \\\n --enable-reasoning", + "summary": "Classify text columns in Hugging Face datasets using vLLM with structured outputs.", + "declared": true + }, + { + "kind": "run", + "task": "Classification", + "script": "classification/classify-gliner2.py", + "url": "https://huggingface.co/datasets/uv-scripts/classification/raw/main/classify-gliner2.py", + "models": [ + "fastino/gliner2.5-multi-v1" + ], + "support": null, + "source": "header", + "flavor": "t4-small", + "timeout": "1h", + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/classification/raw/main/classify-gliner2.py \\\n fancyzhx/ag_news your-username/ag-news-gliner2 \\\n --labels World Sports Business \"Science and technology\" --max-samples 1000", + "summary": "Classify a text column of a Hub dataset with GLiNER2 — zero-shot, or with your fine-tuned model.", + "declared": true + }, + { + "kind": "train", + "task": "Fine-tuning", + "script": "classification/train-classifier.py", + "url": "https://huggingface.co/datasets/uv-scripts/classification/raw/main/train-classifier.py", + "models": [ + "LiquidAI/LFM2.5-Encoder-350M" + ], + "support": null, + "source": "header", + "flavor": "l4x1", + "timeout": "1h", + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/classification/raw/main/train-classifier.py \\\n fancyzhx/ag_news your-username/my-news-classifier \\\n --max-samples 2000 --epochs 1", + "summary": "Fine-tune a text-classification encoder on a Hub dataset and push the trained model to the Hub.", + "declared": false + }, + { + "kind": "train", + "task": "Fine-tuning", + "script": "classification/train-gliner2.py", + "url": "https://huggingface.co/datasets/uv-scripts/classification/raw/main/train-gliner2.py", + "models": [ + "fastino/gliner2.5-multi-v1" + ], + "support": null, + "source": "header", + "flavor": "t4-small", + "timeout": "1h", + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/classification/raw/main/train-gliner2.py \\\n biglam/blbooksgenre your-username/gliner2-blbooks-genre \\\n --dataset-config title_genre_classifiction --text-column title", + "summary": "Fine-tune GLiNER2 into a text classifier — a small model (74M to 287M parameters, depending on the base checkpoint) that already works zero-shot.", + "declared": false + }, + { + "kind": "train", + "task": "Fine-tuning", + "script": "classification/train-setfit.py", + "url": "https://huggingface.co/datasets/uv-scripts/classification/raw/main/train-setfit.py", + "models": [ + "sentence-transformers/all-MiniLM-L6-v2" + ], + "support": null, + "source": "documented", + "flavor": null, + "timeout": null, + "command": "hf jobs uv run --flavor cpu-basic --secrets HF_TOKEN \\\n https://huggingface.co/datasets/uv-scripts/classification/raw/main/train-setfit.py \\\n fancyzhx/ag_news your-username/ag-news-setfit \\\n --num-samples 8", + "summary": "Few-shot text classification with SetFit — train on 8-64 labelled examples per class, on CPU or GPU.", + "declared": false + }, + { + "kind": "run", + "task": "Embeddings", + "script": "embeddings/embed-to-lance.py", + "url": "https://huggingface.co/datasets/uv-scripts/embeddings/raw/main/embed-to-lance.py", + "models": [ + "BAAI/bge-base-en-v1.5" + ], + "support": null, + "source": "documented", + "flavor": null, + "timeout": null, + "command": "hf jobs uv run --flavor l4x1 -s HF_TOKEN \\\n https://huggingface.co/datasets/uv-scripts/embeddings/raw/main/embed-to-lance.py \\\n stanfordnlp/imdb your-username/imdb-vecdb --column text --model BAAI/bge-base-en-v1.5 --private", + "summary": "Embed a Hugging Face dataset and push it back as a Lance vector index — a Hub dataset that IS a searchable vector database.", + "declared": false + }, + { + "kind": "run", + "task": "Embeddings", + "script": "embeddings/generate-embeddings-vllm.py", + "url": "https://huggingface.co/datasets/uv-scripts/embeddings/raw/main/generate-embeddings-vllm.py", + "models": [ + "Qwen/Qwen3-Embedding-0.6B" + ], + "support": null, + "source": "documented", + "flavor": null, + "timeout": null, + "command": "hf jobs uv run --flavor l4x1 -s HF_TOKEN \\\n https://huggingface.co/datasets/uv-scripts/embeddings/raw/main/generate-embeddings-vllm.py \\\n stanfordnlp/imdb your-username/imdb-embeddings --column text --model Qwen/Qwen3-Embedding-0.6B --private", + "summary": "High-throughput embedding generation with vLLM pooling mode — the \"scale\" variant of generate-embeddings.py, for large decoder embedding models (e.g.", + "declared": false + }, + { + "kind": "run", + "task": "Embeddings", + "script": "embeddings/generate-embeddings.py", + "url": "https://huggingface.co/datasets/uv-scripts/embeddings/raw/main/generate-embeddings.py", + "models": [ + "sentence-transformers/all-MiniLM-L6-v2" + ], + "support": null, + "source": "documented", + "flavor": null, + "timeout": null, + "command": "hf jobs uv run --flavor l4x1 -s HF_TOKEN \\\n https://huggingface.co/datasets/uv-scripts/embeddings/raw/main/generate-embeddings.py \\\n stanfordnlp/imdb your-username/imdb-embeddings \\\n --column text --model sentence-transformers/all-MiniLM-L6-v2", + "summary": "Generate embeddings for a Hugging Face dataset (text OR images) with sentence-transformers, and push the result back to the Hub as a new dataset with…", + "declared": true + }, + { + "kind": "run", + "task": "Embeddings", + "script": "embeddings/launch-embedding-fleet.py", + "url": "https://huggingface.co/datasets/uv-scripts/embeddings/raw/main/launch-embedding-fleet.py", + "models": [ + "sentence-transformers/all-MiniLM-L6-v2" + ], + "support": null, + "source": "flags-needed", + "flavor": null, + "timeout": null, + "command": null, + "summary": "Fan one embedding run out across N Hugging Face Jobs, then consolidate.", + "declared": false + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/deepseek-ocr-vllm.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/deepseek-ocr-vllm.py", + "models": [ + "deepseek-ai/DeepSeek-OCR" + ], + "support": "core", + "source": "header", + "flavor": "a10g-small", + "timeout": null, + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/deepseek-ocr-vllm.py \\\n my-dataset my-output --max-samples 10", + "summary": "Convert document images to markdown using DeepSeek-OCR with vLLM.", + "declared": true + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/deepseek-ocr2-vllm.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/deepseek-ocr2-vllm.py", + "models": [ + "deepseek-ai/DeepSeek-OCR-2" + ], + "support": "core", + "source": "header", + "flavor": "a10g-small", + "timeout": null, + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/deepseek-ocr2-vllm.py \\\n my-dataset my-output --max-samples 10", + "summary": "Convert document images to markdown using DeepSeek-OCR-2 with vLLM.", + "declared": true + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/dots-mocr.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/dots-mocr.py", + "models": [ + "rednote-hilab/dots.mocr" + ], + "support": "core", + "source": "header", + "flavor": "a10g-small", + "timeout": null, + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/dots-mocr.py \\\n INPUT_DATASET OUTPUT_DATASET", + "summary": "Convert document images to markdown using dots.mocr with vLLM.", + "declared": true + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/dots-ocr.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/dots-ocr.py", + "models": [ + "rednote-hilab/dots.ocr" + ], + "support": "core", + "source": "header", + "flavor": "a10g-small", + "timeout": null, + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/dots-ocr.py \\\n INPUT_DATASET OUTPUT_DATASET", + "summary": "Convert document images to markdown using DoTS.ocr with vLLM.", + "declared": true + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/glm-ocr.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/glm-ocr.py", + "models": [ + "zai-org/GLM-OCR" + ], + "support": "core", + "source": "header", + "flavor": "a10g-small", + "timeout": null, + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/glm-ocr.py \\\n {source_dataset} \\\n \\\n --image-column {image_column} \\\n --batch-size {batch_size} \\\n --task {task}", + "summary": "Convert document images to markdown using GLM-OCR with vLLM.", + "declared": true + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/hunyuan-ocr-1.5.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/hunyuan-ocr-1.5.py", + "models": [ + "tencent/HunyuanOCR" + ], + "support": "core", + "source": "header", + "flavor": "a10g-small", + "timeout": null, + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/hunyuan-ocr-1.5.py \\\n INPUT_DATASET OUTPUT_DATASET", + "summary": "Convert document images to markdown using HunyuanOCR-1.5 with vLLM.", + "declared": true + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/lift-extract.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/lift-extract.py", + "models": [ + "datalab-to/lift" + ], + "support": "core", + "source": "header", + "flavor": "a100-large", + "timeout": null, + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/lift-extract.py \\\n INPUT_DATASET OUTPUT_DATASET \\\n --schema '{\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\"}}}' \\\n --max-samples 5 --shuffle --seed 42", + "summary": "Extract structured JSON from document images OR multi-page PDFs using Datalab's lift model (datalab-to/lift, 9B, Qwen3.5-based).", + "declared": true + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/lighton-ocr2-saturate.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/lighton-ocr2-saturate.py", + "models": [ + "lightonai/LightOnOCR-2-1B" + ], + "support": "core", + "source": "header", + "flavor": "a10g-small", + "timeout": null, + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/lighton-ocr2-saturate.py \\\n ", + "summary": "Convert document images to markdown using LightOnOCR-2 via saturate.", + "declared": true + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/lighton-ocr2.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/lighton-ocr2.py", + "models": [ + "lightonai/LightOnOCR-2-1B" + ], + "support": "core", + "source": "header", + "flavor": "a10g-small", + "timeout": null, + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/lighton-ocr2.py \\\n ", + "summary": "Convert document images to markdown using LightOnOCR-2 with vLLM.", + "declared": true + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/lighton-ocr3-bucket.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/lighton-ocr3-bucket.py", + "models": [ + "lightonai/LightOnOCR-3-4B" + ], + "support": "core", + "source": "header", + "flavor": "a10g-small", + "timeout": null, + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/lighton-ocr3-bucket.py \\\n hf://buckets/biglam/britannica/source/pages/00070476bsb \\\n hf://buckets/your-username/ocr-output/britannica --mode grounding --max-samples 20", + "summary": "OCR images and PDFs from an HF Bucket (or a directory) using LightOnOCR-3, writing markdown files.", + "declared": true + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/lighton-ocr3.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/lighton-ocr3.py", + "models": [ + "lightonai/LightOnOCR-3-4B" + ], + "support": "core", + "source": "header", + "flavor": "a10g-small", + "timeout": null, + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/lighton-ocr3.py \\\n INPUT_DATASET OUTPUT_DATASET --max-samples 10", + "summary": "Convert document images to markdown (or layout blocks) using LightOnOCR-3 with vLLM.", + "declared": true + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/minicpm-ocr.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/minicpm-ocr.py", + "models": [ + "openbmb/MiniCPM-OCR-1.0" + ], + "support": "experimental", + "source": "header", + "flavor": "a10g-large", + "timeout": "2h", + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/minicpm-ocr.py \\\n INPUT_DATASET OUTPUT_DATASET --max-samples 20", + "summary": "Convert document images to markdown with OpenBMB's MiniCPM-OCR 1.0 pipeline.", + "declared": false + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/nuextract3.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/nuextract3.py", + "models": [ + "numind/NuExtract3" + ], + "support": "core", + "source": "header", + "flavor": "a10g-small", + "timeout": null, + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/nuextract3.py \\\n INPUT_DATASET OUTPUT_DATASET --max-samples 5 --shuffle --seed 42", + "summary": "Convert document images to markdown OR extract structured JSON using NuExtract3 with vLLM.", + "declared": true + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/olmocr2-vllm.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/olmocr2-vllm.py", + "models": [ + "allenai/olmOCR-2-7B-1025-FP8" + ], + "support": "core", + "source": "header", + "flavor": "a10g-small", + "timeout": null, + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/olmocr2-vllm.py \\\n INPUT_DATASET OUTPUT_DATASET", + "summary": "Convert document images to markdown using olmOCR-2 with vLLM.", + "declared": true + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/ovis-ocr2-saturate.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/ovis-ocr2-saturate.py", + "models": [ + "ATH-MaaS/OvisOCR2" + ], + "support": "core", + "source": "header", + "flavor": "a10g-small", + "timeout": null, + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/ovis-ocr2-saturate.py \\\n ", + "summary": "Convert document images to markdown using OvisOCR2 via saturate.", + "declared": true + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/ovis-ocr2.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/ovis-ocr2.py", + "models": [ + "ATH-MaaS/OvisOCR2" + ], + "support": "core", + "source": "header", + "flavor": "a10g-small", + "timeout": null, + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/ovis-ocr2.py \\\n INPUT_DATASET OUTPUT_DATASET --batch-size 16", + "summary": "Convert document images to markdown using OvisOCR2 with vLLM.", + "declared": true + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/paddleocr-vl-1.6.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/paddleocr-vl-1.6.py", + "models": [ + "PaddlePaddle/PaddleOCR-VL-1.6" + ], + "support": "core", + "source": "header", + "flavor": "a10g-small", + "timeout": null, + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/paddleocr-vl-1.6.py \\\n INPUT_DATASET OUTPUT_DATASET", + "summary": "Convert document images to text/tables/formulas using PaddleOCR-VL-1.6 with vLLM.", + "declared": true + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/qianfan-ocr.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/qianfan-ocr.py", + "models": [ + "baidu/Qianfan-OCR" + ], + "support": "core", + "source": "header", + "flavor": "a10g-small", + "timeout": null, + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/qianfan-ocr.py \\\n INPUT_DATASET OUTPUT_DATASET --max-samples 10", + "summary": "Convert document images to markdown using Qianfan-OCR with vLLM.", + "declared": true + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/surya-ocr.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/surya-ocr.py", + "models": [ + "datalab-to/surya-ocr-2" + ], + "support": "core", + "source": "header", + "flavor": "a10g-small", + "timeout": null, + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/surya-ocr.py \\\n INPUT_DATASET OUTPUT_DATASET \\\n --max-samples 5 --shuffle --seed 42", + "summary": "Document intelligence on images OR multi-page PDFs with Datalab's Surya OCR 2 (datalab-to/surya-ocr-2, 650M, Qwen3.5-style).", + "declared": true + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/unlimited-ocr-vllm.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/unlimited-ocr-vllm.py", + "models": [ + "baidu/Unlimited-OCR" + ], + "support": "core", + "source": "header", + "flavor": "a10g-small", + "timeout": null, + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/unlimited-ocr-vllm.py \\\n your-input-dataset your-output-dataset --max-samples 10", + "summary": "Convert document images to markdown using Baidu Unlimited-OCR with vLLM.", + "declared": true + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/unlimited-ocr2-live.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/unlimited-ocr2-live.py", + "models": [ + "baidu/Unlimited-OCR-2" + ], + "support": "experimental", + "source": "header", + "flavor": "rtx-pro-6000", + "timeout": "4h", + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/unlimited-ocr2-live.py \\\n \"hf://buckets/biglam/britannica/source/pages/1771EncyclopediaBritannicaNLS/*.jp2\" \\\n /britannica-1771 --live-port 7860", + "summary": "unlimited-ocr2.py with a live view you can watch (and screen-record) while the Job runs.", + "declared": false + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/unlimited-ocr2-saturate.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/unlimited-ocr2-saturate.py", + "models": [ + "baidu/Unlimited-OCR-2" + ], + "support": "experimental", + "source": "header", + "flavor": "rtx-pro-6000", + "timeout": "4h", + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/unlimited-ocr2-saturate.py \\\n \"hf://buckets/biglam/britannica/source/pages/1771EncyclopediaBritannicaNLS/*.jp2\" \\\n /britannica-1771-uocr2", + "summary": "OCR whole documents (book volumes, PDFs) with Baidu Unlimited-OCR-2 via saturate.", + "declared": false + }, + { + "kind": "run", + "task": "OCR", + "script": "ocr/unlimited-ocr2.py", + "url": "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/unlimited-ocr2.py", + "models": [ + "baidu/Unlimited-OCR-2" + ], + "support": "core", + "source": "header", + "flavor": "rtx-pro-6000", + "timeout": "4h", + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/ocr/raw/main/unlimited-ocr2.py \\\n \"hf://buckets/biglam/britannica/source/pages/1771EncyclopediaBritannicaNLS/*.jp2\" \\\n /britannica-1771", + "summary": "OCR whole documents (book volumes, folders of scans) with Baidu Unlimited-OCR-2.", + "declared": true + }, + { + "kind": "run", + "task": "Segmentation", + "script": "sam3/detect-objects.py", + "url": "https://huggingface.co/datasets/uv-scripts/sam3/raw/main/detect-objects.py", + "models": [ + "facebook/sam3" + ], + "support": null, + "source": "documented", + "flavor": null, + "timeout": null, + "command": "hf jobs uv run --flavor a100-large \\\n -s HF_TOKEN \\\n https://huggingface.co/datasets/uv-scripts/sam3/raw/main/detect-objects.py \\\n INPUT_DATASET OUTPUT_DATASET \\\n --class-name photograph \\\n --confidence-threshold 0.5", + "summary": "Detect objects in images using Meta's SAM3 (Segment Anything Model 3).", + "declared": true + }, + { + "kind": "run", + "task": "Segmentation", + "script": "sam3/segment-objects.py", + "url": "https://huggingface.co/datasets/uv-scripts/sam3/raw/main/segment-objects.py", + "models": [ + "facebook/sam3" + ], + "support": null, + "source": "documented", + "flavor": null, + "timeout": null, + "command": "hf jobs uv run --flavor a100-large \\\n -s HF_TOKEN \\\n https://huggingface.co/datasets/uv-scripts/sam3/raw/main/segment-objects.py \\\n INPUT_DATASET OUTPUT_DATASET \\\n --class-name photograph", + "summary": "Segment objects in images using Meta's SAM3 (Segment Anything Model 3).", + "declared": true + }, + { + "kind": "run", + "task": "Transcription", + "script": "transcription/cohere-transcribe-vllm.py", + "url": "https://huggingface.co/datasets/uv-scripts/transcription/raw/main/cohere-transcribe-vllm.py", + "models": [ + "CohereLabs/cohere-transcribe-03-2026" + ], + "support": null, + "source": "flags-needed", + "flavor": null, + "timeout": null, + "command": null, + "summary": "Transcribe audio files from a directory using Cohere Transcribe via vLLM.", + "declared": false + }, + { + "kind": "run", + "task": "Transcription", + "script": "transcription/cohere-transcribe.py", + "url": "https://huggingface.co/datasets/uv-scripts/transcription/raw/main/cohere-transcribe.py", + "models": [ + "CohereLabs/cohere-transcribe-03-2026" + ], + "support": null, + "source": "documented", + "flavor": null, + "timeout": null, + "command": "hf jobs uv run --flavor l4x1 \\\n -s HF_TOKEN \\\n -v hf://buckets/your-username/audio-input:/input:ro \\\n -v hf://buckets/your-username/transcripts:/output \\\n https://huggingface.co/datasets/uv-scripts/transcription/raw/main/cohere-transcribe.py \\\n /input /output --language en --compile", + "summary": "Transcribe audio files from a directory using Cohere Transcribe via transformers.", + "declared": false + }, + { + "kind": "run", + "task": "Transcription", + "script": "transcription/easytranscriber-transcribe.py", + "url": "https://huggingface.co/datasets/uv-scripts/transcription/raw/main/easytranscriber-transcribe.py", + "models": [ + "CohereLabs/cohere-transcribe-03-2026", + "distil-whisper/distil-large-v3.5", + "facebook/mms-1b-all" + ], + "support": null, + "source": "documented", + "flavor": null, + "timeout": null, + "command": "hf jobs uv run --flavor l4x1 -s HF_TOKEN \\\n -e UV_TORCH_BACKEND=cu128 \\\n -v hf://buckets/your-username/audio-files:/input:ro \\\n -v hf://buckets/your-username/transcripts-aligned:/output \\\n https://huggingface.co/datasets/uv-scripts/transcription/raw/main/easytranscriber-transcribe.py \\\n /input /output \\\n --language en --emit-txt --emit-srt", + "summary": "Transcribe audio with word-level timestamps using kb-labb/easytranscriber.", + "declared": false + }, + { + "kind": "run", + "task": "Transcription", + "script": "transcription/granite-turboctc-transcribe.py", + "url": "https://huggingface.co/datasets/uv-scripts/transcription/raw/main/granite-turboctc-transcribe.py", + "models": [ + "ibm-granite/granite-speech-5.0-470m-turboctc" + ], + "support": null, + "source": "documented", + "flavor": null, + "timeout": null, + "command": "hf jobs uv run --flavor l4x1 -s HF_TOKEN \\\n -e UV_TORCH_BACKEND=cu128 \\\n -v hf://buckets/your-username/audio-input:/input:ro \\\n -v hf://buckets/your-username/transcripts:/output \\\n https://huggingface.co/datasets/uv-scripts/transcription/raw/main/granite-turboctc-transcribe.py \\\n /input /output", + "summary": "Transcribe English audio files with IBM Granite Speech 5.0 TurboCTC (470M).", + "declared": false + }, + { + "kind": "run", + "task": "Transcription", + "script": "transcription/moss-transcribe-diarize-server.py", + "url": "https://huggingface.co/datasets/uv-scripts/transcription/raw/main/moss-transcribe-diarize-server.py", + "models": [ + "OpenMOSS-Team/MOSS-Transcribe-Diarize" + ], + "support": null, + "source": "flags-needed", + "flavor": null, + "timeout": null, + "command": null, + "summary": "High-throughput transcription + diarization via an in-job sgl-omni server.", + "declared": false + }, + { + "kind": "run", + "task": "Transcription", + "script": "transcription/moss-transcribe-diarize.py", + "url": "https://huggingface.co/datasets/uv-scripts/transcription/raw/main/moss-transcribe-diarize.py", + "models": [ + "OpenMOSS-Team/MOSS-Transcribe-Diarize" + ], + "support": null, + "source": "documented", + "flavor": null, + "timeout": null, + "command": "hf jobs uv run --flavor l4x1 -s HF_TOKEN \\\n -e UV_TORCH_BACKEND=cu128 \\\n -v hf://buckets/your-username/audio-files:/input:ro \\\n -v hf://buckets/your-username/transcripts:/output \\\n https://huggingface.co/datasets/uv-scripts/transcription/raw/main/moss-transcribe-diarize.py \\\n /input /output --emit-txt --emit-srt", + "summary": "Transcribe + diarize audio files using MOSS-Transcribe-Diarize (0.9B).", + "declared": false + }, + { + "kind": "run", + "task": "Transcription", + "script": "transcription/nemotron-transcribe.py", + "url": "https://huggingface.co/datasets/uv-scripts/transcription/raw/main/nemotron-transcribe.py", + "models": [ + "nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B" + ], + "support": null, + "source": "header", + "flavor": "rtx-pro-6000", + "timeout": null, + "command": "hf jobs uv run \\\n -v hf://buckets/your-username/audio-files:/input:ro \\\n -v hf://buckets/your-username/transcripts:/output \\\n https://huggingface.co/datasets/uv-scripts/transcription/raw/main/nemotron-transcribe.py \\\n /input /output", + "summary": "Transcribe audio with NVIDIA Nemotron 3.5 Transcribe via an in-job vLLM server.", + "declared": true + }, + { + "kind": "run", + "task": "Video", + "script": "video/marlin-caption.py", + "url": "https://huggingface.co/datasets/uv-scripts/video/raw/main/marlin-caption.py", + "models": [ + "NemoStation/Marlin-2B" + ], + "support": null, + "source": "header", + "flavor": "a10g-small", + "timeout": "2h", + "command": "hf jobs uv run \\\n -v hf://buckets/your-username/my-videos:/input:ro \\\n https://huggingface.co/datasets/uv-scripts/video/raw/main/marlin-caption.py \\\n /input hf://buckets/your-username/my-videos/captions", + "summary": "Caption videos with timestamped events using Marlin-2B, writing a resumable dataset.", + "declared": true + }, + { + "kind": "run", + "task": "Video", + "script": "video/qwen38-caption.py", + "url": "https://huggingface.co/datasets/uv-scripts/video/raw/main/qwen38-caption.py", + "models": [ + "Qwen/Qwen3.8-27B" + ], + "support": null, + "source": "header", + "flavor": "a100-large", + "timeout": null, + "command": "hf jobs uv run \\\n -v hf://buckets/your-username/my-videos:/input:ro \\\n https://huggingface.co/datasets/uv-scripts/video/raw/main/qwen38-caption.py \\\n /input hf://buckets/your-username/my-videos/captions", + "summary": "Caption whole videos — no chunking — with Qwen3.8-27B, writing a resumable dataset.", + "declared": true + }, + { + "kind": "run", + "task": "Text generation", + "script": "vllm/generate-responses.py", + "url": "https://huggingface.co/datasets/uv-scripts/vllm/raw/main/generate-responses.py", + "models": [ + "Qwen/Qwen3-30B-A3B-Instruct-2507" + ], + "support": null, + "source": "header", + "flavor": "a100-large", + "timeout": "2h", + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/vllm/raw/main/generate-responses.py \\\n input output --max-model-len 32768", + "summary": "Generate responses for prompts in a dataset using vLLM for efficient GPU inference.", + "declared": false + }, + { + "kind": "run", + "task": "Text generation", + "script": "vllm/vlm-classify.py", + "url": "https://huggingface.co/datasets/uv-scripts/vllm/raw/main/vlm-classify.py", + "models": [ + "Qwen/Qwen2-VL-7B-Instruct" + ], + "support": null, + "source": "header", + "flavor": "l40sx1", + "timeout": "1h", + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/vllm/raw/main/vlm-classify.py \\\n your-username/INPUT_DATASET \\\n your-username/OUTPUT_DATASET \\\n --classes \"title-page,content,index,other\"", + "summary": "Classify images using Vision Language Models with vLLM.", + "declared": true + }, + { + "kind": "run", + "task": "Object detection", + "script": "vlm-object-detection/qwen3vl-detect-fewshot.py", + "url": "https://huggingface.co/datasets/uv-scripts/vlm-object-detection/raw/main/qwen3vl-detect-fewshot.py", + "models": [ + "Qwen/Qwen3.6-35B-A3B" + ], + "support": null, + "source": "header", + "flavor": "a100-large", + "timeout": "1h", + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/vlm-object-detection/raw/main/qwen3vl-detect-fewshot.py \\\n INPUT_DATASET OUTPUT_DATASET", + "summary": "Instruction-oriented object detection with Qwen3-VL via vLLM.", + "declared": true + }, + { + "kind": "run", + "task": "Object detection", + "script": "vlm-object-detection/qwen3vl-detect.py", + "url": "https://huggingface.co/datasets/uv-scripts/vlm-object-detection/raw/main/qwen3vl-detect.py", + "models": [ + "Qwen/Qwen3.6-35B-A3B" + ], + "support": null, + "source": "header", + "flavor": "a100-large", + "timeout": "1h", + "command": "hf jobs uv run \\\n https://huggingface.co/datasets/uv-scripts/vlm-object-detection/raw/main/qwen3vl-detect.py \\\n INPUT_DATASET OUTPUT_DATASET", + "summary": "Instruction-oriented object detection with Qwen3-VL via vLLM.", + "declared": true + } + ], + "data_tools": [ + { + "repo": "uv-scripts/object-detection", + "description": "Check and convert detection datasets" + }, + { + "repo": "uv-scripts/deduplication", + "description": "Remove near-duplicate text" + }, + { + "repo": "uv-scripts/build-atlas", + "description": "Build an embedding atlas" + }, + { + "repo": "uv-scripts/iiif-tiles", + "description": "Make IIIF deep-zoom tiles" + }, + { + "repo": "uv-scripts/data-processing", + "description": "Convert uploads to optimized Parquet" + } + ] +} diff --git a/tools/build-catalog.py b/tools/build-catalog.py new file mode 100755 index 0000000..a387360 --- /dev/null +++ b/tools/build-catalog.py @@ -0,0 +1,447 @@ +#!/usr/bin/env python3 +"""Generate the Run on Jobs Space from the recipes in this repo. + + python3 tools/build-catalog.py # write run-on-jobs/ + python3 tools/build-catalog.py --check # fail if run-on-jobs/ is stale + python3 tools/build-catalog.py --verify-models # also check each declared model on the Hub + +Scripts are only parsed, never imported or run. Standard library only. +""" + +import argparse +import ast +import html +import json +import re +import sys +import tomllib +import urllib.error +import urllib.request +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +OUTPUT_DIR = ROOT / "run-on-jobs" +HUB_BASE = "https://huggingface.co/datasets/uv-scripts" + +TASKS = { + "ocr": "OCR", + "transcription": "Transcription", + "sam3": "Segmentation", + "vlm-object-detection": "Object detection", + "video": "Video", + "classification": "Classification", + "gliner": "Entities", + "embeddings": "Embeddings", + "vllm": "Text generation", +} +TRAINING_TASK = "Fine-tuning" +TASK_ORDER = [*TASKS.values(), TRAINING_TASK] + +DATA_TOOLS = { + "object-detection": "Check and convert detection datasets", + "deduplication": "Remove near-duplicate text", + "build-atlas": "Build an embedding atlas", + "iiif-tiles": "Make IIIF deep-zoom tiles", + "data-processing": "Convert uploads to optimized Parquet", +} + +# Recipes without a [tool.hf-jobs] header whose documented command passed a smoke Job with checked output. +# Only these and header recipes are declared on model pages. +SMOKE_TESTED_EXAMPLES = { + "embeddings/generate-embeddings.py", + "sam3/detect-objects.py", + "sam3/segment-objects.py", +} + +# Left out of the Space until they run again. +BROKEN = { + "gliner/extract-entities.py", # the default model's tokenizer needs tiktoken, which the script doesn't declare +} + +# Listed on the page, but not declared on model pages until the plain command works for the model. +NOT_DECLARED = { + "vllm/generate-responses.py", # the default model needs --max-model-len on the header's a100-large +} + +# Recipe metadata that names a family rather than a Hub repo. +NOT_HUB_MODELS = {"PaddlePaddle/PP-OCRv6"} + +OCR_SUPPORT_LEVELS = {"core", "experimental"} +SKIP_SCRIPT = re.compile(r"(^_|^test_|conftest|setup|benchmark|judge|compare)", re.IGNORECASE) +ALTERNATIVE_SCRIPT = re.compile(r"saturate|live|server|fleet|fewshot", re.IGNORECASE) +PATH_ARGUMENT = re.compile(r"^(input|output)$|_dir$|_path$|_folder$") +MODEL_ID = re.compile(r"^[A-Za-z0-9][\w.-]*/[\w.+-]+$") +PLACEHOLDER_OWNERS = {"org", "datasets", "models", "hf", "bucket", "buckets", "input", "output", "user", "your", "username", "path", "dir", "my-org", "uv-scripts"} + +STATUS = {"header": "command ready", "documented": "example command", "flags-needed": "read the script first"} +ROW_STATUS = {"header": "command ready", "documented": "example command", "flags-needed": "no command yet"} + + +def read_header(text): + match = re.search(r"(?m)^# /// script\n((?:#.*\n)*?)# ///", text) + if not match: + return {} + lines = [line[2:] if line.startswith("# ") else line[1:] for line in match.group(1).splitlines()] + try: + return tomllib.loads("\n".join(lines)).get("tool", {}).get("hf-jobs", {}) + except tomllib.TOMLDecodeError: + return {} + + +def read_arguments(tree): + """Return module string constants and the add_argument calls, using only the syntax tree.""" + constants = {} + for node in tree.body: + is_single_assign = isinstance(node, ast.Assign) and len(node.targets) == 1 and isinstance(node.targets[0], ast.Name) + if is_single_assign and isinstance(node.value, ast.Constant) and isinstance(node.value.value, str): + constants[node.targets[0].id] = node.value.value + + arguments = [] + for node in ast.walk(tree): + if not (isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) and node.func.attr == "add_argument"): + continue + names = [arg.value for arg in node.args if isinstance(arg, ast.Constant) and isinstance(arg.value, str)] + if not names: + continue + keywords = {keyword.arg: keyword.value for keyword in node.keywords if keyword.arg} + default = keywords.get("default") + if isinstance(default, ast.Constant): + default = default.value + elif isinstance(default, ast.Name): + default = constants.get(default.id) + else: + default = None + required = isinstance(keywords.get("required"), ast.Constant) and keywords["required"].value is True + arguments.append({"name": names[0], "default": default, "positional": not names[0].startswith("-"), "required": required}) + return constants, arguments + + +def find_models(constants, arguments): + candidates = [] + for argument in arguments: + if isinstance(argument["default"], str) and "model" in argument["name"].lower() and MODEL_ID.match(argument["default"]): + candidates.append(argument["default"]) + for name, value in constants.items(): + if "MODEL" in name.upper() and MODEL_ID.match(value): + candidates.append(value) + unique = dict.fromkeys(candidates) + return [model for model in unique if model.split("/")[0].lower() not in PLACEHOLDER_OWNERS] + + +def first_documented_command(text, script_name, url): + """Return the first documented example that launches this script, as (job_flags, script_arguments).""" + for match in re.finditer(r"hf jobs uv run(?:[^\n]*\\\n)*[^\n]*", text): + if script_name not in match.group(0): + continue + lines = [re.sub(r"^\s*(#\s*)?", "", line).rstrip(" \\") for line in match.group(0).split("\n")] + command = " \\\n ".join(line for line in lines if line) + if url not in command: + command = re.sub(r"(?
{html.escape(command)}
' + + +def recipe_html(recipe): + status = STATUS[recipe["source"]] + if recipe["support"] == "experimental": + status += ", experimental" + status_class = "warn" if recipe["source"] == "flags-needed" else "muted" + flavor = f" · {html.escape(recipe['flavor'])}" if recipe["flavor"] else "" + summary = f'

{html.escape(recipe["summary"])}

' if recipe["summary"] else "" + if recipe["source"] == "flags-needed": + body = f'

Read the script for its options. There is no ready-made command yet.

' + else: + body = command_block(recipe["command"]) + name = recipe["script"].split("/")[-1] + return ( + f'

{html.escape(name)} ' + f'{status}{flavor}

{summary}{body}
' + ) + + +def model_row_html(task, model, recipes): + ranked = sorted(recipes, key=rank) + best = ranked[0] + row_status = ROW_STATUS[best["source"]] + (f" · {best['flavor']}" if best["flavor"] else "") + row_class = "warn" if best["source"] == "flags-needed" else "muted" + search_text = " ".join([model, task] + [f"{recipe['script']} {recipe['summary']}" for recipe in ranked]).lower() + others = ranked[1:] + more = "" + if others: + label = f"{len(others)} more recipe{'s' if len(others) > 1 else ''}" + more = f'
{label}' + "".join(recipe_html(recipe) for recipe in others) + "
" + return ( + f'
  • ' + f'{html.escape(model)}{html.escape(row_status)}' + + recipe_html(best) + more + "
    • " + ) + + +def render_index(recipes): + template = (ROOT / "tools" / "run-on-jobs-template.html").read_text() + entries = {} + for recipe in recipes: + for model in recipe["models"]: + entries.setdefault((recipe["task"], model), []).append(recipe) + + groups = [] + used_tasks = [] + for task in TASK_ORDER: + models = sorted((model for entry_task, model in entries if entry_task == task), key=str.lower) + if not models: + continue + used_tasks.append(task) + rows = "".join(model_row_html(task, model, entries[(task, model)]) for model in models) + groups.append(f'

    {html.escape(task)}

      {rows}
    ') + + filters = '' + "".join( + f'' for task in used_tasks + ) + tools = "".join( + f'
  • {html.escape(label)}
    • ' for folder, label in DATA_TOOLS.items() + ) + return template.replace("{{FILTERS}}", filters).replace("{{GROUPS}}", "".join(groups)).replace("{{TOOLS}}", tools) + + +def render_readme(recipes): + model_lines = "\n".join(f" - {model}" for model in declared_models(recipes)) + return f"""--- +title: Run on Jobs +emoji: ⚡ +colorFrom: blue +colorTo: gray +sdk: static +pinned: false +short_description: Batch-run this model on your own data with HF Jobs +models: +{model_lines} +--- + +# Run on Jobs + +A catalogue of [uv-scripts](https://huggingface.co/uv-scripts) recipes: one command runs a model over your own data on [Hugging Face Jobs](https://huggingface.co/docs/hub/jobs). + +This Space is generated from [davanstrien/uv-scripts](https://github.com/davanstrien/uv-scripts). To change it, edit the recipes or `tools/build-catalog.py` there. Agents should read [AGENTS.md](./AGENTS.md). +""" + + +def render_agents(): + return """# Run a model on Hugging Face Jobs + +Each recipe is one script, run from its URL with `hf jobs uv run`. There is nothing to clone or install except the `hf` CLI. + +## Set up + +- Check `hf version` (needs 1.32+). If it's missing: `curl -LsSf https://hf.co/cli/install.sh | bash`, then `hf auth login`. +- Jobs need pay-as-you-go credit, and each job costs money. Confirm with the user before starting one. +- The CLI is the current reference: `hf --help`, `hf jobs uv run --help`, `hf jobs hardware`. To install the Hugging Face CLI skill, run `hf skills add`. + +## Find a recipe + +- `hf download uv-scripts/run-on-jobs recipes.json --repo-type space`, then look for the model in `recipes[].models`. +- `kind`: `run` applies a model to data; `train` fine-tunes one. +- `source`: `header` carries its launch settings; `documented` is an example from the script's docs; `flags-needed` has no command, so read the script first. +- Read a script's docstring and options: `curl -s | head -100`. Running it locally may fail on GPU-only dependencies. +- Each task's README explains its recipes, for example https://huggingface.co/datasets/uv-scripts/ocr. OCR also has `models.json`, with language coverage and support levels. +- Fuller guides: [uv-recipes](https://github.com/davanstrien/uv-scripts/tree/main/skills/uv-recipes) and, for few-shot fine-tuning, [setfit-on-jobs](https://github.com/davanstrien/uv-scripts/tree/main/skills/setfit-on-jobs). + +## Run it + +Start from the recipe's `command` field. Most recipes go from dataset to dataset: + + hf jobs uv run --max-samples 10 + +Recipes that read files use mounts. A local folder also works as a source: + + hf jobs uv run -v hf://buckets//:/input:ro -v hf://buckets//:/output /input /output + +- Preview for free with `--dry-run`, and check the flavor: without a header, a GPU recipe can resolve to `cpu-basic`. +- Flags before the URL configure the job and override the header, such as `--flavor` and `--timeout`. Flags after it go to the script. If a script flag has the same name as a job flag, put `--` before the URL. +- Start with a small `--max-samples` if the script has it, and add `--private` if the output shouldn't be public. + +## Check it + +- Use `--detach` for long runs, then `hf jobs logs -f `, `hf jobs wait ` or `hf jobs inspect `. `hf jobs cancel ` stops one. +- Look at the output rows. A completed job doesn't prove they're correct. +""" + + +def generate(): + recipes = load_recipes() + catalog = { + "recipes": sorted(recipes, key=lambda recipe: recipe["script"]), + "data_tools": [{"repo": f"uv-scripts/{folder}", "description": label} for folder, label in DATA_TOOLS.items()], + } + return { + "index.html": render_index(recipes), + "README.md": render_readme(recipes), + "AGENTS.md": render_agents(), + "recipes.json": json.dumps(catalog, indent=1, ensure_ascii=False) + "\n", + } + + +def verify_models(recipes): + missing = [] + for model in declared_models(recipes): + try: + urllib.request.urlopen(f"https://huggingface.co/api/models/{model}", timeout=30) + except urllib.error.HTTPError as error: + missing.append(f"{model} (HTTP {error.code})") + return missing + + +def main(): + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--check", action="store_true", help="Fail if run-on-jobs/ does not match the recipes") + parser.add_argument("--verify-models", action="store_true", help="Check that declared models exist on the Hub") + args = parser.parse_args() + + files = generate() + if args.check: + stale = [name for name, content in files.items() if not (OUTPUT_DIR / name).exists() or (OUTPUT_DIR / name).read_text() != content] + if stale: + print(f"Stale: {', '.join(stale)}. Run python3 tools/build-catalog.py and commit the result.") + return 1 + else: + for name, content in files.items(): + (OUTPUT_DIR / name).write_text(content) + + recipes = load_recipes() + print(f"{len(recipes)} recipes, {len(declared_models(recipes))} declared models") + if args.verify_models: + missing = verify_models(recipes) + if missing: + print("Declared models not found:", ", ".join(missing)) + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/run-on-jobs-template.html b/tools/run-on-jobs-template.html new file mode 100644 index 0000000..5fee821 --- /dev/null +++ b/tools/run-on-jobs-template.html @@ -0,0 +1,298 @@ + + + + + +Run on Jobs + + + + + +
    +

    Use Hub models on your own data

    +

    Run a model over your whole dataset, or fine-tune it on your own examples. Each command starts a job on Hugging Face Jobs and saves the results to the Hub. There is nothing to clone.

    +
    +
    +

    Read seven scanned NASA pages with GLM-OCR and save the text to a new public dataset on your account. Add --private to keep it private.

    hf jobs uv run \
+    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/glm-ocr.py \
+    uv-scripts/ocr-demo your-username/ocr-demo-results
    The scan
    Scanned page listing Space Shuttle foods
    What the Job wrote
    Space Shuttle Food and Beverage List
+
+Foods*
+
+Applesauce (T)
+Apricots, dried (IM)
+Asparagus (R)
+Bananas (FD)
+Beef almondine (R)
+Beef, corned (I) (T)
+Beef and gravy (T)
+Beef, ground w/pickle sauce (T)
+Beef jerky (IM)
+Beef patty (R)
+Beef, slices w/barbeque sauce (T)
+Beef steak (I) (T)
+Beef stroganoff w/noodles (R)
+Bread, seedless rye (I) (NF)
+Broccoli au gratin (R)
+Breakfast roll (I) (NF)
+Candy, Life Savers*, assorted flavors (NF)
+Cauliflower w/cheese (R)
+Cereal, bran flakes (R)
+Cereal, cornflakes (R)
+Cereal, granola (R)
+Cereal, granola w/blueberries (R)
+Cereal, granola w/raisins (R)
+Cheddar cheese spread (T)
+Chicken a la king (T)
+
+Chicken and noodles (R)
+Chicken and rice (R)
+Chili mac w/beef (R)
+Cookies, pecan (NF)
+Cookies, shortbread (NF)
+Crackers, graham (NF)
+Eggs, scrambled (R)
+Food bar, almond crunch (NF)
+Food bar, chocolate chip (NF)
+Food bar, granola (NF)
+Food bar, granola/raisin (NF)
+Food bar, peanut butter/granola (NF)
+Frankfurters (Vienna sausage) (T)
+Fruitcake (NF)
+Fruit cocktail (T)
+Green beans, french w/mushrooms (R)
+Green beans and broccoli (R)
+Ham (I) (T)
+Jam/Jelly (T)
+Macaroni and cheese (R)
+Meatballs w/barbeque sauce (T)
+Nuts, almonds (NF)
+Nuts, cashews (NF)
+Nuts, peanuts (NF)
+
+Peach ambrosia (R)
+Peaches, dried (IM)
+Peaches, (T)
+Peanut butter
+Pears (FD)
+Pears (T)
+Peas w/butter sauce (R)
+Pineapple, crushed (T)
+Pudding, butterscotch (T)
+Pudding, chocolate (R) (T)
+Pudding, lemon (T)
+Pudding, vanilla (R) (T)
+Rice pilaf (R)
+Salmon (T)
+Sausage patty (R)
+Shrimp creole (R)
+Shrimp cocktail (R)
+Soup, cream of mushroom (R)
+Spaghetti w/meatless sauce (R)
+Strawberries (R)
+Tomatoes, stewed (T)
+Tuna (T)
+Turkey and gravy (T)
+Turkey, smoked/sliced (I) (T)
+Turkey tetrazzini (R)
+Vegetables, mixed Italian (R)
+
+Beverages
+
+Apple drink
+Cocoa
+Coffee, black
+Coffee w/cream
+Coffee w/cream and sugar
+Coffee w/sugar
+Grape drink
+Grapefruit drink
+Instant breakfast, chocolate
+Instant breakfast, strawberry
+
+Instant breakfast, vanilla
+Lemonade
+Orange drink
+Orange-grapefruit drink
+Orange-pineapple drink
+Strawberry drink
+Tea
+Tea w/lemon and sugar
+Tea w/sugar
+Tropical punch
+
+Condiments
+
+Barbeque sauce
+Catsup
+Mustard
+Pepper
+Salt
+Hot pepper sauce
+Mayonnaise
+
+*Abbreviations in parentheses indicate type of food T = thermostabilized, I = irradiated, IM = intermediate moisture, FD = freeze dried, R = rehydratable, and NF = natural form.
+
+Shuttle Menu
+
+The Space Shuttle menu currently features more than 70 food items and 20 beverages. Shuttle travelers will have a varied menu every day for six days rather than the personal preference meals used in previous space missions. Each day, three meals are allowed, with a repeat of menus after six days. The pantry also provides plenty of foods for snacks and between meal beverages and for individual menu changes. The pantry also stores additional contingency food to last 96 hours.
+
+Food System Constraints
+
+The primary objective in designing a food system for the Space Shuttle is to provide food that is safe and nutritious, light in weight and compact, and is packaged in a convenient form that allows easy manipulation in the weightless environment of an orbiting spacecraft. To achieve this objective requires a careful consideration of three important factors: biological, operational, and engineering. Under each factor are many constraints that affect the final choice of food and how it is

    Our run: davanstrien/ocr-demo-results. 7 pages, 2.4 minutes of processing on a10g-small, after a few minutes of setup.

    +

    Transcribe four minutes of a NASA podcast, with speaker tags and word timestamps, into your own Bucket.

    hf buckets create your-username/transcripts --private
+hf jobs uv run --timeout 45m \
+    -v hf://buckets/uv-scripts/transcription-demo/podcast:/input:ro \
+    -v hf://buckets/your-username/transcripts/podcast:/output \
+    https://huggingface.co/datasets/uv-scripts/transcription/raw/main/nemotron-transcribe.py \
+    /input /output --timestamps
    <spk:0> Jim, thank you so much for coming on the podcast today. ... <spk:3> It is, uh, the mission control room, of course, is now open for business ...

    Our run: about 7.5 minutes on rtx-pro-6000, about $0.34. Most of that was model setup.

    +

    Embed the first 100 IMDb reviews and save them as a private dataset.

    hf jobs uv run --flavor l4x1 -s HF_TOKEN \
+    https://huggingface.co/datasets/uv-scripts/embeddings/raw/main/generate-embeddings.py \
+    stanfordnlp/imdb your-username/imdb-embeddings \
+    --column text --max-samples 100 --private

    One L4 GPU embeds about 900 rows per second with all-MiniLM-L6-v2.

    +

    Train a news topic classifier from 16 examples, with no GPU, and save the model to your account.

    hf jobs uv run --flavor cpu-basic --secrets HF_TOKEN \
+    https://huggingface.co/datasets/uv-scripts/classification/raw/main/train-setfit.py \
+    fancyzhx/ag_news your-username/ag-news-setfit \
+    --num-samples 4 --max-train-pool 2000 --max-eval-samples 100 \
+    --max-seq-length 128 --sampling-strategy undersampling --max-minutes 5

    Our run: 0.77 accuracy on 100 test articles, against 0.32 for always guessing the most common topic. Training took 20 seconds on cpu-basic.

    +
    +

    You need the hf CLI (1.32 or newer), a login and pay-as-you-go credit. Add --dry-run to see the GPU and settings without starting a job.

    +

    Find your model

    +

    Replace the capitalised placeholders and your-username. Check the script if you're unsure what a value should be.

    + + + +{{GROUPS}} +
    +

    Data tools with no model

      {{TOOLS}}
    +

    For agents

    +

    Read AGENTS.md and recipes.json.

    +
    +
    + + +