Add HuggingFace Jobs backend and adopt it as the integration-test substrate #118

Part of #108 · Phase 2 · Unblocks real testing for every other phase

Opportunity

HuggingFace Jobs has been verified working today for the contextlab org (academia plan). A real container ran and returned:

CLUSTRIX_HF_OK 3.12.14 x86_64

Available flavors: cpu-basic, cpu-upgrade, cpu-xl, t4-small, t4-medium, l4x1, l4x4, l40sx1, l40sx4, l40sx8, a10g-small, a10g-large, a10g-largex2, a10g-largex4, a100-large, h100, h100x8.

Setup notes for whoever picks this up: the token must be fine-grained with job.write scoped to the contextlab org, not just the user. A user-only scope yields 403 ... missing permissions: job.write on the org namespace; a personal-namespace attempt yields 402 Payment Required because the personal account is not Pro. (403 = token can't act here; 402 = token is fine, nobody's paying. Getting a 402 is progress.)

Two deliverables

1. Clustrix currently targets the wrong HF primitive

clustrix/cloud_providers/huggingface_spaces.py targets Spaces — long-lived web applications. It is also a stub in practice: it defines create_space(), but dispatch requires create_instance (executor_cloud.py:222), so it dies at executor_cloud.py:234 with NotImplementedError.

Jobs is exactly clustrix's execution model: hand over a container, run a function, collect a result, exit. Implement an HF Jobs backend — it is both easier to get right than Spaces and immediately useful as infrastructure.

• huggingface_jobs backend implementing the real provider interface
• Function serialization -> job payload -> result retrieval, end to end
• GPU flavor selection wired to the existing @cluster resource arguments
• Decide the fate of the Spaces provider: fix it to satisfy the interface, or remove it rather than ship a NotImplementedError

2. Adopt it as the default integration-test substrate

The scheduled real-world workflow has failed on all of its last 60 runs because it depends on ndoli.dartmouth.edu and tensor01.dartmouth.edu — hosts CI cannot reach (VPN, credentials, availability). That is why "real-world testing" has been theoretical.

HF Jobs needs no cluster reservation, no VPN, no institutional SSH credentials, and costs cents.

• Real-world CI job runs against HF Jobs on a schedule and on demand
• HF_TOKEN stored as a GitHub Actions secret
• Institutional-host tests remain, but as an opt-in tier that does not gate CI

Cost guardrails (required, not optional)

• Every CI-facing job pins --flavor cpu-basic and an explicit --timeout
• GPU flavors are manual/opt-in only; h100x8 on the academia plan is real money
• A hard ceiling on concurrent jobs, and teardown asserted in CI
• Spend is observable — a periodic check that fails loudly on anomaly

Why this matters strategically

The cloud path stayed broken for months because verifying it meant provisioning, IAM, billing, and teardown — so nobody did, and a first-line KeyError (utils.py:151 vs executor_cloud.py:390) went unnoticed. HF Jobs collapses that feedback loop to one CLI call. Choosing a test backend by how fast it fails loudly is worth more than choosing the one that most resembles production.