Pin ROCm TorchAO nightly separately to unbreak ROCm CI - #23264
Conversation
#23184 bumped TORCHAO_NIGHTLY_VERSION to 0.19.0.dev20260907, which has no rocm7.2 build, so both ROCm jobs fail with a 403 when fetching the TorchAO wheel. Pin ROCm to 0.19.0.dev20260805, the newest rocm7.2 nightly with compiled kernels, and keep the voxtral job's full ExecuTorch install from trying to satisfy the torchao>=0.19.0.dev20260907 bound.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23264
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 New Failure, 1 Pending, 3 Unrelated FailuresAs of commit 6389b0b with merge base 96c621d ( NEW FAILURE - The following job has failed:
BROKEN TRUNK - The following jobs failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
This PR needs a
|
This branch was successfully deployed
2 participants
Summary
Since #23184, every run of
unittest-rocm-gfx950-rocm7.2andtest-voxtral-realtime-rocm-gfx950-rocm7.2on main fails while installing TorchAO:(Green ROCm workflow runs on HUD since then are runs where the path filter skipped these jobs.)
Both ROCm scripts build the wheel URL from
TORCHAO_NIGHTLY_VERSION, but0.19.0.dev20260907has no rocm7.2 build. On the rocm7.2 nightly index, the lastcp310-abi3(compiled-kernel) wheel is0.19.0.dev20260805; later ROCm nightlies arepy3-none-anyonly, and nothing is published after0.19.0.dev20260831.Changes:
ROCM_TORCHAO_NIGHTLY_VERSION = "0.19.0.dev20260805"ininstall_requirements.pyand use it in both ROCm scripts.torchao>=0.19.0.dev20260907,<0.20requirement would otherwise make pip try to replace the ROCm wheel. Install ExecuTorch with--no-deps, then install its declared dependencies except torchao. The AOTI job usesEXECUTORCH_BUILD_MINIMAL=1, which has no torchao dependency, so it only needs the pin change.Follow-up: switch ROCm back to
TORCHAO_NIGHTLY_VERSIONonce TorchAO publishes rocm7.2 nightlies again.Test plan
torchao-0.19.0.dev20260805+rocm7.2-cp310-abi3-manylinux_2_28_x86_64.whlis downloadable and0.19.0.dev20260907+rocm7.2is not.bash -non both scripts; checked the dependency filter locally (drops torchao and non-matching markers/extras).