Repository navigation
feat(bench): hold the ROCm guard lock for a whole gate or session - #2249
Merged
Merged
Conversation
Units starved on rocm_gpu_guard.sh because every guarded run, and every unguarded verify-rocm, competed separately for idle windows on the shared host. rocm_gpu_guard.sh --hold takes the lock through the existing acquire_lock (same --max-wait, exit 75), skips the idle wait and monitor, exports ROCM_GPU_GUARD_LOCK_HELD and runs the command in the foreground, so guards inside keep their own idle wait and monitor but take no lock. --idle-secs, --max-attempts and --log are rejected with --hold. --status reports held or free from a ROCM_GPU_GUARD_LOCK.holder sidecar that every holder writes on acquire and removes on exit (stale pids are flagged). make verify-rocm-held builds unlocked and then runs verify-rocm under one hold. Existing guard behavior is unchanged. Validated with the GuardTests additions (28 guard tests) and make verify-python-tooling verify-versions verify-fmt, plus a demo on a private lock. Refs #2244.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
rocm_gpu_guard.sh --hold [--max-wait SECS] -- COMMANDtakes the host-wide lock through the existingacquire_lock(exit 75 on--max-wait), skips the idle wait and monitor, exportsROCM_GPU_GUARD_LOCK_HELDand runs COMMAND in the foreground with its status passed back. Guards inside keep their idle wait and monitor but take no lock.--idle-secs,--max-attemptsand--logare rejected (exit 2); under an outer guard it just runs COMMAND; INT and TERM give 130 and 143.--statusprints held or free (exit 0 free, 1 held) and the holder from$LOCK.holder(pid, start, mode, command), written by every holder after it takes the lock and removed on exit; a dead pid is reported as stale.make verify-rocm-heldbuilds the test, release and clippy artifacts without the lock, then runsmake verify-rocmunder one hold.verify-rocmis unchanged.docs/benchmarks.mdgains "Sessions on a shared host". FIFO fairness stays out of scope as the issue says.Verification
Run on gfx1151, private lock and private KFD dir, never the host lock: 11 new
GuardTests(28 guard tests pass),make verify-python-tooling verify-versions verify-fmt,cargo test --features rocm --test dead_doc_pointers, and a hand demo of--holdand--statusagainst a private lock.make verify-rocm-heldandmake verify-rocmwere not run, as instructed for this unit; the orchestrator's gate run covers the end-to-end acceptance item. shellcheck is not installed here (bash -npasses).Not verified: Metal and CUDA are not available on this host; nothing on those paths changes.
Closes #2244