Conversation
…river The buffer load check is on by default and makes every profiled launch call the real jit_fn.warmup first. On CPU-only hosts Triton has no backend driver, so warmup fails with "0 active drivers ([]). There should only be one." and the launch never runs. Probe triton's driver.active in Profiler.pre_warmup_callback; when no driver is available, skip warmup, disable the buffer load check for that profiler, and say so in the finalize report instead of failing or reporting a spurious buffer load issue.
Performance Benchmark
Iterations: 1 warmup + 20 measured |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On CPU-only hosts, every launch traced with the Profiler failed before the kernel ran:
The buffer load check is enabled by default (
PROFILER_DISABLE_BUFFER_LOAD_CHECK=0), soProfiler.pre_warmup_callbackasksTritonTrace.runto call the realjit_fn.warmupbefore each launch. Warmup compiles the kernel throughtriton.runtime.driver.active, which raises when no backend driver is available. The Profiler is the only client that requests warmup. The existing profiler tests all setprofiler_disable_buffer_load_check = True, so CPU CI never ran this path.Changes
Profiler.pre_warmup_callbackchecksdriver.activefirst. With no driver, it skips warmup and turns off the buffer load check for that profiler, so the launch runs normally.Skipped: no active Triton driver to compile the kernel (e.g. CPU-only host).instead of dropping the section or flagging a buffer load issue without any ASM to check.Tests
test_buffer_load_check_without_active_driverintests/end_to_end/test_profiler.py. It monkeypatches Triton's driver factory to simulate a driverless host, so it also covers this path on GPU machines. Without the fix it fails with the same0 active driverserror.pytest tests/end_to_end/test_profiler.py tests/unit/test_profiler.py: 24 passed.CUDA_VISIBLE_DEVICES=""and a masked add kernel under@tilelens.trace("profiler"): it used to fail with0 active driversand now completes with the skip note. With the GPU visible (RTX 4090), the output is the same as before.pre-commit(ruff, ruff-format, mypy, codespell, …) passes on the changed files.