Add inference_batch for sharing const-prop results across compilations - #917
Closed
AntonOresten wants to merge 1 commit into
Closed
Add inference_batch for sharing const-prop results across compilations#917AntonOresten wants to merge 1 commit into
inference_batch for sharing const-prop results across compilations#917AntonOresten wants to merge 1 commit into
Conversation
Author
|
Closing for now. Wasn't sold on the idea and it's very targetted |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Each compilation's interpreter starts with an empty local inference cache, where Julia keeps const-prop results for callees, which never reach the global code cache. A batch of compilations of one method with different constants (an autotuning sweep) re-derives them every time. cuTile worked around this by threading one cache through a sweep's interpreters via a package-local
ScopedValue(JuliaGPU/cuTile.jl@6dcda23): 9.7 → 2.4 ms per config.inference_batch(f)is that in GPUCompiler, bounded to where sharing is sound: interpreters constructed within the scope share a cache per task,cache_ownerand world. Outside a batch nothing changes;get_interpreteris untouched, and back-ends with their own interpreter can callinference_cache(owner, world).Toy measurement (24 const-propagated inlineable stages, a per-variant
@noinlinecallee, 25 variants): inference 8.8 → 4.9 ms per variant. End-to-endcompile(:llvm)is unchanged here, where LLVM dominates; the gain is where inference is the compile.Needs JuliaGPU/CompilerCaching.jl#24 to be effective: without it the callee walk clears the shared cache in place after any compilation with a source-less callee (measured: no gain). Adds ScopedValues.jl as a dependency (also in #915).
Made with Fable