Skip to content

Add inference_batch for sharing const-prop results across compilations - #917

Closed
AntonOresten wants to merge 1 commit into
JuliaGPU:mainfrom
AntonOresten:inference_batch
Closed

Add inference_batch for sharing const-prop results across compilations#917
AntonOresten wants to merge 1 commit into
JuliaGPU:mainfrom
AntonOresten:inference_batch

Conversation

@AntonOresten

Copy link
Copy Markdown

Each compilation's interpreter starts with an empty local inference cache, where Julia keeps const-prop results for callees, which never reach the global code cache. A batch of compilations of one method with different constants (an autotuning sweep) re-derives them every time. cuTile worked around this by threading one cache through a sweep's interpreters via a package-local ScopedValue (JuliaGPU/cuTile.jl@6dcda23): 9.7 → 2.4 ms per config.

inference_batch(f) is that in GPUCompiler, bounded to where sharing is sound: interpreters constructed within the scope share a cache per task, cache_owner and world. Outside a batch nothing changes; get_interpreter is untouched, and back-ends with their own interpreter can call inference_cache(owner, world).

Toy measurement (24 const-propagated inlineable stages, a per-variant @noinline callee, 25 variants): inference 8.8 → 4.9 ms per variant. End-to-end compile(:llvm) is unchanged here, where LLVM dominates; the gain is where inference is the compile.

Needs JuliaGPU/CompilerCaching.jl#24 to be effective: without it the callee walk clears the shared cache in place after any compilation with a source-less callee (measured: no gain). Adds ScopedValues.jl as a dependency (also in #915).

Made with Fable

@AntonOresten

Copy link
Copy Markdown
Author

Closing for now. Wasn't sold on the idea and it's very targetted

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant