Simplify and speed up param packer - #2904
Merged
Merged
Conversation
This simplifies the param packer and stays in Cython. The original motivation was a (honestly) minor speed regression, but this also speeds up the same cases by a large factor for many arguments (up to 10x). Many arguments are not too interesting, but also 16 have a 16% speedup at least. Some other optimizations are certainly plausible and may become more relevant as kernel launch time decreases.
This comment has been minimized.
This comment has been minimized.
mdboom
approved these changes
Sep 17, 2026
mdboom
left a comment
Contributor
There was a problem hiding this comment.
LGTM. My agent found no additional issues.
This is the kind of thing we would normally port to 12.9.x as well, if these patches apply cleanly there. The generated changes will automatically make it there eventually, but they seem independent enough that if the rest weren't ported it would be ok.
Contributor
Author
Hmmm, I suspect the generated changes would break without the rest unfortunately. But this should apply cleanly (enough), just means need to follow-up on that. |
Contributor
Author
|
Created a manual backport in gh-2911, hopefully that is enough to not worry about that! |
Contributor
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This simplifies the param packer and stays in Cython. The original motivation was a small regression found by the benchmarks. But this also speeds up the same cases by a large factor. For many arguments up to 10x.
That many arguments are not too interesting, but also 16 have a 16% speedup at least.
Some further optimizations are certainly plausible and may become more relevant as kernel launch time decreases. I limited it to only a few things:
isinstance()checksThere are a bunch of smaller things I could think of (but didn't do to keep this a more mechanical "move" and I am not sure how hot these are):
isinstance()checks mostly for exact type checksgetattr(type(value), 'getPtr', None)on the type (avoid binding and stricter contract).Benchmark
This really changes all except the last two lines (the last two pass an existing buffer or )
main(µs/launch)bench_launch_512_doublesbench_launch_512_intsbench_launch_512_boolsbench_launch_512_args(all-Nonetypes)bench_launch_16_argsbench_launch_512_args_pre_packedbench_launch_empty_kernelDetails
AMD Ryzen Threadripper PRO 7975WX / NVIDIA RTX PRO 6000 Blackwell (
sm_120),driver 580.173.02, CUDA 13.0, CPython 3.14.7 (GIL build), Cython 3.2.9,
g++ -O3. Both sides built through an identical pipeline from a detached-HEADworktree and the PR branch. 1500 launches per repetition, min of 5 repetitions,
7 rounds interleaved, median of per-round minima; process pinned to one core on
an otherwise idle machine.
The last two rows are controls:
bench_launch_empty_kernelpasseskernelParams=0andbench_launch_512_args_pre_packedsupplies analready-packed buffer, so neither reaches the packing path and both are
expected to be flat.