NEON-vectorized union2by2 - #538
Open
gitRasheed wants to merge 2 commits into
Open
Conversation
128-bit SIMD kernel for array-container unions, dispatched when both inputs have at least 384 elements; smaller inputs keep the scalar assembly. Based on CRoaring's union_vector16, with a transposed bitonic merge network (shorter dependency chain than the SSE rotate network, which measured slower than the scalar code on Neoverse cores), a two-multiply movemask replacement, and a skip over the merge network when the incoming block cannot interleave with the carried values. Differential tests cover the caller contracts: iorArray's aliased buffer, lazyorArray's zero-length buffer, block-boundary duplicates, and store overrun guards.
When the disjoint check passes, stay in a small loop that selects blocks with a predicted branch instead of returning to the main loop's CSEL select. On run-structured data the branch predicts and the untaken cursor stays off the loop-carried dependency chain; the first interleaving block merges and rejoins the main loop. Run-structured unions improve about 2.5x over the base kernel; random data never enters the loop.
gitRasheed
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
NEON implementation of
union2by2for arm64. When both arrays have at least 384 elements, unions run through a 128-bit SIMD kernel that processes eight values per step; smaller inputs keep using the scalar assembly from #287.The kernel is based on CRoaring's SSE
union_vector16, adapted for ARM: a transposed bitonic merge network (the SSE rotate network is latency-bound on modern ARM cores and measured slower than the scalar code), a two-multiply replacement for PMOVMSKB, and a fast path that skips the merge network while incoming blocks do not interleave with the carried values.Type of Change
Changes Made
What was changed?
setutil_neon_arm64.s: the NEON kernel.setutil_arm64.go: threshold dispatch, the shuffle table, and scalar tails that keep the existing caller contracts (iorArrayaliasing,lazyorArray's zero-length buffer).setutil_arm64.s: Union2by2 arm64 assembly #287's routine renamedunion2by2scalar, unchanged.BenchmarkUnion2By2, following thepopcnt_bench_test.golayout. It rotates 16 fixed-seed datasets; a single fixed pair lets the branch predictor memorize the scalar path and understates the difference.Why was it changed?
Closes #288.
How was it changed?
Hand-written NEON assembly in Go's assembler syntax plus a small Go wrapper, same approach as
popcnt_neon_arm64.s.Testing
go testpasses.Formatting
go fmtclean.Fuzzing
-fuzz=FuzzSmatcannot run as written: the function is referenced by this template, the README, and smat.go, but its definition is not in the tree (it seems to have been lost in the October 2025 fuzzer rework). I ran the smat state machine through a smallsmat.Fuzzwrapper instead: 300 seconds, 1,373,418 executions on Graviton 4, no failures. Happy to send the wrapper as a follow-up PR.Performance Impact
Measured on Graviton 4 (c8g.xlarge), Go 1.26.5, medians of repeated runs.
BenchmarkUnion2By2from this PR; scalar rows are the #287 code:2.5x to 4x on random and run-structured shapes, no allocation changes.
On the real-roaring-datasets corpus (
BENCH_REAL_DATA=1 go test -bench=BenchmarkRealDataFastOr), census1881 improves 1.25x. It is the one dataset dominated by large, mostly disjoint array unions, which is the shape this kernel targets. The other eleven datasets stay within a few percent of master; their unions are small, dense, or rare, so the kernel barely runs.Trade-off: when two arrays are nearly identical, the scalar loop consumes both elements on every equal pair and its branch becomes predictable, so it does about half the work. The kernel does the same fixed work per block regardless, so that one shape favors scalar. None of the existing datasets that I benchmarked on have this shape.
Breaking Changes
None. arm64 only; output is identical to the scalar path.
Related Issues
Closes #288.
Related to #287.