perf(runend): Scan short-run filter masks sequentially

[lwwmanning]

Summary

RunEnd filtering currently counts selected rows separately inside every run. For short runs, each BitBuffer::count_range call repeats alignment and range setup.

The kernel now scans the mask once with a sequential cursor when the average run length is at most 64 rows. Longer runs retain the existing range-popcount path. Sparse filters retain the existing direct-take path.

Changes

The benchmark compares production dispatch, direct take, the previous range scan, and the sequential scan. It covers production-width run ends, multiple array lengths, uniform and skewed runs, selectivity, clustered masks, and Dictionary values.

On an Apple M4 Max, the sequential path was 8% to 35% faster through an average run length of 64. It was neutral at 128 rows and 2% to 41% slower at 256 and 512 rows. Skewed runs showed the same crossover, so the threshold preserves the old path before the regression.

Conformance and oracle tests cover both branches, bitmap offsets, irregular runs, nulls, and sliced arrays. CodSpeed will provide the x86_64 comparison before this draft becomes ready.

🤖 Generated with OpenAI Codex

[codspeed-hq]

Merging this PR will regress 5 benchmarks

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 6 improved benchmarks
❌ 5 regressed benchmarks
✅ 2188 untouched benchmarks
🆕 84 new benchmarks
⏩ 206 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	filter_run_end[len=16384_run=16_density=0.1]	131.4 µs	175.2 µs	-24.98%
❌	Simulation	random_i16[0.95]	76.6 µs	94.7 µs	-19.12%
❌	Simulation	filter_run_end[len=16384_run=16_density=0.9]	130.8 µs	159.4 µs	-17.95%
❌	Simulation	filter_run_end[len=16384_run=16_density=0.5]	131.6 µs	159.4 µs	-17.46%
❌	Simulation	filter_run_end[len=4096_run=16_density=0.9]	42.8 µs	47.9 µs	-10.67%
⚡	WallTime	arrow_checked_add_u32_neon[16384]	20.5 µs	12.7 µs	+60.52%
⚡	Simulation	random_i8[0.5]	90.6 µs	67 µs	+35.25%
⚡	WallTime	arrow_checked_add_u32_avx2[16384]	21.4 µs	17.6 µs	+21.35%
⚡	Simulation	allocate_drop_arrow[0]	456.9 ns	402.7 ns	+13.45%
⚡	Simulation	allocate_drop_bytes[0]	520.2 ns	466 ns	+11.62%
⚡	WallTime	mul_u32_nonnull_avx512	6.3 µs	5.7 µs	+11.16%
🆕	WallTime	filter_materialized_range_neon[len128_run64_density50_primitive_offset0]	N/A	1.6 µs	N/A
🆕	WallTime	filter_materialized_range_neon[len4096_run64_density50_primitive_offset0]	N/A	2.7 µs	N/A
🆕	WallTime	filter_materialized_range_neon[len65536_run128_density50_dictionary_offset0]	N/A	14.8 µs	N/A
🆕	WallTime	filter_materialized_range_neon[len65536_run128_density50_primitive_offset0]	N/A	13.1 µs	N/A
🆕	WallTime	filter_materialized_range_neon[len65536_run16_density50_primitive_offset0]	N/A	91 µs	N/A
🆕	WallTime	filter_materialized_range_neon[len65536_run256_density50_primitive_offset0]	N/A	9.5 µs	N/A
🆕	WallTime	filter_materialized_range_neon[len65536_run32_density50_primitive_offset0]	N/A	38.5 µs	N/A
🆕	WallTime	filter_materialized_range_neon[len65536_run512_density50_primitive_offset0]	N/A	7.3 µs	N/A
🆕	WallTime	filter_materialized_range_neon[len65536_run64_density1_primitive_offset0]	N/A	14.2 µs	N/A
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing wm/runend-filtering (30869b8) with develop (265b705)}

Footnotes

1. 206 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[lwwmanning]

Closing this prototype because the benchmark results do not justify its complexity.

The sequential scan was 9–17% faster for 16–32-row runs on AArch64. The gain fell to 1–6% around 64-row runs. It became slower for longer runs. Results on AVX2 and AVX-512 were neutral or slower.

The implementation adds a second filter path and about 500 lines. We did not measure an improvement in an end-to-end workload. We will keep the branch as a record of the experiment.

[robert3005]

I think you want to adjust the compressor to give RLE edge despite bigger size