Repository navigation
Reduce V2 scan filtering, planning, and repeated file reads - #10844
Draft
joseph-isaacs wants to merge 25 commits into
Draft
joseph-isaacs wants to merge 25 commits into
joseph-isaacs wants to merge 25 commits into
Conversation
…tion pieces Filtered V2 scans used fixed 1,048,576-row chunks. A DataFusion byte-range partition of a ~1M-row file then became a single chunk, so when the surviving rows sat in one partition (PolarSignals time ranges) one thread did all the work. Size filtered chunks by the serialized bytes the projection reads per row and by the partition's rows per thread, keeping 1M-row chunks for cheap projections over large ranges. ProjectionMorsel also wrapped its pieces in a ChunkedArray, which the Arrow export concatenated, re-merging nested dictionaries. Emit pieces as separate batches, grouping only pieces under 4,096 rows. PolarSignals SF1 on DataFusion, V2 vs V1 geomean: 1.79x -> 0.93x. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>
Exporting a non-dictionary Vortex array to an Arrow dictionary type went through arrow_cast, which hashes each primitive value with SipHash and builds an ArrowError per value. PolarSignals requests its nested `locations` fields as dictionaries, so this was ~90% of scan CPU for both V1 and V2. Encode primitive values directly, hashing their bits with FxHash and keeping nulls in the codes, and fall back to arrow_cast for other types or when the codes overflow. PolarSignals SF1 on DataFusion (V2): q00-q03 ~3.6x faster, total 4.28s -> 1.90s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>
…ictionaries Hash-map growth and hashing still dominated exporting PolarSignals' nested dictionary fields. Keys spanning under 65,536 values (line numbers) are now looked up in a dense table. When a sample of the values is nearly all distinct (addresses), the values become the dictionary with sequential codes, reusing their buffer, since Arrow dictionaries may hold duplicates. Other inputs hash into a map sized from the sample. PolarSignals SF1 on DataFusion (V2): q00-q03 another 22-31% faster. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>
A bit-packed array without patches only holds values in 0..2^bit_width, so its unpacked values are already codes into that range. Exporting it to an Arrow dictionary now unpacks straight into the codes type and uses 0..2^bit_width as the values, instead of decoding to the values type and dictionary-encoding the result. Applies up to 16-bit widths when the dictionary is no larger than the array. To unpack into narrower codes, BitPacked's cast kernel now also handles narrowing integer casts when the array has no patches and the target holds every value the bit width allows, skipping the generic bounds check. PolarSignals SF1 on DataFusion (V2): q00-q03 another ~2x faster. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>
…ange The primitive dictionary export scanned every key for its range before choosing an encoding, even for high-cardinality columns such as addresses that can never use the dense table. Sample first, and scan the full range only when the sampled keys fit the dense table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>
The run-end filter kernel called BitBuffer::count_range once per run. Each call sets up a SIMD popcount, which dominates when runs are a few rows long: TPC-DS store_sales columns average about 6 rows per run, and the kernel was the top CPU cost of TPC-DS q13 for both scans. Short runs now rank their ends in one pass over the mask's 64-bit words, one word popcount per run. Arrays whose runs average at least 256 rows keep one bulk popcount per run. run_end_filter benchmark: 4-row runs 300us -> 73us, 16-row runs 3.2us -> 1.2us, 1024-row runs 1.6us -> 0.8us. TPC-DS SF1 DataFusion geomean: V1 -3.6%, V2 -1.5%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>
A V2 scan reads each chunk through a session of its own, and the file's segment source shares a read only while it is in flight. A stored segment that spans a chunk boundary was therefore read again by every chunk that started after its neighbour had finished, and by every partition and every concurrent scan of the file. TPC-DS q28 on SF1 read 188 MB where V1 reads 114 MB, every segment V1 reads up to two and a half times as often. The file's ScanCache now retains up to 64 MiB of recently read host segments in least-recently-used order, and every scan of the file reads through it. ScanCache::with_retained_bytes sets the budget; zero disables it. q28 now reads 125 MB. TPC-DS SF1 on DataFusion, V2 vs V1 geomean: 1.052 -> 1.002. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>
TPC-H already defaults to the planner-driven V2 scan. Extend the default to TPC-DS and ClickBench so the SQL benchmarks measure V2 on them too. `--scan-v2 false` or `VORTEX_SCAN_V2=0` still selects V1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com>
Avoid expanding whole structs for single-field predicates and evaluate infallible cross-column OR predicates before applying sparse selections. Increase only unfiltered scan and output batch limits. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs
added
the
action/bench-sql
ChatGPT Codex Connector
Keep infallible OR predicate results in the original row domain and intersect them directly with the selection. Remove the now-redundant physical filter hint. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Share segment sources and scan caches across concurrent scans from one format and object store instance. Key file state by full object metadata and retain weak entries so plans and segment bytes are released when scans finish. Keep custom reader factories isolated. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Detect simple single-field expressions without allocating per-node annotation maps. Reuse the expanded expression annotations for multi-field partitioning. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs
changed the title
Cache a lazy code filter before splitting a dictionary to align packed columns, avoiding repeated filter execution for each output slice. Preserve dictionary values and cover sparse selections with nullable codes and values. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Key scan projections, filters, and byte estimates by expression structure without hashing lazy dtypes. DataFusion binds a fresh tree per partition, so identity keys miss equivalent expressions. Exercise zone-proof reuse with independently bound filters and projections. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Align proportional partition boundaries to 65536 rows, preserving complete disjoint coverage and exact file endpoints. This reduces repeated planning and fragmented small-table outputs. Compared V1 and V2 across ten TPC-DS queries at SF1, SF10 and SF30, using three rounds of nine iterations. Q28 remains faster and Q49 approaches V1 at SF10 and beats it at SF30. Q41 and Q58 remain open regressions. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Cache lazy filtered values before splitting them across packed columns, avoiding repeated filtering of the same selected rows. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Detect data and row-index references before partitioning. Data-only, row-index-only, and constant expressions retain their existing plans without allocating annotations for every expression node. Mixed expressions use the existing partitioner. SF10/SF30 Q41 profiles reduce worker CPU by 3.8%/2.7%; paired latency improves from 16.49/15.71 ms to 16.09/14.69 ms. Q28 guard remains faster than V1. Unit tests, lint, and formatting were not run. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Resolve conflicts with the V2 scan composition refactor (#10708): port adaptive filtered chunk sizing and retained segments onto ScanBuilder::prepare()/PreparedScan, adopt develop's pruning terminology and public planner API, and move ScanRun-based tests onto prepare(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBwd35JJjjdEU17qcW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBwd35JJjjdEU17qcW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBwd35JJjjdEU17qcW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs
removed
the
action/bench-sql
joseph-isaacs
added
the
action/bench-sql
Claude
github-actions
Bot
removed
the
action/bench-sql
Merging this PR will degrade performance by 3.73%2 warnings
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing 
Footnotes
|
Contributor
Polar Signals Profiling ResultsLatest Run
Previous Runs (14)
Powered by Polar Signals Cloud |
Contributor
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.005x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.011x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.991x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.885x ✅, 8↑ 1↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.961x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.960x ➖, 5↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.006x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.006x ➖, 4↑ 5↓)
datafusion / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.010x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.067x ➖, 1↑ 2↓)
datafusion / parquet / ns (1.048x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (0.972x ➖, 2↑ 1↓)
duckdb / parquet / ns (1.009x ➖, 0↑ 0↓)
|
Keep dictionary caches scoped to live file streams, reuse scan planning and filter work, and preserve sparse string dictionary decoding. Exclude inherited Arrow export and run-end kernel experiments. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs
changed the title
joseph-isaacs
marked this pull request as ready for review
October 27, 2026 17:19
joseph-isaacs
added
the
action/bench-sql
ChatGPT Codex Connector
github-actions
Bot
removed
the
action/bench-sql
joseph-isaacs
added
the
changelog/performance
ChatGPT Codex Connector
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs
added
the
action/bench-sql
ChatGPT Codex Connector
github-actions
Bot
removed
the
action/bench-sql
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Use the execution value cache during segment discovery, preserving dictionary reuse across splits. Check the optimized childless empty projection directly. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs
added
the
action/bench-sql
ChatGPT Codex Connector
github-actions
Bot
removed
the
action/bench-sql
joseph-isaacs
marked this pull request as draft
October 27, 2026 18:02
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
V2 repeats predicate and planning work across scan batches and can reread segments across overlapping file streams. This change shares that work and keeps sparse string projections from decoding an entire dictionary.
The diff is scoped to scanning, DataFusion integration, and that benchmark default. Inherited Arrow export and run-end kernel experiments were removed. No mmap or Arrow coalescer dependency patch is included. The latest 1,024-code selection-cap experiment was excluded after a Q41 regression.
Measured pre-merge evidence: focused comparison measured Q28 at SF10 523.996 → 250.270 ms (52% faster) and SF30 1,555.519 → 699.656 ms (55% faster), with eight workers, rotating comparisons, and warm-cache non-mmap reads. Full-suite comparison confirmed SF10 Q28 at 528.097 → 249.643 ms. Those binaries predate this develop merge and scope cleanup; their timings are not measurements of the current head.
All-query parity is not established. Repeated gaps include TPC-DS Q58 and ClickBench Q24/Q26/Q28. Q58 profiles and a separate diagnostic point to downstream Arrow join-output coalescing; that change is outside this PR.
Validation: CI on b0caf5a4 passed (29 jobs succeeded, one skipped). The ARM workspace run passed 12,052 tests, with 685 skipped, including dictionary-value reuse between splits, file-cache lifetime, and empty-projection pruning. Formatting used nightly-2026-09-10; git diff --check passed. Local compilation was unavailable because the pinned Rust toolchain was not installed and /opt/rustup was not writable. Merged develop 088118b and resolved conflicts.
Segment discovery now consults the execution value cache: prefetch must not reread dictionary values already retained by a live file scan. The existing dictionary read-count regression failed before this fix and passed after it. The focused comparison measures this fix against V1 and preceding V2 at SF10/SF30 and on ClickBench, including IO metrics and CPU profiles. SF10 and SF30 completed with valid provenance and coverage. Current-head Q28 improved from 540.270 to 250.542 ms at SF10 (53.6%) and 1,617.646 to 701.101 ms at SF30 (56.7%), without mmap. SF10 Q28 CPU time fell from 16,665 to 9,119 ms across 20 queries. Separate metric captures show output batches falling from 2,010 to 410 and reads from 2,311 to 145. Q58 remains 5.7% slower at SF10 and 6.2% slower at SF30; SF30 Q32 remains 2.8% slower. The focused ClickBench comparison also completed: Q22 1,933.097 → 1,579.578 ms (18.3% lower), Q24 136.569 → 143.428 ms (+5.0%), Q26 138.577 → 146.015 ms (+5.4%), and Q28 15,639.255 → 15,825.604 ms (+1.2%). Q24/Q26 CPU profiles show extra bit counting and compressed filtering; the prefetch fix improves the preceding V2 slightly but does not remove these gaps. The current-head Q58 diagnostic completed both scales: SF10 Q58 637.898 → 661.598 ms (+3.7%), SF30 1,732.919 → 1,837.185 ms (+6.0%). Across 20 SF30 queries, total sampled CPU increased from 66.70 to 70.31 seconds, while inclusive join-output coalescing increased from 1.23 to 6.83 seconds; string copying accounts for 0.85 versus 5.26 seconds within that coalescing time. These inclusive categories overlap and must not be summed. This revalidates the downstream batching interaction on the current head. The same run measured SF30 Q28 at 1,577.424 → 699.435 ms (55.7% lower elapsed time).
The requested SQL-label benchmark run also passed, including its build and all scheduled SQL benchmark jobs. The workflow removes the SQL label after triggering.
The post-merge full-suite comparison uses the same 83eadcb binary for V1/V2, before the prefetch fix. SF1 completed all 99 queries and 43 repeated guards without missing or failed measurements. Repeated Q28 improved from 65.499 to 45.568 ms (30.4%). Q76 remains 5.5% slower, and Q38/Q57/Q22/Q49 trail by 0.4–1.2%. ClickBench also completed all 43 queries and 26 repeated guards with no missing or invalid measurements. Repeated gaps are Q24 +4.2%, Q26 +4.3%, Q13 +1.1%, Q28 +0.7%, and Q0 +0.06%. SF10 completed all 99 queries and 43 repeated guards without missing, invalid, or failed measurements. Repeated gaps are Q58 +3.8%, Q99 +2.2%, and Q32 +0.4%; the initial Q20 +4.0% sweep gap did not repeat. Q72 was 1.1% slower in its single long-query sample, which is not a repeated warm median. SF30 also completed all 99 queries and 43 repeated guards without missing, invalid, or failed measurements. Repeated gaps are Q58 +5.0%, Q22 +1.7%, Q16 +0.9%, Q32 +0.8%, and Q47 +0.2%. These pre-prefetch-fix results do not establish parity for the latest head.
The current-head Q99 follow-up did not reproduce the earlier full-suite gap: SF10 235.844 → 234.615 ms (-0.5%) and SF30 531.117 → 522.692 ms (-1.6%). Sampled CPU across 100 queries fell by 1.9% and 2.5%, respectively. Timing variation across rounds is larger than these small elapsed differences, so this supports no repeatable Q99 regression in these runs rather than a new optimization claim.
A separate cast-filter sharing candidate completed targeted tests and SF10/SF30/ClickBench comparisons. It is not included in this PR: ClickBench Q24/Q26 improved only 1.0%/0.5% versus this head, while Q40 regressed 3.3%; SF30 Q41 also showed a 3.4% median regression with overlapping rounds. The evidence is insufficient to promote it. All-query parity remains unresolved.