Repository navigation
Reuse deserialized segments across partial V2 scans - #10849
Draft
joseph-isaacs wants to merge 1 commit into
Draft
joseph-isaacs wants to merge 1 commit into
joseph-isaacs wants to merge 1 commit into
Conversation
Keep full encoded arrays in a per-file cache with four FIFO shards and a 16 MiB total weighted capacity. Partial scans can reuse deserialization and skip read requests on a hit; whole-segment scans bypass the cache. Decode, weigh, and release evicted arrays outside shard locks. Cover independent row selections, plan identity, disabled caching, failed reads, and eviction. Preserve existing dictionary-value reuse and update read-count expectations for overlapping chunks. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs
added
the
action/bench-sql
github-actions
Bot
removed
the
action/bench-sql
Merging this PR will improve performance by 12.79%4 warnings, 1 note
Performance Changes
Tip Curious why performance improved? Comment Comparing 
Footnotes
|
Contributor
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Contributor
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.981x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.006x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.985x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.991x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.008x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.994x ➖, 2↑ 1↓)
No file size changes detected. |
Contributor
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.992x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.097x ➖, 1↑ 3↓)
datafusion / parquet / ns (1.054x ➖, 0↑ 3↓)
duckdb / vortex-file-compressed / ns (1.017x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.056x ➖, 0↑ 3↓)
|
joseph-isaacs
added
the
changelog/performance
ChatGPT Codex Connector
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
V2 scan chunks that overlap the same stored segment can repeatedly deserialize its encoded array and issue another segment request. Reuse the full encoded array across partial scans of a file, then apply each reader's own slice and mask.
This is a focused follow-up rebased directly onto
developata59294e35e7109fac894a7a38a68f42db0c3c967. It contains only the segment-reuse change; #10799's unmerged changes and the completed dynamic-filter pushdown work are not included.Changes
Performance evidence before rebase
These are local DataFusion measurements against
02d9b085a27c965cb6f1f9b652062e262dd11de0from #10799, with eight workers/partitions and V2 enabled on both sides. They have not been rerun against currentdevelop.The focused comparison used three alternating rounds of nine executions, discarding the first two executions in each process. Broader comparisons used three rounds of seven executions with the same warm-up exclusion. ClickBench Q2's profile showed 7.2% less worker CPU and segment-source compute CPU falling from 3.29 s to 0.94 s over 200 executions.
Known tradeoffs: isolated confirmation found FineWeb Q7 about 3.7% slower and TPC-H Q6 about 1.0% slower. One TPC-DS SF10 Q9 memory comparison increased peak process RSS by about 51 MiB. Cold TPC-DS results remained inconclusive after 11 pairs. DuckDB and explicit segment-cache configurations remain unmeasured.
Validation
vortex-layoutlibrary tests passed, including five new regression cases; pinned formatting and whitespace checks passed.git range-diff; only conflict resolution was updated documentation terminology.git diff --check origin/develop...HEADpasses and the worktree is clean.