Repository navigation
Unwrap top level slice and filter in duckdbexport - #10624
Open
robert3005 wants to merge 4 commits into
Open
robert3005 wants to merge 4 commits into
robert3005 wants to merge 4 commits into
Conversation
Merging this PR will improve performance by 12.78%4 warnings, 1 note
Performance Changes
Tip Curious why performance improved? Comment Comparing 
Footnotes
|
myrrc
added
the
action/bench-sql-extended
github-actions
Bot
removed
the
action/bench-sql-extended
myrrc
added
changelog/performance
Contributor
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Contributor
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.088x ➖, 0↑ 2↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.112x ❌, 3↑ 5↓)
duckdb / vortex-compact / ns (0.917x ➖, 2↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.004x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.006x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (0.989x ➖, 2↑ 1↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.004x ➖, 0↑ 2↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.924x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.940x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.122x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (0.906x ➖, 1↑ 0↓)
duckdb / vortex-compact / ns (0.955x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.088x ➖, 0↑ 1↓)
|
Contributor
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.979x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.978x ➖, 2↑ 0↓)
duckdb / parquet / ns (0.984x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.023x ➖, 0↑ 1↓)
File Size Changes (201 files changed, +0.0% overall, 98↑ 103↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.995x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.989x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.993x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Contributor
Benchmarks: JOB on NVME 📖Commits: PR No baseline is available for this benchmark yet; PR measurements are shown without comparison. duckdb / vortex-file-compressed / ns (no group data, 0↑ 0↓)
duckdb / parquet / ns (no group data, 0↑ 0↓)
duckdb / duckdb / ns (no group data, 0↑ 0↓)
No baseline file sizes found for base commit. |
Contributor
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.957x ➖, 2↑ 0↓)
datafusion / vortex-compact / ns (1.022x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.993x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.982x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 3↑ 2↓)
duckdb / parquet / ns (1.017x ➖, 0↑ 3↓)
duckdb / duckdb / ns (1.003x ➖, 0↑ 1↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 1↑ 2↓)
duckdb / vortex-compact / ns (0.998x ➖, 3↑ 1↓)
duckdb / parquet / ns (0.991x ➖, 4↑ 2↓)
duckdb / duckdb / ns (1.001x ➖, 2↑ 6↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.015x ➖, 0↑ 2↓)
datafusion / vortex-compact / ns (0.991x ➖, 1↑ 2↓)
datafusion / parquet / ns (0.950x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed / ns (1.045x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (0.909x ➖, 2↑ 0↓)
duckdb / parquet / ns (0.898x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Vortex queries 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.016x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.008x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.010x ➖, 0↑ 0↓)
No file size changes detected. |
robert3005
changed the title
Contributor
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.068x ➖, 2↑ 4↓)
datafusion / vortex-compact / ns (1.048x ➖, 1↑ 2↓)
datafusion / parquet / ns (1.019x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.961x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.962x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.963x ➖, 0↑ 0↓)
|
Splits of 65,536 rows over 131,072-row run-end chunks wrapped each chunk in a lazy slice. The rule that evaluates scalar functions on run-end values only matches a bare run-end array, and optimizing cannot remove the slice because run-end slicing is an execute-time kernel that reads the run ends. So every filter and the export decoded the whole column, four times per chunk for cs_order_number in TPC-DS q16. Add ArrayRef::execute_slice, which runs the slice kernel of the sliced encoding and optimizes the result, and use it in the flat reader. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011JxDQs8L3TZ6VhZV9Bt6cr (cherry picked from commit 5b0caae)
A scan filters each projected column lazily, so the exporter received a filter over a run-end array and decoded it through the canonical path, although the run-end filter kernel keeps the runs and the exporter can export runs as they are. Run the kernels of lazy filters and slices before choosing an exporter. In TPC-DS q16 this exported cs_order_number as runs, saving 8% of the query's CPU. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011JxDQs8L3TZ6VhZV9Bt6cr (cherry picked from commit 2a0defd)
Exporting resolves lazy slices now, so a slice of a run-length array is exported as runs like the unsliced array, as a dictionary vector. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011JxDQs8L3TZ6VhZV9Bt6cr (cherry picked from commit a5c3155)
robert3005
force-pushed
the
rk/duckdb-exporter
branch
from
October 26, 2026 23:10
3483279 to
16996b9
Compare
Resolve lazy filters and slices before choosing an exporter with execute_until instead of a hand-written loop, and slice flat reader chunks lazily again, which removes ArrayRef::execute_slice. Copy validity bits from the bitmap's own offset when the bitmap cannot be shared with DuckDB. A bitmap that starts inside its first byte was read from the wrong bit. Signed-off-by: Robert Kruszewski <github@robertk.io> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
robert3005
force-pushed
the
rk/duckdb-exporter
branch
from
October 26, 2026 23:26
16996b9 to
d89448d
Compare
robert3005
changed the title
Contributor
|
@robert3005 I don't see improvements in our benchmarks |
Contributor
Author
|
I was investigating this pr yesterday that's why I removed a bunch of code. From my benchmarking it should help tpc-ds sf=10. but looks like fineweb is sad for some reason |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.