[core] Fix file index pushdown for bitmap64 deletion vectors - #9141
[core] Fix file index pushdown for bitmap64 deletion vectors#9141QuakeWang wants to merge 3 commits into
Conversation
File index limit and TopN selections only excluded 32-bit deletion vectors, so bitmap64 deletions could reduce pushed-down results after reading. Large files could also force unsupported 32-bit position selections. Project bounded bitmap64 positions for file indexes, fall back when positions cannot be represented, and enforce limits after deletion-vector filtering. Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Resolve the FileIndexEvaluator overlap with apache#9088 and preserve file metadata through limit readers. Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
| RoaringBitmap32 selection = new RoaringBitmap32(); | ||
| long position = 0; | ||
| int remaining = limit; | ||
| while (remaining > 0 && position < file.rowCount()) { |
There was a problem hiding this comment.
This eagerly scans physical positions one by one until it finds limit live rows. With a bitmap64 deletion vector containing a long deleted prefix, even LIMIT 1 can perform millions of isDeleted lookups before the reader is opened; for a >32-bit file it can scan as far as Integer.MAX_VALUE before falling back. Could we use the projected bitmap plus andNot(...).limit(limit) within the 32-bit bound, and return REMAIN outside it? The final LimitRecordReader already preserves correctness on fallback.
There was a problem hiding this comment.
Fixed. For files within the 32-bit bound, the limit selection now projects the bitmap64 DV, applies andNot, and then limits the result. Larger files return REMAIN and rely on the final limit after DV filtering. I also added tests to ensure this path does not scan positions one by one.
| return new BitmapIndexResult( | ||
| () -> { | ||
| RoaringBitmap32 result = new RoaringBitmap32(); | ||
| Iterator<Integer> iterator = candidates.get().iterator(); |
There was a problem hiding this comment.
This rebuilds the result one candidate at a time. A low-selectivity predicate that matches nearly every row turns reader creation into O(row count) Java-level lookups and insertions while retaining both bitmaps. Could we use projectToBitmap32 followed by a bitmap andNot for broad candidate sets, keeping point checks only for sparse results, for example via a cardinality-based choice?
There was a problem hiding this comment.
Updated this as well. Sparse candidates still use point checks, while results larger than 4096 entries use projectToBitmap32 followed by bitmap andNot. The cutoff matches Roaring’s array-container limit. Added coverage to ensure broad candidates do not call isDeleted one by one.
Use bitmap projection for limit selection and broad filter candidates, while preserving point checks for sparse results. Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
|
I feel the changes are a bit too extensive; hardly anyone uses 64-bit bitmaps in a production environment—they are primarily used to generate Iceberg-compatible tables from primary key tables. |
@JingsongLi Thanks, I agree that the current scope has grown beyond the original correctness issue. I’ll simplify the PR to use a conservative fallback whenever Bitmap64 would require position-based file-index pushdown, instead of adding Bitmap64-specific projection and optimization logic. I’ll retain only the minimal post-DV LIMIT safeguard needed for correctness and focused regression tests for the original under-return issue and large positions. This should also remove the cardinality heuristic and the Bitmap64-specific filter/TopN optimizations. Would this reduced scope match your expectation? |
2 participants
Purpose
FileIndexEvaluatoronly excluded 32-bit deletion vectors when building limit and TopN selections. With bitmap64 deletion vectors, pushed-down selections could include deleted rows and return fewer rows than requested. Large files could also require positions that cannot be represented by 32-bit bitmaps.This change adds bounded bitmap64 projection with a safe fallback, filters deletion-vector positions during file-index evaluation, and reapplies the final limit after deletion-vector filtering.
Tests
paimon-coretests, Checkstyle, Spotless, Enforcer, and RAT.