[branch-4.2][fix](zonemap) Do not trust a cut string bound to prune or to answer MIN/MAX (#67642) - #68324
Open
selectdb-robot wants to merge 1 commit into
Open
[branch-4.2][fix](zonemap) Do not trust a cut string bound to prune or to answer MIN/MAX (#67642)#68324selectdb-robot wants to merge 1 commit into
selectdb-robot wants to merge 1 commit into
Conversation
…MIN/MAX (apache#67642) 1. Write side: a max cut to 512 bytes was raised with str[511] += 1. A string column holds arbitrary bytes, so a last byte of 0xff wraps to 0x00 and leaves the max below the rows it covers — pruning then skips pages that do hold matching rows. The raise now carries into the preceding byte until one does not wrap. 2. Read side: a max of all 0xff carries past its first byte and ends up all zero, standing above nothing. ZoneMap::from_proto() spots that and turns pass_all on for the zone, giving up its range instead of ruling rows out with it — which also covers segments written before this fix. 3. MIN/MAX push-down: a cut bound is not a value the column holds (the min is a prefix, the max is that prefix raised), so segment_zone_maps_can_answer_agg() rejects a string bound reaching the 512-byte cut and reads the rows instead. The FE length blacklist is dropped with it — it was both too strict (a VARCHAR(65533) of short values was never pushed down) and too loose (a VARCHAR(512) filled to 512 bytes is cut just the same, yet was answered with a value never inserted). 4. Switch: enable_pushdown_string_minmax → force_pushdown_zonemap_minmax (old name kept as an alias), now meaning "force MIN/MAX onto the zone map even when its bound is not a value the data holds right now" — a cut bound, or one still covering rows a delete predicate removed. Statistics collection turns it on, every other query leaves it off. It applies to MIN/MAX only; COUNT and MIX keep the delete-predicate guard. The new thrift field defaults to false, so an old FE leaves BE behaving as before. (cherry picked from commit b0f266a) Adapt Schema and test writer APIs plus session-variable annotations to branch-4.2. Keep Thrift field ID 1006 without importing the unrelated master field 1005.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Related PR: #67642
Backport #67642 to
branch-4.2, cherry-picked fromb0f266a8ebc1a333f806831f5d2209f837fda5d5.Fix string ZoneMap bounds at the 512-byte cutoff: carry when raising the maximum, discard an all-zero overflowed range on read, and fall back to reading rows for exact MIN/MAX when bounds were truncated. Preserve the original statistics-collection override and the old SQL variable name as an alias.
Branch compatibility adaptations
segment.cpp: retain branch-4.2'sSchemasignature and column-ID iteration; useschema.column(schema.column_id(i))for the new string-type check. Add the original MIN/MAX forcing and fallback logic.SessionVariable.java: use branch-4.2'sVariableMgr.VarAttrwith Chinese/English descriptions, preserving the new variable name, old alias, default, and Thrift mapping.PaloInternalService.thrift: add onlyforce_pushdown_zonemap_minmaxwith the original field ID1006and defaultfalse. Do not import the unrelated1005SNII option from master conflict context.vgeneric_iterators_test.cpp: although this file auto-merged, adapt the added fixture to branch-4.2'sSchema/SchemaSPtr,SegmentWriterOptions,TestSegmentWriter, andfinalize()APIs. All original test cases remain.Validation
.outretains the original final blank separator, which defaultgit diff --checkreports as a blank line at EOF.TQueryOptionsfield IDs are unique.Release note
Fix incorrect string ZoneMap pruning and MIN/MAX results for truncated string bounds.
Check List (For Author)
Check List (For Reviewer who merge this PR)