Skip to content

branch-4.2: [fix](nested column pruning) Keep sub-field predicate access paths - #68331

Draft
hubgeter wants to merge 1 commit into
apache:branch-4.2from
hubgeter:fix/variant-predicate-access-paths-branch-4.2
Draft

hubgeter wants to merge 1 commit into
apache:branch-4.2from
hubgeter:fix/variant-predicate-access-paths-branch-4.2

Conversation

@hubgeter

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: #68214

Problem Summary:

retainPredicatePathsInFinalAllAccessPaths, added by #68214, removes every predicate access path that is not literally one of the final all access paths. That is what NULL/OFFSET paths need, because they are stripped from the all paths on purpose, but it also removes ordinary sub-field paths whenever the all paths collapse to the whole-column path:

SELECT s FROM tbl WHERE struct_element(s, 'city') = 'x';
-- all access paths: [s], predicate access paths: [] (was [s.city])

BE then cannot tell which sub-column the predicate reads, so it loses the eager/lazy split and reads the column as one unit. On branch-4.2 this shows up in test_iceberg_variant_read, where the lazy-read check sees FilteredRowsByLazyRead = 0 for

SELECT CAST(v AS STRING) FROM variant_page_pruning WHERE CAST(v['n'] AS INT) > 3000

Only file scans hit this: on OLAP tables a variant sub-path predicate gets its own sub-column slot, so its all paths already contain the predicate path. That is why the existing unit tests did not catch it, and why the new test uses a struct column, which keeps one slot for the whole column.

This PR keeps the NULL/OFFSET cleanup, since BE switches the whole iterator to NULL_MAP_ONLY/OFFSET_ONLY once such a path shows up and skips the children. Every other predicate path is kept, and added to the all paths when no wider path covers it — the behaviour of master's addPredicatePathsToFinalAllAccessPaths.

Release note

Fix nested/variant sub-field predicates losing lazy materialization on branch-4.2 when the whole column is read.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes. Sub-field predicates keep their access paths, restoring predicate-first reads.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

…ess paths

apache#68214 added retainPredicatePathsInFinalAllAccessPaths, which removes every
predicate access path that is not literally one of the final all access
paths. That is right for NULL/OFFSET paths, which are stripped from the all
paths on purpose, but it also removes ordinary sub-field paths whenever the
all paths collapse to the whole-column path:

  SELECT s FROM tbl WHERE struct_element(s, 'city') = 'x'
  -- all: [s], predicate: [] (was [s.city])

BE then cannot tell which sub-column the predicate reads, loses the
eager/lazy split and reads the column as one unit. On branch-4.2 this shows
up in test_iceberg_variant_read as FilteredRowsByLazyRead = 0 for

  SELECT CAST(v AS STRING) FROM variant_page_pruning WHERE CAST(v['n'] AS INT) > 3000

Only file scans hit this: on OLAP tables a variant sub-path predicate gets
its own sub-column slot, which is why the existing tests did not catch it.

Keep the NULL/OFFSET cleanup, since BE switches the whole iterator to
NULL_MAP_ONLY/OFFSET_ONLY once such a path shows up. Any other predicate
path is kept, and added to the all paths when no wider path covers it, as
master's addPredicatePathsToFinalAllAccessPaths does.
@hubgeter
hubgeter requested a review from yiguolei as a code owner September 21, 2026 09:31
@hubgeter

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter
hubgeter marked this pull request as draft September 21, 2026 09:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants