Skip to content

fix(index): push down deeply nested scalar filters - #8186

Merged
Xuanwo merged 1 commit into
mainfrom
gatekeeper/fix-8170-1
Aug 4, 2026
Merged

fix(index): push down deeply nested scalar filters#8186
Xuanwo merged 1 commit into
mainfrom
gatekeeper/fix-8170-1

Conversation

@lance-gatekeeper

Copy link
Copy Markdown
Contributor

Summary

  • consume every trailing string literal when extracting paths from variadic get_field expressions
  • preserve chained get_field support and canonical nested field formatting
  • cover the production planner path for a two-level nested scalar index filter

Root cause

DataFusion optimization combines deeply nested field access into one variadic get_field call. The scalar-index path extractor required exactly two arguments, returned no path for depth two or greater, and left the filter as a full-scan refinement.

Validation

  • cargo fmt --all -- --check
  • cargo test -p lance-index (925 passed, 2 ignored; 7 doctests passed)
  • cargo clippy --all --tests --benches -- -D warnings

Fixes #8170

@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer bug Something isn't working labels Aug 3, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The change fixes the optimizer-boundary mismatch at the narrow path-extraction boundary while preserving chained access, canonical nested-field formatting, and safe fallback for malformed expressions. This is preferable to broader planner normalization because it covers every expression producer without changing planner or format contracts.

@Xuanwo Xuanwo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this fix!

@Xuanwo
Xuanwo merged commit ff90523 into main Aug 4, 2026
40 checks passed
@Xuanwo
Xuanwo deleted the gatekeeper/fix-8170-1 branch August 4, 2026 07:30
@Xuanwo Xuanwo added the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: scalar index predicate pushdown silently fails for nested fields deeper than one level

1 participant