perf: preserve left functional dependencies for ASOF joins - #24799
Draft
Xuanwo wants to merge 4 commits into
Draft
perf: preserve left functional dependencies for ASOF joins#24799Xuanwo wants to merge 4 commits into
Xuanwo wants to merge 4 commits into
Conversation
github-actions
Bot
added
sql
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #24799 +/- ##
=========================================
Coverage 81.47% 81.48%
=========================================
Files 1122 1123 +1
Lines 403629 406506 +2877
Branches 403629 406506 +2877
=========================================
+ Hits 328866 331239 +2373
- Misses 55510 55880 +370
- Partials 19253 19387 +134 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
github-merge-queue Bot
pushed a commit
that referenced
this pull request
Sep 4, 2026
## Which issue does this PR close? - Part of #318. - Umbrella PR: #23738. - Depends on #23828 (merged). ## Rationale for this change This is the logical-planning layer of the ASOF JOIN stack. It defines the logical contract and planner behavior separately from the SQL frontend and serialization formats. #23828 is merged, so this PR's diff against `main` is the isolated logical layer. It no longer depends on the optional floating-point follow-up #24375. ## What changes are included in this PR? - Add `LogicalPlan::AsOfJoin`, `AsOfJoin`, and `AsOfMatch`. - Validate deterministic expressions, input ownership, supported match operators, equality-key types, and USING constraints. - Add `LogicalPlanBuilder` entry points and schema construction that preserves both qualified `USING` keys while exposing one unqualified wildcard key. - Integrate ASOF joins with tree transforms, display, type coercion, projection pruning, row bounds, and physical planning. - Plan logical ASOF joins to the broadcast-based `AsOfJoinExec` from #23828. - Fail closed at proto, SQL unparser, and Substrait boundaries until their owning stack layers add explicit support. - Defer ASOF-specific functional-dependency refinement to #24799 and filter pushdown to #24801 so each optimization can be reviewed independently. ## Are these changes tested? Yes: - `cargo fmt --all` - `cargo clippy --all-targets --all-features -- -D warnings` - `cargo test -p datafusion-expr min_rows_of_joins --all-features` - `cargo test -p datafusion-substrait asof_join_fails_closed_until_substrait_has_an_extension --all-features` - The extended workspace test command from the contributor guide ## Are there any user-facing changes? This adds logical-plan and builder APIs for ASOF joins. SQL syntax, DataFrame APIs, and plan serialization are intentionally left to dependent stack PRs. Floating equality keys remain rejected by the merged physical operator unless the independent follow-up #24375 is also included. As with any new public `LogicalPlan` variant, downstream exhaustive matches must add an arm. The variant is appended so existing variants retain their `PartialOrd` ordering; maintainers should still treat the enum addition as a Rust source-compatibility break. This PR can be reviewed independently now that #23828 has merged. The optimization follow-ups #24799 and #24801 are not required by the core ASOF stack.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
An ASOF join emits exactly one output row for every left row. Unlike a general
left join, it cannot duplicate left rows, so functional dependencies from the
left input keep their original dependency mode. Dependencies from the right
input are not preserved because one right row may match multiple left rows.
Keeping this metadata lets downstream optimizer rules continue to use left-side
uniqueness after an ASOF join.
GitHub shows the cumulative stack because fork branches cannot be selected as
upstream PR bases. The isolated diff for this layer is:
Xuanwo/datafusion@xuanwo/asof-logical...xuanwo/asof-functional-dependencies
What changes are included in this PR?
output schema.
Singledependency and rejectspropagation of the corresponding right dependency.
Are these changes tested?
cargo fmt --allcargo clippy --all-targets --all-features -- -D warningscargo test -p datafusion-expr asof_join_schema_preserves_only_left_dependencies --all-featuresAre there any user-facing changes?
No result semantics or public APIs change. This only improves optimizer metadata
for logical ASOF joins.