Skip to content

fix: declare Hive partitioned scans as Range - #25279

Open
gstamatakis95 wants to merge 1 commit into
apache:mainfrom
gstamatakis95:issue-23436-hive-range-partitioning
Open

gstamatakis95 wants to merge 1 commit into
apache:mainfrom
gstamatakis95:issue-23436-hive-range-partitioning

Conversation

@gstamatakis95

@gstamatakis95 gstamatakis95 commented Sep 13, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

  • Hive grouped scans declare Partitioning::Hash, but no hash ever ran.
  • The optimizer treats any two Hash partitionings with equal counts as co-partitioned, so it skips the repartition.
  • Partitioned hash joins then drop rows. Fact {A, B, C} joined to dimension {A, C, D} returns only A, because fact group 1 holds B while dimension group 1 holds C.
  • Joining a Hive scan to a VALUES list also loses rows, since the planner hash repartitions the other side.

What changes are included in this PR?

  • Add FileGroup::group_by_partition_values_with_split_points, returning groups plus the SplitPoints between them.
  • Merge values into contiguous chunks rather than round robin. Only contiguous chunks can be described by split points. Group counts are unchanged.
  • Add range_partitioning_from_partition_fields and deprecate output_partitioning_from_partition_fields.
  • Give ListFilesResult a partition_split_points field and declare Range from ListingTable::scan.
  • Declare nothing when split_file_groups_by_statistics re-cuts the groups. Comparing counts misses this, since the re-cut can return the same number.
  • Drop the dynamic filter guard in HashJoinExec since itt existed only because Hive groups reported Hash.

What is the testing strategy for this PR?

  • TEST 14 and 15 in preserve_file_partitioning.slt cover the two failures above.
  • TEST 16 self joins the merged high cardinality table. TEST 17 asserts the statistics re-cut declares nothing.
  • Reverting only table.rs makes TEST 14 return one row instead of two and breaks TEST 12.
  • Six unit tests route every file through its split points and assert it lands in its own group, covering NULLs, compound keys, a NULL inside a split point, and unordered split points.
  • Two datafusion-proto round trips cover Dictionary(UInt16, Utf8) and Utf8View split points.

Are there any user-facing changes?

  • EXPLAIN prints Range([col ASC], [(v1), (v2)], N) instead of Hash([col], N).
  • Joins between Hive tables with different values gain a RepartitionExec(Range) rather than returning wrong results.
  • Partitioned dynamic filter pushdown works again under preserve_file_partitions.
  • File group membership changes when distinct values exceed target_partitions. Byte balance can shift on date partitions with larger recent days. Set preserve_file_partitions = 0 to restore the old behaviour.
  • API break: ListFilesResult gains a public field. Add partition_split_points: vec![] or use ...
  • Providers declaring Hash over value grouped files should move to the new function. They already return missing rows today when the other join input is hash repartitioned.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) catalog Related to the catalog crate proto Related to proto crate datasource Changes to the datasource crate physical-plan Changes to the physical-plan crate labels Sep 13, 2026
@gstamatakis95 gstamatakis95 changed the title fix: declare Hive partitioned scans as Range instead of Hash partitio… fix: declare Hive partitioned scans as Range Sep 13, 2026
@gstamatakis95
gstamatakis95 marked this pull request as ready for review September 14, 2026 21:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

catalog Related to the catalog crate datasource Changes to the datasource crate physical-plan Changes to the physical-plan crate proto Related to proto crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Represent Hive/value partitioning without claiming Partitioning::Hash

1 participant