Skip to content

fix: write each batch's partitions in first-row order for the clustered writer - #81

Open
CaptainAni187 wants to merge 1 commit into
apache:mainfrom
CaptainAni187:fix/clustered-split-order-39
Open

CaptainAni187 wants to merge 1 commit into
apache:mainfrom
CaptainAni187:fix/clustered-split-order-39

Conversation

@CaptainAni187

Copy link
Copy Markdown

Closes #39

With fanout disabled, TaskWriter passes the pieces from RecordBatchPartitionSplitter::split straight to ClusteredWriter, but the splitter groups rows in a HashMap, so they come out in arbitrary order. When a partition spans a batch boundary, ClusteredWriter can see it closed and then reopened, and fails with "Cannot write to partition that was previously closed" even though the input is sorted.

This went with the TaskWriter side of the two options in the issue, so no iceberg-rust change is needed. On the clustered path the pieces are now sorted by the first row their partition value appears in the batch. The values come from the batch's _partition column through arrow_struct_to_literal, the same conversion the splitter uses, so the grouping matches. Fanout and unpartitioned writes are unchanged.

This doesn't touch #38: the optimizer can still drop the sort before the writer, so fanout-disabled inserts through SQL need that one too.

Test: test_task_writer_clustered_partition_spans_batches writes 50 sorted partitions of 3 rows in batches of 8, so most batches split a partition. Without the fix it fails every run with the error from the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Clustered (fanout disabled) writes fail on sorted input because batches are split in HashMap order

1 participant