Skip to content

Honor identity sort orders for Arrow table writes - #3830

Open
adonm wants to merge 1 commit into
apache:mainfrom
adonm:feat/identity-sorted-writes
Open

Honor identity sort orders for Arrow table writes#3830
adonm wants to merge 1 commit into
apache:mainfrom
adonm:feat/identity-sorted-writes

Conversation

@adonm

@adonm adonm commented Aug 21, 2026

Copy link
Copy Markdown

Related: #271

What changes?

Honors table sort orders for materialized pyarrow.Table writes when every sort field uses an identity transform and one consistent null placement:

  • sorts unpartitioned tables before bin packing
  • sorts each final Iceberg partition independently before bin packing
  • carries the table sort-order ID through WriteTask
  • writes the truthful DataFile.sort_order_id

Unsupported transforms, nested/missing fields, mixed null placement, and streaming RecordBatchReader writes preserve current behavior: data is not claimed as sorted and the file sort-order ID remains null. A warning explains why.

Why?

PyIceberg accepts table sort metadata but currently writes unsorted files and hard-codes sort_order_id=None. This prevents readers from safely using sort-order-aware pruning and leaves manifest metadata inconsistent with users' write intent.

Testing

  • Unit test verifies physical ascending order, nulls-last behavior, and returned sort-order ID
  • Integration test verifies scan order and manifest sort_order_id
  • Local InMemoryCatalog append/scan/manifest round trip passed
  • PYTHONPATH=. uv run pytest tests/io/test_pyarrow.py -k sort_table_for_identity_sort_order -q — passed
  • ruff check on changed files — passed
  • git diff --check — passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant