Skip to content

fix: delegate Arrow schema conversion to PyIceberg - #343

Closed
konstantinoscs wants to merge 1 commit into
feature/tow-2494-iceberg-filter-expressionsfrom
codex/tow-2494-native-arrow-schema
Closed

konstantinoscs wants to merge 1 commit into
feature/tow-2494-iceberg-filter-expressionsfrom
codex/tow-2494-native-arrow-schema

Conversation

@konstantinoscs

@konstantinoscs konstantinoscs commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Pass the original pyarrow.Schema to PyIceberg's public catalog APIs.
  • Let PyIceberg assign nested field IDs, preserve docs/nullability, and reject unsupported or lossy types.
  • Remove Tower's hand-written schema converter.
  • Document timestamp, time, date, decimal, and timezone precision behavior.

Compatibility and S3/BYO

The already-selected catalog receives the same Arrow schema object. Tests compose default S3 Tables/BYO string routing with create() and prove no Tower vending occurs, preserving those customer paths.

PyIceberg's native validation errors now propagate. Nanosecond timestamps reject by default rather than silently downcasting; the docs list the accepted and rejected Arrow types.

Verification

  • 20 schema-specific tests at the lower bound.
  • Completed compatibility suite: 103 tests passed with PyArrow 25.0.1 / PyIceberg 0.11.1.
  • Full Stack A suite: 225 passed, 10 skipped; Black and mypy clean.

Stack

5 of 5 in Stack A. Base: #342.

@coderabbitai

coderabbitai Bot commented Aug 13, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: d3534afa-09d8-45b3-a1a9-32135ecb2298

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant