Repository navigation
Avro: define named types once and read references to them - #4119
Open
ozenyunus35 wants to merge 2 commits into
Open
ozenyunus35 wants to merge 2 commits into
ozenyunus35 wants to merge 2 commits into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4116
Rationale for this change
As described in the issue, an Avro named type (
decimal_P_S,fixed_L,uuid_fixed) may only be defined once per schema; later uses must refer to it by name. PyIceberg couldn't read the reference form (TypeError: Type not recognized: decimal_10_2), so it failed on manifests written by Java and iceberg-rust, and it wrote the full definition every time, which strict readers reject. This follows the approach in the issue: lenient on read, strict on write.AvroSchemaConversionremembers each named type (record,enum,fixed) it converts, and resolves a later string type with that name to the same Iceberg type. A repeated full definition is still accepted, so existing manifests keep working. The map is reset at the start ofavro_to_iceberg.ConvertSchemaToAvroemits the definition of a fixed, decimal or uuid type the first time its name comes up and the bare name after that. The visitor walks fields in the order they are serialized, so the definition always comes before its references.Are these changes tested?
Yes, in the existing test files:
tests/utils/test_schema_conversion.py: reading the reference form (the repro from the issue, plusfixed_3anduuid_fixed), reading the repeated form, an unknown name still raisingTypeError, and writing a schema with repeated decimal, fixed (one inside a struct) and uuid types: the first use is a definition, later uses are names, a different decimal gets its own definition, and the result reads back to the same schema.tests/avro/test_file.py:test_repeated_named_types_round_tripwrites a file with twodecimal(10,2)and two uuid fields throughAvroOutputFileand reads it back with PyIceberg and with fastavro. Without the change, fastavro refuses the file with aSchemaParseExceptionfor the redefined name.The new tests fail without the change.
tests/utils/test_schema_conversion.py,tests/avro,tests/test_avro_sanitization.pyandtests/utils/test_manifest.pypass. In a full-m unitrun, the same 50 tests fail before and after the change in my environment (optional extras I didn't install, such as the Entra and Hive dependencies). ruff 0.14.3 and mypy 1.18.2 (the pre-commit versions) are clean.Are there any user-facing changes?
Manifests and manifest lists with repeated named types can now be read when written in the reference form, and PyIceberg now writes the reference form.
I used an AI assistant to help draft parts of the code and tests; I reviewed the changes and ran the tests myself.