Avro: can't read named type references, and writes repeated definitions (e.g. multiple decimal partition fields) #4116

Apache Iceberg version

main (development). This affects all versions.

Please describe the bug 🐞

When two or more partition fields have the same decimal(P,S) type (e.g. identity(a), identity(b) where both are decimal(10,2)), the type decimal_10_2 shows up twice in the manifest's Avro schema. The Avro spec only allows one definition per name, so the second use should be a reference, i.e. just "decimal_10_2".

PyIceberg gets this backwards:

• ❌ Can't read the reference form. AvroSchemaConversion treats "decimal_10_2" as an unknown type, so it fails on manifests written by Java, and by iceberg-rust since perf(manifest): upgrade to apache-avro 0.22 and read manifest entries without Value iceberg-rust#3354.
• ❌ Writes the repeated form. iceberg_to_avro writes the full definition every time, which strict readers like apache-avro (Rust) 0.22 reject.

Repro:

from pyiceberg.utils.schema_conversion import AvroSchemaConversion

dec = {"type": "fixed", "name": "decimal_10_2", "size": 5, "logicalType": "decimal", "precision": 10, "scale": 2}
AvroSchemaConversion().avro_to_iceberg({"type": "record", "name": "r102", "fields": [
    {"name": "a", "type": ["null", dec], "field-id": 1000, "default": None},
    {"name": "b", "type": ["null", "decimal_10_2"], "field-id": 1001, "default": None}]})
# TypeError: Type not recognized: decimal_10_2

The same applies to fixed(L) (fixed_L) and uuid (uuid_fixed).

We should match Java and iceberg-rust: lenient on read, strict on write.

• read: accept both the reference form and the repeated form, since manifests in the repeated form already exist
• write: define each named type once, then refer to it by name

More background, including how Java and iceberg-rust handle this, is in apache/iceberg-rust#3466.

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time