[rustyconover]

Rationale for this change

This PR was split out of #53994 at review request, so that the specification change can be reviewed on its own.

In IPC, a dictionary-encoded column is a single Field whose type is the value type. The only metadata slot was Field.custom_metadata, so dictionary<values=extension<X>> and extension<X, storage=dictionary<...>> serialized identically. Arrow C++ reads both as the latter, so pyarrow cannot read back its own stream of pa.dictionary(pa.int32(), pa.uuid()). The dev@ thread (https://lists.apache.org/thread/4xj0yrl2vn0lbrv39p3dvd28okvkjxyz) reached lazy consensus on adding KeyValue metadata for the value type to DictionaryEncoding (option 1).

This is a format change, so it needs a [VOTE] on dev@ and two reference implementations (see below).

What changes are included in this PR?

• format/Schema.fbs: an optional custom_metadata: [KeyValue] appended to DictionaryEncoding. It is the custom_metadata of the dictionary value type, with the same semantics as Field.custom_metadata (including the reserved ARROW:extension:* keys) but applied to the dictionary values. It is the IPC counterpart of ArrowSchema.dictionary->metadata in the C Data Interface.
• Columnar.rst: a new "Dictionary-encoded extension types" section. The format version is bumped to 1.6.
  • Writers should put a value extension type's ARROW:extension:name/ARROW:extension:metadata there rather than in Field.custom_metadata.
  • Readers apply it to the value type before wrapping it in the dictionary type, then apply Field.custom_metadata as before.
  • Extension types whose storage type is a dictionary keep using Field.custom_metadata, as today.
• Versioning.rst: the version 1.6 entry.
• CDataInterface.rst: a note that ArrowSchema.dictionary->metadata holds the metadata of the dictionary value type, and how it maps to the two IPC custom_metadata slots.
• Integration.rst: a matching optional "metadata" member in the "dictionary" object of the integration JSON format, with an example.

The member is appended, so it is backward compatible. MetadataVersion stays V5, and the file footer and Flight need no other change. Readers that predate it ignore it and read a dictionary of the storage type.

Generated flatbuffers code is not included. Each implementation regenerates its bindings in its own PR.

Are these changes tested?

This PR changes only the specification. It is exercised by the implementations and a new dictionary_extension archery integration case:

• C++: GH-49704: [C++] Read and write DictionaryEncoding.custom_metadata for dictionary value types #53994, stacked on this PR
• Go, the second reference implementation: feat(arrow/ipc): store dictionary value type metadata in DictionaryEncoding arrow-go#1381. C++ ⇄ Go passes the new case over IPC.
• Rust, which reads the new member: Add custom_metadata to IPC DictionaryEncoding arrow-rs#11622
• nanoarrow, which maps the member to ArrowSchema.dictionary->metadata: feat(ipc): Read and write DictionaryEncoding.custom_metadata for dictionary values arrow-nanoarrow#972

Are there any user-facing changes?

Yes, this is a format addition (version 1.6). Writers that adopt it move a dictionary value type's extension keys from Field.custom_metadata to DictionaryEncoding.custom_metadata. Readers that predate it read such columns as a plain dictionary of the storage type and lose the extension type.

Was AI used for this PR?

PR code and description written by:

• Human
• AI

Reviewed before submission by:

• Human
• AI
• Not reviewed

• GitHub Issue: [Format] Add custom_metadata to DictionaryEncoding for the dictionary value type #54958

🤖 Generated with Claude Code

[github-actions]

Thanks for opening a pull request!

This pull request has been automatically closed because you currently have 4 open pull requests, which is more than the limit of 3.

Due to the increase in pull requests opened by AI bots, and in order to keep the review queue manageable, Apache Arrow limits contributors without repository access to at most 3 concurrently open pull requests. This helps make sure each pull request gets the attention it needs and that work in progress does not go stale.

Once one of your other open pull requests has been merged or closed, you are welcome to reopen this one.

See also:

• Contribution Guidelines - Limit on concurrent pull requests
• Contribution Guidelines - Contributing Overview

[github-actions]

⚠️ GitHub issue #54958 has been automatically assigned in GitHub to PR creator.

[Copilot]

🔵 Needs a closer look

This shared-format change needs maintainer confirmation of the required dev@ vote and interoperability evidence from two reference implementations.

No open issues 🎉

What changed in this PR

This specification-only PR introduces Arrow format 1.6 to distinguish dictionary value extension types from extension types whose storage is a dictionary.

Changes:

• Adds optional value-type metadata to DictionaryEncoding.
• Defines metadata placement, reader behavior, and compatibility with older readers.
• Documents equivalent metadata in the C Data Interface and integration JSON format.

File	Description
format/​Schema.fbs	Appends optional DictionaryEncoding.custom_metadata and records version 1.6.
docs/​source/​format/​Versioning.rst	Adds the format 1.6 release entry.
docs/​source/​format/​Integration.rst	Defines optional dictionary metadata with a UUID example.
docs/​source/​format/​Columnar.rst	Specifies extension metadata semantics and backward compatibility.
docs/​source/​format/​CDataInterface.rst	Maps dictionary value and parent metadata to their IPC counterparts.

_{Review effort: Balanced}

[rustyconover]

Hi @pitrou, @rok, or @raulcd,

Can you please reopen this PR, apparently I'm not in the club that can have more than 3 open pull requests:

This pull request has been automatically closed because you currently have 4 open pull requests, which is more than the limit of 3.

Thanks,

Rusty

[rok]

@rustyconover sorry about that, it's the zeitgeist. #54960 will give you unlimited PRs.

[rustyconover]

Thanks @rok!