[Format] Add custom_metadata to DictionaryEncoding for the dictionary value type #54958

Describe the enhancement requested

In IPC, a dictionary-encoded column is a single Field whose type is the dictionary value type, and the only metadata slot is Field.custom_metadata. So dictionary<values=extension<X>> and extension<X, storage=dictionary<...>> serialize identically, and readers cannot tell them apart. For example, pyarrow cannot read back its own IPC stream of pa.dictionary(pa.int32(), pa.uuid()) (#49704).

The dev@ discussion (https://lists.apache.org/thread/4xj0yrl2vn0lbrv39p3dvd28okvkjxyz) reached lazy consensus on adding an optional custom_metadata: [KeyValue] to DictionaryEncoding. It would carry the custom metadata of the dictionary value type, as the IPC counterpart of ArrowSchema.dictionary->metadata in the C Data Interface.

This issue tracks the format/specification change (format version 1.6). The implementations are tracked separately:

• C++: [C++][Format] Can't roundtrip Dictionary of ExtensionType over IPC #49704 / GH-49704: [C++] Read and write DictionaryEncoding.custom_metadata for dictionary value types #53994
• Go: feat(arrow/ipc): store dictionary value type metadata in DictionaryEncoding arrow-go#1381
• Rust (reader): Add custom_metadata to IPC DictionaryEncoding arrow-rs#11622
• nanoarrow: feat(ipc): Read and write DictionaryEncoding.custom_metadata for dictionary values arrow-nanoarrow#972

Component(s)

Format