From 505f322c7e1124b0dd1e30ab563e06c4f1aad2fc Mon Sep 17 00:00:00 2001 From: Rusty Conover Date: Thu, 22 Oct 2026 12:46:06 -0400 Subject: [PATCH] [Format] Add custom_metadata to DictionaryEncoding for the dictionary value type In IPC, a dictionary-encoded field is a single Field whose type is the dictionary value type, and the only metadata slot was Field.custom_metadata. As a result, dictionary> and extension> were serialized identically. Add an optional custom_metadata member at the end of the DictionaryEncoding table. It holds the custom metadata of the dictionary value type, with the same semantics as Field.custom_metadata (including the reserved ARROW:extension:* keys), and is the IPC counterpart of ArrowSchema.dictionary->metadata in the C Data Interface. Document it in the columnar format (format version 1.6), the versioning history, the C Data Interface, and the integration JSON format, which gains a matching optional "metadata" member in the "dictionary" object. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/source/format/CDataInterface.rst | 8 ++++ docs/source/format/Columnar.rst | 61 +++++++++++++++++++++++++-- docs/source/format/Integration.rst | 37 +++++++++++++++- docs/source/format/Versioning.rst | 9 +++- format/Schema.fbs | 4 ++ 5 files changed, 113 insertions(+), 6 deletions(-) diff --git a/docs/source/format/CDataInterface.rst b/docs/source/format/CDataInterface.rst index 0c71b0cd8b0b..82b9eefd0c48 100644 --- a/docs/source/format/CDataInterface.rst +++ b/docs/source/format/CDataInterface.rst @@ -537,6 +537,14 @@ For dictionary-encoded arrays, the :c:member:`ArrowSchema.format` string encodes the *index* type. The dictionary *value* type can be read from the :c:member:`ArrowSchema.dictionary` structure. +The :c:member:`ArrowSchema.metadata` of the :c:member:`ArrowSchema.dictionary` +structure holds the metadata of the dictionary value type, such as the +extension type of the dictionary values, while the parent structure's +:c:member:`ArrowSchema.metadata` applies to the dictionary-encoded type as a +whole. In the IPC format, these correspond to the ``custom_metadata`` of the +``DictionaryEncoding`` and of the ``Field``, respectively (see +:ref:`format_metadata_extension_types_dictionary`). + The same holds for :c:member:`ArrowArray` structure: while the parent structure points to the index data, the :c:member:`ArrowArray.dictionary` points to the dictionary values array. diff --git a/docs/source/format/Columnar.rst b/docs/source/format/Columnar.rst index f95d7e80769d..2578f973f6dc 100644 --- a/docs/source/format/Columnar.rst +++ b/docs/source/format/Columnar.rst @@ -21,7 +21,7 @@ Arrow Columnar Format ********************* -*Version: 1.5* +*Version: 1.6* .. seealso:: :ref:`Additions to the Arrow columnar format since version 1.0.0 ` @@ -1289,7 +1289,10 @@ array. This includes: * A ``dictionary`` property indicating whether the field is dictionary-encoded or not. If it is, a dictionary "id" is assigned to allow matching a subsequent dictionary IPC message with the - appropriate field. + appropriate field. In this case, the field's data type is the + dictionary value type, and the ``dictionary`` property may carry the + ``custom_metadata`` of the dictionary value type (see + :ref:`format_metadata_extension_types_dictionary`). We additionally provide both schema-level and field-level ``custom_metadata`` attributes allowing for systems to insert their @@ -1677,7 +1680,10 @@ Custom Application Metadata We provide a ``custom_metadata`` field at three levels to provide a mechanism for developers to pass application-specific metadata in Arrow protocol messages. This includes ``Field``, ``Schema``, and -``Message``. +``Message``. The ``DictionaryEncoding`` of a dictionary-encoded ``Field`` +additionally has a ``custom_metadata`` field, with the same semantics as +the ``Field``'s ``custom_metadata`` but applied to the dictionary value +type (see :ref:`format_metadata_extension_types_dictionary`). The colon symbol ``:`` is to be used as a namespace separator. It can be used multiple times in a key. @@ -1732,6 +1738,55 @@ extension types: metadata indicating the market trading calendar the data corresponds to +.. _format_metadata_extension_types_dictionary: + +Dictionary-encoded extension types +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A dictionary-encoded field is serialized as a single ``Field`` whose data +type is the dictionary value type and whose ``dictionary`` property is a +``DictionaryEncoding`` giving the index type. The ``Field``'s +``custom_metadata`` describes the field's own type, so an extension type +whose storage type is a dictionary type stores its extension keys in the +``Field``'s ``custom_metadata``, as any other extension type does. + +The ``custom_metadata`` of the ``DictionaryEncoding`` is the +``custom_metadata`` of the dictionary value type. It has the same semantics +as a ``Field``'s ``custom_metadata``, including the reserved +``'ARROW:extension:*'`` keys, but applies to the dictionary values rather +than to the field. It is the IPC counterpart of the metadata of the +dictionary value ``ArrowSchema`` in the :ref:`C Data Interface +` (``ArrowSchema.dictionary->metadata``), while the +``Field``'s ``custom_metadata`` corresponds to the metadata of the parent +``ArrowSchema`` (``ArrowSchema.metadata``). + +If the dictionary *value* type is an extension type, for example a +dictionary of ``arrow.uuid`` values, writers should store the +``'ARROW:extension:name'`` and ``'ARROW:extension:metadata'`` keys of the +value type in the ``custom_metadata`` of the ``DictionaryEncoding`` rather +than in the ``Field``'s ``custom_metadata``. Both may be present, such as +for an extension type whose storage type is a dictionary of extension +values. Writers should omit the ``DictionaryEncoding``'s ``custom_metadata`` +when the dictionary value type has no metadata. + +When reading a dictionary-encoded field, the dictionary value type is +reconstructed from the ``Field``'s data type and the ``DictionaryEncoding``'s +``custom_metadata``. The result is wrapped in the dictionary type given by +the ``DictionaryEncoding``, and then the ``Field``'s ``custom_metadata`` is +applied to that dictionary type. As for other extension types, a reader +that does not recognize the extension name in the ``DictionaryEncoding``'s +``custom_metadata`` uses the dictionary value's storage type. + +Readers that predate this ``custom_metadata`` field (added in format version +1.6) ignore it and read a dictionary of the storage type. The field is +additive: the ``MetadataVersion`` remains ``V5``, and the IPC file footer +and Arrow Flight, which embed the same ``Schema``, need no other change. + +.. note:: + Before format version 1.6, a dictionary whose value type is an extension + type could not be distinguished from an extension type whose storage type + is a dictionary, as only the ``Field``'s ``custom_metadata`` was available. + .. seealso:: :ref:`format_canonical_extensions` diff --git a/docs/source/format/Integration.rst b/docs/source/format/Integration.rst index 5038132241c3..3129c95a1942 100644 --- a/docs/source/format/Integration.rst +++ b/docs/source/format/Integration.rst @@ -198,14 +198,19 @@ present if there are dictionary type fields in the schema. "dictionary": { "id": /* integer */, "indexType": /* Type */, - "isOrdered": /* boolean */ + "isOrdered": /* boolean */, + "metadata" : /* Metadata */ }, "metadata" : /* Metadata */ } The ``dictionary`` attribute is present if and only if the ``Field`` corresponds to a dictionary type, and its ``id`` maps onto a column in the ``DictionaryBatch``. In this -case the ``type`` attribute describes the value type of the dictionary. +case the ``type`` attribute describes the value type of the dictionary, and the +optional ``metadata`` attribute of ``dictionary`` holds the custom metadata of the +dictionary value type, as the ``custom_metadata`` of ``DictionaryEncoding`` does in +the IPC format (see :ref:`format_metadata_extension_types_dictionary`). It may be +omitted when the dictionary value type has no metadata. For primitive types, ``children`` is an empty array. @@ -424,6 +429,34 @@ would be represented:: ] } +A dictionary whose value type is an extension type carries the extension +metadata in the ``metadata`` attribute of its ``dictionary`` object instead of +the field's ``metadata``. For example, a dictionary of ``arrow.uuid`` values +with ``int8`` indices would be represented as:: + + { + "name" : "name_of_the_field", + "nullable" : /* boolean */, + "type" : { + "name" : "fixedsizebinary", + "byteWidth" : 16 + }, + "children" : [], + "dictionary": { + "id": 0, + "indexType": { + "name": "int", + "bitWidth": 8, + "isSigned": true + }, + "isOrdered": false, + "metadata" : [ + {"key": "ARROW:extension:name", "value": "arrow.uuid"}, + {"key": "ARROW:extension:metadata", "value": ""} + ] + } + } + **RecordBatch**:: { diff --git a/docs/source/format/Versioning.rst b/docs/source/format/Versioning.rst index 52cf207b959b..6e8e0e04d21c 100644 --- a/docs/source/format/Versioning.rst +++ b/docs/source/format/Versioning.rst @@ -76,7 +76,7 @@ that new clients can read serialized data produced by library version Post-1.0.0 Format Versions ========================== -Since version 1.0.0, there have been five new minor versions and zero new +Since version 1.0.0, there have been six new minor versions and zero new major versions of the Arrow format. Each new minor version added new features. When these new features are not used, the new minor format versions are compatible with format version 1.0.0. The new features added in each minor @@ -110,3 +110,10 @@ Version 1.5 ----------- * Expanded Decimal type bit widths to allow 32-bit and 64-bit types. + +Version 1.6 +----------- + +* Added ``custom_metadata`` to ``DictionaryEncoding``, carrying the custom + metadata of the dictionary value type, such as an extension type (see + :ref:`format_metadata_extension_types_dictionary`). diff --git a/format/Schema.fbs b/format/Schema.fbs index 67b771912828..46586f65bd11 100644 --- a/format/Schema.fbs +++ b/format/Schema.fbs @@ -25,6 +25,7 @@ /// Version 1.4 - Add BinaryView, Utf8View, variadicBufferCounts, ListView, and /// LargeListView. /// Version 1.5 - Add 32-bit and 64-bit as allowed bit widths for Decimal +/// Version 1.6 - Add custom_metadata to DictionaryEncoding. namespace org.apache.arrow.flatbuf; @@ -511,6 +512,9 @@ table DictionaryEncoding { isOrdered: bool; dictionaryKind: DictionaryKind; + + /// Metadata for the dictionary value type, e.g. the extension type of the values + custom_metadata: [ KeyValue ]; } /// ----------------------------------------------------------------------