Describe the enhancement requested
Add content-defined chunking (CDC) to the Parquet writer. It is already available in Parquet C++ (apache/arrow#45750, PyArrow >= 21) and parquet-rs (apache/arrow-rs#9450). With CDC, data page boundaries come from a rolling hash of the column values rather than fixed sizes. Pages then stay stable when rows are inserted or appended, so content-addressable storage (e.g. Hugging Face Xet) can deduplicate them. The output is a regular Parquet file, with no format change.
It would help to use the same parameters as the other implementations:
min_chunk_size (default 256 KiB)max_chunk_size (default 1 MiB)norm_level (default 0)
Motivation: Iceberg is adding write.parquet.content-defined-chunking.* table properties (apache/iceberg-rust#2375, apache/iceberg-python#3889). Iceberg Java cannot honor them until parquet-java supports CDC.
Component(s)
Core
Describe the enhancement requested
Add content-defined chunking (CDC) to the Parquet writer. It is already available in Parquet C++ (apache/arrow#45750, PyArrow >= 21) and parquet-rs (apache/arrow-rs#9450). With CDC, data page boundaries come from a rolling hash of the column values rather than fixed sizes. Pages then stay stable when rows are inserted or appended, so content-addressable storage (e.g. Hugging Face Xet) can deduplicate them. The output is a regular Parquet file, with no format change.
It would help to use the same parameters as the other implementations:
min_chunk_size(default 256 KiB)max_chunk_size(default 1 MiB)norm_level(default 0)Motivation: Iceberg is adding
write.parquet.content-defined-chunking.*table properties (apache/iceberg-rust#2375, apache/iceberg-python#3889). Iceberg Java cannot honor them until parquet-java supports CDC.Component(s)
Core