GH-47752: [C++][Python][Parquet] Add minimum compression savings for V2 pages by MashrafeeAryan · Pull Request #52580 · apache/arrow

MashrafeeAryan · 2026-10-09T17:39:40Z

Rationale for this change

Parquet's V2 pages currently keep compression whenever the compressed values are smaller than the original values. Even if it saves very little space, it still does it. For instance, it can compress 100 KB to 99 KB and save it. Even though that's 1% storage efficient, it might not be worth the decompression overhead while reading the file.

What changes are included in this PR?

In the PR, I added an optional min_space_savings setting between 0.0 and 1.0. It gives users an option to select the minimum threshold to compress V2 page data values. For instance, if the original file is 100 KB and the user has min_space_savings = 0.1, it means the compressed file needs to save at least 10% space. As a result, unless the file is <= 90 KB, it won't get saved.

I keep the current behavior when the user does not set min_space_savings.

Added C++ and Python implementation.

Are these changes tested?

C++ implementation has been tested with unit tests.
PyArrow implementation has been tested with unti tests.

Are there any user-facing changes?

C++ and Python users can set min_space_savings when writing V2 pages. For example, 0.1 requires at least 10% space savings before keeping compression.

Was AI used for this PR?

ChatGPT helped to set up the build for PyArrow.

Codex helped find a bug in dataset_writing which ignored min_space_savings.

PR code and description written by:

Human (I wrote it. Used ChatGPT to fix my grammar.)
AI

Reviewed before submission by:

Human
AI
Not reviewed

GitHub Issue: [C++][Parquet] Make compression adaptive with V2 data pages #47752

github-actions · 2026-10-09T17:40:24Z

⚠️ GitHub issue #47752 has been automatically assigned in GitHub to PR creator.

github-actions · 2026-10-09T17:43:30Z

⚠️ GitHub issue #47752 has been automatically assigned in GitHub to PR creator.

Add min_space_savings to control how much space compression must save. V2 pages below the threshold are written uncompressed. Keep the default behavior and V1 pages unchanged.

Add an option to choose how much space compression must save when writing Parquet files from Python. Add tests as well.

Dataset writing ignored min_space_savings. Move the argument to the end of the call so it reaches the writer. Add a test to check that the setting changes the file size.

github-actions · 2026-10-10T09:56:31Z

⚠️ GitHub issue #47752 has been automatically assigned in GitHub to PR creator.

rok

This looks great @MashrafeeAryan!
I left some inline comments, noting major.
Besides addressing the linter errors, we should update the Python docs. A new option is added to write_table, ParquetWriter, write_to_dataset and ParquetFileWriteOptions. We should update _parquet_writer_arg_docs in python/pyarrow/parquet/core.py and the data-page-V2 paragraph in docs/source/cpp/parquet.rst.

Throw an error when users set min_space_savings for V1 pages. Update C++ and Python tests.

Require compressed data to be smaller, even when min_space_savings is 0.0. Simplify the compression check and update the test.

Check min_space_savings when users set it. Update the test and Python error handling.

Check compressed and original sizes in the existing test. Keep checking that the data reads back correctly.

Explain min_space_savings in the Python and C++ docs.

Fix spacing and line wrapping in the C++ changes.

rok · 2026-10-23T11:48:13Z

      return this;
    }

+    /// Used to set minimum threshold for V2 page data values compression


Perhaps?

Suggested change

/// Used to set minimum threshold for V2 page data values compression

/// Used to set minimum compression ration threshold for V2 page data values compression

rok · 2026-10-23T12:02:00Z

Doc issue appears to be from wrong ordering in _parquet_writer_arg_docs. min_space_savings must com last before kwargs.

PR03: Wrong parameters order. Actual: ('table', 'where', 'row_group_size', 'version', 'use_dictionary', 'compression', 'write_statistics', 'use_deprecated_int96_timestamps', 'coerce_timestamps', 'allow_truncated_timestamps', 'data_page_size', 'flavor', 'filesystem', 'compression_level', 'use_byte_stream_split', 'column_encoding', 'data_page_version', 'use_compliant_nested_type', 'encryption_properties', 'write_batch_size', 'dictionary_pagesize_limit', 'store_schema', 'write_page_index', 'write_page_checksum', 'sorting_columns', 'store_decimal_as_integer', 'write_time_adjusted_to_utc', 'max_rows_per_page', 'bloom_filter_options', 'use_content_defined_chunking', 'min_space_savings', '**kwargs'). Documented: ('table', 'where', 'row_group_size', 'version', 'use_dictionary', 'compression', 'write_statistics', 'use_deprecated_int96_timestamps', 'coerce_timestamps', 'allow_truncated_timestamps', 'data_page_size', 'flavor', 'filesystem', 'compression_level', 'use_byte_stream_split', 'column_encoding', 'data_page_version', 'min_space_savings', 'use_compliant_nested_type', 'encryption_properties', 'write_batch_size', 'dictionary_pagesize_limit', 'store_schema', 'write_page_index', 'write_page_checksum', 'sorting_columns', 'store_decimal_as_integer', 'write_time_adjusted_to_utc', 'max_rows_per_page', 'bloom_filter_options', 'use_content_defined_chunking', '**kwargs')

rok · 2026-10-23T12:08:54Z

+    else:
+        assert column.total_compressed_size < column.total_uncompressed_size
+
+


How about a threshold test too? Note this requires adding a random import.

Suggested change

def _v2_column_chunk(table, **options):

sink = pa.BufferOutputStream()

pq.write_table(table, sink, compression="snappy", use_dictionary=False,

data_page_version="2.0", **options)

return pq.ParquetFile(sink.getvalue()).metadata.row_group(0).column(0)

def test_min_space_savings_threshold():

rng = random.Random(42)

values = [rng.randrange(16) for _ in range(1000)]

table = pa.table({"v": pa.array(values, pa.int32())})

base = _v2_column_chunk(table)

savings = 1 - base.total_compressed_size / base.total_uncompressed_size

assert 0.1 < savings < 0.9

kept = _v2_column_chunk(table, min_space_savings=savings - 0.05)

assert kept.total_compressed_size == base.total_compressed_size

dropped = _v2_column_chunk(table, min_space_savings=savings + 0.05)

assert dropped.total_compressed_size == dropped.total_uncompressed_size

assert dropped.compression == "SNAPPY"

rok · 2026-10-23T12:12:32Z

@MashrafeeAryan this is looking good, some minor suggestions from me, but I think it's ready for another reviewer. Feel free to take it out of draft.

@pitrou could we ask you for a review here?

github-actions Bot added Component: Parquet Component: C++ Component: Python awaiting review Awaiting review labels Oct 9, 2026

MashrafeeAryan added 3 commits October 9, 2026 20:43

Add adaptive compression for Parquet V2 pages

688cfcd

Add min_space_savings to control how much space compression must save. V2 pages below the threshold are written uncompressed. Keep the default behavior and V1 pages unchanged.

Let Python users set minimum compression savings

01512cc

Add an option to choose how much space compression must save when writing Parquet files from Python. Add tests as well.

Fix ignored compression savings in dataset writing

89bf1b2

Dataset writing ignored min_space_savings. Move the argument to the end of the call so it reaches the writer. Add a test to check that the setting changes the file size.

MashrafeeAryan force-pushed the adaptive-compression branch from 65825cc to 89bf1b2 Compare October 10, 2026 09:57

rok reviewed Oct 16, 2026

View reviewed changes

Comment thread cpp/src/parquet/properties.h Outdated

Comment thread cpp/src/parquet/properties.h

Comment thread cpp/src/parquet/column_writer.cc Outdated

Comment thread cpp/src/parquet/properties.h Outdated

Comment thread python/pyarrow/tests/parquet/test_parquet_writer.py

MashrafeeAryan added 6 commits October 18, 2026 09:12

Reject min_space_savings for V1 pages

7d705a9

Throw an error when users set min_space_savings for V1 pages. Update C++ and Python tests.

Skip compression when it saves no space

d8d4642

Require compressed data to be smaller, even when min_space_savings is 0.0. Simplify the compression check and update the test.

Reject bad savings values right away

f4cc427

Check min_space_savings when users set it. Update the test and Python error handling.

Check that Python uses compression savings

f20f66c

Check compressed and original sizes in the existing test. Keep checking that the data reads back correctly.

Explain minimum compression savings

adef6d4

Explain min_space_savings in the Python and C++ docs.

Fix C++ formatting

e4953da

Fix spacing and line wrapping in the C++ changes.

github-actions Bot added the Component: Documentation label Oct 18, 2026

rok reviewed Oct 23, 2026

View reviewed changes

github-actions Bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Oct 23, 2026

Navigation Menu

Repository navigation

GH-47752: [C++][Python][Parquet] Add minimum compression savings for V2 pages - #52580

GH-47752: [C++][Python][Parquet] Add minimum compression savings for V2 pages#52580
MashrafeeAryan wants to merge 9 commits into
apache:mainfrom
MashrafeeAryan:adaptive-compression

MashrafeeAryan commented Oct 9, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 9, 2026

Uh oh!

github-actions Bot commented Oct 9, 2026

Uh oh!

github-actions Bot commented Oct 10, 2026

Uh oh!

rok left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rok Oct 23, 2026 •

edited

Loading

Uh oh!

rok commented Oct 23, 2026

Uh oh!

rok Oct 23, 2026

Uh oh!

rok commented Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	/// Used to set minimum threshold for V2 page data values compression
	/// Used to set minimum compression ration threshold for V2 page data values compression

		else:
		assert column.total_compressed_size < column.total_uncompressed_size

+def _v2_column_chunk(table, **options):
+    sink = pa.BufferOutputStream()
+    pq.write_table(table, sink, compression="snappy", use_dictionary=False,
+                   data_page_version="2.0", **options)
+    return pq.ParquetFile(sink.getvalue()).metadata.row_group(0).column(0)
+def test_min_space_savings_threshold():
+    rng = random.Random(42)
+    values = [rng.randrange(16) for _ in range(1000)]
+    table = pa.table({"v": pa.array(values, pa.int32())})
+    base = _v2_column_chunk(table)
+    savings = 1 - base.total_compressed_size / base.total_uncompressed_size
+    assert 0.1 < savings < 0.9
+    kept = _v2_column_chunk(table, min_space_savings=savings - 0.05)
+    assert kept.total_compressed_size == base.total_compressed_size
+    dropped = _v2_column_chunk(table, min_space_savings=savings + 0.05)
+    assert dropped.total_compressed_size == dropped.total_uncompressed_size
+    assert dropped.compression == "SNAPPY"

Conversation

MashrafeeAryan commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Was AI used for this PR?

Uh oh!

github-actions Bot commented Oct 9, 2026

Uh oh!

github-actions Bot commented Oct 9, 2026

Uh oh!

github-actions Bot commented Oct 10, 2026

Uh oh!

rok left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rok Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rok commented Oct 23, 2026

Uh oh!

rok Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

rok commented Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MashrafeeAryan commented Oct 9, 2026 •

edited

Loading

rok left a comment •

edited

Loading

rok Oct 23, 2026 •

edited

Loading