Skip to content

[spark][flink] Support bucket-level compaction for fixed-bucket tables - #9219

Open
lilei1128 wants to merge 2 commits into
apache:masterfrom
lilei1128:bucket-compaction
Open

[spark][flink] Support bucket-level compaction for fixed-bucket tables#9219
lilei1128 wants to merge 2 commits into
apache:masterfrom
lilei1128:bucket-compaction

Conversation

@lilei1128

@lilei1128 lilei1128 commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Purpose

For non-partitioned tables with many buckets, or partitioned tables where each partition
contains many buckets, compacting all buckets in one job may consume excessive planning
and execution resources and eventually cause OOM errors.

This PR introduces bucket-level compaction for fixed-bucket tables in Flink and Spark.
Users can split a large compaction into multiple jobs by selecting individual bucket IDs
or bucket ranges.

For example:

    CALL sys.compact(
      `table` => 'database.large_bucket_table',
      compact_strategy => 'full',
      buckets => '0-999'
  );

A table with many buckets can be compacted in batches such as 0-999, 1000-1999, and 2000-2999.

Fix #9218

Tests

  • ParameterUtilsTest#testParseIntegerRanges

  • ParameterUtilsTest#testInvalidIntegerRanges

  • CompactActionITCase#testCompactSpecifiedBucketRangesFromAction

  • CompactProcedureITCase#testCompactSpecifiedBucketRanges

  • CompactProcedureTestBase.scala

    • Paimon Procedure: compact specified bucket ranges
    • Paimon Procedure: reject buckets for dynamic bucket table

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support bucket-level compaction for fixed-bucket tables

1 participant