Skip to content

[flink] Filter undo recovery records before row materialization - #5005

Open
platinumhamburg wants to merge 1 commit into
apache:mainfrom
platinumhamburg:undo-recovery-filtering
Open

platinumhamburg wants to merge 1 commit into
apache:mainfrom
platinumhamburg:undo-recovery-filtering

Conversation

@platinumhamburg

Copy link
Copy Markdown
Contributor

Purpose

Closes #5004.

Undo recovery currently copies every changelog row before discarding changes that cannot produce an undo. Filter scanner-backed rows before materialization to avoid unnecessary column copying on wide tables.

Brief change log

  • Add internal filtered polling that copies only selected rows while preserving consumed offsets and the scanned-record poll limit.
  • Select the first relevant change per primary key within each recovery poll, without prematurely updating persistent recovery deduplication.
  • Preserve ordinary polling for other LogScanner implementations and keep returned row values independent of scanner buffers.

Tests

  • 52 focused unit tests covering fetch buffers, filtered poll limits and offset progress, selected-row ownership, undo selection, and executor behavior.
  • 5 local UndoRecoveryManagerITCase tests covering full and partial updates, partitioned tables, idempotency, and scanner failure.
  • 8 local Flink120UndoRecoveryITCase tests covering checkpoint failover, producer-offset recovery, savepoint rescaling, and bounded jobs.
  • Checkstyle, Spotless, and RAT checks passed during the targeted Maven build.

API and Format

Adds an @Internal method to LogScannerImpl. The LogScanner interface and storage format are unchanged.

Documentation

No user-facing configuration changes.

@platinumhamburg
platinumhamburg marked this pull request as ready for review November 2, 2026 12:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement] Filter undo recovery records before row materialization

1 participant