Skip to content

[#18690] Port Iceberg IO tests off deprecated DoFnTester - #40909

Open
Supersaiyan101 wants to merge 2 commits into
apache:masterfrom
Supersaiyan101:port-iceberg-dofntester
Open

Supersaiyan101 wants to merge 2 commits into
apache:masterfrom
Supersaiyan101:port-iceberg-dofntester

Conversation

@Supersaiyan101

Copy link
Copy Markdown
Contributor

This PR addresses #18690 by porting all remaining tests in sdks/java/io/iceberg/ off of the deprecated DoFnTester:

  • BundleLifterTest: replaced DoFnTester with direct invocation of DoFn lifecycle methods (startBundle, processElement, finishBundle) along with TestOutputReceiver and MultiOutputReceiver.
  • ApplyWatermarkColumnTest: replaced DoFnTester with direct invocation of fn.process(row, receiver) to verify expected exception throwing.
  • LocalResolveDoFnTest & ResolveChangesTest: replaced DoFnTester with direct DoFn invocation collecting ValueInSingleWindow<Row> with timestamp and pane information.

All changed tests were verified and formatted with ./gradlew :sdks:java:io:iceberg:spotlessApply.

@github-actions github-actions Bot added java io iceberg Iceberg IO connector size/m M: 50-249 lines changed labels Oct 30, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR cleanly migrates four test classes in sdks/java/io/iceberg off the deprecated DoFnTester by directly invoking DoFn lifecycle and element processing methods with lightweight OutputReceiver and MultiOutputReceiver implementations.

🔍 General Feedback

  • In LocalResolveDoFnTest and ResolveChangesTest, the custom OutputReceiver implementation fails to propagate ValueKind when elements are emitted via OutputBuilder (calling the 4-arg ValueInSingleWindow.of overload which hardcodes ValueKind.INSERT). This is a critical bug that will cause CDC update/delete assertions (UPDATE_BEFORE, UPDATE_AFTER, DELETE) to fail.
  • Since OutputReceiver already provides default implementations for output and outputWithTimestamp that delegate to builder(output), overriding those methods in the anonymous OutputReceiver classes is redundant.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

.setReceiver(
wv ->
outputs.add(
ValueInSingleWindow.of(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Critical: LocalResolveDoFn emits CDC records using out.builder(row).setValueKind(kind).output(), and the tests assert on value.getValueKind() (via kindAndProjectedRow). However, the 4-argument ValueInSingleWindow.of(value, timestamp, window, paneInfo) overload hardcodes ValueKind.INSERT, discarding wv.getValueKind(). As a result, all emitted records will have ValueKind.INSERT instead of UPDATE_BEFORE, UPDATE_AFTER, or DELETE, causing the tests in this class to fail.

Use the 9-argument ValueInSingleWindow.of(...) overload and pass wv.getValueKind() (and wv.causedByDrain()). You can also remove the redundant output and outputWithTimestamp overrides since OutputReceiver already provides default methods that delegate to builder(...).

.setReceiver(
wv ->
outputs.add(
ValueInSingleWindow.of(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Critical: Similar to LocalResolveDoFnTest, ResolveChanges sets the CDC kind via .setValueKind(kind) on the OutputBuilder, and kindAndData inspects value.getValueKind(). Calling the 4-argument ValueInSingleWindow.of(...) overload drops wv.getValueKind() and defaults to ValueKind.INSERT, which breaks updatePairAndExtraRowsPreserveKindsAndTimestamp.

Pass wv.getValueKind() using the 9-argument ValueInSingleWindow.of(wv.getValue(), wv.getTimestamp(), wv.getWindows().iterator().next(), wv.getPaneInfo(), wv.getRecordId(), wv.getRecordOffset(), wv.causedByDrain(), wv.getOpenTelemetryContext(), wv.getValueKind()) overload (and optionally drop the redundant output/outputWithTimestamp overrides).

}

@SuppressWarnings("unchecked")
public <T> java.util.List<T> getOutput(TupleTag<T> tag) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Nit: Prefer importing java.util.Collections and java.util.List at the top of the file rather than using fully-qualified names here and on line 67.

@Supersaiyan101

Copy link
Copy Markdown
Contributor Author

/gemini review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR cleanly migrates four test classes in sdks/java/io/iceberg off the deprecated DoFnTester by directly invoking the DoFn lifecycle and @ProcessElement methods using TestOutputReceiver and WindowedValues.builder(). All review comments from the initial revision regarding preserving ValueKind on ValueInSingleWindow outputs and import cleanup have been properly addressed.

🔍 General Feedback

  • The replacements in BundleLifterTest, ApplyWatermarkColumnTest, LocalResolveDoFnTest, and ResolveChangesTest accurately preserve the original test semantics (including bundle lifecycle, output tags, timestamps, and CDC ValueKind metadata).

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @chamikaramj for label iceberg.
Backup reviewer: ahmedabu98 (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@codecov

codecov Bot commented Oct 30, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.84%. Comparing base (12bbb17) to head (aec65ec).
⚠️ Report is 12 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #40909   +/-   ##
=======================================
  Coverage   79.83%   79.84%           
=======================================
  Files         494      494           
  Lines       83575    83525   -50     
=======================================
- Hits        66722    66690   -32     
+ Misses      16853    16835   -18     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

iceberg Iceberg IO connector io java Next Action: Reviewers size/m M: 50-249 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant