Skip to content

test: cover native Iceberg write handoff cleanup - #6117

Draft
sam-1112 wants to merge 3 commits into
apache:mainfrom
sam-1112:test-iceberg-native-jvm-handoff-cleanup-5646
Draft

sam-1112 wants to merge 3 commits into
apache:mainfrom
sam-1112:test-iceberg-native-jvm-handoff-cleanup-5646

Conversation

@sam-1112

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes part of #5646 (native-to-JVM handoff cleanup).

Rationale for this change

A native Iceberg writer can successfully return its task payload, then fail on the JVM while decoding the manifest and building the task commit message. At that point, no commit message reaches the driver, so the task's data files must be deleted by its failure listener.

What changes are included in this PR?

  • Add a scoped test-only failpoint after WrittenFileCleanup owns native written_file_locations and before manifest decoding.
  • Add an end-to-end native Iceberg write regression test that injects a post-native handoff failure.
  • Verify all failed task attempts report non-empty native payload locations, no snapshot is committed, and no failed-task Parquet file remains on disk or in Iceberg metadata.

How are these changes tested?

make core

./mvnw test -Pspark-3.5 -Dtest=none \
  '-Dsuites=org.apache.comet.CometIcebergWriteActionSuite post-native handoff failure'

./mvnw test -Dtest=none \
  '-Dsuites=org.apache.comet.CometIcebergWriteActionSuite post-native handoff failure'

./mvnw spotless:check -Pspark-3.5

The targeted regression passes on Spark 3.5 and the default Spark 4.1 profile.

@github-actions github-actions Bot added enhancement New feature or request test Testing related area:writer Native Parquet writer area:Iceberg labels Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Iceberg area:writer Native Parquet writer enhancement New feature or request test Testing related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant