Skip to content

test: cover Iceberg driver commit cleanup - #6119

Open
sam-1112 wants to merge 2 commits into
apache:mainfrom
sam-1112:test-iceberg-driver-commit-failure-cleanup-5646
Open

sam-1112 wants to merge 2 commits into
apache:mainfrom
sam-1112:test-iceberg-driver-commit-failure-cleanup-5646

Conversation

@sam-1112

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes part of #5646 (driver-side Iceberg commit failure cleanup).

Rationale for this change

When every native write task succeeds but the driver-side Iceberg commit fails, Comet must abort the complete set of task commit messages. Otherwise, files produced by the failed write can be left behind as orphans.

What changes are included in this PR?

  • Add a scoped, test-only pre-commit hook to IcebergCommitExec.
  • Add a native Iceberg DELETE regression test that creates a real commit-time conflict after all task commit messages have been collected.
  • Verify that the original Iceberg conflict reaches the caller, the concurrent append is the only committed change, and the failed DELETE’s files are removed from both storage and Iceberg metadata.
  • Verify the failure path reaches both CometIcebergWriteExec and IcebergCommitExec.

How are these changes tested?

make core

./mvnw test -Pspark-3.5 -Dtest=none \
    '-Dsuites=org.apache.comet.CometIcebergWriteActionSuite driver commit failure aborts'

./mvnw test -Dtest=none \
    '-Dsuites=org.apache.comet.CometIcebergWriteActionSuite driver commit failure aborts'

./mvnw spotless:check -Pspark-3.5

Before merge, apply the run-iceberg-tests label to validate the Iceberg version matrix.

@github-actions github-actions Bot added enhancement New feature or request test Testing related area:writer Native Parquet writer area:Iceberg labels Sep 22, 2026
@sam-1112
sam-1112 force-pushed the test-iceberg-driver-commit-failure-cleanup-5646 branch from 0767de1 to a8a4f0a Compare September 22, 2026 18:03
@sam-1112
sam-1112 marked this pull request as ready for review September 23, 2026 03:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Iceberg area:writer Native Parquet writer enhancement New feature or request test Testing related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant