Found while enabling spark.comet.exec.localTableScan.enabled by default in draft PR #6780 (Spark 4.1 SQL tests and [scans] job). Part of #4347.
5 tests fail in the Spark 4.1 [scans] job because a LocalTableScan-sourced write is now fully native (CometIcebergWrite over CometLocalTableScan) instead of IcebergWriteExec. Tests: "registration tags a fall-back reason on the write exec", "SparkWrite reflection helpers all resolve...", "no fall-back reason is recorded when the iceberg write feature is disabled", "fall-back: BatchWrite that is not an Iceberg SparkWrite", "Compatible when every partition field is void...". Related to #6664. Once default-on, the explicit localTableScan.enabled in docs and Iceberg diffs becomes redundant.
Found while enabling
spark.comet.exec.localTableScan.enabledby default in draft PR #6780 (Spark 4.1 SQL tests and[scans]job). Part of #4347.5 tests fail in the Spark 4.1 [scans] job because a LocalTableScan-sourced write is now fully native (CometIcebergWrite over CometLocalTableScan) instead of IcebergWriteExec. Tests: "registration tags a fall-back reason on the write exec", "SparkWrite reflection helpers all resolve...", "no fall-back reason is recorded when the iceberg write feature is disabled", "fall-back: BatchWrite that is not an Iceberg SparkWrite", "Compatible when every partition field is void...". Related to #6664. Once default-on, the explicit localTableScan.enabled in docs and Iceberg diffs becomes redundant.