You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
input_file_name() returns empty values over a native Iceberg scan #6707
With the native Iceberg scan, which is on by default (spark.comet.scan.icebergNative.enabled=true), input_file_name(), input_file_block_start() and input_file_block_length() return "", -1 and -1 for every row of an Iceberg table. Spark returns each row's data file and block offsets. The query succeeds, so the wrong values are silent.
These expressions read InputFileBlockHolder, which Iceberg's Spark reader sets as it opens each data file. The native scan never sets it. #3312 made CometScanRule fall back from the native Parquet scan when the plan uses these expressions, but transformV2Scan has no such check, and it is not handed the plan to make one.
No Comet operator is needed between the scan and the projection:
*(1) Project [input_file_name() AS input_file_name()#27, input_file_block_start() AS input_file_block_start()#28L, input_file_block_length() AS input_file_block_length()#29L, id#26L]
+- *(1) CometColumnarToRow
+- CometIcebergNativeScan [id#26L], .../db/t/metadata/v4.metadata.json
Steps to reproduce
spark.sql("CREATE TABLE cat.db.t (id BIGINT) USING iceberg")
for (i <-0 until 3) {
spark.sql(s"INSERT INTO cat.db.t SELECT id FROM range(${i *1000}, ${(i +1) *1000})")
}
spark
.sql("SELECT input_file_name(), input_file_block_start(), input_file_block_length(), id FROM cat.db.t")
.show(3, false)
Comet returns ["", -1, -1, <id>] for all 3000 rows. SELECT input_file_name(), id FROM cat.db.t WHERE id >= 0, which puts a CometFilter above the scan, returns "" for every row too.
Expected behavior
Each row reports the data file it was read from, as Spark does. Falling back to Spark's Iceberg reader when the plan uses these expressions, as the native Parquet scan does, would also be acceptable.
Additional context
Reproduced on main at b80bf4e08 with the default Spark 4.1 profile and a Hadoop catalog, comparing against the same query with spark.comet.enabled=false.
From reading branch-1.0 and branch-1.1, the native Iceberg scan is on by default in both and neither has a guard, so this most likely ships in 1.0 and 1.1.
Describe the bug
With the native Iceberg scan, which is on by default (
spark.comet.scan.icebergNative.enabled=true),input_file_name(),input_file_block_start()andinput_file_block_length()return"",-1and-1for every row of an Iceberg table. Spark returns each row's data file and block offsets. The query succeeds, so the wrong values are silent.These expressions read
InputFileBlockHolder, which Iceberg's Spark reader sets as it opens each data file. The native scan never sets it. #3312 madeCometScanRulefall back from the native Parquet scan when the plan uses these expressions, buttransformV2Scanhas no such check, and it is not handed the plan to make one.No Comet operator is needed between the scan and the projection:
Steps to reproduce
Comet returns
["", -1, -1, <id>]for all 3000 rows.SELECT input_file_name(), id FROM cat.db.t WHERE id >= 0, which puts aCometFilterabove the scan, returns""for every row too.Expected behavior
Each row reports the data file it was read from, as Spark does. Falling back to Spark's Iceberg reader when the plan uses these expressions, as the native Parquet scan does, would also be acceptable.
Additional context
mainatb80bf4e08with the default Spark 4.1 profile and a Hadoop catalog, comparing against the same query withspark.comet.enabled=false.branch-1.0andbranch-1.1, the native Iceberg scan is on by default in both and neither has a guard, so this most likely ships in 1.0 and 1.1.CometScanRule.readsInputFileBlock(plan)could be reused here oncetransformV2Scanhas the plan.docs/source/user-guide/latest/iceberg.mdshould name the fallback once there is one.spark.comet.scan.csv.v2.enabled, off by default) goes through the sametransformV2Scan. I have not tested it.