Upsert fails after adding in a new column, target field not found

[saul-data]

Apache Iceberg version

0.10.0 (latest release)

Please describe the bug 🐞

The upsert works perfectly fine until I needed to add a new field in the table.

Add the new column in the table

# Add in a column to an existing table

from pyiceberg.types import TimestamptzType, TimestampType

table = catalog.load_table(table_identifier)

(
    table.update_schema()
         .add_column("created_at", TimestamptzType(), doc="UTC created time", required=False) 
         .commit()
)

print("New schema:", table.schema())

Upsert the records

# Batch the records in 1000s
for rb in arrow_table_fixed.to_batches(max_chunksize=1000):
    batch_tbl = pa.Table.from_batches([rb])

# Upsert the data into the Iceberg table
try:
    upd = iceberg_table.upsert(batch_tbl)
    print("Upserted data into the Iceberg table.")
    print(upd)
except Exception as e:
    print(f"An error occurred during upsert: {e}")

Error message saying that the target schema doesn't have the new column

An error occurred during upsert: Target schema's field names are not matching the table's field names: ['cik_str', 'ticker', 'title', 'created_at'], ['cik_str', 'ticker', 'title']

Checked the target schema on Iceberg and the column is definitely there

# Get the schema from the Iceberg table
iceberg_table = catalog.load_table(table_identifier)
# 2) Get the PyArrow schema directly from the Iceberg schema
arrow_schema = iceberg_table.schema().as_arrow()
print(arrow_schema.schema)

output

cik_str: large_string not null
  -- field metadata --
  PARQUET:field_id: '1'
ticker: large_string not null
  -- field metadata --
  PARQUET:field_id: '2'
title: large_string
  -- field metadata --
  PARQUET:field_id: '3'
created_at: timestamp[us, tz=UTC]
  -- field metadata --
  doc: 'UTC created time'
  PARQUET:field_id: '5'

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[sumanth-manchala]

The issue seems to be related to the projection function below where if a snapshot_id is present, it returns the schema related to that snapshot which is an older schema.

iceberg-python/pyiceberg/table/__init__.py

Lines 1682 to 1691 in 8275b13

	def projection(self) -> Schema:
	current_schema = self.table_metadata.schema()
	if self.snapshot_id is not None:
	snapshot = self.table_metadata.snapshot_by_id(self.snapshot_id)
	if snapshot is not None:
	if snapshot.schema_id is not None:
	try:
	current_schema = next(
	schema for schema in self.table_metadata.schemas if schema.schema_id == snapshot.schema_id
	)

Snapshot ID is updated on the DataScan class via these lines, which defaults to current snapshot ID always (through the default Main ref)

iceberg-python/pyiceberg/table/__init__.py

Lines 802 to 811 in 8275b13

	matched_iceberg_record_batches_scan = DataScan(
	table_metadata=self.table_metadata,
	io=self._table.io,
	row_filter=matched_predicate,
	case_sensitive=case_sensitive,
	)
	if branch in self.table_metadata.refs:
	matched_iceberg_record_batches_scan = matched_iceberg_record_batches_scan.use_ref(branch)

Maintainers, please let me know your thoughts on handling this - I can take up the PR

[rudolfix]

We have the same problem when running automated tests with upsert and pyiceberg 0.10. I was able to track it down to
https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/__init__.py#L824

a DataScan used to generate batched does not see newly added columns. Note that we didn't append any data to the table in between schema evolution and upsert transaction.

previously (0.9) used table.scan didn't have this problem.

I hope it helps a little...

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[MardanovTimur]

Any progress ? pyiceberg ver 0.11 same problem

[kevinjqliu]

Thank you for reporting this issue @saul-data. I did not see this before but codex helped me flagged this important issue 😄
I opened #3816 to address this

[kevinjqliu]

I'm hoping we get get this in as part of our next release, 0.12