Skip to content

feat: Support difference_cols for upsert update detection#3611

Open
minhduc29 wants to merge 1 commit into
apache:mainfrom
minhduc29:feat-upsert-difference-cols
Open

feat: Support difference_cols for upsert update detection#3611
minhduc29 wants to merge 1 commit into
apache:mainfrom
minhduc29:feat-upsert-difference-cols

Conversation

@minhduc29

Copy link
Copy Markdown

Closes #3598

Rationale for this change

upsert compares every non-key column of each matched row in Python to decide whether it needs updating, which is expensive for wide tables or complex types (structs, lists). This adds an optional difference_cols parameter to Table.upsert / Transaction.upsert / get_rows_to_update that limits the comparison to a user-provided subset of columns (e.g. a hash column that reflects any change to the row), as proposed in #3598.

Design notes:

  • difference_cols only affects change detection: rows detected as changed are still written with all of their columns.
  • Columns missing from the source table, join columns, or an empty list raise ValueError, validated up front in Transaction.upsert to fail fast. The issue proposed either raising or falling back to comparing all columns — raising seemed safer, but happy to change if the fallback is preferred.
  • Behavior without the parameter is unchanged.

Are these changes tested?

Yes. Unit tests cover detection limited to the listed columns (a matched row changed only outside difference_cols is skipped), updated rows being written with all columns, the three validation errors, and an end-to-end upsert via the in-memory catalog.

Are there any user-facing changes?

Yes, a new optional difference_cols parameter on Table.upsert and Transaction.upsert, documented in the API docs. No behavior change when it is not provided.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow user to define a subset of columns for update detection in UPSERT

1 participant