Skip to content

feat: add transform-driven column updates - #83

Merged
rchowell merged 9 commits into
daft-engine:mainfrom
FANNG1:feat/update-columns-transform
Oct 14, 2026
Merged

rchowell merged 9 commits into
daft-engine:mainfrom
FANNG1:feat/update-columns-transform

Conversation

@FANNG1

@FANNG1 FANNG1 commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Closes #84.

Summary

  • add a transform-driven update_columns API that does not require a prepared DataFrame
  • support Lance SQL expressions, BatchUDF, and row-preserving Arrow callables
  • support an optional where predicate evaluated before the transform
  • schedule work by fragment ID without grouping or shuffling user data
  • allow transform workers to request Daft/Ray CPU, GPU, process, and custom Ray resources
  • commit rewritten fragments atomically with the existing RewriteColumns transaction path

Correctness

  • pin every worker and the final commit to the same Lance snapshot
  • validate predicates, target columns, transform schemas, row counts, nullability, and safe casts
  • keep no-match updates as no-ops and preserve OCC conflict detection
  • request matched physical offsets with fragment.update_columns(..., with_offsets=True) and propagate them through updated_fragment_offsets, including for stable-row-ID datasets
  • allow transform-driven updates of a real business column named fragment_id while preserving the grouping-key restriction in update_columns_df
  • always begin writes from the current snapshot; the update APIs do not accept version, asof, or default_scan_options

Tests

  • uv run pytest tests/io/lance/test_update_columns.py -q (40 passed)
  • uv run pytest tests/ -q (492 passed, 10 skipped, 2 xfailed, 2 xpassed)
  • Ruff format and lint passed
  • focused mypy checks for changed files passed

@FANNG1
FANNG1 force-pushed the feat/update-columns-transform branch 2 times, most recently from 9b931ae to 60ca79b Compare October 10, 2026 07:28
@FANNG1
FANNG1 marked this pull request as ready for review October 10, 2026 07:55
@FANNG1

FANNG1 commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor Author

@rchowell @universalmind303 could you take a look when you get a chance? This is the transform-driven half of the column-update work (#84), built on the same RewriteColumns commit path as #58.

@FANNG1
FANNG1 force-pushed the feat/update-columns-transform branch from 60ca79b to d88c189 Compare October 12, 2026 22:48
Comment on lines +246 to +249
max_concurrency: int | None,
) -> list[str]:
if max_concurrency is not None and max_concurrency <= 0:
raise ValueError("max_concurrency must be a positive integer.")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: it's a bit odd to have concurrency checks in this method

@rchowell
rchowell merged commit 8b6a708 into daft-engine:main Oct 14, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support transform-driven updates of existing Lance columns

2 participants