fix(query): support update over parted tables and prevent column duplication by: - #406
fix(query): support update over parted tables and prevent column duplication by:#406belowzeroff wants to merge 2 commits into
Conversation
…ication by: Two bugs in ray_update hit parted (and, for the second one, flat) tables: 1. MODIFYING an existing column of a parted table failed because the update path read the RAY_PARTED_BASE wrapper type / MAPCOMMON segment shape through ray_vec_new, ray_data and the per-group gather, none of which understands a parted column (error: 'expression type I64 does not match ? column', by: path: 'group: argument must be a vector'). Flatten the whole input table once, the way select does. 2. A by: update on an EXISTING target column appended the aggregate as a second column with the same name instead of replacing the original, so the schema became [k v w w] and 'at' kept reading the stale value. ray_table_add_col always appends, so skip source columns that the update dict replaces when copying the initial schema. Adds regression coverage for both in update_parted.rfl and corrects the by: broadcast expectations in query_update_coverage (the aggregate now lands on every row of its group, kdb style).
singaraiona
left a comment
There was a problem hiding this comment.
The parted-table materialization is a sensible reuse of the existing helper, and the focused cases look good. I found three issues to address before merge:
-
test/rfl/query/update_parted.rfl:93 calls (exit 0). rayforce.test evaluates this file inside the test-runner process, so this terminates the entire runner before it records this test or executes later tests. A filtered run prints only the test name and exits successfully; removing the line produces the normal PASS summary. This can mask later failures while leaving CI green.
-
src/ops/query.c:11412-11437 skips replaced columns and appends them later. That removes duplicate names, but it also changes schema order: updating v in [k v w] produces [k w v], and updating k produces [v w k]. Existing targets should be substituted in their original slots; only genuinely new columns should be appended. Please add a regression where the updated target is not last.
-
Empty grouped updates lose the target type. For an empty table with k:I64, v:F64, w:F64, (update {w: (sum v) from: E by: k}) returns w:I64. With no groups, first_group never retypes the provisional I64 output. Please preserve or infer the type for the zero-group case and add a regression.
What a user sees today
Two distinct
updatefailures hit tables loaded from a partitioned database with.db.parted.get— and one hits any table usingupdatewithby:.1. Updating an existing column of a parted table crashes
Every modifier of an existing column aborts the script:
error: type: update: expression type I64 does not match ? columnwhere:-masked write →error: type: vec_new: type must be a positive concrete vector type, got ?by:write →error: type: group: argument must be a vector or list, got ?Adding a new column (
{new_col: ...}) worked, so the failure lookedinconsistent and was hard to diagnose. Root cause: parted data columns carry
the
RAY_PARTED_BASEwrapper type (printed as?) plus a MAPCOMMON partitionkey; the update pipeline read them through
ray_vec_new/ray_data/ theper-group gather, none of which understands the parted/segmented shape.
selectalready handled this by materialising parted columns —updatenowflattens the input table once instead of failing.
2.
update ... by:silently does nothing to the target columnThe query "succeeds" (no error) but appears to produce no change: the
aggregate is appended as a duplicate column, so the schema becomes
[k v w w],(key T)suddenly lists the column twice, and(at T 'w)keeps returning the stale original values. Affects flat tables too —
not just parted ones. Root cause:
ray_table_add_colalways appends, andthe
by:branch never skipped the source columns that the update dictreplaces.
Fix
ray_updateflattens a parted input once throughquery_materialize_parted_col(mirroringselect), so every branch belowsees wrapper-free vectors.
by:branch skips source columns that the update dict replaces whencopying the initial schema, so the aggregate lands as the single (correct)
target column. Aggregates now broadcast to every row of their group (kdb
by:semantics).Tests
test/rfl/query/update_parted.rfl— regression for the parted cases(modify / scalar / where / by: / mixed add+modify / in-place) with flat
oracles.
test/rfl/query/query_update_coverage.rfl— correctedby:expectations(the old test passed for the wrong reason: it read the duplicate original
column).
Full suite:
./rayforce.testpasses (exit 0).