Skip to content

fix(query): support update over parted tables and prevent column duplication by: - #406

Open
belowzeroff wants to merge 2 commits into
RayforceDB:devfrom
belowzeroff:fix/update-parted-columns
Open

fix(query): support update over parted tables and prevent column duplication by:#406
belowzeroff wants to merge 2 commits into
RayforceDB:devfrom
belowzeroff:fix/update-parted-columns

Conversation

@belowzeroff

Copy link
Copy Markdown
Contributor

What a user sees today

Two distinct update failures hit tables loaded from a partitioned database with .db.parted.get — and one hits any table using update with by:.

1. Updating an existing column of a parted table crashes

(set trades (.db.parted.get "/data/market" 'trades))
(update {v: (+ v 100) from: trades})          ;; error
(update {v: 5 from: trades})                  ;; error
(update {v: 99 from: trades where: (> k 1)})  ;; error

Every modifier of an existing column aborts the script:

  • plain / scalar broadcast →
    error: type: update: expression type I64 does not match ? column
  • where:-masked write →
    error: type: vec_new: type must be a positive concrete vector type, got ?
  • grouped by: write →
    error: type: group: argument must be a vector or list, got ?

Adding a new column ({new_col: ...}) worked, so the failure looked
inconsistent and was hard to diagnose. Root cause: parted data columns carry
the RAY_PARTED_BASE wrapper type (printed as ?) plus a MAPCOMMON partition
key; the update pipeline read them through ray_vec_new / ray_data / the
per-group gather, none of which understands the parted/segmented shape.
select already handled this by materialising parted columns — update now
flattens the input table once instead of failing.

2. update ... by: silently does nothing to the target column

(update {w: (sum v) from: T by: k})   ;; T already has column w

The query "succeeds" (no error) but appears to produce no change: the
aggregate is appended as a duplicate column, so the schema becomes
[k v w w], (key T) suddenly lists the column twice, and (at T 'w)
keeps returning the stale original values. Affects flat tables too —
not just parted ones. Root cause: ray_table_add_col always appends, and
the by: branch never skipped the source columns that the update dict
replaces.

Fix

  • ray_update flattens a parted input once through
    query_materialize_parted_col (mirroring select), so every branch below
    sees wrapper-free vectors.
  • The by: branch skips source columns that the update dict replaces when
    copying the initial schema, so the aggregate lands as the single (correct)
    target column. Aggregates now broadcast to every row of their group (kdb
    by: semantics).

Tests

  • New test/rfl/query/update_parted.rfl — regression for the parted cases
    (modify / scalar / where / by: / mixed add+modify / in-place) with flat
    oracles.
  • test/rfl/query/query_update_coverage.rfl — corrected by: expectations
    (the old test passed for the wrong reason: it read the duplicate original
    column).

Full suite: ./rayforce.test passes (exit 0).

belowzeroff and others added 2 commits August 15, 2026 09:02
…ication by:

Two bugs in ray_update hit parted (and, for the second one, flat) tables:

1. MODIFYING an existing column of a parted table failed because the
   update path read the RAY_PARTED_BASE wrapper type / MAPCOMMON segment
   shape through ray_vec_new, ray_data and the per-group gather, none of
   which understands a parted column (error: 'expression type I64 does
   not match ? column', by: path: 'group: argument must be a vector').
   Flatten the whole input table once, the way select does.

2. A by: update on an EXISTING target column appended the aggregate as a
   second column with the same name instead of replacing the original,
   so the schema became [k v w w] and 'at' kept reading the stale value.
   ray_table_add_col always appends, so skip source columns that the
   update dict replaces when copying the initial schema.

Adds regression coverage for both in update_parted.rfl and corrects the
by: broadcast expectations in query_update_coverage (the aggregate now
lands on every row of its group, kdb style).

@singaraiona singaraiona left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parted-table materialization is a sensible reuse of the existing helper, and the focused cases look good. I found three issues to address before merge:

  1. test/rfl/query/update_parted.rfl:93 calls (exit 0). rayforce.test evaluates this file inside the test-runner process, so this terminates the entire runner before it records this test or executes later tests. A filtered run prints only the test name and exits successfully; removing the line produces the normal PASS summary. This can mask later failures while leaving CI green.

  2. src/ops/query.c:11412-11437 skips replaced columns and appends them later. That removes duplicate names, but it also changes schema order: updating v in [k v w] produces [k w v], and updating k produces [v w k]. Existing targets should be substituted in their original slots; only genuinely new columns should be appended. Please add a regression where the updated target is not last.

  3. Empty grouped updates lose the target type. For an empty table with k:I64, v:F64, w:F64, (update {w: (sum v) from: E by: k}) returns w:I64. With no groups, first_group never retypes the provisional I64 output. Please preserve or infer the type for the zero-group case and add a regression.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants