.db.parted.fill fails on nested (LIST) columns

[vbanas]

Rayforce issue report: .db.parted.fill fails on nested (LIST) columns

Engine	rayforce v2.5.14, commit 6067f3a7 (first observed on v2.5.13)
Verified with	freshly built dev binary (make, ASan+UBSan, -O0 -DDEBUG) on 2026-08-14
Trigger	any date-parted root where a table carrying a LIST column is missing from at least one partition day
Severity	fill (and therefore any parted read that gap-fills first) fails for the whole root; the error is misleading; the failed fill leaves partial state
Repro scripts	/tmp/rayissue/repro{1..5}*.rfl (inlined in the appendix; self-contained)

1. Summary

.db.parted.fill cannot create the empty copy of any table that carries a
LIST column (for example LIST-of-DICT — per-row dictionaries such as
rate schedules). The failure is not in the storage format and not in the
read path — both handle 0-row LIST columns correctly. It is a single
guard in ray_vec_new, reached through fill's empty_table_like, which
refuses RAY_LIST because the type code is 0:

empty_table_like (part.c:563)          ray_vec_new (vec.c:155-157)
    ray_vec_new(col->type, 0)   ──►    if (type <= 0 || type >= RAY_TYPE_COUNT)
                                           return ray_error("type", ...)
                                       /* RAY_LIST == 0  →  "type" error */

Three aggravating factors documented below: the real error is flattened
to a bare io before it reaches the caller (§6a), the failed fill
leaves partial on-disk state (§6b), and the case has no test
coverage (§6d).

2. How this arises

A parted root is one directory per partition value with one splayed table
dir inside each. Per-table partition sets diverge in ordinary use: a
table is added after the root already has history; a quiet table writes
nothing across a partition boundary while its neighbors advance; a
backfill lands historical partitions for one table beside tables that
only grow forward. A parted read requires every table present in every
partition dir, so the standard remedy is .db.parted.fill, which creates
the missing 0-row copies.

The moment any table in the root carries a LIST column — e.g. a
reference-data table whose cells are DICTs — the first such fill fails
for the whole root:

error: io: parted <root>: fill failed

with no hint of which table, which column, or the real cause. Any
application that gap-fills on read (or at startup) is now unable to open
the root.

Workarounds available to applications, both costly: keep LIST-column
tables in a separate parted root (second sym domain, duplicated
registry/plumbing), or forbid LIST columns in parted stores outright.
Neither protects against the quiet-table-crossing-a-day case inside a
root that already mixes both — only the engine fix does.

3. Minimal reproductions (all verified on v2.5.14)

3.1 The failure

Two day dirs; flat table A in both; SCHED (sym + LIST-of-DICT) only in
the newest day. Fill must create SCHED's empty copy in the older day:

(set 'FLAT (table [s v] (list ['x 'y] [1 2])))
(set 'SCHED (table [acct sched]
  (list ['a 'b]
        (list (dict [0] [5.0])
              (dict [0 250000] [15.0 10.0])))))
(.db.splayed.set "/tmp/rayissue/r1/2024.01.01/A/" FLAT)
(.db.splayed.set "/tmp/rayissue/r1/2024.01.02/A/" FLAT)
(.db.splayed.set "/tmp/rayissue/r1/2024.01.02/SCHED/" SCHED)
(.db.parted.fill "/tmp/rayissue/r1/")

error: io: parted /tmp/rayissue/r1/: fill failed

No SCHED dir appears under 2024.01.01/. The message names neither the
table, nor the column, nor the real cause.

3.2 Control: flat columns fill fine

The identical layout with the dict column replaced by f64 succeeds:

(.db.parted.fill "/tmp/rayissue/r2/")   →  [2024.01.01]
(count (.db.parted.get "/tmp/rayissue/r2/" 'SCHED2))   →  2

3.3 Fill is not atomic — partial state on failure

Tables are processed in (alphabetical) union order; everything filled
before the failing table stays on disk, everything after never runs:

day 2024.01.02: A_FLAT  M_FLAT  N_SCHED  Z_FLAT     day 2024.01.01: M_FLAT
(.db.parted.fill …)  →  error
day 2024.01.01 afterwards:  A_FLAT  M_FLAT          ← A_FLAT was CREATED
                                                      Z_FLAT never attempted

A failed fill therefore mutates the root.

3.4 The read path is NOT broken — 0-row nested columns round-trip

A 0-row table with i64 + sym + LIST columns, written and read with a
dir-local symfile, round-trips perfectly:

(set 'E (table [id who sched] (list (take [0] 0) (take ['x] 0) (list))))
(.db.splayed.set "/tmp/rayissue/r4/T/" E "/tmp/rayissue/r4/T/.sym")
(.db.splayed.get "/tmp/rayissue/r4/T/" "/tmp/rayissue/r4/T/.sym")
→  <table 0x3>, count 0, keys [id who sched]

So the fix only needs to make fill construct the empty table; the
save/load machinery already supports it end to end (§5).

3.5 A false lead worth recording

An earlier diagnosis ("0-row splayed dirs with dict columns cannot be
read back") came from a repro that wrote with an external symfile
path and read with none:

(.db.splayed.set "/tmp/rayissue/r5/T/" E "/tmp/rayissue/r5/external.sym")   ; sym OUTSIDE the dir
(.db.splayed.get "/tmp/rayissue/r5/T/")                                     ; no sym argument
→ error (sym resolution)

splay_resolve_sym finds no <dir>/.sym and the parent is not
partition-shaped, so the first SYM column raises the loud "sym" error
(col.c require_dom path). That is correct behavior — symfile
resolution, not a nested-column defect.

4. Root cause

ray_parted_fill (src/store/part.c:644-773) builds each missing empty by
reading the newest partition holding the table and cloning it row-less:

/* part.c:556-571 */
static ray_t* empty_table_like(ray_t* tmpl) {
    ...
    for (int64_t c = 0; c < ncols; c++) {
        ray_t* col = ray_table_get_col_idx(tmpl, c);
        ...
        ray_t* ecol = ray_vec_new(col->type, 0);          /* :563 — the bug */
        if (!ecol || RAY_IS_ERR(ecol)) { ray_release(out); return ...; }

/* src/vec/vec.c:155-157 */
ray_t* ray_vec_new(int8_t type, int64_t capacity) {
    if (type <= 0 || type >= RAY_TYPE_COUNT)
        return ray_error("type", "vec_new: type must be a positive concrete vector type, got %s", ...);

RAY_LIST == 0 (include/rayforce.h:70), so a LIST column trips
type <= 0 and empty_table_like returns a "type" error. (RAY_TABLE
98 / RAY_DICT 99 would trip type >= RAY_TYPE_COUNT (15) the same
way, but neither can occur as a stored column: save preflight refuses
bare DICT columns — test_splay_save_preflight_preserves_generation —
and nothing writes nested-TABLE columns.)

The correct constructor for the LIST case exists:
ray_list_new(0) (include/rayforce.h:550, src/vec/list.c) accepts
capacity 0 and returns an empty RAY_LIST.

5. Why nothing else needs fixing (on-disk + read path facts)

• Writer: col_save_impl (src/store/col.c) dispatches LIST columns to
  the LSTG container; but is_str_list (col.c:209) is vacuously true
  for a 0-element LIST, so an empty LIST column is written as a
  12-byte STRL file — magic + i64 count 0:

  populated LIST-of-DICT (LSTG):  4c53 5447 0002 0000 0000 0000 0063 0501 …
                                  L S T G  count=2      DICT(0x63) elems…
  0-row LIST (STRL, 12 bytes):    5354 524c 0000 0000 0000 0000
                                  S T R L  count=0
  

• Reader: the mmap validator rejects extended-magic files with nyi,
  and the splayed loader falls back to the buffered loader
  (src/store/splay.c:426-433) which magic-dispatches STRL →
  col_load_str_list → ray_list_new(0) for count 0. Verified
  empirically by §3.4.

• Parted read: ray_read_parted reuses the same splayed loader per
  partition (part.c:283) and its cross-partition schema check compares
  name + type per index — empty RAY_LIST == populated RAY_LIST. Empty
  segments contribute 0 rows.

So a fill that merely constructs the empty LIST column correctly will
save, load, and integrate with parted reads with no further changes.

6. Secondary findings

(a) Error flattening. Two of the three failure sites inside the fill
loop overwrite the real cause with RAY_ERR_IO before the generic wrap:

/* part.c:719-723 — template read failed */        err = RAY_ERR_IO;  /* real code lost */
/* part.c:725-731 — empty_table_like failed */     err = RAY_ERR_IO;  /* "type" lost   */
/* part.c:739-740 — ray_splay_save failed */       err = se;          /* real code KEPT */
/* part.c:768-770 — the wrap */
return ray_error(ray_err_code_str(err), "parted %s: fill failed", db_root);

The surfaced error is io: parted <root>: fill failed for a "type"
bug, a "corrupt" template, or a torn partition alike — with no table,
column, or cause. Diagnosing this issue required reading the source; an
operator would have nothing to act on.

(b) Non-atomicity. §3.3: fills committed before the failure remain on
disk; later tables are skipped. Fill is idempotent-by-design (a re-run
after a fix completes the rest), so partial state is not corrupting — but
combined with (a), a failed fill leaves the root changed with no
indication of how far it got.

(c) The sym_path edge. Fill passes a symfile only if <root>/.sym
already exists (part.c:664-670). A sym-bearing template in a root with no
.sym yet fails at the template read (sym error, flattened to
io). In practice the root .sym predates any fill (the writes that
created the partitions create it); noted for completeness when reading
the code.

(d) Test coverage gap. The only fill test,
test/rfl/system/db_parted_fill.rfl, uses flat SYM+I64 tables. The
nested-column splay roundtrip test (test_store.c
splay_dict_column_roundtrip) covers only the non-empty case. Nothing
covers: 0-row nested columns, fill over a nested-column table, or the
error text of a failed fill.

(e) Same guard elsewhere. Other ray_vec_new(<dynamic type>, …)
sites exist in the ops layer (datalog.c, query.c, collection.c,
sort.c, idxop.c) with the same LIST landmine, but none is reachable
from fill; they are out of scope for this issue (worth a separate audit).

7. Impact on applications

• Any parted root mixing LIST-column tables with tables whose partition
  sets can diverge — added-later tables, quiet tables crossing a
  partition boundary, historical backfills — becomes unopenable at the
  first gap, because gap-filling reads fail root-wide.
• The opaque io gives operators nothing to act on; the natural (wrong)
  conclusions are disk trouble or store corruption.
• Available workarounds (separate root per column-shape, or banning LIST
  columns from parted stores) push complexity into every application and
  still leave the mixed root exposed to the quiet-table case.

8. Proposed fix (engine)

1. empty_table_like (part.c:563): dispatch on the column type —
   RAY_LIST → ray_list_new(0); RAY_TABLE/RAY_DICT → explicit
   ray_error("nyi", …) naming the column (defensive; cannot occur from
   disk); everything else → existing ray_vec_new (SYM already routes to
   ray_sym_vec_new internally).
2. Error fidelity (part.c:719-731, 768-770): capture the cause code +
   message at the failure sites (the message must be copied immediately —
   it lives in the thread-local ray_last_err_msg rewritten by every
   ray_error call) and wrap as
   ray_error(cause_code, "parted %s: fill failed: %s", db_root, cause) —
   the outer parted <root>: fill failed shape is preserved (a prefix of
   the new message) for any caller that matches on it, while the code
   becomes the real cause (type/corrupt/sym) and the detail names
   the failing table/column.
3. Leave the sym_path edge as is (§6c) — after (2) it self-describes.

Proposed tests

• Extend test/rfl/system/db_parted_fill.rfl: sym + LIST-of-DICT table
  present only in the newest day → fill backfills the older day; parted
  read-back (row count, 'DICT cell type, a value probe); idempotent
  second fill returns []; an error-fidelity case (garbage .d in a
  template → !- corrupt, which pre-fix reported io).
• New C test beside splay_dict_column_roundtrip: 0-row i64+sym+LIST
  table, dir-local .sym, ray_splay_save + ray_read_splayed
  roundtrip (pins §3.4 so the vacuous-is_str_list/STRL behavior can
  never regress silently).

9. Affected versions / verification

• Present in v2.5.13 and v2.5.14 (6067f3a7); empty_table_like is
  unchanged between them (v2.5.14's store changes are STR hash-cache
  validation in col.c, unrelated).
• Verification: engine make test (ASan) green including the new cases;
  repros §3.1 and §3.3 flip from error to [2024.01.01]-style answers;
  §3.2/§3.4 stay green (regression guards).

Appendix: repro scripts

Run from the engine repo root (./rayforce <script>); each script is
self-contained under /tmp/rayissue/.

repro1_fill_fails.rfl — §3.1

(set 'FLAT (table [s v] (list ['x 'y] [1 2])))
(set 'SCHED (table [acct sched]
  (list ['a 'b]
        (list (dict [0] [5.0])
              (dict [0 250000] [15.0 10.0])))))
(.db.splayed.set "/tmp/rayissue/r1/2024.01.01/A/" FLAT)
(.db.splayed.set "/tmp/rayissue/r1/2024.01.02/A/" FLAT)
(.db.splayed.set "/tmp/rayissue/r1/2024.01.02/SCHED/" SCHED)
(println (.db.parted.fill "/tmp/rayissue/r1/"))

repro2_flat_ok.rfl — §3.2 (control): as repro 1 with
(table [acct rate] (list ['a 'b] [5.0 10.0])) in place of SCHED —
fill answers [2024.01.01], parted count 2.

repro3_partial.rfl — §3.3

(set 'FLAT (table [s v] (list ['x 'y] [1 2])))
(set 'SCHED (table [acct sched] (list ['a] (list (dict [0] [5.0])))))
(.db.splayed.set "/tmp/rayissue/r3/2024.01.01/M_FLAT/" FLAT)
(.db.splayed.set "/tmp/rayissue/r3/2024.01.02/A_FLAT/" FLAT)
(.db.splayed.set "/tmp/rayissue/r3/2024.01.02/M_FLAT/" FLAT)
(.db.splayed.set "/tmp/rayissue/r3/2024.01.02/N_SCHED/" SCHED)
(.db.splayed.set "/tmp/rayissue/r3/2024.01.02/Z_FLAT/" FLAT)
(println (try (.db.parted.fill "/tmp/rayissue/r3/") (fn [e] 'FILL_FAILED)))
;; afterwards 2024.01.01 holds A_FLAT (created) + M_FLAT (pre-existing); Z_FLAT never attempted

repro4_empty_roundtrip.rfl — §3.4

(set 'E (table [id who sched]
  (list (take [0] 0) (take ['x] 0) (list))))
(.db.splayed.set "/tmp/rayissue/r4/T/" E "/tmp/rayissue/r4/T/.sym")
(println (count (.db.splayed.get "/tmp/rayissue/r4/T/" "/tmp/rayissue/r4/T/.sym")))

repro5_false_lead.rfl — §3.5

(set 'E (table [id who sched]
  (list (take [0] 0) (take ['x] 0) (list))))
(.db.splayed.set "/tmp/rayissue/r5/T/" E "/tmp/rayissue/r5/external.sym")
(println (try (.db.splayed.get "/tmp/rayissue/r5/T/") (fn [e] 'GET_FAILED)))