Add Utf8 types for RowFn - #9715
Conversation
Add `Utf8Column` and `Utf8Sink` to the unstable row function types. `Utf8Column` decodes a `Utf8` column once per batch and hands each row callback a `Utf8View`. The view dereferences to `str`, so a callback calls `str` methods on it directly. It also exposes the string-view length, prefix and inline flag, which a comparison kernel can read before it reads the bytes. The column also accepts a batch constant. `Utf8Sink` builds a `VarBinViewArray` output. It initializes every row to an empty string, so its skipped-row initializer does no work and skip-invalid execution needs no input filter. The decode step validates every view and replaces the view of each null row with an empty view. A dense callback can therefore read the payload of a null row, and `Utf8View::as_str` can skip the UTF-8 check. These types come from the SpiralDB engine, where the `spql.str.*` functions use them. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
8a7667a to
0c141b1
Compare
Merging this PR will regress 2 benchmarks
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | arrow_checked_add_u32_neon[16384] |
12.4 µs | 20.4 µs | -39.24% |
| ❌ | Simulation | compact[(2048, 90)] |
1.4 µs | 1.5 µs | -10.63% |
| ⚡ | WallTime | add_shapes_neon[(128, PerRowPerRow)] |
3.7 µs | 1.9 µs | +90.91% |
| ⚡ | WallTime | multiply_shapes_neon[(128, PerRowPerRow)] |
2.9 µs | 2 µs | +47.49% |
| ⚡ | WallTime | subtract_shapes_neon[(128, PerRowPerRow)] |
2.9 µs | 2 µs | +47.01% |
| ⚡ | WallTime | mul_u64_nonnull_neon |
21.2 µs | 15.2 µs | +39.44% |
| ⚡ | WallTime | mul_i64_nonnull_neon |
20.9 µs | 17.2 µs | +21.79% |
| ⚡ | WallTime | multiply_shapes_neon[(16384, PerRowPerRow)] |
20.7 µs | 17 µs | +21.53% |
| ⚡ | WallTime | arrow_checked_add_u32_avx512[16384] |
21.3 µs | 17.6 µs | +20.5% |
| ⚡ | WallTime | arrow_checked_add_u32_avx2[16384] |
21.3 µs | 17.7 µs | +20.25% |
| ⚡ | WallTime | add_u32_nonnull_neon |
7.9 µs | 6.6 µs | +20.22% |
| ⚡ | WallTime | add_shapes_neon[(16384, PerRowPerRow)] |
11.3 µs | 9.8 µs | +16.29% |
| ⚡ | WallTime | add_i32_nonnull_neon |
9 µs | 7.8 µs | +15.29% |
| ⚡ | WallTime | add_i64_nonnull_neon |
11.3 µs | 9.8 µs | +15.28% |
| ⚡ | WallTime | compare_int_nullable_neon |
5.6 µs | 4.9 µs | +14.29% |
| ⚡ | WallTime | words_gather_scalar_avx2[65536] |
9.4 µs | 8.2 µs | +13.61% |
| ⚡ | Simulation | cold_misaligned[(16, 64)] |
392.4 µs | 346.1 µs | +13.38% |
| ⚡ | WallTime | add_i64_nullable_neon |
12.8 µs | 11.5 µs | +11.42% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ct/row-fn-string (0c141b1) with develop (a67cd1b)
Footnotes
-
206 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
4 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩
connortsui20
changed the title
Utf8 types for RowFn
1 participant
Summary
Adds
Utf8column and view types for theRowFnframework.(This is getting upstreamed from https://github.com/spiraldb/spiraldb/pull/6858)
Changes
Adds
Utf8ColumnandUtf8ViewasInputElementand decode targets for use in Vortex.We should figure ou tif we want to ship a string library with Vortex (since there is not equivalent string library in Arrow)