[connortsui20]

Summary

• Tracking Issue: Tracking Issue: Fix Canonical Decimal #10701

Every decimal precision now has one canonical storage width, DecimalDType::canonical_values_type(), and the Arrow and DuckDB imports produce arrays at that width. Before this, the Arrow import kept whatever width Arrow used (a Decimal128 column at precision 10 arrived as i128), and the DuckDB import narrowed precision 1 and 2 with a lossy as cast.

This doesn't make the canonical width an invariant yet. DecimalArray still accepts any storage width, and every kernel, writer and exporter already handles mixed widths, so only the two imports change behavior. Enforcing the width in try_new comes later in #10701, once DecimalByteParts decoding, the read path for old files and the compressor stop producing other widths.

Changes

canonical_values_type() replaces DecimalType::smallest_decimal_value_type with the same precision-to-width table, so the migrated call sites behave as before. Two shared helpers in vortex_array::arrays::decimal do the copies that the imports need now and that the later steps reuse. copy_to_canonical_width copies values into the canonical width, and a narrowing copy fails on a valid value outside the precision and writes zeros in null slots. check_precision checks values in place.

The Arrow import stays zero-copy when the Arrow width already matches and copies otherwise. Either way, every valid value is checked against the precision, so a value outside it now fails at import instead of reaching scalar_at or min_max, which can panic on it (#10749). The error points at arrow-rs PrimitiveArray::null_if_overflow_precision for callers that would rather get nulls. The dtype is built with DecimalDType::try_new, so an invalid precision or scale from FFI is an error rather than a panic. The DuckDB import goes through the same helpers.

This overlaps with #10305, which touches many of the same files. For review, read the commits one at a time.

API Changes

DecimalType::smallest_decimal_value_type is removed in favor of DecimalDType::canonical_values_type(). Arrow and DuckDB decimal imports return arrays at the canonical width and fail on a valid value outside the declared precision.

[codspeed]

Merging this PR will degrade performance by 14.94%

3 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 3 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

⚡ 5 improved benchmarks
❌ 11 regressed benchmarks
✅ 2192 untouched benchmarks
⏩ 366 skipped benchmarks¹
🗄️ 73 archived benchmarks run²

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	CPU Simulation	filter_powerlaw_by_random[10000]	71.7 ns	812.7 ns	-91.18%
❌	CPU Simulation	test_repeated_scan_scalar_at[int_str]	336.2 µs	412.1 µs	-18.41%
❌	CPU Simulation	bench_compare_varbin[(50000, 2)]	134.5 µs	164.8 µs	-18.38%
❌	CPU Simulation	bench_compare_varbin[(50000, 64)]	136.2 µs	166.4 µs	-18.14%
❌	CPU Simulation	bench_compare_sliced_dict_varbinview[(7500, 10000)]	195.6 µs	230.2 µs	-15.02%
❌	CPU Simulation	bench_compare_sliced_dict_varbinview[(1000, 10000)]	81.2 µs	92.5 µs	-12.22%
❌	CPU Simulation	bench_compare_sliced_dict_varbinview[(10000, 10000)]	207.9 µs	236.3 µs	-12.02%
❌	Walltime	add_constant_shapes_neon[(32768, PerRowConstant)]	14.4 µs	16.1 µs	-10.92%
❌	Walltime	add_constant_shapes_neon[(32768, PerRowNullableConstant)]	14.9 µs	16.7 µs	-10.65%
❌	Walltime	add_constant_shapes_neon[(32768, ConstantPerRow)]	14.4 µs	16.1 µs	-10.48%
❌	Walltime	starts_with_avx512[8, VarBinView]	68.4 µs	76.3 µs	-10.34%
⚡	Walltime	mul_u64_nonnull_neon	39.1 µs	20.9 µs	+87.38%
⚡	CPU Simulation	missing_stats[16]	88.2 µs	71.6 µs	+23.12%
⚡	CPU Simulation	encode_primitives[u8, (2000, 4)]	164.3 µs	135.6 µs	+21.19%
⚡	Walltime	starts_with_avx512[8, VarBin]	46.1 µs	39.4 µs	+17.08%
⚡	Walltime	starts_with_avx512[64, VarBin]	60.6 µs	53.2 µs	+13.87%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ct/decimal-width-owner (9836fef) with develop (487bdba)}

Footnotes

1. 366 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. 73 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩