Repository navigation
Build precision 19 decimal scalars at i128 - #10903
connortsui20 wants to merge 1 commit into
Conversation
| /// The [`DecimalValue`] variant of the canonical width of `dtype`'s precision. | ||
| fn canonical_decimal_value(value: i256, dtype: &DecimalDType) -> PyResult<DecimalValue> { |
There was a problem hiding this comment.
This might blow things up but I think that is fine because it is a scalar, not an array?
The Python and JNI decimal scalar factories picked the value width from
`DecimalDType::required_bit_width()`, which counts magnitude bits without a
sign bit. At precision 19 it returns 64, so both factories chose `i64`, and a
valid value above `i64::MAX` failed with an overflow error. For example,
`vx.scalar(decimal.Decimal("9999999999999999999"))` failed.
Both factories now match on `DecimalDType::canonical_values_type()`, which
gives `i128` at precision 19. Every other precision keeps its width.
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
81cc7b1 to
41073b9
Compare
Merging this PR will degrade performance by 13.54%3 warnings
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
Summary
vx.scalar(decimal.Decimal("9999999999999999999"))fails with an overflow error, and so does a JNI decimal literal at precision 19 with a value abovei64::MAX. Both factories picked the value width fromDecimalDType::required_bit_width(), which counts magnitude bits without a sign bit. At precision 19 that is 64, so they chosei64, which can't hold the largest 19-digit values.Changes
Both factories now match on
DecimalDType::canonical_values_type()from #10897, which givesi128at precision 19. Every other precision keeps its width, because the sign bit still fits there. The Python precision inference test and a new JNI unit test cover the largest and smallest values at precision 19.This is stacked on #10897 and retargets to
developafter that merges.