fix: reject decimal promotion that changes the scale#3613
Open
anxkhn wants to merge 1 commit into
Open
Conversation
The DecimalType handler in promote() guarded scale equality with `file_type.scale == file_type.scale`, which compares the file scale to itself and is always true. As a result any decimal-to-decimal promotion with a widening precision was accepted regardless of the scale. Per the Iceberg spec a decimal may only be promoted when the scale is unchanged and the precision widens (decimal(P, S) to decimal(P2, S) with P2 > P), matching TypeUtil.isPromotionAllowed in the Java implementation. The bug affected both paths that use promote(): on read a differing-scale promotion built a reader at the wrong scale and reinterpreted the stored unscaled integers (silent data corruption), and on write it let a DataFrame column with a different scale pass the compatibility check. Compare the file scale to the read scale instead. The identical tautology in the test oracle masked the defect, so fix it too and add a decimal with a different scale to the promotion matrix, plus explicit regression tests for the schema and read paths. Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
The
DecimalTypehandler inpromote()(pyiceberg/schema.py) guarded scaleequality with
file_type.scale == file_type.scale, which compares the file'sscale to itself and is therefore always true. As a result any decimal-to-decimal
promotion with a widening precision was accepted regardless of whether the scale
matched.
Per the Iceberg spec, a
decimal may only be promoted when the scale is unchanged and the precision widens
(
decimal(P, S)todecimal(P2, S)withP2 > P; "scale cannot change"). Thismatches
TypeUtil.isPromotionAllowedin the Java reference implementation, whichrequires
fromDecimal.scale() == toDecimal.scale()andfromDecimal.precision() <= toDecimal.precision().The defect affected both code paths that call
promote():pyiceberg/avro/resolver.py), a differing-scale promotion was acceptedand a
DecimalReaderwas built at the read scale, reinterpreting the file'sstored unscaled integers at the wrong scale. For example a value stored as
1.23(unscaled123, scale2) would read back as0.0123at scale4.This is silent data corruption rather than an error.
_check_schema_compatibleinpyiceberg/schema.py), a DataFramecolumn of
decimal(9, 2)was accepted as compatible with a table column ofdecimal(18, 4).The fix compares the file scale to the read scale instead. The identical tautology
in the test oracle (
should_promoteintests/test_schema.py) masked the defect,so it is corrected as well.
Are these changes tested?
Yes.
tests/test_schema.py: fixed the mirrored tautology in theshould_promoteoracle, and added
DecimalType(10, 4)toTEST_PRIMITIVE_TYPESso the existingparametrized
test_promotionnow exercises differing-scale pairs (previously alldecimal fixtures were scale 2, so this case was never covered). Added
test_decimal_promotionwith explicit cases: widening precision at fixed scalesucceeds, equal precision/scale resolves, changing the scale raises, and reducing
the precision raises.
tests/avro/test_resolver.py: addedtest_resolve_decimal_to_decimal_change_scalecovering the read path (the data-corruption vector), and updated the existing
reduce-precision assertion to the generalized error message.
Reverting only the source change (keeping the tests) turns the new and
differing-scale cases red with "DID NOT RAISE ResolveError" (the exact corruption
symptom); with the fix the targeted suites pass (388 passed).
make lint(ruff,ruff-format, mypy, uv-lock, license/codespell) passes.
Integration tests that need Docker + Spark were not run in this environment; the
behavior is covered by the unit tests above.
Are there any user-facing changes?
Yes, a behavioral correctness change. A decimal-to-decimal promotion that changes
the scale is now correctly rejected with a
ResolveError(previously it wassilently accepted). This brings PyIceberg in line with the Iceberg spec and the
Java implementation. The error message in the else branch was generalized from
"Cannot reduce precision from {file_type} to {read_type}" to
"Cannot promote {file_type} to {read_type}", since that branch now also fires for
scale changes (matching the wording of the sibling promote handlers).