Repository navigation
fix: align Hive decimal hashing and extend unit coverage - #510
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fix two Hive Decimal hashing differences from Java: include every 32-bit word of the unscaled magnitude in the hash, and apply HALF_UP once when reducing scale by multiple digits. For example, Decimal 4294967296 now hashes to 961, and unscaled=149, scale=40 hashes to 69.
Add focused coverage for GenericArray, default file-index predicates, histogram windows, object-store filesystem operations, REST message serialization, MergeTreeWriter error propagation, and Parquet filtering/fallback paths. The Parquet tests cover missing page indexes, disjoint predicate/bitmap ranges, no-predicate index I/O, and recovery from a synchronous prebuffer exception. Decimal overflow tests document the existing C++ zero fallback and Java null/NullPointerException behavior.
Tests
Add
GenericArrayTestcoverage for typed getters, string/binary representations, borrowed-value lifetimes, primitive-array conversion, empty arrays, and rejection of null primitive elements.Add
HiveHasherTestcoverage for integer/long/byte hashing, Decimal normalization, complete magnitude hashing, and precision overflow. AddHiveBucketFunctionTest.TestDecimalHighBitsCompatibleWithJavato verify Java-compatible bucket assignment.Extend
HistogramWindowingImplTestfor window rotation and expiration, reset at or beyond the maximum span, merge-window alignment, clock rollback, zero windows/duration, sample-count gating, and clone independence.Extend
FileIndexReaderTest.TestDefaultIndexReaderto verify conservative results for starts-with, ends-with, contains, and LIKE predicates.Add object-store existence/directory-status and seek/empty-read tests; extend directory-listing coverage and assert error messages for invalid reads, closed streams, and directory opens.
Add
RestMessagesTest.ConfigResponseMissingOrNullMapsand extend snapshot-list, database, and table response tests to cover serialization round trips.Add Parquet tests for bitmap trim falling back to whole-row-group reads without page indexes, disjoint predicate/bitmap page ranges producing EOF, and successful data reads after a synchronous prebuffer exception. Extend the page-index test to verify that a null predicate returns the complete row group without additional index I/O.
Update
WithOptionalIOManager/MergeTreeWriterTest.TestSortedReaderFailureto verify the IOError type, original error message, and reader cleanup with and without the optional IOManager, including builds withPAIMON_EXTRA_ERROR_CONTEXT.API and Format
No public API, file format, or protocol changes. Corrected Decimal hashes can change Hive bucket assignments for affected values to match Java.
Documentation
Inline comments explain the Decimal hash calculation and Java overflow behavior. No new user-facing feature documentation is needed.
Generative AI tooling
Generated-by: OpenAI Codex 0.163.0