Skip to content

fix: align Hive decimal hashing and extend unit coverage - #510

Merged
lxy-9602 merged 4 commits into
apache:mainfrom
zjw1111:codex/add-targeted-unit-tests
Nov 4, 2026
Merged

lxy-9602 merged 4 commits into
apache:mainfrom
zjw1111:codex/add-targeted-unit-tests

Conversation

@zjw1111

@zjw1111 zjw1111 commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Purpose

Fix two Hive Decimal hashing differences from Java: include every 32-bit word of the unscaled magnitude in the hash, and apply HALF_UP once when reducing scale by multiple digits. For example, Decimal 4294967296 now hashes to 961, and unscaled=149, scale=40 hashes to 69.

Add focused coverage for GenericArray, default file-index predicates, histogram windows, object-store filesystem operations, REST message serialization, MergeTreeWriter error propagation, and Parquet filtering/fallback paths. The Parquet tests cover missing page indexes, disjoint predicate/bitmap ranges, no-predicate index I/O, and recovery from a synchronous prebuffer exception. Decimal overflow tests document the existing C++ zero fallback and Java null/NullPointerException behavior.

Tests

  • Add GenericArrayTest coverage for typed getters, string/binary representations, borrowed-value lifetimes, primitive-array conversion, empty arrays, and rejection of null primitive elements.

  • Add HiveHasherTest coverage for integer/long/byte hashing, Decimal normalization, complete magnitude hashing, and precision overflow. Add HiveBucketFunctionTest.TestDecimalHighBitsCompatibleWithJava to verify Java-compatible bucket assignment.

  • Extend HistogramWindowingImplTest for window rotation and expiration, reset at or beyond the maximum span, merge-window alignment, clock rollback, zero windows/duration, sample-count gating, and clone independence.

  • Extend FileIndexReaderTest.TestDefaultIndexReader to verify conservative results for starts-with, ends-with, contains, and LIKE predicates.

  • Add object-store existence/directory-status and seek/empty-read tests; extend directory-listing coverage and assert error messages for invalid reads, closed streams, and directory opens.

  • Add RestMessagesTest.ConfigResponseMissingOrNullMaps and extend snapshot-list, database, and table response tests to cover serialization round trips.

  • Add Parquet tests for bitmap trim falling back to whole-row-group reads without page indexes, disjoint predicate/bitmap page ranges producing EOF, and successful data reads after a synchronous prebuffer exception. Extend the page-index test to verify that a null predicate returns the complete row group without additional index I/O.

  • Update WithOptionalIOManager/MergeTreeWriterTest.TestSortedReaderFailure to verify the IOError type, original error message, and reader cleanup with and without the optional IOManager, including builds with PAIMON_EXTRA_ERROR_CONTEXT.

API and Format

No public API, file format, or protocol changes. Corrected Decimal hashes can change Hive bucket assignments for affected values to match Java.

Documentation

Inline comments explain the Decimal hash calculation and Java overflow behavior. No new user-facing feature documentation is needed.

Generative AI tooling

Generated-by: OpenAI Codex 0.163.0

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lxy-9602
lxy-9602 merged commit becc48a into apache:main Nov 4, 2026
28 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants