Skip to content

[python] Fix GenericVariant UUID decoding for Java compatibility and support datetime/UUID - #9204

Open
juntaozhang wants to merge 2 commits into
apache:masterfrom
juntaozhang:pr-variant-datetime-uuid-encoding-compatibility
Open

[python] Fix GenericVariant UUID decoding for Java compatibility and support datetime/UUID#9204
juntaozhang wants to merge 2 commits into
apache:masterfrom
juntaozhang:pr-variant-datetime-uuid-encoding-compatibility

Conversation

@juntaozhang

Copy link
Copy Markdown
Contributor

Purpose

  • This PR extends pypaimon's GenericVariant to support encoding native Python datetime and uuid.UUID values, and fixes the UUID binary decoding byte order to align with Java.
  • Completes the Python API documentation for supported shredding schema types

Tests

  • Unit tests: paimon-python/pypaimon/tests/variant_test.py
  • Java-Python E2E tests: JavaPyE2ETest and java_py_read_write_test.py

@JingsongLi

Copy link
Copy Markdown
Contributor
  • Python datetime encoding relies on a floating-point calculation (total_seconds() * 1_000_000), which leads to a loss of microsecond precision for dates far removed from the epoch.
  • Empirical testing shows a round-trip for 9998-07-08 ...654321 results in ...654336 (an error of 15μs); errors of ±1μs also occur for the years 1600, 1900, 2500, and 5000.
  • The implementation should be changed to use pure integer arithmetic based on days, seconds, and microseconds.

@juntaozhang

Copy link
Copy Markdown
Contributor Author
  • Python datetime encoding relies on a floating-point calculation (total_seconds() * 1_000_000), which leads to a loss of microsecond precision for dates far removed from the epoch.

    • Empirical testing shows a round-trip for 9998-07-08 ...654321 results in ...654336 (an error of 15μs); errors of ±1μs also occur for the years 1600, 1900, 2500, and 5000.

    • The implementation should be changed to use pure integer arithmetic based on days, seconds, and microseconds.

@JingsongLi Thanks for your review. I have fixed the issue, PTAL.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants