[core] Read expire_tags older_than as a wall clock, not as an instant - #9263
Open
PDGGK wants to merge 1 commit into
Open
[core] Read expire_tags older_than as a wall clock, not as an instant#9263PDGGK wants to merge 1 commit into
PDGGK wants to merge 1 commit into
Conversation
parseTimestampData(s, p, tz) returns a Timestamp holding a true epoch milli, and Timestamp.toLocalDateTime() renders that milli as a UTC wall clock. Composing them shifts the cutoff by the JVM's UTC offset, while what it is compared against -- TagManager stores LocalDateTime.now() -- is a local wall clock. For the documented example the cutoff becomes 15:00 in America/New_York and 03:00 in Asia/Shanghai instead of the 11:00 the user typed, deleting tags that were meant to be kept or keeping ones meant to expire. The existing tests fed the procedure a string produced by fromLocalDateTime().getMillisecond() rendered through java.sql.Timestamp, which pre-shifts by exactly the offset the procedure re-applies, so they passed in every timezone. They now pass the plain wall clock a user types.
PDGGK
force-pushed
the
fix-expire-tags-timezone
branch
from
August 17, 2026 18:02
9ee74ac to
d0a994e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
expire_tags'solder_thanargument is shifted by the JVM's UTC offset, so on any non-UTC deployment the cutoff is not the time the user typed.The two halves disagree about what the value is:
parseTimestampData(s, p, tz)isTimestamp.fromInstant(local.atZone(tz).toInstant())(DateTimeUtils:547-552), so the resultingTimestampholds a true epoch milli;Timestamp.toLocalDateTime()(Timestamp:92-103) divides that milli byMILLIS_PER_DAYwith no zone at all — it renders the instant as a UTC wall clock. It is the exact inverse offromLocalDateTime, not offromInstant.What it is compared against is a local wall clock: tags are created with
LocalDateTime.now()(TagManager:168), the fallback create time isDateTimeUtils.toLocalDateTime(modificationTime)=atZone(systemDefault())(TagTimeExpire:81), and the sibling retention check usesLocalDateTime.now()(:88). The comparison isolderThanTime.isAfter(createTime)at:89.Measured, for the documented example
older_than => '2024-09-06 11:00:00'(docs/flink/procedures.md):2024-09-06 11:002024-09-06 15:002024-09-06 03:00It is wrong in both directions: west of UTC it deletes tags the user asked to keep — and once a tag's snapshot has expired,
TagManager.deleteTagfalls through to cleaning the data files, so that is not recoverable. East of UTC it keeps tags the user asked to expire, which is quieter but still not what was requested.What changes
Three call sites —
paimon-flink-common,paimon-flink-1.18andpaimon-spark-common— parse the argument as the wall clock it is:DateTimeUtils.toLocalDateTime(String, int)(:563) isfromTemporalAccessor(...)with no zone conversion, so the parsed wall clock is compared against wall clocks.Deliberately not touched: the other
parseTimestampData(..., TimeZone.getDefault())call sites are correct, because they keep the instant rather than re-rendering it —ProcedureUtils:91takes.getMillisecond()and subtracts it fromSystem.currentTimeMillis(), andOrphanFilesClean:477compares theTimestampagainstTimestamp.fromEpochMillis(System.currentTimeMillis()). Both sides are epoch millis there. The defect is specificallyparseTimestampData(..., tz)followed by.toLocalDateTime().Why the existing tests did not catch it
ExpireTagsProcedureITCaseandExpireTagsActionTestdo not pass the procedure a wall clock. They take a tag's create time and put it through a round-trip that happens to cancel the bug out:fromLocalDateTime(...).getMillisecond()encodes a local wall clock as if it were UTC, andjava.sql.Timestamp.toString()then renders that instant back in the local zone — so the string handed to the procedure is already pre-shifted by exactly the offset the procedure re-applies. Measured, the composition returns the original11:00in UTC,America/New_YorkandAsia/Shanghaialike.A user typing the documented form does not do that dance, so the tests were green while the feature was wrong. Both are updated here to pass the plain wall clock, which is what the procedure's argument is.
Test evidence
masterbehaviour restored, tests as updatedThe failure names the symptom directly —
expire_tagsreturned tags that the assertion lists under "elements not expected".The UTC column is why this is worth flagging rather than assuming CI would have caught it: under
-Duser.timezone=UTCthe change is a no-op and every test passes either way.API and Format
No change to any option, on-disk format or public signature. The interpretation of
older_thanchanges on non-UTC JVMs, which is the point; a caller that had empirically compensated for the shift would need to stop doing so.