Repository navigation
[spark] Fix read batch time for the first batch and active splits - #10835
Open
wangzhigang1999 wants to merge 1 commit into
Open
wangzhigang1999 wants to merge 1 commit into
wangzhigang1999 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #10834
Purpose
Spark's
read batch timecan omit the first batch and the split being read when a task finishes.PaimonRecordReaderIteratorinitializes its counter after the firstreadBatch()call, clearing that batch's elapsed time.PaimonPartitionReaderonly reports completed splits, while Spark takes its final metric snapshot before closing the reader. A query that stops within a split, such asLIMIT, can therefore report zero or incomplete batch time.Initialize the counter before the first read and include the active split in metric snapshots. Use the same helper when finishing a split and closing the partition reader: close the reader, add its elapsed time to the completed total, then clear the reference. If closing throws, retain the reference and completed total so a later cleanup call can retry. Successful repeated closes do not add the time again.
The change preserves the metric name, timing scope, and per-split conversion from nanoseconds to milliseconds.
Tests
Add two tests to the existing
PaimonMetricTestsuite:The timing tests delay the first batch by 25 ms and assert a 20 ms lower bound. The EOF cases take their comparison snapshot after the EOF read, since that read can add elapsed time.
Local results for the final test source:
spark3spark4The 13 tests comprise 11 existing tests and 2 new tests; the second new test runs all four close/EOF combinations. Both runs used JDK 17 and the normal build checks. Dependencies, including the updated common reader bundled into the Spark version module, were installed before running the target suite.