Repository navigation
[FLINK-41265][formats] Add a metric for JSON messages with ignored parse errors. - #29670
Open
danzhewuju wants to merge 1 commit into
Open
danzhewuju wants to merge 1 commit into
danzhewuju wants to merge 1 commit into
Conversation
…rse errors. Signed-off-by: danzhewuju <danyuhao@qq.com> Generated-by: GPT-6 Astra
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
This PR addresses FLINK-41265 by exposing a metric for input messages affected by ignored JSON parsing or field conversion errors.
In production,
json.ignore-parse-errorsmay have been enabled long before an incident. Users investigating missing records or unexpected null fields may no longer be aware of this configuration and suspect data loss in Flink while the job continues running.The new counter makes these events visible through monitoring dashboards and alerts. Users can identify when errors start occurring, observe their frequency over time, and determine whether the problem is still ongoing.
Brief change log
json.numMessagesWithParseErrorsthrough the deserialization initialization context when the plain JSON format usesjson.ignore-parse-errors=true.The scope is limited to the plain JSON format. CDC formats do not report this metric. Connector-specific metric context propagation and key/value metric-group isolation are outside this PR.
Verifying this change
Added
JsonParseErrorMetricsTest, parameterized for both JSON decoders, covering:Updated
JsonFormatFactoryTestto account for metric activation by the format factory.Local test execution and the full
./mvnw clean verifyhave not been completed.Does this pull request potentially affect one of the following parts:
@Public(Evolving): No.Documentation
The documentation explains that the counter measures affected input messages, including messages that still emit rows with null fields, rather than the number of dropped output rows.
Was generative AI tooling used to co-author this PR?
Generated-by: GPT-6 Astra