Skip to content

[FLINK-41265][formats] Add a metric for JSON messages with ignored parse errors. - #29670

Open
danzhewuju wants to merge 1 commit into
apache:masterfrom
danzhewuju:add-metrics-for-ignore-parse-errro
Open

danzhewuju wants to merge 1 commit into
apache:masterfrom
danzhewuju:add-metrics-for-ignore-parse-errro

Conversation

@danzhewuju

Copy link
Copy Markdown
Contributor

What is the purpose of the change

This PR addresses FLINK-41265 by exposing a metric for input messages affected by ignored JSON parsing or field conversion errors.

In production, json.ignore-parse-errors may have been enabled long before an incident. Users investigating missing records or unexpected null fields may no longer be aware of this configuration and suspect data loss in Flink while the job continues running.

The new counter makes these events visible through monitoring dashboards and alerts. Users can identify when errors start occurring, observe their frequency over time, and determine whether the problem is still ongoing.

Brief change log

  • Register json.numMessagesWithParseErrors through the deserialization initialization context when the plain JSON format uses json.ignore-parse-errors=true.
  • Track ignored errors in both the JsonNode and streaming JsonParser decoders, including errors handled inside field converters.
  • Increment the counter at most once per input message. A top-level JSON array counts as one message, even if multiple elements contain errors.
  • Include messages that still produce rows with null fields while preserving existing deserialization behavior.
  • Add focused tests and document the metric in the English and Chinese JSON format documentation.

The scope is limited to the plain JSON format. CDC formats do not report this metric. Connector-specific metric context propagation and key/value metric-group isolation are outside this PR.

Verifying this change

Added JsonParseErrorMetricsTest, parameterized for both JSON decoders, covering:

  • Counting malformed messages and field conversion failures, with at most one increment per input message.
  • Keeping the counter unchanged for valid messages, accepted null or missing fields, and empty inputs.
  • Preserving strict-mode exception behavior without registering the ignored-error counter.

Updated JsonFormatFactoryTest to account for metric activation by the format factory.

Local test execution and the full ./mvnw clean verify have not been completed.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): No.
  • The public API, i.e., is any changed class annotated with @Public(Evolving): No.
  • The serializers: Yes, the JSON deserialization implementation is updated. No state serializer or checkpoint format is changed.
  • The runtime per-record code paths (performance sensitive): Yes, error tracking and message-level counter updates are added to JSON deserialization.
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: No.
  • The S3 file system connector: No.

Documentation

  • Does this pull request introduce a new feature? Yes, a JSON format metric.
  • If yes, how is the feature documented? In the English and Chinese SQL JSON format documentation.

The documentation explains that the counter measures affected input messages, including messages that still emit rows with null fields, rather than the number of dropped output rows.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: GPT-6 Astra

…rse errors.

Signed-off-by: danzhewuju <danyuhao@qq.com>
Generated-by: GPT-6 Astra
@flinkbot

flinkbot commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants