Skip to content

Fix MAL line attribution: separate the operator and machine coordinates - #13971

Open
wu-sheng wants to merge 4 commits into
masterfrom
feat/mal-line-attribution
Open

Fix MAL line attribution: separate the operator and machine coordinates#13971
wu-sheng wants to merge 4 commits into
masterfrom
feat/mal-line-attribution

Conversation

@wu-sheng

@wu-sheng wu-sheng commented Aug 8, 2026

Copy link
Copy Markdown
Member

Fix MAL generated-class and debug-record line attribution

  • If this is non-trivial feature, paste the links/URLs to the design doc.

  • Update the documentation to include this new feature.

    • docs/en/setup/backend/admin-api/dsl-debugging-mal.md (the sourceLine wire shape), docs/en/operation/dynamic-code-generation-debugging.md (the SourceFile format and the two coordinate spaces)
  • Tests(including UT, IT, E2E) are added to verify the new feature.

    • 31 new unit tests across 6 classes; see the verification table below.
  • If it's UI related, attach the screenshots below.

  • If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.

  • Update the CHANGES log.


The problem

A MAL rule's compiled expression lives in two independent coordinate spaces, and a single
overloaded yamlSource string was serving as class label, file identifier and line coordinate at
once. All four resulting artifacts were wrong:

artifact before why it was wrong
class name vm_L0_cpu_total_percentage L0 is the rules-list index, not a line — every file's first rule was L0
SourceFile (vm.yaml:0)cpu_total_percentage.java names a file that is never written (the file is named after the class), so IDE source-attach could never resolve — independent of the line being wrong
LineNumberTable 1,2,3,4,5,6,7 statement ordinals, matching neither the YAML nor the generated source
debug sourceLine omitted hardcoded 0 at all five probe sites; the docs said "omitted for MAL (no per-line mapping)" — true only because the mapping was never built

The fix

Operator coordinate — the rule .yaml the operator wrote — now reaches the dsl-debugging API
as a per-stage sourceLine. Per stage, not per rule, because formatExp splices expPrefix,
exp and expSuffix into one expression. Real generated output for a rule whose exp: is on line
38 and whose file-level expSuffix: is on line 32:

captureInput(..., "node_cpu",                         _node_cpu, 38);   // exp:
captureStage(..., "sum(['host'])",                    _node_cpu, 38);   // exp:
captureStage(..., "rate('PT1M')",                     _node_cpu, 38);   // exp:
captureStage(..., "service(['host'], Layer.GENERAL)", _node_cpu, 32);   // expSuffix:

A per-rule implementation emits 38 on all four and looks perfectly plausible.

Machine coordinate — the generated .java — is now self-consistent: SourceFile equals the
file actually written, and LineNumberTable points at real statements in it.

SourceFile        (vm.yaml:0)cpu_total_percentage.java  →  vm_L37_cpu_total_percentage.java
LineNumberTable   1,2,3,4,5,6,7                         →  11,12,13,14,15,16,17

File line 16 really is .service(...), 17 is return.

MalSourceRef carries both coordinates with distinct accessors, so substituting one for the other
is a visible mistake rather than the default. An unresolvable line is -1, never 0 or omitted, so
a resolution failure stays visible in the artifact; it renders as unknown inside class names,
where - is not legal in an identifier.

Deliberate decisions worth reviewing

  • Companion closure classes now emit no LineNumberTable. No .java is written for them, and
    the store-scan is not a statement-boundary detector there (one real companion had a single entry
    spanning 204 bytes). An absent table makes the JVM report "unknown line", which is honest; the
    previous one attributed every frame in the closure to line 1.
  • Source anchors are excluded from equals/hashCode. Otherwise adding a comment above a rule
    shifts its line, changes its identity, and triggers a spurious STRUCTURAL re-apply in
    runtime-rule — dropping and recreating BanyanDB measures for a semantically identical edit.
  • MALDebugRecorder's five append* methods gained a sourceLine parameter. One in-tree
    implementor; not a public API/SPI.
  • zabbix-rules is untouched. It is not a runtime-rule catalog, has no debug binding, and
    ZabbixConfig does not override getSourceName() — so its classes were never source-named and
    are unaffected.

Verification

check result
clean checkstyle:check (full repo) exit 0
license-eye header check 0 invalid / 4981
clean install javadoc:javadoc -Pall exit 0 — 90 jars, 12 apidocs
shipped MAL rules (MALExpressionExecutionTest) 1362 / 1362
shipped filters / LAL / parser 35 / 55 / 22
module suites 213, 206, 149, 49, 23, 10

The 22 parser tests confirm the injectExpPrefix rewrite (right-to-left replace → left-to-right
build, so splice ranges are recoverable) produces byte-identical output.

New tests, by coordinate:

coordinate tests
machine MalGeneratedSourceLinesTest (4), MalLineAttributionTest (4 — compiles for real, reads the written .java/.class, asserts SourceFile == filename and every table entry indexes a statement)
operator MalYamlLineIndexTest (6), MalPrefixInjectionRangesTest (4), MalSourceMapTest (5), MalProbeSourceLineTest (5), MalProbeEmissionTest (3 — runs with injection enabled and asserts the literal in the emitted probe)

The wrapper-geometry guard was mutation-tested: shifting the constant by one fails
MalLineAttributionTest with line 17 should hold a statement but was: '}'.

Both shipped MAL file-format variants are covered by test-scope fixtures (no bundled rules are read):
the zabbix metrics: rules key, and folded/literal block scalars, which anchor on their key line and
must not shift the rule that follows.

Not run locally: the dsl-debugging/mal e2e. Everything up to the emitted probe literal is
unit-verified, but a live debug session showing sourceLine in an API response is CI's first look.

A MAL rule's compiled expression lives in two independent coordinate spaces,
and one overloaded `yamlSource` string was serving as class label, file
identifier and line coordinate at once. It was wrong in all three roles:

- the `_L<n>_` segment in a generated class name was the rules-list INDEX,
  not a line, so every rule file's first rule was labelled `L0`;
- `SourceFile` embedded that same fake line AND named a file that is never
  written (it used the metric name, while the generated source file is named
  after the class), so IDE source-attach could never resolve a MAL stack
  frame regardless of whether the line was right;
- `LineNumberTable` held statement ordinals matching neither the YAML nor
  the generated source;
- MAL debug records carried no line at all: the recorder hardcoded 0 at all
  five probe sites, and the wire shape documented `sourceLine` as "omitted
  for MAL (no per-line mapping)" -- true only because the mapping had never
  been built.

The fix splits the three roles apart.

OPERATOR COORDINATE (the rule .yaml an operator wrote) now reaches the
dsl-debugging API as a per-STAGE `sourceLine`. Per stage rather than per
rule, because `formatExp` splices `expPrefix`, `exp` and `expSuffix` into
one expression: a stage contributed by the file-level `expSuffix:` must
report the suffix's line, not the rule's. A per-rule implementation gets
that silently wrong, since the rule's line does exist in the file.
`MalYamlLineIndex` resolves the four anchors from a second snakeyaml compose
pass over the same text bean binding discards; `MalSourceMap` attributes an
offset to the fragment that authored it, using splice ranges
`injectExpPrefix` already computed but discarded.

MACHINE COORDINATE (the generated .java) is now self-consistent:
`SourceFile` equals the file actually written, and `LineNumberTable` points
at real statements in it. `MalGeneratedSourceLines` derives the boundaries
from the same generated text the bytecode scan walks, so the two cannot
drift; a mismatch logs a warning. Companion closure classes emit no
`LineNumberTable` at all -- no source file is written for them and the
store-scan is not a boundary detector there, so an absent table (JVM reports
"unknown line") beats a degenerate one.

The class-name label keeps the YAML line, but the real one.

`MalSourceRef` carries both coordinates with distinct accessors so
substituting one for the other is a visible mistake rather than the default.
An unresolvable line is `-1`, never 0 or omitted, so a resolution failure
stays visible in the artifact instead of silently degrading; it renders as
`unknown` inside class names, where `-` is not legal.

Contract change: `MALDebugRecorder`'s five `append*` methods take an
additional `sourceLine`.

Rule behaviour is unchanged: 1362 shipped-rule execution tests, 35 filter
tests and 22 parser tests pass, the last confirming the `injectExpPrefix`
rewrite is byte-identical.
writeSourceFile used new FileWriter(file), which encodes with whatever the
JVM default charset is. Nothing pins that: project.build.sourceEncoding is
compile-time only, and no file.encoding or LANG is set in the poms or the
workflows, so the bytes of the generated source depended on the runner's
locale.

The header it emits carried an em dash, so every one of the 1402 generated
.java files had a non-ASCII byte on line 1. On a JVM whose default charset
encodes U+2014 outside ASCII, reading that file back with a strict UTF-8
decoder throws MalformedInputException -- which is how MalLineAttributionTest
failed in CI while passing locally.

Two layers: encode explicitly as UTF-8, which is required anyway because a
MAL string literal may legitimately be non-ASCII and would otherwise be
mangled; and drop the em dash from the header so the common path has no
charset dependency at all.

Verified by running the affected tests under -Dfile.encoding=GBK, which
reproduces the original failure without the fix.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes MAL operator-facing YAML attribution and generated Java debug coordinates.

Changes:

  • Adds YAML source indexing and per-stage debug line propagation.
  • Aligns generated SourceFile and LineNumberTable metadata.
  • Adds tests, documentation, and changelog updates.

Reviewed changes

Copilot reviewed 34 out of 34 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
MalFileApplier.java Stamps runtime-rule source lines.
MALDebugRecorderImpl.java Records probe source lines.
MalYamlLineIndexTest.java Tests YAML anchor indexing.
MalProbeSourceLineTest.java Tests per-stage attribution.
MalSourceMapTest.java Tests expression source mapping.
MalProbeEmissionTest.java Tests emitted probe literals.
MalPrefixInjectionRangesTest.java Tests prefix splice ranges.
MalLineAttributionTest.java Validates generated class metadata.
MalGeneratedSourceLinesTest.java Tests statement-line detection.
RuleSourceLines.java Applies source anchors to rules.
Rules.java Indexes disk-loaded rule text.
Rule.java Stores file-level anchors.
MetricsRule.java Stores per-rule anchors.
MetricRuleConfig.java Exposes source-line contracts.
MetricConvert.java Builds and propagates source maps.
MalYamlLineIndex.java Resolves YAML node lines.
MALDebugRecorder.java Adds source-line parameters.
MALDebug.java Forwards probe source lines.
DSL.java Adds source-mapped parsing.
MalSourceRef.java Models YAML/generated coordinates.
MalSourceMap.java Maps expression offsets to YAML.
MALScriptParser.java Tracks prefix and method offsets.
MALMethodChainCodegen.java Attributes chain-stage probes.
MalGeneratedSourceLines.java Finds generated statement lines.
MALExpressionModel.java Stores method-call source offsets.
MALExprCodegen.java Emits attributed probes.
MALCodegenHelper.java Adds line literals to probes.
MALClosureCodegen.java Removes misleading companion tables.
MALClassGenerator.java Aligns generated source metadata.
MALBytecodeHelper.java Builds generated-source line tables.
Analyzer.java Attributes filter/output probes.
dsl-debugging-mal.md Documents MAL sourceLine.
dynamic-code-generation-debugging.md Documents coordinate spaces.
changes.md Records the attribution fix.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Two review findings, both the same defect class this change set exists to
remove -- a number that is wrong but looks right.

MALBytecodeHelper wrote a LineNumberTable even when the source-line list and
the bytecode boundaries disagreed, having just logged that the result would
be wrong. That is worse than it reads: line_number is an unsigned u2, so an
UNRESOLVED (-1) serializes as 65535 and the JVM reports it as a real line
rather than an unknown one, and any shifted boundary keeps a plausible
mapping. The attribute is now omitted entirely on a count mismatch or an
unresolvable entry, matching the companion-class policy of preferring no
attribution to false attribution.

MethodCall.sourceStartIndex mixed two index spaces. ANTLR indexes its
CharStream by CODE POINT while MalSourceMap builds segments from
String.length()/substring, i.e. UTF-16 code units; for
`x.tagEqual('n','<emoji>').sum(['h'])` those are 30 and 31. A single
supplementary character before a stage shifted every later offset, so a
stage authored in expSuffix could resolve against the exp segment and report
a line that exists but is not its own. Token positions are now converted to
Java String offsets at the boundary, with a fast path when the two spaces
coincide.

Both are covered by tests that fail without the fix (offset 21 vs 20 for the
supplementary-character case), including the non-BMP expression the review
asked for and a BMP control confirming the conversion is a no-op for
ordinary rules.
@wu-sheng wu-sheng added bug Something isn't working and you are sure it's a bug! backend OAP backend related. labels Aug 8, 2026
@wu-sheng wu-sheng added this to the 11.0.0 milestone Aug 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend OAP backend related. bug Something isn't working and you are sure it's a bug!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants