Fix MAL line attribution: separate the operator and machine coordinates - #13971
Open
wu-sheng wants to merge 4 commits into
Open
Fix MAL line attribution: separate the operator and machine coordinates#13971wu-sheng wants to merge 4 commits into
wu-sheng wants to merge 4 commits into
Conversation
A MAL rule's compiled expression lives in two independent coordinate spaces, and one overloaded `yamlSource` string was serving as class label, file identifier and line coordinate at once. It was wrong in all three roles: - the `_L<n>_` segment in a generated class name was the rules-list INDEX, not a line, so every rule file's first rule was labelled `L0`; - `SourceFile` embedded that same fake line AND named a file that is never written (it used the metric name, while the generated source file is named after the class), so IDE source-attach could never resolve a MAL stack frame regardless of whether the line was right; - `LineNumberTable` held statement ordinals matching neither the YAML nor the generated source; - MAL debug records carried no line at all: the recorder hardcoded 0 at all five probe sites, and the wire shape documented `sourceLine` as "omitted for MAL (no per-line mapping)" -- true only because the mapping had never been built. The fix splits the three roles apart. OPERATOR COORDINATE (the rule .yaml an operator wrote) now reaches the dsl-debugging API as a per-STAGE `sourceLine`. Per stage rather than per rule, because `formatExp` splices `expPrefix`, `exp` and `expSuffix` into one expression: a stage contributed by the file-level `expSuffix:` must report the suffix's line, not the rule's. A per-rule implementation gets that silently wrong, since the rule's line does exist in the file. `MalYamlLineIndex` resolves the four anchors from a second snakeyaml compose pass over the same text bean binding discards; `MalSourceMap` attributes an offset to the fragment that authored it, using splice ranges `injectExpPrefix` already computed but discarded. MACHINE COORDINATE (the generated .java) is now self-consistent: `SourceFile` equals the file actually written, and `LineNumberTable` points at real statements in it. `MalGeneratedSourceLines` derives the boundaries from the same generated text the bytecode scan walks, so the two cannot drift; a mismatch logs a warning. Companion closure classes emit no `LineNumberTable` at all -- no source file is written for them and the store-scan is not a boundary detector there, so an absent table (JVM reports "unknown line") beats a degenerate one. The class-name label keeps the YAML line, but the real one. `MalSourceRef` carries both coordinates with distinct accessors so substituting one for the other is a visible mistake rather than the default. An unresolvable line is `-1`, never 0 or omitted, so a resolution failure stays visible in the artifact instead of silently degrading; it renders as `unknown` inside class names, where `-` is not legal. Contract change: `MALDebugRecorder`'s five `append*` methods take an additional `sourceLine`. Rule behaviour is unchanged: 1362 shipped-rule execution tests, 35 filter tests and 22 parser tests pass, the last confirming the `injectExpPrefix` rewrite is byte-identical.
writeSourceFile used new FileWriter(file), which encodes with whatever the JVM default charset is. Nothing pins that: project.build.sourceEncoding is compile-time only, and no file.encoding or LANG is set in the poms or the workflows, so the bytes of the generated source depended on the runner's locale. The header it emits carried an em dash, so every one of the 1402 generated .java files had a non-ASCII byte on line 1. On a JVM whose default charset encodes U+2014 outside ASCII, reading that file back with a strict UTF-8 decoder throws MalformedInputException -- which is how MalLineAttributionTest failed in CI while passing locally. Two layers: encode explicitly as UTF-8, which is required anyway because a MAL string literal may legitimately be non-ASCII and would otherwise be mangled; and drop the em dash from the header so the common path has no charset dependency at all. Verified by running the affected tests under -Dfile.encoding=GBK, which reproduces the original failure without the fix.
There was a problem hiding this comment.
Pull request overview
Fixes MAL operator-facing YAML attribution and generated Java debug coordinates.
Changes:
- Adds YAML source indexing and per-stage debug line propagation.
- Aligns generated
SourceFileandLineNumberTablemetadata. - Adds tests, documentation, and changelog updates.
Reviewed changes
Copilot reviewed 34 out of 34 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
MalFileApplier.java |
Stamps runtime-rule source lines. |
MALDebugRecorderImpl.java |
Records probe source lines. |
MalYamlLineIndexTest.java |
Tests YAML anchor indexing. |
MalProbeSourceLineTest.java |
Tests per-stage attribution. |
MalSourceMapTest.java |
Tests expression source mapping. |
MalProbeEmissionTest.java |
Tests emitted probe literals. |
MalPrefixInjectionRangesTest.java |
Tests prefix splice ranges. |
MalLineAttributionTest.java |
Validates generated class metadata. |
MalGeneratedSourceLinesTest.java |
Tests statement-line detection. |
RuleSourceLines.java |
Applies source anchors to rules. |
Rules.java |
Indexes disk-loaded rule text. |
Rule.java |
Stores file-level anchors. |
MetricsRule.java |
Stores per-rule anchors. |
MetricRuleConfig.java |
Exposes source-line contracts. |
MetricConvert.java |
Builds and propagates source maps. |
MalYamlLineIndex.java |
Resolves YAML node lines. |
MALDebugRecorder.java |
Adds source-line parameters. |
MALDebug.java |
Forwards probe source lines. |
DSL.java |
Adds source-mapped parsing. |
MalSourceRef.java |
Models YAML/generated coordinates. |
MalSourceMap.java |
Maps expression offsets to YAML. |
MALScriptParser.java |
Tracks prefix and method offsets. |
MALMethodChainCodegen.java |
Attributes chain-stage probes. |
MalGeneratedSourceLines.java |
Finds generated statement lines. |
MALExpressionModel.java |
Stores method-call source offsets. |
MALExprCodegen.java |
Emits attributed probes. |
MALCodegenHelper.java |
Adds line literals to probes. |
MALClosureCodegen.java |
Removes misleading companion tables. |
MALClassGenerator.java |
Aligns generated source metadata. |
MALBytecodeHelper.java |
Builds generated-source line tables. |
Analyzer.java |
Attributes filter/output probes. |
dsl-debugging-mal.md |
Documents MAL sourceLine. |
dynamic-code-generation-debugging.md |
Documents coordinate spaces. |
changes.md |
Records the attribution fix. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Two review findings, both the same defect class this change set exists to
remove -- a number that is wrong but looks right.
MALBytecodeHelper wrote a LineNumberTable even when the source-line list and
the bytecode boundaries disagreed, having just logged that the result would
be wrong. That is worse than it reads: line_number is an unsigned u2, so an
UNRESOLVED (-1) serializes as 65535 and the JVM reports it as a real line
rather than an unknown one, and any shifted boundary keeps a plausible
mapping. The attribute is now omitted entirely on a count mismatch or an
unresolvable entry, matching the companion-class policy of preferring no
attribution to false attribution.
MethodCall.sourceStartIndex mixed two index spaces. ANTLR indexes its
CharStream by CODE POINT while MalSourceMap builds segments from
String.length()/substring, i.e. UTF-16 code units; for
`x.tagEqual('n','<emoji>').sum(['h'])` those are 30 and 31. A single
supplementary character before a stage shifted every later offset, so a
stage authored in expSuffix could resolve against the exp segment and report
a line that exists but is not its own. Token positions are now converted to
Java String offsets at the boundary, with a fast path when the two spaces
coincide.
Both are covered by tests that fail without the fix (offset 21 vs 20 for the
supplementary-character case), including the non-BMP expression the review
asked for and a BMP control confirming the conversion is a no-op for
ordinary rules.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix MAL generated-class and debug-record line attribution
If this is non-trivial feature, paste the links/URLs to the design doc.
Update the documentation to include this new feature.
docs/en/setup/backend/admin-api/dsl-debugging-mal.md(thesourceLinewire shape),docs/en/operation/dynamic-code-generation-debugging.md(theSourceFileformat and the two coordinate spaces)Tests(including UT, IT, E2E) are added to verify the new feature.
If it's UI related, attach the screenshots below.
If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.
Update the
CHANGESlog.The problem
A MAL rule's compiled expression lives in two independent coordinate spaces, and a single
overloaded
yamlSourcestring was serving as class label, file identifier and line coordinate atonce. All four resulting artifacts were wrong:
vm_L0_cpu_total_percentageL0is the rules-list index, not a line — every file's first rule wasL0SourceFile(vm.yaml:0)cpu_total_percentage.javaLineNumberTable1,2,3,4,5,6,7sourceLine0at all five probe sites; the docs said "omitted for MAL (no per-line mapping)" — true only because the mapping was never builtThe fix
Operator coordinate — the rule
.yamlthe operator wrote — now reaches the dsl-debugging APIas a per-stage
sourceLine. Per stage, not per rule, becauseformatExpsplicesexpPrefix,expandexpSuffixinto one expression. Real generated output for a rule whoseexp:is on line38 and whose file-level
expSuffix:is on line 32:A per-rule implementation emits
38on all four and looks perfectly plausible.Machine coordinate — the generated
.java— is now self-consistent:SourceFileequals thefile actually written, and
LineNumberTablepoints at real statements in it.File line 16 really is
.service(...), 17 isreturn.MalSourceRefcarries both coordinates with distinct accessors, so substituting one for the otheris a visible mistake rather than the default. An unresolvable line is
-1, never0or omitted, soa resolution failure stays visible in the artifact; it renders as
unknowninside class names,where
-is not legal in an identifier.Deliberate decisions worth reviewing
LineNumberTable. No.javais written for them, andthe store-scan is not a statement-boundary detector there (one real companion had a single entry
spanning 204 bytes). An absent table makes the JVM report "unknown line", which is honest; the
previous one attributed every frame in the closure to line 1.
equals/hashCode. Otherwise adding a comment above a ruleshifts its line, changes its identity, and triggers a spurious STRUCTURAL re-apply in
runtime-rule — dropping and recreating BanyanDB measures for a semantically identical edit.
MALDebugRecorder's fiveappend*methods gained asourceLineparameter. One in-treeimplementor; not a public API/SPI.
zabbix-rulesis untouched. It is not a runtime-rule catalog, has no debug binding, andZabbixConfigdoes not overridegetSourceName()— so its classes were never source-named andare unaffected.
Verification
clean checkstyle:check(full repo)license-eye header checkclean install javadoc:javadoc -PallMALExpressionExecutionTest)The 22 parser tests confirm the
injectExpPrefixrewrite (right-to-leftreplace→ left-to-rightbuild, so splice ranges are recoverable) produces byte-identical output.
New tests, by coordinate:
MalGeneratedSourceLinesTest(4),MalLineAttributionTest(4 — compiles for real, reads the written.java/.class, assertsSourceFile== filename and every table entry indexes a statement)MalYamlLineIndexTest(6),MalPrefixInjectionRangesTest(4),MalSourceMapTest(5),MalProbeSourceLineTest(5),MalProbeEmissionTest(3 — runs with injection enabled and asserts the literal in the emitted probe)The wrapper-geometry guard was mutation-tested: shifting the constant by one fails
MalLineAttributionTestwithline 17 should hold a statement but was: '}'.Both shipped MAL file-format variants are covered by test-scope fixtures (no bundled rules are read):
the zabbix
metrics:rules key, and folded/literal block scalars, which anchor on their key line andmust not shift the rule that follows.
Not run locally: the
dsl-debugging/male2e. Everything up to the emitted probe literal isunit-verified, but a live debug session showing
sourceLinein an API response is CI's first look.