Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 58 additions & 4 deletions TRACKER.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
## Status

- **Current phase:** 6F — Field hardening, feedback round 1 (runs before 6.1–6.5)
- **Next step:** ship **0.4.0** to the tester for a re-validation round; then 6F.6 test-coverage hardening (blocked: needs the field's actual failing variant — the fixture's custom-wrapper shape already passes; get a failing test file from the tester or ship the defensive `coverage-unmapped` half only)
- **Done:** 0.1–0.4, 1.1–1.6, 2.1–2.5, 3.1–3.6, 4.1–4.6, 5.1–5.7, 6F.1–6F.5, 6F.7–6F.8 · **release 0.4.0 prepared** (all packages bumped, changelog + version strings, `npm pack` → ui-lineage-0.4.0.tgz verified, publish pending)
- **Next step:** publish 0.4.1 to the tester (all shippable fixes: 6F.6–6F.10 + visualizer). 6F.6 detection half remains blocked on a real failing test file — its defensive `coverage-unmapped` half shipped.
- **Done:** 0.1–0.4, 1.1–1.6, 2.1–2.5, 3.1–3.6, 4.1–4.6, 5.1–5.7, 6F.1–6F.5, 6F.7–6F.10 + 6F.6 (defensive half) · **0.4.1 prepared** (all packages bumped, changelog + version strings, `npm pack` → ui-lineage-0.4.1.tgz verified: both bins, no @coderadar leak; publish to npm pending). 0.4.0 tagged/merged earlier. **Self-validated on Grafana frontend** (6,461 files, 15,334 nodes / 18,367 edges in 72 s): 55 RTK-query data sources, 32 routes, 1,009 coverage edges, gibberish declines — all previously 0/1 in the field run.
- **Gates passed:** Gate 0 (CI + red-path, #5/#6) · Gate 1 (precision 1.000, recall 0.895, zero poison) · Gate 2 (C1 instance attribution 1.000 · B1 4-level handler chains · C6 store writers↔readers · A9 portals — scorecard 137/0/0, precision & recall 1.000) · Gate 3 (B3 action effects · B4 routers · B6 cyclic journeys terminate · B7/B8 form & non-JSX events · G5 flag/role conditions — precision & recall 1.000) · Gate 4 (A4 rarity · A10 fuzzy/OCR · A1 structural · A6 subtree · E3 vision annotations · E2 aliases · G4 corrections — high-conf correct 1.000, ambiguity honesty 1.000, poison rate 0.000) · Gate 5 (F1 context bundle · F2 blast radius · F3 test coverage · F4 response schema · F5 git history · MCP server over stdio — scorecard 265/0/0, all honesty metrics 1.000; **M5 reached** — ticket in → budgeted context bundle out, over MCP)

## What CodeRadar is
Expand Down Expand Up @@ -477,7 +477,7 @@ unmarked, so **the fixture is fully green with zero marks and Gate 6F's extracto
are met**. 4 new parser tests (129 total); eval 290 pass / 0 fail / 0 xfail / 0
unexpected-pass, gate OK, metrics 1.000.

### [ ] 6F.6 Test-coverage detection hardening
### [~] 6F.6 Test-coverage detection hardening
**Failure modes:** F3
**Build:** the field run found 1 `covered-by` edge app-wide, making `untested` warnings
near-universal noise. Handle custom render wrappers (`renderWithProviders(<X/>)` — resolve
Expand All @@ -488,6 +488,16 @@ through to the JSX argument), test files importing through the same alias/barrel
**Accept:** field-patterns fixture: wrapped renders produce `covered-by` edges; a near-empty
coverage graph emits `coverage-unmapped` instead of per-component `untested`; enable the 6F.2
checks.
**Done (defensive half):** the `coverage-unmapped` downgrade shipped — `buildBundle` now, when
test files exist but < 5% of components carry a `covered-by` edge, emits one graph-level
`coverage-unmapped — only N/M components have mapped test coverage` note instead of a
near-universal false `untested`. A genuinely test-free repo (no test nodes) keeps the accurate
per-component `untested`. 3 agent-sdk unit tests (near-empty → downgrade · healthy → keep ·
no-tests → keep); verified on the real Grafana graph (34% coverage → per-component `untested`
preserved). **Detection half still open** (custom-wrapper/alias/outside-`__tests__` resolution):
could NOT reproduce the field failure — the fixture's `renderWithProviders` shape already maps,
and Grafana produced 1,009 `covered-by` edges, so detection works on real code. Blocked on a
real failing test file from the tester before building the wrong thing.

### [x] 6F.7 Scoring & result-surface polish
**Failure modes:** A4, D2, D6
Expand Down Expand Up @@ -537,9 +547,53 @@ round-trips, dangling-edge drop, `</script>` breakout guard, self-contained asse
README documents the command. eval unaffected (290/0/0/0). **Gate 6F extractor criteria met;
only 6F.6 (coverage, data-blocked) remains before the gate fully closes.**

### [x] 6F.9 Stopword & rare-literal scoring noise
**Failure modes:** A15 (new), A4, D6
**Build:** self-found while validating 0.4.0 on Grafana's frontend — `find "Find silences by
matcher"` ranked `OrderBySection` (renders a bare `BY`) top-1 over `SilencesFilter`, because a
lone stopword is a *rare literal* with high IDF; and `find "The"` returned confident matches on
"the". New `isLowSignal(normalized)` in core/text.ts (extends the A14 `hasMatchSignal` guard):
true when a string is empty, punctuation-only, or entirely stopwords (folded through
`foldPlural` so it compares to normalized text). `matchComponents` now drops stopword-only query
terms and skips stopword-only rendered-text targets, so the exact-phrase component wins and a
stopword-only query declines `no-signal`.
**Accept:** new fixture `a15-stopword-noise` (a `BY` component, an exact-phrase component, a
`the`-rendering component): the phrase query ranks the exact-phrase component top-1; a stopword
query declines; a stopword mixed with a real term ignores the stopword-rendering component.
**Done:** implemented + 5 unit tests (2 text, 3 matcher) + fixture. **Verified on the real
Grafana graph:** `Find silences by matcher` → `SilencesFilter` top-1 (`OrderBySection` gone),
`The` → `no-signal`, and `resolve` on a stopword-heavy ticket went from confidently-wrong to an
honest `no-signal`. eval 297/0/0/0, gate OK, all metrics 1.000. A16 (HTML-entity rendered text)
spun out to 6F.10 — out of scope for a scoring PR.

### [x] 6F.10 HTML-entity rendered-text noise
**Failure modes:** A16 (new)
**Build:** self-found on Grafana — rendered text that is an HTML entity (`&nbsp;`, `&#34;`,
`&gt;`) normalizes to a junk token (`nbsp`, `34`, `gt`); numeric entities make gibberish
containing those digits match. Decode/strip HTML entities in the parser's text-extraction pass
(parser-react), or treat entity-only rendered text as low-signal. Not a scoring fix — belongs in
extraction, hence a separate step from 6F.9.
**Accept:** fixture with entity-only rendered text: it produces no match target; gibberish that
shares digits with a numeric entity declines. eval green.
**Done:** new `entities.ts` in parser-react — `decodeEntities(text)` resolves the HTML entities
React decodes at render time (numeric decimal `&#34;` / hex `&#x22;` generically, named entities
from a curated map: markup, whitespace, punctuation, symbols, currency, accented Latin-1;
unknown names left verbatim, matching React). `extractRenderedText` decodes JSX text and quoted
attribute values — the two surfaces React HTML-decodes — while JS string/template literals stay
untouched (React renders `{"&nbsp;"}` literally). Decoded entities become the character React
renders, which the normalizer strips: `&nbsp;`→space→dropped, `&gt;`/`&#34;`/`&middot;`→
punctuation that normalizes to empty, so an entity-only component yields **no discriminating
target** (verified: `EntitySpacer.renderedText` = `["\"", ">", "·", "<", "›"]`, zero
alphanumeric tokens). New fixture `a16-html-entities` (entity-only `EntitySpacer` + real
`QuotaNotice`): the named-entity token `nbsp`, the numeric-entity token `34`, and a gibberish
query sharing those digits (`zzqwxnomatch12345`, which pre-fix matched via `&#34;`→"34") all
decline `no-signal`, while the real query still lands on `QuotaNotice` and isn't poisoned by the
digit-sharing gibberish. 7 new tests (4 unit decode + 3 fixture integration), 136 parser-react
total; eval 304/0/0/0, gate OK, all metrics 1.000.

**Gate 6F:** field-patterns fixture fully green (skip list empty ✅) · instance resolution ≥ 95%
(✅ 100%) · RTK data sources > 0 (✅) · route nodes > 0 (✅) · gibberish queries decline
`no-signal` (✅) · `pnpm eval` green end-to-end (✅ 290/0/0/0). Remaining before the gate is
`no-signal` (✅) · `pnpm eval` green end-to-end (✅ 304/0/0/0). Remaining before the gate is
formally stamped: 6F.6 test-coverage hardening (blocked on a real failing sample).

---
Expand Down
2 changes: 2 additions & 0 deletions docs/failure-modes.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ every downstream agent. When in doubt, return ranked candidates with evidence, o
| A12 | **Non-text UI** — charts, icons, canvases | Nothing to text-match | Graceful degradation: structural candidates + explicit low confidence | 4 |
| A13 | **Third-party embeds** — Intercom, iframes | Content isn't ours | Detect and report "not in this codebase" | 4 |
| A14 | **Empty-normalizing rendered text** — components rendering bare `\|` / `/` / `-` | Normalization strips punctuation → empty target treated as a substring of every query; matcher becomes a universal wildcard (field-found in v0.3.0) | Discard targets that normalize to empty; minimum alphanumeric token length; `no-signal` decline when only such targets match | 6F |
| A15 | **Stopword & rare-literal noise** — a component renders a bare `BY`; a ticket says "The …" | A stopword is a rare literal with high IDF, so it outranks the exact-phrase match; stopword-only queries return confident junk (field-found on Grafana 0.4.0) | Drop stopword-only query terms; reject stopword-only rendered-text targets (extends A14's low-signal guard) | 6F |
| A16 | **HTML-entity rendered text** — a component renders `&nbsp;` / `&#34;` / `&gt;` | Entities normalize to junk tokens (`nbsp`, `34`, `gt`); numeric entities let gibberish sharing those digits match (self-found on Grafana 0.4.0) | Decode/strip HTML entities in the text-extraction pass, or treat entity-only rendered text as low-signal | 6F |

## B. User-journey extraction

Expand Down
3 changes: 3 additions & 0 deletions eval/fixtures/a15-stopword-noise/app/AckPanel.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
export function AckPanel() {
return <p>Before you continue, acknowledge the following</p>;
}
8 changes: 8 additions & 0 deletions eval/fixtures/a15-stopword-noise/app/OrderBySection.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
export function OrderBySection() {
return (
<div>
<label>Sort</label>
<span>BY</span>
</div>
);
}
7 changes: 7 additions & 0 deletions eval/fixtures/a15-stopword-noise/app/SilencesFilter.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
export function SilencesFilter() {
return (
<div>
<input placeholder="Find silences by matcher" />
</div>
);
}
17 changes: 17 additions & 0 deletions eval/fixtures/a15-stopword-noise/golden.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"failureMode": "A15",
"note": "Stopword & rare-literal noise (field-found on Grafana, 0.4.0): a component rendering a bare stopword ('BY') is a rare literal whose high IDF let it outrank the component that renders the whole phrase, and a stopword-only query ('The') returned confident matches. Fix (6F.9): drop stopword-only query terms and reject stopword-only rendered-text targets, so the exact-phrase component wins and a stopword query declines no-signal.",
"expect": {
"components": [
{ "name": "OrderBySection", "instances": 0 },
{ "name": "SilencesFilter", "instances": 0 },
{ "name": "AckPanel", "instances": 0 }
],
"queries": [
{ "terms": ["Find silences by matcher"], "status": "ok", "top": "SilencesFilter" },
{ "terms": ["silences"], "status": "ok", "top": "SilencesFilter" },
{ "terms": ["The"], "status": "declined" },
{ "terms": ["the", "acknowledge"], "status": "ok", "top": "AckPanel" }
]
}
}
16 changes: 16 additions & 0 deletions eval/fixtures/a16-html-entities/app/EntitySpacer.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
// Its only rendered text is HTML entities — the shape the Grafana frontend hit
// (26 &nbsp;, 4 &quot;, plus &#34;/&gt;/&lt;/&middot;/&rsaquo;). Each decodes to
// punctuation/whitespace, so after decoding there is no match target at all.
export function EntitySpacer() {
return (
<div aria-hidden="true">
<span>&nbsp;</span>
<span>&#34;</span>
<span>&gt;</span>
<span>&quot;</span>
<span>&middot;</span>
<span>&lt;</span>
<span>&rsaquo;</span>
</div>
);
}
6 changes: 6 additions & 0 deletions eval/fixtures/a16-html-entities/app/QuotaNotice.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
// A real component with distinctive text, so the fixture isn't trivially
// all-declining: a real query must still land here, and gibberish that shares
// digits with a numeric entity must not poison it.
export function QuotaNotice() {
return <p>Storage quota exceeded</p>;
}
17 changes: 17 additions & 0 deletions eval/fixtures/a16-html-entities/golden.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"failureMode": "A16",
"note": "HTML-entity rendered text (self-found on Grafana, 0.4.0): a component whose only rendered text is HTML entities (&nbsp;, &#34;, &gt;, &quot;, &middot;, &lt;, &rsaquo;) normalized to junk tokens ('nbsp', '34', 'gt'), creating spurious matches — and a numeric entity like &#34; let a gibberish query sharing its digits ('...12345') match. Fix (6F.10): decode HTML entities in the parser's text-extraction pass so they become the character React renders, which normalizes away as punctuation. The entity-only component then produces no match target: a named-entity token ('nbsp'), a numeric-entity token ('34'), and a gibberish query that shares those digits all decline no-signal, while a real query still lands on the real component and isn't poisoned by the digit-sharing gibberish.",
"expect": {
"components": [
{ "name": "EntitySpacer", "instances": 0 },
{ "name": "QuotaNotice", "instances": 0 }
],
"queries": [
{ "terms": ["34"], "status": "declined" },
{ "terms": ["nbsp"], "status": "declined" },
{ "terms": ["zzqwxnomatch12345"], "status": "declined" },
{ "terms": ["Storage quota exceeded"], "status": "ok", "top": "QuotaNotice" },
{ "terms": ["zzqwxnomatch12345", "Storage quota exceeded"], "status": "ok", "top": "QuotaNotice" }
]
}
}
2 changes: 1 addition & 1 deletion eval/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@coderadar/eval",
"version": "0.4.0",
"version": "0.4.1",
"private": true,
"description": "CodeRadar eval harness — scans failure-mode fixtures, diffs against golden outputs, emits the scorecard that gates every phase.",
"license": "MIT",
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "coderadar",
"private": true,
"version": "0.4.0",
"version": "0.4.1",
"description": "Static analysis toolkit that maps UI components to their data sources (APIs, state, events) — enabling AI agents to trace any screenshot back to the code and data behind it.",
"license": "MIT",
"packageManager": "pnpm@11.2.2",
Expand Down
2 changes: 1 addition & 1 deletion packages/agent-sdk/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@coderadar/agent-sdk",
"version": "0.4.0",
"version": "0.4.1",
"private": true,
"description": "resolveContext orchestrator — classifies a ticket and runs the matching engine. Deterministic, no LLM in the node. Bundled into the published ui-lineage package.",
"license": "MIT",
Expand Down
77 changes: 77 additions & 0 deletions packages/agent-sdk/src/bundle.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -97,4 +97,81 @@ describe("buildBundle (TRACKER 5.2, F1)", () => {
expect(ood.match).toEqual([]);
expect(ood.warnings.some((w) => w.includes("out-of-scope"))).toBe(true);
});

it("reports a plain 'untested' when the repo genuinely has no tests", () => {
// graphWithLineage has 1 component, 0 test nodes → untested is accurate.
const bundle = buildBundle(graph, { text: "the Big card overview" }, { budgetTokens: 8000 });
expect(bundle.warnings.some((w) => w.startsWith("untested —"))).toBe(true);
expect(bundle.warnings.some((w) => w.startsWith("coverage-unmapped"))).toBe(false);
});
});

describe("coverage-unmapped downgrade (TRACKER 6F.6, F3)", () => {
// A graph where test files EXIST but almost none map to a component — the
// field-found signature (tests present, ~0 covered-by edges). The matched
// component's missing coverage must read as "unmapped", not a false "untested".
function sparselyCoveredGraph(componentCount: number, coveredCount: number): LineageGraph {
const nodes: LineageGraph["nodes"] = [];
const edges: LineageEdge[] = [];
for (let i = 0; i < componentCount; i += 1) {
const name = i === 0 ? "SilencePanel" : `Widget${i}`;
const text = i === 0 ? "Silence alert notifications" : `Widget ${i} label`;
nodes.push({
id: nodeId("component", `${name}.tsx`, name),
kind: "component",
name,
loc: loc(`${name}.tsx`),
exportName: name,
props: [],
renderedText: [{ text, source: "jsx" }],
rendersComponents: [],
structure: {
table: 0, columns: 0, form: 0, input: 0, button: 0,
link: 0, image: 0, heading: 0, list: 0, repeated: 0,
},
});
}
// Several test files exist, but they only cover the LAST few components —
// never the matched SilencePanel (index 0).
for (let t = 0; t < 5; t += 1) {
const testId = nodeId("test", `Widget.test.tsx`, `test${t}`);
nodes.push({
id: testId,
kind: "test",
name: `test${t}`,
loc: loc(`__tests__/Widget${t}.test.tsx`),
framework: "vitest",
});
if (t < coveredCount) {
edges.push({ from: nodes[componentCount - 1 - t]!.id, to: testId, kind: "covered-by" });
}
}
return {
version: 2,
root: "/app",
generatedAt: "2026-01-01T00:00:00.000Z",
generator: "test",
nodes,
edges,
};
}

it("emits coverage-unmapped instead of untested when coverage is near-empty", () => {
// 40 components, tests present, only 1 covered (2.5% < 5% floor).
const g = sparselyCoveredGraph(40, 1);
const bundle = buildBundle(g, { text: "Silence alert notifications" }, { budgetTokens: 8000 });
expect(bundle.match[0]?.component).toBe("SilencePanel");
expect(bundle.warnings.some((w) => w.startsWith("coverage-unmapped"))).toBe(true);
expect(bundle.warnings.some((w) => w.startsWith("untested —"))).toBe(false);
});

it("keeps per-component untested when coverage is healthy", () => {
// 10 components, 5 covered (50% ≥ 5%) — the matched SilencePanel is not one
// of them, so untested is the accurate, useful signal.
const g = sparselyCoveredGraph(10, 5);
const bundle = buildBundle(g, { text: "Silence alert notifications" }, { budgetTokens: 8000 });
expect(bundle.match[0]?.component).toBe("SilencePanel");
expect(bundle.warnings.some((w) => w.startsWith("untested —"))).toBe(true);
expect(bundle.warnings.some((w) => w.startsWith("coverage-unmapped"))).toBe(false);
});
});
24 changes: 23 additions & 1 deletion packages/agent-sdk/src/bundle.ts
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,29 @@ export function buildBundle(
if (test !== undefined) testFiles.add(test.loc.file);
}
bundle.tests = [...testFiles].sort().map((file) => ({ file }));
if (!topCovered) bundle.warnings.push(`untested — no test renders ${top.component.name}`);
if (!topCovered) {
// Graph-level coverage health (6F.6, F3): when test files exist but almost
// no component carries a covered-by edge, coverage mapping is effectively
// blind — a per-component "untested" is then a near-universal FALSE signal
// (the field-found noise: 1 covered-by edge across a whole app). Downgrade
// to one honest graph-level note. A genuinely test-free repo (no test
// nodes at all) keeps the accurate "untested".
let componentCount = 0;
let testCount = 0;
for (const node of graph.nodes) {
if (node.kind === "component") componentCount += 1;
else if (node.kind === "test") testCount += 1;
}
const coveredComponents = new Set<string>();
for (const edge of graph.edges) if (edge.kind === "covered-by") coveredComponents.add(edge.from);
const coverageUnmapped =
testCount > 0 && componentCount > 0 && coveredComponents.size / componentCount < 0.05;
bundle.warnings.push(
coverageUnmapped
? `coverage-unmapped — only ${coveredComponents.size}/${componentCount} components have mapped test coverage; "untested" is unreliable for this graph`
: `untested — no test renders ${top.component.name}`,
);
}

// Git history (5.6): recent commits touching the matched component's files.
const files = new Set<string>([top.component.loc.file]);
Expand Down
Loading
Loading