feat(match): honest structural fallback + A3/A12 fixtures#32
Merged
Conversation
Close the A3/A12 coverage gap from Phase 4.2. Structural matching already handled text-free UIs, but nothing proved the "graceful degradation + explicit low confidence" contract: - A structure-only match (no text/alias/correction evidence) is now capped at medium confidence — shape alone is never "high". - Fixtures a3-api-text (labels served by API/CMS → matched by structure, medium confidence; a text query honestly declines) and a12-non-text (chart + icon buttons → text declines, structure descriptor returns a medium-confidence candidate). - GoldenQuery gains a `confidence` assertion. 41 core tests pass; eval green (honesty metrics still 1.0/1.0/0.0). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gap-closing: A3 & A12 (structural fallback honesty)
The catalog files A3 (text served by API/CMS) and A12 (non-text UI — charts/canvases/icon buttons) under Phase 4.2, but only A1 had a fixture. This proves the "graceful degradation + explicit low confidence" contract.
high.a3-api-text: labels are all{data.*}(no literals) → matched by structure at medium confidence; a text query honestlydeclineds.a12-non-text: a chart + two icon buttons → a text querydeclineds (no false match), a structure descriptor returns a medium-confidence candidate.GoldenQuery.confidenceassertion added.Verification
41 core tests pass;
pnpm evalgreen — honesty metrics unchanged (high-conf correct 1.0, ambiguity honesty 1.0, poison 0.0).🤖 Generated with Claude Code