fix(security-audit): lift the deciding lines above the truncation point - #764
Conversation
Run 35842217451 composed a 226,302-character report and posted 31,662 of it. What reached the issue was `VERDICT: INCONCLUSIVE` naming `audit-ci-secrets.md` and a note telling the reader to read its `UNVERIFIABLE` checks — while that domain's one `UNVERIFIABLE` line, and the whole of `## Application security`, `## Hosted accounts` and `## Summary`, sat past the cut. The clamp keeps the head, and the lines that decide a verdict are not positionally predictable, so lift them: every fragment's verdict line plus its `UNVERIFIABLE`, `FAIL:`, `BLOCKER` and `WARNING` lines are emitted ahead of the report, where the clamp cannot reach them. Matched at line start after an optional heading or bullet marker, so the `FAIL IF` vocabulary the fragments are written in is not lifted as a finding.
Deploying mouseterm with
|
| Latest commit: |
1edea60
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://764e6dca.mouseterm.pages.dev |
| Branch Preview URL: | https://fix-audit-lift-deciding-line.mouseterm.pages.dev |
dormouse-bot
left a comment
There was a problem hiding this comment.
Feedback on work in progress, not a merge verdict.
The lift aborts the reporting step on a fragment with no marker line, so nothing reaches the issue at all. The step opens with set -eo pipefail, and grep exits 1 when it matches nothing — so grep … | cut | sed fails the pipeline and set -e kills the step inside the for loop, before the { … } > audit-comment.md group and before any gh issue create/comment. The verdict-reading loop above already names the state that triggers it: "an unrecognized first line — a subagent that ignored the preamble, a stray heading or blank line". A nonempty fragment carrying none of VERDICT:, FAIL:, UNVERIFIABLE, BLOCKER or WARNING at line start — prose, a stray heading, or a lone newline, since [ -s "$f" ] passes on one byte — matches nothing.
So the failure mode this PR exists to fix arrives through the fix, in a worse form: last night the reader got a truncated report; here they get a red run and an artifact.
docs/specs/security-audit.md also states the lift without its bound: "Every fragment's verdict line and its … lines are lifted into the head". head -n 40 takes the first 40 in AUDIT_FRAGMENTS order, so a first domain emitting 40 or more WARNING/FAIL: lines pushes a later domain's VERDICT: line out — the same loss, at the bound. Last night's four fragments produced 8 lines, so there is headroom, but the spec claims what the code does not do. The inline suggestion states the bound above the fold, which AGENTS.md → "Specs" wants anyway ("facts such as constants, bounds, directions … are not evidence"), and drops the trailing why-clause the (rationale) marker replaces. Emitting every fragment's verdict line ahead of its findings and capping only the findings would make the unconditional claim true instead — that one is a design call, not a defect.
How the abort was confirmed
The shipped Surface result, file or close issue block, extracted with workflowRunBlock exactly as scripts/security-audit.test.mjs does, run over a fixture with audit-status.txt = PASS, an ordinary audit-report.md, three VERDICT: PASS fragments, and audit-application.md set to # audit-application.md\n\nI reviewed the specs but could not finish.\n:
| This head | Merge base | |
|---|---|---|
| Exit status | 1 (stderr empty) | 1 |
audit-comment.md written |
no | yes |
gh calls |
label create, artifact lookup |
+ issue list, issue comment 23 |
The existing cases cannot reach it: every fixture fragment is written as VERDICT: ${verdict}\n…, so line 1 always matches. A third case with a markerless fragment, asserting the body is still composed and gh issue comment still called, would pin it — and would go red against this head.
The review on this PR caught that the lift's `grep` exits 1 when a fragment carries no marker line, and this step's `set -eo pipefail` turned that into an abort before `audit-comment.md` was composed and before any `gh issue` call — no report at all, which is worse than the truncation the lift fixes. Reproduced against the shipped step with a markerless `audit-application.md`: exit 1, no body written, no issue call. Both passes now end in `|| true`, and a new case pins it. Verdicts and findings are collected separately so the cap falls on the findings alone. One verdict line per fragment is bounded by the fragment count; a single domain's findings are not, so the shared `head -n 40` could have let the first domain's findings push the last domain's verdict out — the loss the lift exists to prevent. The spec now states the bound and the unconditional verdict claim above the fold.
dormouse-bot
left a comment
There was a problem hiding this comment.
Feedback on work in progress, not a merge verdict.
The new spec rule states the split cap inaccurately in two ways. "followed by its UNVERIFIABLE, FAIL:, BLOCKER and WARNING lines" reads as each verdict followed by that domain's own findings, but the step emits two blocks — cat "$VERDICTS" for every fragment's verdict, then head -n 40 "$FINDINGS" for every fragment's findings. And "Only the findings are capped" is false of the character cap: cut -c1-500 runs on both passes, and only the 40-line cap is findings-only. The inline suggestion states both; it counts 2047 against the 2050 budget already ratcheted here, so it needs no re-ratchet.
Nothing pins the rule this split exists to establish. No case reaches the 40-line boundary, before this commit or after, so "only the findings are capped" is a claim the suite does not check — I restored a shared cap (both greps into one file, verdicts interleaved per fragment, head -n 40 over the result) and all 48 tests stayed green. The second inline suggestion adds the case that goes red against that mutation. The || true half is pinned precisely by contrast: dropping it from the verdict pass alone fails exactly one test, the new one.
The verdict pass is now unbounded, and the rationale still claims otherwise — "Bounded at 40 lines and 500 characters a line so the lift cannot itself exhaust the budget it protects" is now true of the findings block only. What replaces the bound is "one verdict line per fragment", which .github/audit/_preamble.md states as a contract — "Its very first line must be literally VERDICT: PASS" — and which the grep does not enforce: it matches VERDICT: at any line of the fragment, after an optional heading or bullet marker. That latitude is the same reason the findings pass needs a cap at all, and it is agent-written text, which is why every other guard in this step exists rather than living in the prompt. Capping the verdict block generously, or anchoring the lift to the first line the guard loop above already reads, would close it. Four fragments make leaving it open defensible — but then the rationale sentence should say which half it describes.
…ccurately Applies the two review suggestions: the spec now says verdicts and findings are two blocks, both cut to 500 characters, with only the findings carrying the 40-line cap; a new case puts 42 findings in the first fragment and goes red when the cap is shared. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Resolves the spec-word-budgets.json conflict with #761. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Both PRs add a rule to security-audit.md, so its budget is re-ratcheted to 2100. Also corrects the rationale: only the findings carry the 40-line cap; the 500-character cut applies to every lifted line. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Last night's audit told nobody why it stopped. Run 35842217451 composed a 226,302-character report and posted 31,662 of it to #747: the reader got
VERDICT: INCONCLUSIVEnamingaudit-ci-secrets.md, and a note instructing them to "read itsUNVERIFIABLEchecks" — while that domain's oneUNVERIFIABLEline, and the whole of## Application security,## Hosted accountsand## Summarywith their verdicts, sat past the cut. The posted body contains zeroUNVERIFIABLElines outside that note.clampIssueBodykeeps the head, and the lines that decide a verdict are not positionally predictable — the one that decided this run is line 60 of a 70-line fragment that is itself the second of four. So this lifts them instead of teaching the clamp to find them. Every fragment's verdict line is emitted between the notes and the report, then itsUNVERIFIABLE,FAIL:,BLOCKERandWARNINGlines, in the head the clamp keeps by construction. The full sections still follow for as far as the body reaches, and the transcript artifact is unchanged.Only the findings are capped (40 lines, 500 characters each). One verdict line per fragment is bounded by the fragment count; a single domain's findings are not, so a shared cap could let the first domain's findings push the last domain's verdict out — the loss this exists to prevent. Neither pass may fail the step:
grepexits 1 when it matches nothing, and this step runs underset -eo pipefail, so an unguarded pipeline would abort beforeaudit-comment.mdis composed and post nothing at all. That is how the first revision of this PR behaved on a markerless fragment; the review on it caught that, and both passes now end in|| true.The match is anchored at line start after an optional heading or bullet marker, because every fragment is written in
FAIL IFvocabulary:### FAIL IF resultsheads the passing list, and a- PASS:bullet quotes the clause it passed. A bareFAILmatch would lift the passing evidence as findings.FAIL:takes its colon for the same reason; a domain-level failure arrives asVERDICT: FAILand theVERDICT:alternative catches it.Replayed against last night's four real fragments it produces 8 lines, 1,571 characters — all four domain verdicts, the
UNVERIFIABLEclause, andaudit-application.md's threeWARNINGentries, none of which posted.Verified by three new cases in
scripts/security-audit.test.mjs, which runs the workflow's actualSurface result, file or close issueblock, each mutation-checked red:UNVERIFIABLEline and the last domain's verdict survive a report long enough to clamp — and that the body really was clamped, so it cannot pass vacuouslyFAIL IFclause is not lifted as a findingFAILgh issue commentis still calledAll 48 reporting/redaction/orchestrator tests pass,
clamp-issue-body-selftestpasses,spec-lintis clean, the workflow re-parses as YAML, and the touchedrun:block passesbash -nandshellcheck.This is the reporting path only — it does not change any verdict. The clause that made last night inconclusive (
docs/specs/security-ci.md's provider-token scope obligation) is already removed on #748's head, so tonight's audit should get past it; this is what makes the next unreadable report legible.How the truncation was confirmed, and the one merge conflict to expect
Posted comment vs. the
audit-transcriptartifact for the same run:audit-report.mdin the artifact## Supply chain,## CI and secrets(cut mid-list)## Application security,## Hosted accounts,## SummaryUNVERIFIABLElinesaudit-ci-secrets.md:60)The deciding line, from the fragment:
- UNVERIFIABLE (partial): "preview credentials can reach production/TTR/marketing resources" — GitHub-side placement is fully verified … The spec also asks to "inspect… Cloudflare/Neon token scope" directly … that requires holding the live CLOUDFLARE_API_TOKEN/NEON_API_KEY values, which this audit correctly never reads or prints.This is occurrence 2 of a class first recorded on 2026-09-19 in #511, where a
BLOCKERreached the public record as a dangling pointer for the same reason. That entry pre-registered the signature this run matched. Gate 1: High, structural — the clamp cuts the tail every time a report is long, so the failure is not a model decision. Gate 2: targeted fix, normal bar, met.Merge conflict with #748: one line in
scripts/spec-word-budgets.json— this ratchetssecurity-audit.md1950 → 2050 for the new rule, #748 ratchets it to 2000. Whichever lands second takes the higher number. Nothing else overlaps.