Skip to content

fix: audit gate decisions against pipeline answer provenance - #305

Merged
dnth merged 5 commits into
mainfrom
fm/fm-gate-provenance
Nov 13, 2026
Merged

dnth merged 5 commits into
mainfrom
fm/fm-gate-provenance

Conversation

@dnth

@dnth dnth commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Summary

The decision audit now asks the pipeline who answered each gate instead of counting status-log strings.

 PR-ready (bin/fm-pr-check.sh) ─┐
 done acceptance (fm-crew-state) ─┴─> fm_nm_ask_user_decisions
        │
        ├─ pipeline record: axi status steps ─> axi rounds --json per step
        │    events: (step, round N, finding, action, response_origin)
        └─ firstmate record: state/<id>.status
-            bare-key count only; no answering origin
+            gate openings + answered records, bound per step, round, and suffix
  • bin/fm-nm-run-lib.sh reads decision events from axi rounds --json, refuses interactive/auto_yes answers without a firstmate record, and refuses missing, unparseable, or origin-less data naming exactly what is missing.
  • bin/fm-classify-lib.sh emits gate openings and canonical answers; bare, -r<N>, and -detail records bind to their own gate response, so surplus records never migrate across rounds.
  • One firstmate answer covers every finding decided in that gate response; answer excerpts are not authority (fm-send truncates them).
  • bin/fm-pr-check.sh and bin/fm-crew-state.sh share the one verdict; the PR-ready and done-time failures now name the run, step, finding, and origin.

Evidence

  • Before: run 01M796S134JFXVZXT5JVHEMWAY recorded interactive review decisions with no firstmate record and no refusal; older origin-less runs silently fell back to the text-only count.
    After: the same run with an empty status file refuses: run 01M796S134JFXVZXT5JVHEMWAY step review: 1 recorded ask-user gate decision(s) ... finding R25 resolved as fix (round 1, answered interactive). Origin-less run 01M6TA5HW285X0ZF457AB648YK refuses with round 1 records an ask-user decision with no response_origin (captured in the task's real-run-audit.txt).
  • bash tests/fm-nm-run-lib.test.sh -> all passed, including grouped approval, round-suffixed keys, cross-round surplus refusal, and a rounds document over 128 KiB.
  • bash tests/fm-pr-check-handoff.test.sh -> all passed, including the worker-answered refusal naming run/step/finding/origin and PR-ready/crew-state agreement.
  • bash tests/fm-crew-state.test.sh -> all passed.
  • shellcheck -x on every changed script: clean.
  • git diff --name-only origin/main...HEAD: no path under .omp/extensions/.

Merge Danger

Door: two-way. No data migration; the audit is read-only and the branch can be reverted.

Blast Radius: decision gates only. PR-ready and done acceptance refuse until a firstmate resolved record exists for each recorded gate response, so pre-existing runs with origin-less pipeline data now stop loudly instead of passing on a text count. CI and the pipeline remain the only full-suite owners.

Pipeline

Updates from git push no-mistakes

✅ **Intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 2 issues found → auto-fixed (3) ✅
  • 🚨 bin/fm-nm-run-lib.sh:457 - Step-wide totals can accept an unmatched worker decision. For example, firstmate records both the bare key and -r30 for round 30's R1; the worker then answers R2 in round 31. need=2 and have=2, so both gates pass despite R2 lacking authority. Match records to decision events rather than allowing surplus records for one event to cover another. Related sites: bin/fm-classify-lib.sh:717 and :722 count these records independently; bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163 consume the verdict. The fixtures at tests/fm-nm-run-lib.test.sh:157 and tests/fm-pr-check-handoff.test.sh:335 already contain overlapping records but never assert refusal before adding R2's record.
  • ⚠️ bin/fm-nm-run-lib.sh:417 - Passing the complete rounds document through FM_NM_ROUNDS_DOC imposes an operating-system argument/environment size limit. On Linux, a single environment entry cannot exceed roughly 128 KiB; a review history with 30 rounds containing about 5 KiB of findings and summaries per round exceeds this. The python3 launch at bin/fm-nm-run-lib.sh:284 then fails before parsing valid evidence, blocking PR-ready and done acceptance even when every decision matches. Stream the JSON through a file descriptor or stdin instead. Both consumers are affected: bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163.

🔧 Fix applied.
4 issues (3 errors, 1 warning) still open:

  • 🚨 bin/fm-nm-run-lib.sh:457 - Step-wide totals can accept an unmatched worker decision. For example, firstmate records both the bare key and -r30 for round 30's R1; the worker then answers R2 in round 31. need=2 and have=2, so both gates pass despite R2 lacking authority. Match records to decision events rather than allowing surplus records for one event to cover another. Related sites: bin/fm-classify-lib.sh:717 and :722 count these records independently; bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163 consume the verdict. The fixtures at tests/fm-nm-run-lib.test.sh:157 and tests/fm-pr-check-handoff.test.sh:335 already contain overlapping records but never assert refusal before adding R2's record.
  • ⚠️ bin/fm-nm-run-lib.sh:417 - Passing the complete rounds document through FM_NM_ROUNDS_DOC imposes an operating-system argument/environment size limit. On Linux, a single environment entry cannot exceed roughly 128 KiB; a review history with 30 rounds containing about 5 KiB of findings and summaries per round exceeds this. The python3 launch at bin/fm-nm-run-lib.sh:284 then fails before parsing valid evidence, blocking PR-ready and done acceptance even when every decision matches. Stream the JSON through a file descriptor or stdin instead. Both consumers are affected: bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163.
  • 🚨 bin/fm-nm-run-lib.sh:479 - Round 1 leaves the surplus-record bypass reachable when finding IDs repeat. Record bare-key fix R1 and -r30 fix R1 for round 30, then let the worker answer R1 in round 31 without firstmate approval. The matcher assigns the suffixed record to round 30 and the old bare record to round 31, accepting both. Bind bare/detail records to their actual decision occurrence before matching; surplus records must not migrate between rounds. Related sites: bin/fm-classify-lib.sh:712 accepts unbound records; bin/fm-nm-run-lib.sh:498 assigns them; bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163 consume the incorrect pass.
  • 🚨 bin/fm-nm-run-lib.sh:492 - Round 1 introduces a stricter ledger contract than the existing firstmate answer path supports. AC2 requires: “A run whose every ask-user decision has a matching firstmate decision passes.” Firstmate's documented single message can approve R1 and R2 together, producing one resolved line, but assigned[record] = event permits that line to cover only one finding. tests/fm-nm-run-lib.test.sh:175 explicitly constructs this valid approval and then expects refusal. Additionally, matching IDs in answer excerpts at bin/fm-nm-run-lib.sh:481 rejects legitimate answers when fm-send's 220-character cap truncates the IDs (bin/fm-send.sh:224). Revert the round-1 per-finding excerpt machinery to a minimal occurrence-aware gate-response match that preserves grouped decisions and bounded answer records, while retaining the JSON streaming fix. Related changed sites: bin/fm-classify-lib.sh:720 supplies the excerpt; bin/fm-pr-check.sh:202 imposes the new record requirement; both gate consumers at bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163 inherit the refusal. The remedy needs authorization because it revises the prior fix's deliberate matching contract.

🔧 Fix applied.
5 issues (4 errors, 1 warning) still open:

  • 🚨 bin/fm-nm-run-lib.sh:457 - Step-wide totals can accept an unmatched worker decision. For example, firstmate records both the bare key and -r30 for round 30's R1; the worker then answers R2 in round 31. need=2 and have=2, so both gates pass despite R2 lacking authority. Match records to decision events rather than allowing surplus records for one event to cover another. Related sites: bin/fm-classify-lib.sh:717 and :722 count these records independently; bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163 consume the verdict. The fixtures at tests/fm-nm-run-lib.test.sh:157 and tests/fm-pr-check-handoff.test.sh:335 already contain overlapping records but never assert refusal before adding R2's record.
  • ⚠️ bin/fm-nm-run-lib.sh:417 - Passing the complete rounds document through FM_NM_ROUNDS_DOC imposes an operating-system argument/environment size limit. On Linux, a single environment entry cannot exceed roughly 128 KiB; a review history with 30 rounds containing about 5 KiB of findings and summaries per round exceeds this. The python3 launch at bin/fm-nm-run-lib.sh:284 then fails before parsing valid evidence, blocking PR-ready and done acceptance even when every decision matches. Stream the JSON through a file descriptor or stdin instead. Both consumers are affected: bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163.
  • 🚨 bin/fm-nm-run-lib.sh:479 - Round 1 leaves the surplus-record bypass reachable when finding IDs repeat. Record bare-key fix R1 and -r30 fix R1 for round 30, then let the worker answer R1 in round 31 without firstmate approval. The matcher assigns the suffixed record to round 30 and the old bare record to round 31, accepting both. Bind bare/detail records to their actual decision occurrence before matching; surplus records must not migrate between rounds. Related sites: bin/fm-classify-lib.sh:712 accepts unbound records; bin/fm-nm-run-lib.sh:498 assigns them; bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163 consume the incorrect pass.
  • 🚨 bin/fm-nm-run-lib.sh:492 - Round 1 introduces a stricter ledger contract than the existing firstmate answer path supports. AC2 requires: “A run whose every ask-user decision has a matching firstmate decision passes.” Firstmate's documented single message can approve R1 and R2 together, producing one resolved line, but assigned[record] = event permits that line to cover only one finding. tests/fm-nm-run-lib.test.sh:175 explicitly constructs this valid approval and then expects refusal. Additionally, matching IDs in answer excerpts at bin/fm-nm-run-lib.sh:481 rejects legitimate answers when fm-send's 220-character cap truncates the IDs (bin/fm-send.sh:224). Revert the round-1 per-finding excerpt machinery to a minimal occurrence-aware gate-response match that preserves grouped decisions and bounded answer records, while retaining the JSON streaming fix. Related changed sites: bin/fm-classify-lib.sh:720 supplies the excerpt; bin/fm-pr-check.sh:202 imposes the new record requirement; both gate consumers at bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163 inherit the refusal. The remedy needs authorization because it revises the prior fix's deliberate matching contract.
  • 🚨 bin/fm-nm-run-lib.sh:486 - Round 2 introduced order-dependent recovery matching. With decision rounds 30 and 31 and no ledger openings, append the firstmate recovery answer for -r31, then the documented bare-key recovery answer for the first response (round 30). Updating current to 31 causes the bare answer at bin/fm-nm-run-lib.sh:494 to bind to 31 too, incorrectly refusing round 30 despite both approvals. Explicit recovery answers must not move the implicit gate binding; only gate openings should advance it. Related consumers: bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163. The bare-key recovery contract is documented at bin/fm-nm-run-lib.sh:250.

🔧 Fix applied.
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 10 of 10 scenarios driven live against the product
Scenario Result Live Evidence
Local PR-ready and crew-state reject synthetic interactive and auto_yes decisions without firstmate approval, naming run, step, and finding ✅ pass live unauthorized-worker-interactive-and-auto-yes.log in the evidence directory
Local consumers accept grouped findings approved through round and detail keys across multiple steps ✅ pass live grouped-and-suffixed-approvals.log in the evidence directory
Surplus synthetic approval records cannot authorize a repeated finding in a later round ✅ pass live surplus-records-do-not-cover-repeated-round.log in the evidence directory
Reopened local gate occurrences accept their separate bare and detail approvals ✅ pass live reopened-bare-key-approvals.log in the evidence directory
Local recovery accepts a later round approval followed by the first response's bare-key approval ✅ pass live out-of-order-recovery-round-then-bare.log in the evidence directory
Local recovery accepts a later round approval followed by the first response's detail-key approval ✅ pass live out-of-order-recovery-round-then-detail.log in the evidence directory
Both local consumers refuse synthetic origin-less history and identify the missing response_origin ✅ pass live originless-record-refuses.log in the evidence directory
Both real Firstmate commands refuse an injected external rounds-command failure despite approval records ✅ pass live unavailable-rounds-refuses.log in the evidence directory
Both real Firstmate commands refuse injected malformed rounds JSON without a text-only fallback ✅ pass live unparseable-rounds-refuses.log in the evidence directory
Both local consumers accept an approved synthetic history exceeding 128 KiB through the real No-Mistakes CLI ✅ pass live large-history-streams-without-environment-limit.log in the evidence directory
  • TMPDIR="$PWD/.validation-live/tmp" bash tests/fm-nm-run-lib.test.sh
  • TMPDIR="$PWD/.validation-live/tmp" bash tests/fm-pr-check-handoff.test.sh
  • TMPDIR="$PWD/.validation-live/tmp" bash tests/fm-crew-state.test.sh
  • python3 .validation-live/live.py: exercised real Firstmate commands and No-Mistakes status/rounds readers with isolated synthetic records, a synthetic forge head, and external-boundary fault injection
  • git diff --name-only origin/main...HEAD: confirmed no changes under .omp/extensions/
  • Removed the disposable setup and confirmed a clean working tree; every started check exited successfully.
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

dnth added 5 commits November 13, 2026 11:41
The PR-ready and done-time decision audit compared resolved [key=nm-<run>-<step>]
status records against a gate-resolution count reconstructed from the daemon
state database: it never read who answered a gate, and it counted only the bare
step key, so a worker that answered its own gate passed and round-suffixed
decision keys silently did not count.

Read each step's ask-user decision events and their response_origin from the
pipeline's own `no-mistakes axi rounds --json` record instead. A decision the
pipeline records as answered with no matching firstmate resolved record refuses
PR-ready and done acceptance, naming the run, step, finding, and origin. A
round- or detail-suffixed firstmate record counts for its step. Missing,
unparseable, or origin-less rounds data refuses naming exactly what is missing;
there is no fallback to the text-only audit.
@dnth
dnth merged commit 96dcce9 into main Nov 13, 2026
21 checks passed
@dnth
dnth deleted the fm/fm-gate-provenance branch November 13, 2026 07:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant