Repository navigation
fix: audit gate decisions against pipeline answer provenance - #305
Merged
Merged
Conversation
The PR-ready and done-time decision audit compared resolved [key=nm-<run>-<step>] status records against a gate-resolution count reconstructed from the daemon state database: it never read who answered a gate, and it counted only the bare step key, so a worker that answered its own gate passed and round-suffixed decision keys silently did not count. Read each step's ask-user decision events and their response_origin from the pipeline's own `no-mistakes axi rounds --json` record instead. A decision the pipeline records as answered with no matching firstmate resolved record refuses PR-ready and done acceptance, naming the run, step, finding, and origin. A round- or detail-suffixed firstmate record counts for its step. Missing, unparseable, or origin-less rounds data refuses naming exactly what is missing; there is no fallback to the text-only audit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The decision audit now asks the pipeline who answered each gate instead of counting status-log strings.
PR-ready (bin/fm-pr-check.sh) ─┐ done acceptance (fm-crew-state) ─┴─> fm_nm_ask_user_decisions │ ├─ pipeline record: axi status steps ─> axi rounds --json per step │ events: (step, round N, finding, action, response_origin) └─ firstmate record: state/<id>.status - bare-key count only; no answering origin + gate openings + answered records, bound per step, round, and suffixbin/fm-nm-run-lib.shreads decision events fromaxi rounds --json, refusesinteractive/auto_yesanswers without a firstmate record, and refuses missing, unparseable, or origin-less data naming exactly what is missing.bin/fm-classify-lib.shemits gate openings and canonical answers; bare,-r<N>, and-detailrecords bind to their own gate response, so surplus records never migrate across rounds.bin/fm-pr-check.shandbin/fm-crew-state.shshare the one verdict; the PR-ready and done-time failures now name the run, step, finding, and origin.Evidence
01M796S134JFXVZXT5JVHEMWAYrecordedinteractivereview decisions with no firstmate record and no refusal; older origin-less runs silently fell back to the text-only count.After: the same run with an empty status file refuses:
run 01M796S134JFXVZXT5JVHEMWAY step review: 1 recorded ask-user gate decision(s) ... finding R25 resolved as fix (round 1, answered interactive). Origin-less run01M6TA5HW285X0ZF457AB648YKrefuses withround 1 records an ask-user decision with no response_origin(captured in the task'sreal-run-audit.txt).bash tests/fm-nm-run-lib.test.sh-> all passed, including grouped approval, round-suffixed keys, cross-round surplus refusal, and a rounds document over 128 KiB.bash tests/fm-pr-check-handoff.test.sh-> all passed, including the worker-answered refusal naming run/step/finding/origin and PR-ready/crew-state agreement.bash tests/fm-crew-state.test.sh-> all passed.shellcheck -xon every changed script: clean.git diff --name-only origin/main...HEAD: no path under.omp/extensions/.Merge Danger
Door: two-way. No data migration; the audit is read-only and the branch can be reverted.
Blast Radius: decision gates only. PR-ready and done acceptance refuse until a firstmate
resolvedrecord exists for each recorded gate response, so pre-existing runs with origin-less pipeline data now stop loudly instead of passing on a text count. CI and the pipeline remain the only full-suite owners.Pipeline
Updates from git push no-mistakes
✅ **Intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 2 issues found → auto-fixed (3) ✅
bin/fm-nm-run-lib.sh:457- Step-wide totals can accept an unmatched worker decision. For example, firstmate records both the bare key and -r30 for round 30's R1; the worker then answers R2 in round 31. need=2 and have=2, so both gates pass despite R2 lacking authority. Match records to decision events rather than allowing surplus records for one event to cover another. Related sites: bin/fm-classify-lib.sh:717 and :722 count these records independently; bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163 consume the verdict. The fixtures at tests/fm-nm-run-lib.test.sh:157 and tests/fm-pr-check-handoff.test.sh:335 already contain overlapping records but never assert refusal before adding R2's record.bin/fm-nm-run-lib.sh:417- Passing the complete rounds document through FM_NM_ROUNDS_DOC imposes an operating-system argument/environment size limit. On Linux, a single environment entry cannot exceed roughly 128 KiB; a review history with 30 rounds containing about 5 KiB of findings and summaries per round exceeds this. The python3 launch at bin/fm-nm-run-lib.sh:284 then fails before parsing valid evidence, blocking PR-ready and done acceptance even when every decision matches. Stream the JSON through a file descriptor or stdin instead. Both consumers are affected: bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163.🔧 Fix applied.
4 issues (3 errors, 1 warning) still open:
bin/fm-nm-run-lib.sh:457- Step-wide totals can accept an unmatched worker decision. For example, firstmate records both the bare key and -r30 for round 30's R1; the worker then answers R2 in round 31. need=2 and have=2, so both gates pass despite R2 lacking authority. Match records to decision events rather than allowing surplus records for one event to cover another. Related sites: bin/fm-classify-lib.sh:717 and :722 count these records independently; bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163 consume the verdict. The fixtures at tests/fm-nm-run-lib.test.sh:157 and tests/fm-pr-check-handoff.test.sh:335 already contain overlapping records but never assert refusal before adding R2's record.bin/fm-nm-run-lib.sh:417- Passing the complete rounds document through FM_NM_ROUNDS_DOC imposes an operating-system argument/environment size limit. On Linux, a single environment entry cannot exceed roughly 128 KiB; a review history with 30 rounds containing about 5 KiB of findings and summaries per round exceeds this. The python3 launch at bin/fm-nm-run-lib.sh:284 then fails before parsing valid evidence, blocking PR-ready and done acceptance even when every decision matches. Stream the JSON through a file descriptor or stdin instead. Both consumers are affected: bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163.bin/fm-nm-run-lib.sh:479- Round 1 leaves the surplus-record bypass reachable when finding IDs repeat. Record bare-keyfix R1and-r30fix R1for round 30, then let the worker answer R1 in round 31 without firstmate approval. The matcher assigns the suffixed record to round 30 and the old bare record to round 31, accepting both. Bind bare/detail records to their actual decision occurrence before matching; surplus records must not migrate between rounds. Related sites: bin/fm-classify-lib.sh:712 accepts unbound records; bin/fm-nm-run-lib.sh:498 assigns them; bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163 consume the incorrect pass.bin/fm-nm-run-lib.sh:492- Round 1 introduces a stricter ledger contract than the existing firstmate answer path supports. AC2 requires: “A run whose every ask-user decision has a matching firstmate decision passes.” Firstmate's documented single message can approve R1 and R2 together, producing one resolved line, butassigned[record] = eventpermits that line to cover only one finding. tests/fm-nm-run-lib.test.sh:175 explicitly constructs this valid approval and then expects refusal. Additionally, matching IDs in answer excerpts at bin/fm-nm-run-lib.sh:481 rejects legitimate answers when fm-send's 220-character cap truncates the IDs (bin/fm-send.sh:224). Revert the round-1 per-finding excerpt machinery to a minimal occurrence-aware gate-response match that preserves grouped decisions and bounded answer records, while retaining the JSON streaming fix. Related changed sites: bin/fm-classify-lib.sh:720 supplies the excerpt; bin/fm-pr-check.sh:202 imposes the new record requirement; both gate consumers at bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163 inherit the refusal. The remedy needs authorization because it revises the prior fix's deliberate matching contract.🔧 Fix applied.
5 issues (4 errors, 1 warning) still open:
bin/fm-nm-run-lib.sh:457- Step-wide totals can accept an unmatched worker decision. For example, firstmate records both the bare key and -r30 for round 30's R1; the worker then answers R2 in round 31. need=2 and have=2, so both gates pass despite R2 lacking authority. Match records to decision events rather than allowing surplus records for one event to cover another. Related sites: bin/fm-classify-lib.sh:717 and :722 count these records independently; bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163 consume the verdict. The fixtures at tests/fm-nm-run-lib.test.sh:157 and tests/fm-pr-check-handoff.test.sh:335 already contain overlapping records but never assert refusal before adding R2's record.bin/fm-nm-run-lib.sh:417- Passing the complete rounds document through FM_NM_ROUNDS_DOC imposes an operating-system argument/environment size limit. On Linux, a single environment entry cannot exceed roughly 128 KiB; a review history with 30 rounds containing about 5 KiB of findings and summaries per round exceeds this. The python3 launch at bin/fm-nm-run-lib.sh:284 then fails before parsing valid evidence, blocking PR-ready and done acceptance even when every decision matches. Stream the JSON through a file descriptor or stdin instead. Both consumers are affected: bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163.bin/fm-nm-run-lib.sh:479- Round 1 leaves the surplus-record bypass reachable when finding IDs repeat. Record bare-keyfix R1and-r30fix R1for round 30, then let the worker answer R1 in round 31 without firstmate approval. The matcher assigns the suffixed record to round 30 and the old bare record to round 31, accepting both. Bind bare/detail records to their actual decision occurrence before matching; surplus records must not migrate between rounds. Related sites: bin/fm-classify-lib.sh:712 accepts unbound records; bin/fm-nm-run-lib.sh:498 assigns them; bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163 consume the incorrect pass.bin/fm-nm-run-lib.sh:492- Round 1 introduces a stricter ledger contract than the existing firstmate answer path supports. AC2 requires: “A run whose every ask-user decision has a matching firstmate decision passes.” Firstmate's documented single message can approve R1 and R2 together, producing one resolved line, butassigned[record] = eventpermits that line to cover only one finding. tests/fm-nm-run-lib.test.sh:175 explicitly constructs this valid approval and then expects refusal. Additionally, matching IDs in answer excerpts at bin/fm-nm-run-lib.sh:481 rejects legitimate answers when fm-send's 220-character cap truncates the IDs (bin/fm-send.sh:224). Revert the round-1 per-finding excerpt machinery to a minimal occurrence-aware gate-response match that preserves grouped decisions and bounded answer records, while retaining the JSON streaming fix. Related changed sites: bin/fm-classify-lib.sh:720 supplies the excerpt; bin/fm-pr-check.sh:202 imposes the new record requirement; both gate consumers at bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163 inherit the refusal. The remedy needs authorization because it revises the prior fix's deliberate matching contract.bin/fm-nm-run-lib.sh:486- Round 2 introduced order-dependent recovery matching. With decision rounds 30 and 31 and no ledger openings, append the firstmate recovery answer for -r31, then the documented bare-key recovery answer for the first response (round 30). Updating current to 31 causes the bare answer at bin/fm-nm-run-lib.sh:494 to bind to 31 too, incorrectly refusing round 30 despite both approvals. Explicit recovery answers must not move the implicit gate binding; only gate openings should advance it. Related consumers: bin/fm-pr-check.sh:196 and bin/fm-crew-state.sh:163. The bare-key recovery contract is documented at bin/fm-nm-run-lib.sh:250.🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
TMPDIR="$PWD/.validation-live/tmp" bash tests/fm-nm-run-lib.test.shTMPDIR="$PWD/.validation-live/tmp" bash tests/fm-pr-check-handoff.test.shTMPDIR="$PWD/.validation-live/tmp" bash tests/fm-crew-state.test.shpython3 .validation-live/live.py: exercised real Firstmate commands and No-Mistakes status/rounds readers with isolated synthetic records, a synthetic forge head, and external-boundary fault injectiongit diff --name-only origin/main...HEAD: confirmed no changes under.omp/extensions/Removed the disposable setup and confirmed a clean working tree; every started check exited successfully.✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.