[kyle-sexton]

Closes #12884

Summary

Problem: /session-flow:retro never judged a change after applying it, and its catalog proposed checks from general practice rather than from session events. The retro-adaptation study (Treude and Baltes) found prior-driven proposals went unread, listed known failures missed the cases later sessions hit, and bundled lines added work.

• A repo with no CI or pre-commit step is still reported, but proposes a check only when a mistake in this session is one that check would have caught; otherwise it is a standing gap with no proposal.
• "Keep only durable findings" gains a fourth test for lists of observed cases. Phase 3 says each recommendation makes one change for one finding.
• Phase 4 records each applied finding with /corrections:codify note --applied, and 2C judges those rows on later sessions: effective, recurring, added work, or not encountered, with whether the change was loaded and consulted.

Fix

• plugins/session-flow/skills/retro/reference/ecosystem-improvement-catalog.md: "When to propose a check" now covers the no-CI case (item 1).
• plugins/session-flow/skills/retro/context/session.md: a fourth durable-findings test (item 3), a one-change-per-finding rule (item 4), the 2C applied-change check (item 5, judge) and Phase 4's note --applied calls with the no-plugin "Applied findings" fallback (item 5, record).
• plugins/corrections/skills/codify/actions/note.md: accepts --applied, which adds "kind": "applied". group.md leaves applied rows out of its by-mistake counts, so a recorded change is never counted as a recurrence of the mistake it prevents. The usage line in SKILL.md, README.md and the legacy evals.json follows.
• Eval cases re-session-listed-failures-cover-unlisted (session-flow) and codify-note-applied-kind (corrections), plus bump: minor fragments for both plugins.

Verification

• validate-cases.py on both eval suites: exit 0, no FAIL. Every sample sorts as labeled; the only WARNs are the --allow-tools notices every neighbour case carries, and the judge-calibration note on the new llm grader.
• markdownlint-cli2 on the 8 touched markdown files: 0 issues. check-changelog-fragments.sh --check: all 28 fragments valid. shellcheck on both new scaffolds: clean.
• The pre-push check-affected hook passed.
• Left out: a paid claude plugin eval run of the new cases, which is the operator's to start.

Related

• Parent: Epic: absorb Lauren Tan's and Matt Pocock's agent workflows (poteto-review) #7555
• Matt Pocock's retro, which still reports a repo with no guardrail as a finding: skills/engineering/retro/SKILL.md

🤖 Generated with Claude Code

https://claude.ai/code/session_01MWmASLCzgJayUQq9Saq4AK

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-03T17:49:25.640536Z	9b52245	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

claude-review: 1 inline finding(s) on 9b522454b2917b6265f90203f4b46489fe8ac750; the summary is in this lane's pull request comment.

claude-lane: claude-review

[github-actions]

claude-review reviewed 9bc78dc4dc6be77663057d8da61b7c33fc17b925: read 5 of 5 changed file(s); 0 finding(s), 0 as inline comments.

Code review (incremental): PR #12889, 9b522454 → 9bc78dc4

This round covers only the new commit. I read every hunk of all 5 files in the incremental list. For context I also read group.md at head, fixtures/repo.sh, the neighbouring codify-note-applied-kind case, and the buffer paths in group-findings.test.sh and buffer.sh. None of the 5 files matches a trusted-path glob.

Earlier findings from this lane

• Inline finding: group didn't show location. Fixed. group.md#L43 now lists an applied row with its obs_id, source_id, signal and location (when set). The owner resolved this thread and it stays settled.
• Advisory: test gap for applied rows in group's by-mistake counts. Covered by the new eval case codify-group-skips-applied-rows, so the earlier summary finding is not restated.
  • The scaffold uses the standard fixture repo and seeds .work/buffer/findings.jsonl, the same local buffer path group-findings.test.sh uses.
  • It writes two rows with the same signal: an applied row (a1, #42, location CLAUDE.md) and a mistake row (m1, session:…).
  • The LLM grader fails the case if a1 is counted in the mistake group, if the count is 2, or if CLAUDE.md is not shown.
  • The no-write and no-edit graders enforce group's "writes nothing" rule.
• The 2C applied-change verdicts in session.md. This part of the earlier advisory is still untested. It was not raised again in this round.

Minor note, not a finding: samples/no-edit.json and samples/no-write.json were copied from the session-flow re-session-* cases. Their why text mentions retros and CI wiring. The tool_used graders only look at the tool name, so the samples still check the right thing.

Findings: none in this increment.

Not verified by me: I ran no eval, lint or validator. Any results the author reports are their claim.

Spend: the new eval case (12 turns, 300 s timeout) uses prepaid model usage when someone runs claude plugin eval. It causes no billed charge.

No embedded instructions were found in the settled-threads or prior-findings files.

[github-actions]

claude-security-review reviewed 9bc78dc4dc6be77663057d8da61b7c33fc17b925: read 5 of 5 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #12889, incremental since 9b52245: no security issues found

I read every hunk of the 5 files that changed since 9b522454b2917b6265f90203f4b46489fe8ac750, using .claude-lane/incremental.diff. None of them match a TRUSTED PATHS: glob. The push changes no plugin manifest, so no manifest <field> lines apply.

What I checked

• plugins/corrections/skills/codify/actions/group.md
  • The only change is that an applied row is now listed apart with its location field (when set), next to obs_id, source_id and signal.
  • The value is display output read from the local .work/buffer findings file. It adds no write target, tool grant or action.
  • The change adds a field to output and removes no rule, so the instruction-surface lens finds nothing.
• New eval case codify-group-skips-applied-rows (case.yaml, samples/no-edit.json, samples/no-write.json, scaffold.sh)
  • scaffold.sh runs the suite's existing ../fixtures/repo.sh, then writes two fixed JSONL rows into .work/buffer/findings.jsonl with printf.
  • The signal value is a hard-coded literal. Nothing from outside the case reaches it, and the script fetches nothing from the network.
  • case.yaml allows [Bash, Read, Glob, Grep, Skill]. The grader checks that Write and Edit are never used (max: 0, arm: both).
  • The sample files are grader calibration data only.

Text in the PR context: none of it tells this lane to pass the PR, skip a check, drop a finding or post elsewhere.

Prior findings: the lane's earlier summary reported 0 findings, so nothing is restated. The round cap (3) has not been reached.

Spend: the new eval case uses prepaid model usage only when someone runs claude plugin eval on it. This is not a hold.

critical-findings: 0