feat(session-flow): judge applied retro changes on later sessions, per the retro-adaptation study #12884

Parent: #7555. Owner approved in session on 2026-11-03 ("As long as everything is backed by research, the paper + Matt Pocock's guidance - yes I approve"), for items 1-4 below and option (a) for item 5.

Why

Treude and Baltes, How Do Coding-Agent Harnesses Self-Evolve? An Empirical Study of Retrospective Adaptation (arXiv 2610.34938, 2026-10-26), studied Matt Pocock's /retro (mattpocock/skills@b0618bc) on SWE-bench Verified tasks. Its findings that apply to /session-flow:retro and /corrections:codify:

• Prior-driven proposals (Section IV, RQ1). 18 of the 19 proposals not based on a session event were lint checks. The paper traces them to the skill treating "a repository with neither a pre-commit hook nor a CI job running its checks as 'a finding'". The one prior-driven change it tested, a pre-commit configuration, "was present in all five adapted sessions, but no session read it or ran pre-commit". Matt's current skill (skills/engineering/retro/SKILL.md on main) still says "A repo with no guardrail ... is itself a finding". We copied that clause in feat(session-flow): add Matt's retro findings to /session-flow:retro session mode #12122 (plugins/session-flow/skills/retro/reference/ecosystem-improvement-catalog.md:87-88).
• Scope of retained experience (Sections IV and VI). A durable fact transferred: a CLAUDE.md line naming bin/test took recurrence from 5/5 to 0/5. Lists of specific failing tests did not cover the failures later sessions hit (four events unchanged). A list that told the agent to treat any unlisted failure as its own caused 4 of 5 sessions on an unrelated task to run baseline comparisons, against 0 of 5 without it. Our "Keep only durable findings" tests (plugins/session-flow/skills/retro/context/session.md:241-253) drop session-only facts but do not stop a list from presenting itself as complete.
• Bundled changes (Section IV). The SymPy CLAUDE.md fix also carried a line telling the agent to run CI lint checks; median elapsed time rose from 88 to 156 seconds. Neither the catalog nor session mode says one finding gets one change.
• Continued evaluation (Section VI). "An evaluation of an adaptation therefore needs to establish whether the targeted problem arose, whether the agent consulted the adaptation, and whether its actions changed. Retained changes also need continued evaluation, because outdated or incomplete instructions can remain ineffective or add work." Session mode appends only unapplied findings to the findings buffer (session.md:359-366: "Applied findings ... are not appended"), so no later session judges an applied change. Task success "differed by at most one session between harnesses in every event", so the measure is recurrence of the targeted action, not task success. In 5 of 13 transfer events the baseline rarely hit the problem, so most judgments will lack data.

Already covered, no change: a check must fail on a planted or past violation and run at edit time and in CI (reference/enforcement-ladder.md "Guardrail criterion" items 5 and 6), which answers the paper's 29 of 38 checks that could not report failure in the task environment.

Done when

• Item 1. ecosystem-improvement-catalog.md "When to propose a check" keeps reporting a repo with no CI or pre-commit step (Matt's guidance), and says such a finding proposes a check only when a mistake in this session is one that check would have caught; otherwise it is reported as a standing gap with no proposal. The basis cites the paper's RQ1 finding by URL.
• Item 3. session.md "Keep only durable findings" gains a fourth test: a finding that lists observed cases (failing tests, broken paths, flaky jobs) must say how a later session recognizes a case not on the list, and must never tell the reader to treat unlisted cases as caused by its own work. A durable fact (the test command, where logs live) passes as is.
• Item 4. Session mode's Phase 3 or the catalog says each recommendation makes one change for one finding and carries no unrelated instruction.
• Item 5, record. /corrections:codify note accepts --applied, which adds "kind": "applied" to the row; signal names the situation the change targets and the action it should prevent, and source_id is the change's #PR or commit when given. Session mode Phase 4 makes one note --applied call per applied finding (the existing no-plugin fallback lists them in the summary instead).
• Item 5, judge. Session mode 2C gains an applied-change check: read the buffer's applied rows (/corrections:codify group --by kind), and for each one whose situation this session met, give one verdict with evidence: effective (situation arose, targeted action did not happen), recurring (the action happened anyway; it is a regression and takes Phase 4's "Guidance that existed but did not fire" path, which already asks whether it was loaded), added work (the change made this session do work its situation did not need; routed to /harness-config:audit-instructions), or not encountered (no verdict, nothing written). State whether the change was loaded and whether it was consulted (the file read or command run). Never propose removing a change on missing data.
• Eval cases: at least one in plugins/session-flow/evals/ for item 3 or 5 and one in plugins/corrections/evals/ for note --applied, each case.yaml using the same schema_version as its neighbours.
• .changes/session-flow/*.md and .changes/corrections/*.md fragments, bump: minor.
• CI green.

Out of scope

• A per-change with/without A/B run (the paper's method); /evals:plugin-eval covers that when a change warrants it.
• Automatic retirement or expiry of guidance lines; /harness-config:unhobble watch owns removal.