Repository navigation
feat(evals): background hill-climb that opens draft PRs for skill edits beating the baseline #12035
Description
Refs: #6405
Outcome
#6405's title asks for a background hill-climb: "propose skill edits, keep only those that beat the baseline over enough runs, open each as a draft PR". No issue tracks it, and the lane that runs suites (.github/workflows/eval-browser-dispatch.yml) only measures. This slice builds a scheduled or dispatched run that proposes one skill edit, measures it against the current skill, and opens a draft PR only when the edit wins beyond run-to-run variance.
The design is open, so this slice starts with a short plan comment on #6405 covering the run count, the win margin, which suites are eligible, and the authority limits below. It is labeled work-class: structural to match the parent.
Done when
- A plan comment on feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405 names the trigger, the eligible suites, the run count and win margin with their
Basis:, and the per-run--max-cost-usd. - A dispatched run on one suite either opens one draft PR whose body links the with-versus-without result for the edit and for the baseline, or exits with no PR and says why in the job summary.
- The model step holds no merge right, no
ci-statuswrite, and nochecks: writeorworkflowspermission, and it pushes only to a lane branch, never tomain(ADR 0049,docs/adr/0049-run-ci-lanes-on-github-hosted-runners-under-trigger-and-token-hardening.md). - Eval transcripts that reach the model step are framed as data per
docs/contracts/untrusted-content/README.md. - A case whose health line reads
skill loaded: nois excluded from the win test.
Out of scope
- Auto-merging any hill-climb PR.
- Case repair and the health line, which are their own slices of feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405.
Activity
github-actions commentedon Nov 1, 2026
Proposed work class: work-class: structural
Basis: automated intake triage; a maintainer confirms or changes it.
Size: Large
Complexity: High
Uncertainty: Knowable
Flags: none
Blocked by: none
I read this as a new scheduled or dispatched eval run, tracked under #6405. It would propose one skill edit and measure it against the current skill. It would open a draft PR only when the edit beats the baseline by more than run-to-run variance. The issue makes the first step a plan comment that fixes the run count, win margin, eligible suites and authority limits. That plan is part of the work, so I'm not treating it as an open question. The done-when items are specific. I found no open duplicate. Because the work touches CI permissions and eval transcripts that reach a model step, it likely needs splitting into smaller changes.
Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 1, 2026
Not admitted to the issue lane: Size unset is not one of small, medium; Uncertainty unset is not one of known, knowable. Admission takes the issue once each of those is met, when intake triages it again (
.github/actions/resolve-config/admission-rules.json).kyle-sexton commentedon Nov 4, 2026
Status.
Agent choice, not an owner answer.
Routed to a session builder: Automation Lanes. This slice builds a new dispatched eval lane, so its delivery is a new workflow file plus job permissions, and the issue lane refuses any patch under that directory (
.github/actions/resolve-config/patch-rules.json:3,github-workflows). It stays atstatus: confirmedwithhost-runand noagent-ready.The builder starts with the body's first item, the plan comment on #6405 (trigger, eligible suites, run count and win margin with their
Basis:, per-run--max-cost-usd). Model usage under the subscriptions and the accounts' API credits is capacity, not an owner cost question, so that plan needs no owner answer unless it would bill a GitHub or Azure account.fix(evals): keep eval traces on the pass runner and repin ci-workflows
fix(evals): carry-refuse plugin-eval scripts and fail the verdict on …
chore(source-control): add the changelog fragment for the plugin-eval…
kyle-sexton commentedon Nov 5, 2026
Owner decision, AL-Q11, 2026-11-05T05:47:33Z (AskUserQuestion in the Automation Lanes session 24905f8a).
Asked: may the lanes App get a write grant for
chore/hill-climb-*branches, used only on a run a listed person dispatches? It widens what a lanes token can push. Options: grant it, scoped like maintenance-tidy-code-upkeep (brancheschore/hill-climb-*only, human-dispatched runs only, draft PRs only, and the model step holds no write token), recommended; no grant (the run uploads its patch as an artifact); defer the lane.Owner's pick, verbatim: "Grant it (Recommended)".
This unblocks #12035's push half. The grant is a lane-rules.json row, built by the Automation Lanes session.
Posted by the Automation Lanes session. This was generated by AI.
fix(evals): name each post-win stop in the hill-climb job summary
feat(github-actions-conventions): reserve the maintenance function me…
kyle-sexton commentedon Nov 6, 2026
Re-run after #13640 and #13665 (run 42172606678): the policy probe now passes, but the key leg's
claude plugin evalexited 2 with a partial run (auth_failed). The handoff classifier read it asoutcome=success, so the subscription leg never ran and the verdict stopped atgate: partial: pass a did not finish (auth_failed). No draft PR was opened. A fix for the classifier and the auth failure's cause is in progress onfix/eval-auth-failed-handoff; the next re-run follows its merge.kyle-sexton commentedon Nov 6, 2026
Decision recorded. Owner, in session (2026-11-06), verbatim: "OK go w/ recommendations then", answering Q1 on the eval step's subprocess env scrub with option (a): keep
CLAUDE_CODE_SUBPROCESS_ENV_SCRUB=1on the eval step, so the key leg hands off onauth_failedand the subscription leg runs evals (#13699). No change to the scrub; no upstream report of the plugin-evals / env-vars docs conflict.