Run the eval suites deferred by the upstream-pointer sweep PRs

Description

The upstream-pointer sweep PRs (#7061-#7274) merged with their eval runs deferred: they mostly replace restated upstream Claude Code docs text with pointer records. Each PR body lists the skills whose eval run was owed. This issue collects those lists so the runs are not lost once the PRs close.

Run first: PRs that also changed behavior or output

fix(source-control): point restated upstream docs at live sections and match the proxy's GraphQL refusal #7241 source-control: babysit_gh.py recognizes the cloud proxy's GraphQL refusal wording, which changes how the unattended babysit loop falls back to REST. Evals: check, plus a babysit-prs run against a PR in a cloud session.
docs(work-items): point restated upstream docs at live sections #7239 work-items: work/scripts/preflight.sh prints corrected warning text (one test assertion updated). Evals: attend-queue, setup, ship, work.
docs(harness-ops): point restated upstream docs at live sections #7245 harness-ops: the install-state audit table has new swept rows (uploads, dev-mods, usage-data), a jobs kept row, and a reworded backups note. Evals: audit-install-state, audit-native-overlap, audit-performance, audit-skill-visibility, check, machine-profile, prerequisites.
docs(instruction-placement): point restated upstream docs at live sections #7254 instruction-placement: the migrate shim check reads the AGENTS.md plugin option under both its old and new ID. Evals: migrate, audit, delta, plus architecture map-containers, map-context, map-flow, map-landscape, record-decision.
docs(context-budget): point restated upstream docs at live sections #7253 context-budget: the audit's emitted known-uncovered note text changed. Evals: audit, check.

Pointer-only PRs

Run each with /evals:plugin-eval against the merged main. A regression goes in its own bug issue that links this one.

Affected area

Plugin skills across the marketplace (see the list).

Found during the upstream-pointer sweep (PRs #7061-#7274).

Proposed work class: work-class: read-only
Basis: automated intake triage; the lanes App applied this class.

Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none

Placeholder

_{Automated triage, applied by the lanes App.}

This was generated by AI during triage.

Hold: each listed eval run spends eval budget, and no spend or ceiling has been approved for these runs. The owner has to approve that before an agent runs them.

This was generated by AI.

Parked for capacity, not for the owner. The hold said no spend was approved. That is answered: subscription spend does not stop work (owner Q91), and evals run on the subscription login, only while usage allows (owner, 2026-10-25). The runs go through the plugin eval dispatch lane, so this waits on #9704 (validate, gate and summarize dispatched eval runs), now a blocked-by edge, and on subscription headroom.

Under the owner's 2026-10-26 stop rule, a person is needed only for a change in intent or vision, an implementation finding that changes the intent or primary design, or a high or critical security issue; host, attended and eval runs are agent work. So needs-human comes off, and the issue moves to host-run with status: confirmed, which keeps the issue lane from taking it when #9704 closes: #9704 adds an eval dispatch workflow, not eval capability for the issue lane.

The work-resolve-issue lane stopped on this issue (host-only): the run reports the work needs a host or attended run the lane cannot make.
It added host-run and removed agent-ready: a maintainer session on a host builds it, and it waits on no decision from a person.

In the run's words:

host-only: the issue asks for /evals:plugin-eval runs across about 30 plugins' skill suites against the merged main, plus a babysit-prs run against a PR in a cloud session. This CI lane does not load the evals plugin and cannot run eval suites or attended host sessions, so no part of the work can be done here. The issue carries the host-run label, and the owner's 2026-10-27 comment says it waits on ＃9704 (eval dispatch lane) and on subscription headroom, and that host-run keeps the issue lane from taking it.

What the run says it did not do:

Every eval run the issue lists (the 'Run first' group ＃7241, ＃7239, ＃7245, ＃7254, ＃7253 and the pointer-only group ＃7061 through ＃7274). They need /evals:plugin-eval and a host or eval-dispatch run, which this lane cannot make.
Tidy, review, verify and checks pipeline steps: skipped because the run made no change.

Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/40636250186

To hand it back to the lane, remove host-run and add agent-ready again.

Host run. Round 14 ran the owed suites that had never run. 15 suites ran locally and 26 on GitHub-hosted runners. 20 results are VALID, 1 is VALID but not counted, and 4 cases are left.

What ran

Local: Claude Code 2.1.318 on the subscription login, against origin/main 90d3c6962e.
- Each run used a git archive copy, a capped scope, an empty HOME, auto memory off (CLAUDE_CODE_DISABLE_AUTO_MEMORY=1), --runs 3, under the heavy-run lock, at most 2 suites at once.
- The ~/.docker links were the same before and after each batch.
- Judged by run-validity.py --runs 3 and noise-report.py --threshold 0.8 --grader-agreement.
CI: eval-browser-dispatch with runs=3, allow-tools=Bash,Write,Edit and env-scrub=true, after fix(ci): read a dispatched eval's validity and noise on the runner and upload the verdicts #13313 added a validity step that runs on the runner.
- Live check: run 41885795085. Its validity.txt reads verdict: VALID (warnings: skill fired).
- The artifact carries no traces, so whether Bash ran (R61) cannot be checked on CI results. They are counted on the runner's verdict alone.

Counted (VALID)

Where	Plugin and skill
local	harness-ops `prerequisites`; skill-quality `check`; testing `plan`; docs-hygiene `audit-encapsulation`, `audit-progressive-disclosure`, `rename-references`; ruff-format `check`
CI	typos-format, desktop-notification, markdown-format `check`; improvement `improve`; codebase-health `audit`; review `explain-change`; kindle-dedrm `manage`; mcp-tools `audit-posture`; repo-fleet-hygiene `apply`; computer-use `diagnose`, `setup`; playgrounds `use` (6 of 8 cases); visualization `present`

VALID but not counted: dometrain setup. Bash ran in 0 of 6 traces (R61).

INVALID

harness-config, all 5 owed skills. The eval sandbox denies the skills' Bash fetches to code.claude.com and their calls into the plugin's own directory. Locally, the uh-readd scaffold also failed under the empty HOME.
Network denied:
- knowledge setup (npm registry).
- miro setup (code.claude.com).
Calls into the plugin's own copy, or compound probe chains, denied:
- Local: actionlint, eol-normalizer, go-format and powershell-format check.
- CI: bash-format and biome-format check.
Other denials:
- machine-health setup, disk-hygiene clean, repo-fleet-hygiene audit, plugin-quality audit, visualization visualize.
- ai-slop: all 3 dispatches.
- repo-hygiene clean: one run never started, plus 20 denials.
Cost cap: the playgrounds narration pair stopped partial at the 5 USD cap.

These failures come from the sandbox, not from scores. The next round sorts them into two kinds: case or skill changes that avoid the denial, and denials no case can avoid, which get recorded as not measurable the way distill and running-retro were.

Low with-arm scores in VALID results

None of these can be compared with a run from before the sweep, so none is a proven regression yet. Each is checked next against the case's grader and the skill text.

docs-hygiene rr-review-skill-stragglers: 0.67 in all 3 runs, the same as the no-plugin arm.
kindle-dedrm md-friend-files: 0.67.
computer-use diag-uipi-operator: 0.67.
skill-quality ve-new-evals-route-to-suite: 0.78, one run at 0.33.
visualization pres-*: 0.83 to 0.89.
ruff-format su-check-no-config-inert-info: 0.89 in 3 of 3 runs.

Not run

ai-slop audit-mechanism-free-claim and audit-over-compression-boundary.
playgrounds use-installed-routes-to-upstream-skill and use-shadowed-name-is-not-invoked. They need one dispatch each, because the cost cap stops a run of both.
plugin-quality reported 2 of its 4 cases. The cause is unknown.

Still owed

The 4 cases under Not run, plus plugin-quality's 2 missing cases.
The INVALID set: fix the case, or record it as not measurable.
The cases that round 13's ledger lists as INVALID or partly VALID.
The 6 low with-arm scores above.
Skills with no case: tracked in test(evals): add eval cases for pointer-swept skills that have none #13300, now split into test(evals): add eval cases for pointer-swept skills that have none - part 1 of 9 #13303 to test(evals): add eval cases for pointer-swept skills that have none - part 9 of 9 #13311.

The issue's done-when does not hold yet.

Target labels: add none; remove none.

Host run. Round 15: the four leftover cases, a sort of round 14's INVALID results by cause, and a review of the low with-arm scores. All four runs dispatched on main (eval-browser-dispatch, runs 3).

Leftover runs

Case	Run	With	Without	Verdict
ai-slop `audit-mechanism-free-claim`	41901926105	0.73	0.60	INVALID (1 denial)
ai-slop `audit-over-compression-boundary`	41901980466	0.80	0.60	INVALID (1 denial)
playgrounds `use-installed-routes-to-upstream-skill`	41902033867	0.83	0.50	VALID
playgrounds `use-shadowed-name-is-not-invoked`	41902087018	0.75	0.25	VALID

playgrounds use: 8 of 8 cases are now VALID.
plugin-quality's two missing cases never loaded. pq-session-lists-and-waits and pq-session-no-usage-artifact-stops both set plugins: ["../..", "../../../session-flow"] (case.yaml:5), which points outside the plugin directory. The CLI loads plugins only from inside the directory under eval: round 14's job log says "2 case file(s) failed to load". The result still read partial: false, and validate-cases.py does not catch this before a paid run.

INVALID results by cause

Case-only fix: disk-hygiene clean. Its prompts name an /eval/plugin/... path, so the no-plugin arm reads a path that only exists with the plugin loaded (case.yaml:8).
Not measurable as written (the skill does what the eval sandbox always denies):
- harness-config audit fetches the docs pages on every run (audit/SKILL.md:124-126).
- knowledge setup installs npm packages (setup/SKILL.md:102-105).
- machine-health setup probes the literal plugin-data path (setup/SKILL.md:61-62).
Each has a named test still to run:
- harness-config audit-instructions and audit-pass, and miro: the model follows "fetch live" pointers with Bash curl. Test: rerun with WebFetch granted.
- unhobble writes into .claude/unhobble/. Test: grant a scoped Write.
- go-format, powershell-format and eol-normalizer: every denied call contains a literal ${user_config.*}.
- actionlint and powershell-format: the model Greps the plugin README for a version floor.
No denial data: bash-format, biome-format, visualization visualize, repo-hygiene, repo-fleet-hygiene, plugin-quality, ai-slop. The CI artifact carries only the verdict line, so the cause can't be read from it.

Low with-arm scores

Grader defects:
- docs-hygiene rr-review-skill-stragglers: swept-with-grep requires 3 or more Grep calls (graders/swept-with-grep.md:4).
- computer-use diag-uipi-operator: its regex rejects the skill's own "instead of retrying" wording.
- visualization pres-k2-instruction: its regex is missing "comment" and "ignored".
- visualization pres-missing-figure: the line window around the marker is too narrow.
Skill gaps:
- kindle-dedrm manage never says to name My Kindle Content when it declines (manage/SKILL.md:12, :149).
- ruff-format setup check offers to write a config. The only rule on offering is under apply (setup/SKILL.md:200-201).
Run noise: skill-quality ve-new-evals-route-to-suite and visualization pres-outline-record.

None of these is a regression from the pointer sweep. Each is a defect in a grader, a case or a skill, so each is fixed under this issue rather than filed as a sweep regression bug.

The lane rule counts a result only when its traces show Bash ran. The CI artifact has no traces, and its validity.txt holds only the verdict line.
A VALID verdict alone can hide a run that measured nothing. dometrain setup came out VALID locally with Bash in 0 of 6 traces.
The provisional results are typos-format, desktop-notification and markdown-format check; improvement improve; codebase-health audit; review explain-change; kindle-dedrm manage; mcp-tools audit-posture; repo-fleet-hygiene apply; computer-use diagnose and setup; playgrounds use; and visualization present.
They count once the CI verdict carries a Bash-ran count, which Automation Lanes has been asked to add, or once a local rerun shows it. This issue does not close on them before that.

Basis: the R61 rule (Bash must run in the trace) and round 14's dometrain result.

test(computer-use): accept "retrying would still fail" in retry-wont-…

test(visualization): stop blank-marked window at bold slide labels

docs(evals): point the plugins containment rule at its source record

Navigation Menu

Repository navigation

Run the eval suites deferred by the upstream-pointer sweep PRs #7296

Description

Run first: PRs that also changed behavior or output

Pointer-only PRs

Affected area

47 remaining items

What ran

Counted (VALID)

INVALID

Low with-arm scores in VALID results

Not run

Still owed

Leftover runs

INVALID results by cause

Low with-arm scores

Next

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Run the eval suites deferred by the upstream-pointer sweep PRs #7296

Description

Description

Run first: PRs that also changed behavior or output

Pointer-only PRs

Affected area

Activity

github-actions commented on Oct 12, 2026

kyle-sexton commented on Oct 19, 2026

kyle-sexton commented on Oct 27, 2026

melodic-automation-lanes commented on Oct 28, 2026

47 remaining items

kyle-sexton commented on Nov 4, 2026

What ran

Counted (VALID)

INVALID

Low with-arm scores in VALID results

Not run

Still owed

kyle-sexton commented on Nov 4, 2026

Leftover runs

INVALID results by cause

Low with-arm scores

Next

kyle-sexton commented on Nov 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions