Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
bump: minor
---

### Changed

- `/docs-hygiene:rename-references` apply mode no longer asks before applying the Certain bucket.
It applies those matches by default and reports the count, one or two example matches, and how
to review the bucket one match at a time instead. The proceed/abort gate, the Chain-context
prompt and the per-match Ambiguous prompts are unchanged. (#12592)
4 changes: 2 additions & 2 deletions plugins/docs-hygiene/skills/rename-references/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,8 +126,8 @@ Multi-word old/new is fine. Separator is the only delimiter. Quote characters (`
- **Certain**. Slash-token, path, frontmatter glob (high precision, auto-apply candidate)
- **Chain-context**. Preceded/followed by other known skill names; numbered table rows; chain prose with multiple skill tokens
- **Ambiguous**. Bare-token matches where token is in English-verb blocklist (`confirm`/`test`/`review`/`fix`/`clean`/`build`/`lint`/`verify`/`plan`/etc.)
4. **Confirm**. Present buckets via `AskUserQuestion`:
- Certain → "auto-apply N matches?". Yes/no
4. **Confirm**. Present bucket counts and gate proceed/abort via `AskUserQuestion`; then per bucket:
- Certain → no question: auto-applied, reported with count, 1-2 examples and how to review one-by-one instead
- Chain-context → "review N matches one-by-one?". Yes/skip-bucket/auto-apply-all
- Ambiguous → per-match yes/no with surrounding context
5. **Apply**, Edit each accepted match. Idempotent, running twice is safe.
Expand Down
15 changes: 9 additions & 6 deletions plugins/docs-hygiene/skills/rename-references/context/apply.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,13 +47,16 @@ Classify per [triage.md](triage.md) into Certain / Chain-context / Ambiguous / E

Present bucket counts via the report template from [triage.md](triage.md) "Reporting bucket counts." Then gate the proceed/abort decision via `AskUserQuestion`.

Per-bucket confirmation flow:
Per-bucket flow:

**Certain bucket:**

- Show count + 1-2 example matches
- `AskUserQuestion`: "Auto-apply N matches?" with options "auto-apply", "review one-by-one"
- Default: auto-apply (these forms have empirically near-zero false-positive rate)
- Mark every match accepted for Phase 5 without asking (these forms have empirically near-zero false-positive rate)
- Report the count and 1-2 example matches with the bucket counts, before the proceed/abort gate,
and say how to review the bucket one by one instead: answer "review Certain one by one" in the
reply to the gate or to any later Phase 4 prompt. The Certain matches are then presented per
match with 3 lines of context and the options "rename this", "skip this", "skip remaining
Certain"

**Chain-context bucket:**

Expand All @@ -68,15 +71,15 @@ Per-bucket confirmation flow:
- ALWAYS one-by-one, because batched confirmation defeats the safety purpose

**Record every decline as a confirmed skip, keyed by `(file, line, start, end)`.** "skip bucket",
"skip this" and "skip remaining ambiguous" each produce skip records for the matches they cover.
"skip this", "skip remaining ambiguous" and "skip remaining Certain" each produce skip records for the matches they cover.
Phase 6 subtracts those spans from the actionable count. Without the record the same declined
match re-enters Outcome B on every re-sweep and the loop cannot terminate. See Phase 6.

If user picks "abort" at any prompt, halt and report partial state (no matches edited yet, since the Edit phase has not started).

### Phase 5: Apply

For each accepted match (from Phase 4 user confirmations):
For each accepted match (every Certain match not moved to one-by-one review, plus the matches accepted at the Phase 4 prompts):

- Use Edit tool with exact old/new substitution
- For matches where the form requires partial replacement (e.g., chain prose `→ confirm →` becomes `→ verify →`), construct precise old_string and new_string
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Match patterns where the rename intent is unambiguous regardless of surrounding
Form 15 where the token is not a common noun. The demoted alternatives land in Bucket 2 or
Bucket 3 below; `patterns.md` "Phase 0" carries the per-alternative table

**User flow:** present count and 1-2 example matches via `AskUserQuestion`. Two options: "auto-apply N matches" or "review one-by-one." Default: auto-apply.
**User flow:** auto-apply without asking. Report the count, 1-2 example matches, and how to review the bucket one by one instead: answer "review Certain one by one" in the reply to the proceed/abort gate (`apply.md` Phase 4).

**Why default-apply:** these forms have near-zero false-positive rate empirically, so treating them as ambiguous burdens the user without value.

Expand Down Expand Up @@ -160,6 +160,9 @@ Sweep results across <N> tracked files:
| Ambiguous | <Z> | Per-match confirm |
| Excluded | <W> | Skipped (plan-docs/historical/memory) |

Certain examples: <1-2 matches>. To review them one by one instead, answer
"review Certain one by one".

Proceed?
```

Expand Down
12 changes: 12 additions & 0 deletions plugins/docs-hygiene/skills/rename-references/evals/evals.json
Original file line number Diff line number Diff line change
Expand Up @@ -596,6 +596,18 @@
"`/confirm-changes` is named as what a trailing `\\b` wrongly matches",
"The always-loaded surface is identified as the reason a stale directive there is worse than elsewhere"
]
},
{
"id": 49,
"name": "certain-bucket-auto-applies-and-reports",
"prompt": "Rename the skill /verify to /verify-changes. The sweep finds only slash-token and path references.",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Advisory: this eval uses a blocklisted token, so it can fail for a reason unrelated to the change. verify is on the English-verb blocklist (triage.md L92). SKILL.md L159 says triage "forces these into ambiguous bucket regardless of regex hit position", and L189 says "If a token is in the blocklist, force into ambiguous regardless of position". Only patterns.md Phase 0 (L488) limits the blocklist to Form 2 and the chain forms, which would keep /verify slash-tokens and paths Certain.

How it fails: a model that follows SKILL.md puts every /verify match in Ambiguous. The Certain count is then 0, so expectation 2 ("The report gives the Certain count, one or two example matches…") fails and the per-match Ambiguous prompts run. The eval then fails on a blocklist-precedence question instead of the Certain-bucket behavior it is meant to test.

Fix: use a token that is not on the blocklist, so the eval tests only this change. For example:

Suggested change
"prompt": "Rename the skill /verify to /verify-changes. The sweep finds only slash-token and path references.",
"prompt": "Rename the skill /deploy-docs to /publish-docs. The sweep finds only slash-token and path references.",

The SKILL.md wording conflict is older than this PR. This comment is only about which token the new eval uses.

claude-lane: claude-review

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #12805, the owner-login replacement for this PR: eval 49 now renames /deploy-docs to /publish-docs, which is not on the blocklist (context/triage.md:92), so it tests only the Certain-bucket behavior. The older SKILL.md versus patterns.md Phase 0 wording conflict is left out of this change.

"expected_output": "Every match is a Form 1 slash-token or Form 3 path, so all of them land in the Certain bucket. The skill reports the bucket counts with the Certain count and one or two example matches, says the user can answer \"review Certain one by one\" to review them per match instead, and gates only the proceed/abort decision. It asks no separate \"auto-apply N matches?\" question for the Certain bucket; on proceed, Phase 5 edits every Certain match.",
"files": [],
"expectations": [
"No `AskUserQuestion` asks whether to auto-apply the Certain bucket",
"The report gives the Certain count, one or two example matches, and how to review the bucket one by one instead",
"The proceed/abort gate still runs before any edit"
]
}
]
}
Loading