pstack: strengthen blinded eval bakeoffs

[poteto]

Strengthens poteto-mode's one-shot eval bakeoff with five required checks:

• skill-on versus sanitized skill-off or prior-version ablation arms
• negative organic routing prompts with false-positive cost
• 2-3 trials per arm and prompt with pass-rate reporting
• outcome-first grading, with transcripts used only to explain behavior
• fresh workspaces and sessions with no prior or sibling artifacts

Keeps standing regression packs separate from promotion bakeoffs and bumps pstack from 0.11.3 to 0.11.4.

  

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 6 potential issues.

Autofix Details

Bugbot Autofix prepared fixes for all 6 issues found in the latest run.

• ✅ Fixed: Promotion assumes missing ablation arms
  • The playbook now defines variant and control arms for skill, structural, and prompt changes while holding every non-target input constant.
• ✅ Fixed: Transcript path blocks fresh trials
  • Each trial now records its workspace and transcript identity so synthesis reads only that exact workspace-scoped transcript path.
• ✅ Fixed: Shared fan-out breaks ablation
  • Trials now launch independently in their arm-specific workspaces without shared grounding or candidate-visible files.
• ✅ Fixed: Skill-off ignores install paths
  • Skill-absent arms now isolate or disable workspace, user, and plugin copies and preflight resolved skill visibility.
• ✅ Fixed: Arena phases clash with bakeoff
  • The bakeoff now uses its own run and judge contracts and explicitly forbids arena base selection, grafting, or output synthesis.
• ✅ Fixed: Prompt assigned before authored
  • Organic prompts are now authored before comparison arms and isolated trial workspaces are constructed.

Or push these changes by commenting:

@cursor push 01ad992a57

Preview (01ad992a57)

diff --git a/pstack/skills/poteto-mode/playbooks/eval.md b/pstack/skills/poteto-mode/playbooks/eval.md
--- a/pstack/skills/poteto-mode/playbooks/eval.md
+++ b/pstack/skills/poteto-mode/playbooks/eval.md
@@ -18,13 +18,13 @@
 **Steps:**
 
 1. **Frame.** State the variant and the promote-or-reject claim. Write a judge-only rubric with 3-6 concrete criteria. Grade task success and the intended behavioral shape. Never make a turn-1 skill load, a particular file read, a citation, or "did the skill trigger?" a pass condition.
-2. **Build the ablation arms.** When adding, rewriting, or supplying a skill, compare the variant with that skill absent. Use the prior version when absence would make the task artificial. Give both arms the same organic prompt and project skeleton. The control is just another sanitized label. Promote only if the variant-on arm beats the variant-off arm on the rubric.
-3. **Author an organic prompt set.** Include at least one task where the behavior should apply. If the variant changes a description, routing, sticky behavior, or when-to-apply rule, include at least one task where it should not engage and add false-positive cost to the rubric. Write what a user would type, with no leakage of what is measured.
-4. **Set up isolated trials.** Create a fresh per-trial workspace with only the arm's variant and context an organic task would have. Keep the project skeleton identical across arms. A cheap deterministic preflight may confirm setup facts, but it only aids synthesis. Regex or `should_trigger` checks do not judge task success or decide promotion.
-5. **Run 2-3 one-shot trials per prompt and arm** through the **arena** skill's Phase B. Match the model mix across arms and run candidates in parallel. No retries, coaching, or repair. When budget binds, prefer 2 trials across fewer models over 1 trial across more models.
-6. **Spawn one blinded judge** on a different model family per the **arena** skill's Phase C. In one pass, it scores every output by randomized sanitized label against the same rubric and marks each criterion and output pass or fail.
-7. **Inspect transcripts after scoring to explain how, not to decide pass or fail.** Read each candidate's local transcript under the active workspace's `agent-transcripts/` directory (the system prompt names this path). Do not glob across `~/.cursor/projects/*/`; that crosses workspace boundaries and reads private chats from unrelated projects. Use the transcript to verify isolation, see what the candidate read, and explain the output. It is synthesis evidence, not a pass gate.
-8. **Read every output yourself** end to end. Report pass rates by arm and prompt, then compare your read with the judge's verdict. Promote only when the variant-on arm beats the variant-off arm overall without adding false positives. Otherwise reject it. Explain disagreements as judge bias, contamination, or rubric ambiguity.
+2. **Author an organic prompt set.** Include at least one task where the behavior should apply. If the variant changes a description, routing, sticky behavior, or when-to-apply rule, include at least one task where it should not engage and add false-positive cost to the rubric. Write what a user would type, with no leakage of what is measured. If the task prompt itself is the target, write matched current and proposed versions here; otherwise every arm gets the same prompt.
+3. **Build comparison arms for every target.** The variant arm gets the proposed skill, structure, or prompt; the control gets the current version, or the skill absent only when absence is the realistic baseline. Use the prior skill when absence would make the task artificial. Hold the project skeleton, model mix, and every non-target input constant; when the task prompt is the target, its two authored versions are the sole difference. The control is just another sanitized label. Promotion always compares the variant arm with the control.
+4. **Set up isolated trials.** Create a fresh per-trial workspace containing only that arm's variant and the context an organic task would have. Keep the project skeleton identical across arms. A fresh workspace alone does not remove skills installed through workspace `.cursor/skills/`, user `~/.cursor/skills/`, or plugin `~/.cursor/plugins/` paths: for a skill-absent arm, use an isolated profile or disable every candidate-visible copy, then preflight the resolved skill sources and fail setup if the skill remains visible. Record each trial's exact workspace path and session or transcript ID as orchestrator-only metadata. A cheap deterministic preflight may confirm setup facts, but it only aids synthesis. Regex or `should_trigger` checks do not judge task success or decide promotion.
+5. **Run 2-3 one-shot trials per prompt and arm.** Launch every runner directly in its recorded workspace with that arm's isolated context. Fan out the independent launches in parallel, but never give runners shared grounding or candidate-visible files across workspaces. Match model and trial pairings across arms. Ask only for the organic task's output, not a graft-oriented rationale. Missing output is a failed trial; there are no retries, coaching, or repairs. When budget binds, prefer 2 trials across fewer models over 1 trial across more models.
+6. **Spawn one blinded judge** on a different model family after every trial finishes. In one pass, it scores every output by randomized sanitized label against the same rubric and marks each criterion and output pass or fail. Do not invoke the arena workflow, recommend a base, graft, or combine outputs; this bakeoff ends with arm-level scoring.
+7. **Inspect transcripts after scoring to explain how, not to decide pass or fail.** Read only each recorded trial transcript from its exact workspace's transcript directory, normally `~/.cursor/projects/<trial-workspace-slug>/agent-transcripts/`, using the session or transcript ID recorded during setup. Derive `<trial-workspace-slug>` from that workspace path; do not glob across `~/.cursor/projects/*/` or inspect any unregistered workspace. Use the transcript to verify isolation, see what the candidate read, and explain the output. It is synthesis evidence, not a pass gate.
+8. **Read every output yourself** end to end. Report pass rates by arm and prompt, then compare your read with the judge's verdict. Promote only when the variant arm beats the control overall without adding false positives. Otherwise reject it. Explain disagreements as judge bias, contamination, or rubric ambiguity.
 
 **Related:** Shipped skills may keep a separate standing regression pack of 5-20 cases. It is distinct from this bakeoff.

_{You can send follow-ups to the cloud agent here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Shared skill disable breaks arms
  • Required trial-local disables and prohibited mutating skill sources shared by other arms, preserving variant visibility during parallel trials.

Or push these changes by commenting:

@cursor push 53c57af5dc

Preview (53c57af5dc)

diff --git a/pstack/skills/poteto-mode/playbooks/eval.md b/pstack/skills/poteto-mode/playbooks/eval.md
--- a/pstack/skills/poteto-mode/playbooks/eval.md
+++ b/pstack/skills/poteto-mode/playbooks/eval.md
@@ -20,7 +20,7 @@
 1. **Frame.** State the variant and the promote-or-reject claim. Write a judge-only rubric with 3-6 concrete criteria. Grade task success and the intended behavioral shape. Never make a turn-1 skill load, a particular file read, a citation, or "did the skill trigger?" a pass condition.
 2. **Author an organic prompt set.** Include at least one task where the behavior should apply. If the variant changes a description, routing, sticky behavior, or when-to-apply rule, include at least one task where it should not engage and add false-positive cost to the rubric. Write what a user would type, with no leakage of what is measured. If the task prompt itself is the target, write matched current and proposed versions here; otherwise every arm gets the same prompt.
 3. **Build comparison arms.** Variant gets the proposed skill, structure, or prompt. Control gets the current version. For skill presence or content changes, the control may be skill-absent when absence is the realistic baseline, otherwise the prior skill. Hold the project skeleton, model mix, and every non-target input constant. The control is another sanitized label. Promote only when the variant beats the control on the rubric.
-4. **Set up isolated trials.** Fresh per-trial workspace with only that arm's variant and organic-task context. Identical project skeleton across arms. A fresh workspace does not clear skills from workspace `.cursor/skills/`, user `~/.cursor/skills/`, or plugin installs: for a skill-absent arm, isolate or disable every candidate-visible copy, preflight resolved sources, and fail setup if the skill remains visible. Record each trial's workspace path and transcript ID as orchestrator-only metadata. Cheap deterministic preflights aid synthesis only; they never replace the blinded rubric.
+4. **Set up isolated trials.** Fresh per-trial workspace with only that arm's variant and organic-task context. Identical project skeleton across arms. A fresh workspace does not clear skills from workspace `.cursor/skills/`, user `~/.cursor/skills/`, or plugin installs: for a skill-absent arm, isolate or disable every candidate-visible copy, preflight resolved sources, and fail setup if the skill remains visible. Scope every disable to that trial (for example, with trial-local user and plugin roots); never mutate a skill source shared by other arms. Record each trial's workspace path and transcript ID as orchestrator-only metadata. Cheap deterministic preflights aid synthesis only; they never replace the blinded rubric.
 5. **Run 2-3 one-shot trials per prompt and arm.** Launch each runner directly in its recorded workspace with that arm's isolated context. Fan out in parallel with no shared grounding and no candidate-visible files across workspaces. Match model and trial pairings across arms. Ask only for the organic task output, not a graft rationale. Missing output fails the trial. No retries, coaching, or repair. When budget binds, prefer 2 trials on fewer models over 1 on many.
 6. **Spawn one blinded judge** on a different model family after every trial finishes. In one pass, score every output by randomized sanitized label against the same rubric. Mark each criterion and output pass or fail. Do not run the arena pick/graft workflow. This bakeoff ends at arm-level scoring.
 7. **Inspect transcripts after scoring to explain how, not to decide pass or fail.** Read only the recorded transcript for each trial from that workspace's transcript directory (normally `~/.cursor/projects/<trial-workspace-slug>/agent-transcripts/`), using the session or transcript ID from setup. Derive the slug from the recorded workspace path. Do not glob across `~/.cursor/projects/*/` or open unregistered workspaces. Transcripts verify isolation and explain the output. They are not a pass gate.

_{You can send follow-ups to the cloud agent here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.

Autofix Details

Bugbot Autofix prepared fixes for all 3 issues found in the latest run.

• ✅ Fixed: Unscoped variant skill preflight
  • Scoped resolved-source preflights to skill presence and content changes so prompt-only and structure-only bakeoffs remain valid.
• ✅ Fixed: Disable never restored on failure
  • Required recorded prior state and guaranteed restoration on preflight failure, abort, completion, and before another arm runs.
• ✅ Fixed: Control prior skill unchecked
  • Added a preflight failure when a snapshotted-prior control does not resolve exactly the saved skill snapshot.

Or push these changes by commenting:

@cursor push fcd4eb2ebd

Preview (fcd4eb2ebd)

diff --git a/pstack/skills/poteto-mode/playbooks/eval.md b/pstack/skills/poteto-mode/playbooks/eval.md
--- a/pstack/skills/poteto-mode/playbooks/eval.md
+++ b/pstack/skills/poteto-mode/playbooks/eval.md
@@ -20,7 +20,7 @@
 1. **Frame.** State the variant and the promote-or-reject claim. Write a judge-only rubric with 3-6 concrete criteria. Grade task success and the intended behavioral shape. Never make a turn-1 skill load, a particular file read, a citation, or "did the skill trigger?" a pass condition.
 2. **Author an organic prompt set.** Include at least one task where the behavior should apply. If the variant changes a description, routing, sticky behavior, or when-to-apply rule, include at least one task where it should not engage and add false-positive cost to the rubric. Write what a user would type, with no leakage of what is measured. If the task prompt itself is the target, write matched current and proposed versions here; otherwise every arm gets the same prompt.
 3. **Build comparison arms.** Before editing, snapshot any prior skill contents the control will need. Variant gets the proposed skill, structure, or prompt. Control gets the current version. For skill presence or content changes, the control may be skill-absent when absence is the realistic baseline, otherwise the snapshotted prior skill. Never plant the ablation-target skill into a skill-absent control. Hold the project skeleton, model mix, and every non-target input constant. The control is another sanitized label. Promote only when the variant beats the control on the rubric.
-4. **Set up isolated trials.** Fresh per-trial workspace with only that arm's variant and organic-task context. Identical project skeleton across arms. A fresh workspace does not clear skills from workspace `.cursor/skills/`, user `~/.cursor/skills/`, or plugin installs: for a skill-absent arm, use a workspace-local isolation (or disable that is restored before any other arm runs). Never apply a shared user/plugin disable that also strips the skill from the variant arm. Preflight resolved sources and fail setup if the skill remains visible on a skill-absent arm or missing on a variant arm. Record each trial's workspace path and transcript ID as orchestrator-only metadata. Cheap deterministic preflights aid synthesis only; they never replace the blinded rubric.
+4. **Set up isolated trials.** Fresh per-trial workspace with only that arm's variant and organic-task context. Identical project skeleton across arms. A fresh workspace does not clear skills from workspace `.cursor/skills/`, user `~/.cursor/skills/`, or plugin installs: for a skill-absent arm, prefer workspace-local isolation. If a user/plugin copy must be disabled, record its prior state and restore it in a guaranteed cleanup path before any other arm runs and on preflight failure, abort, or bakeoff completion. Never apply a shared user/plugin disable that also strips the skill from the variant arm. For skill presence or content changes, preflight resolved sources and fail setup if the ablation-target skill remains visible on a skill-absent arm, the proposed skill is missing on the variant, or a snapshotted-prior control does not resolve exactly that snapshot. Record each trial's workspace path and transcript ID as orchestrator-only metadata. Cheap deterministic preflights aid synthesis only; they never replace the blinded rubric.
 5. **Run 2-3 one-shot trials per prompt and arm.** Launch each runner directly in its recorded workspace with that arm's isolated context. Fan out in parallel with no shared grounding and no candidate-visible files across workspaces. Match model and trial pairings across arms. Ask only for the organic task output, not a graft rationale. Missing output fails the trial. No retries, coaching, or repair. When budget binds, prefer 2 trials on fewer models over 1 on many.
 6. **Spawn one blinded judge** on a different model family after every trial finishes. In one pass, score every output by randomized sanitized label against the same rubric. Mark each criterion and output pass or fail. Do not run the arena pick/graft workflow. This bakeoff ends at arm-level scoring.
 7. **Inspect transcripts after scoring to explain how, not to decide pass or fail.** Read only the recorded transcript for each trial from that workspace's transcript directory (normally `~/.cursor/projects/<trial-workspace-slug>/agent-transcripts/`), using the session or transcript ID from setup. Derive the slug from the recorded workspace path. Do not glob across `~/.cursor/projects/*/` or open unregistered workspaces. Transcripts verify isolation and explain the output. They are not a pass gate.

_{You can send follow-ups to the cloud agent here.}

[cursor]

Disable never restored on failure

Medium Severity · Logic Bug

The new shared disable path only says to restore before any other arm runs. There is no restore when preflight fails, the bakeoff aborts, or the run finishes, so a user or plugin skill can stay disabled after setup exits.

 

^{Reviewed by Cursor Bugbot for commit 8d53b5c. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.

There are 4 total unresolved issues (including 1 from previous review).

Autofix Details

Bugbot Autofix prepared fixes for all 3 issues found in the latest run.

• ✅ Fixed: Budget clashes with model rule
  • The budget guidance now explicitly preserves the two-model-family floor for skills that ship across models.
• ✅ Fixed: Control skips production preflight
  • The preflight now requires production-faithful delivery for every non-absent arm, including the control.
• ✅ Fixed: FP passes escape large-N review
  • Large-N review now includes every output from should-not-engage prompts so false positives cannot hide among judge-approved passes.

Or push these changes by commenting:

@cursor push e763243188

Preview (e763243188)

diff --git a/pstack/skills/poteto-mode/playbooks/eval.md b/pstack/skills/poteto-mode/playbooks/eval.md
--- a/pstack/skills/poteto-mode/playbooks/eval.md
+++ b/pstack/skills/poteto-mode/playbooks/eval.md
@@ -20,11 +20,11 @@
 1. **Frame.** State the variant and the promote-or-reject claim. Write a judge-only rubric with 3-6 concrete criteria. Grade task success and the intended behavioral shape. Never make a turn-1 skill load, a particular file read, a citation, or "did the skill trigger?" a pass condition.
 2. **Author an organic prompt set.** Include at least one task where the behavior should apply. If the variant changes a description, routing, sticky behavior, or when-to-apply rule, include at least one task where it should not engage and add false-positive cost to the rubric. Write what a user would type. Never name the behavioral tell the rubric grades (if you measure dated headings, do not say "dated note" in the prompt). No other leakage of what is measured. If the task prompt itself is the target, write matched current and proposed versions here; otherwise every arm gets the same prompt.
 3. **Build comparison arms.** Before editing, snapshot any prior skill contents the control will need. Variant gets the proposed skill, structure, or prompt. Control gets the current version. For skill presence or content changes, the control may be skill-absent when absence is the realistic baseline, otherwise the snapshotted prior skill. Never plant the ablation-target skill into a skill-absent control. Hold the project skeleton, model mix, and every non-target input constant. The control is another sanitized label. Promote only when the variant beats the control on the rubric.
-4. **Set up isolated trials.** Fresh per-trial workspace with only that arm's variant and organic-task context. Identical project skeleton across arms. A fresh workspace does not clear skills from workspace `.cursor/skills/`, user `~/.cursor/skills/`, or plugin installs: for a skill-absent arm, use a workspace-local isolation (or disable that is restored before any other arm runs). Never apply a shared user/plugin disable that also strips the skill from the variant arm. Preflight resolved sources and fail setup if the skill remains visible on a skill-absent arm or missing on a variant arm. For sticky, mode, description, or other always-on triggers, preflight that the variant reaches the candidate the way production does (reminder in context, description always loaded, and so on). If the harness cannot inject it that way, stop: the bakeoff is invalid for that variant class. Record each trial's workspace path and transcript ID as orchestrator-only metadata. Cheap deterministic preflights aid synthesis only; they never replace the blinded rubric.
-5. **Run 2-3 one-shot trials per prompt and arm.** Launch each runner directly in its recorded workspace with that arm's isolated context. Fan out in parallel with no shared grounding and no candidate-visible files across workspaces. Match model and trial pairings across arms. If the skill ships across models, use at least two model families; matched pairings on one family are not enough. Ask only for the organic task output, not a graft rationale. Missing output fails the trial. No retries, coaching, or repair. When budget binds, prefer 2 trials on fewer models over 1 on many.
+4. **Set up isolated trials.** Fresh per-trial workspace with only that arm's variant and organic-task context. Identical project skeleton across arms. A fresh workspace does not clear skills from workspace `.cursor/skills/`, user `~/.cursor/skills/`, or plugin installs: for a skill-absent arm, use a workspace-local isolation (or disable that is restored before any other arm runs). Never apply a shared user/plugin disable that also strips the skill from the variant arm. Preflight resolved sources and fail setup if the skill remains visible on a skill-absent arm or missing on a variant arm. For sticky, mode, description, or other always-on triggers, preflight that each non-absent arm's skill reaches the candidate the way it would in production (reminder in context, description always loaded, and so on). If the harness cannot deliver every non-absent arm's skill that way, stop: the bakeoff is invalid for that variant class. Record each trial's workspace path and transcript ID as orchestrator-only metadata. Cheap deterministic preflights aid synthesis only; they never replace the blinded rubric.
+5. **Run 2-3 one-shot trials per prompt and arm.** Launch each runner directly in its recorded workspace with that arm's isolated context. Fan out in parallel with no shared grounding and no candidate-visible files across workspaces. Match model and trial pairings across arms. If the skill ships across models, use at least two model families; matched pairings on one family are not enough. Ask only for the organic task output, not a graft rationale. Missing output fails the trial. No retries, coaching, or repair. When budget binds, prefer 2 trials on fewer models over 1 on many, but never go below two model families for a skill that ships across models.
 6. **Spawn one blinded judge** on a different model family after every trial finishes. In one pass, score every output by randomized sanitized label against the same rubric. Mark each criterion and output pass or fail. Programmatic checks may filter obvious fails before the judge; they do not decide promote or reject. Do not run the arena pick/graft workflow. This bakeoff ends at arm-level scoring.
 7. **Inspect transcripts after scoring to explain how, not to decide pass or fail.** Read only the recorded transcript for each trial from that workspace's transcript directory (normally `~/.cursor/projects/<trial-workspace-slug>/agent-transcripts/`), using the session or transcript ID from setup. Derive the slug from the recorded workspace path. Do not glob across `~/.cursor/projects/*/` or open unregistered workspaces. Transcripts verify isolation and explain the output. They are not a pass gate.
-8. **Read the outputs yourself.** At small N, read every output end to end. At large N, read every fail plus a stated random sample of passes; silent skim is not enough. Report pass rates by arm and prompt, then compare with the judge. Promote only when the variant beats the control overall without adding false positives. Otherwise reject. Explain disagreements as judge bias, contamination, or rubric ambiguity.
+8. **Read the outputs yourself.** At small N, read every output end to end. At large N, read every fail and every output for prompts where the behavior should not engage, plus a stated random sample of the remaining passes; silent skim is not enough. Report pass rates by arm and prompt, then compare with the judge. Promote only when the variant beats the control overall without adding false positives. Otherwise reject. Explain disagreements as judge bias, contamination, or rubric ambiguity.
 
 **Related:** Shipped skills may keep a separate standing regression pack of 5-20 cases. It is distinct from this bakeoff.

_{You can send follow-ups to the cloud agent here.}

[cursor]

Budget clashes with model rule

Medium Severity · Logic Bug

Step 5 newly requires at least two model families when a skill ships across models, while the same step still says that under budget pressure agents should prefer two trials on fewer models. There is no tie-break, so a verbatim reader can collapse to one family with two matched trials, then promote on evidence that does not generalize.

 

^{Reviewed by Cursor Bugbot for commit 98957c3. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

There are 4 total unresolved issues (including 2 from previous reviews).

Bugbot Autofix prepared fixes for both issues found in the latest run.

• ✅ Fixed: Final promote bar contradicts ablation
  • Updated the final synthesis and reply contract to apply and report both the prior-version and skill-absent comparisons.
• ✅ Fixed: FP gate lacks negative prompts
  • Required negative organic prompts and false-positive rubric costs for every skill-presence or skill-content variant.

Or push these changes by commenting:

@cursor push d78ddff795

Preview (d78ddff795)

diff --git a/pstack/skills/poteto-mode/playbooks/eval.md b/pstack/skills/poteto-mode/playbooks/eval.md
--- a/pstack/skills/poteto-mode/playbooks/eval.md
+++ b/pstack/skills/poteto-mode/playbooks/eval.md
@@ -19,14 +19,14 @@
 **Steps:**
 
 1. **Frame.** State the variant and the promote-or-reject claim. Write a judge-only rubric with 3-6 concrete criteria. Grade task success and the intended behavioral shape. Never make a turn-1 skill load, a particular file read, a citation, or "did the skill trigger?" a pass condition.
-2. **Author an organic prompt set.** Include at least one task where the behavior should apply. If the variant changes a description, routing, sticky behavior, or when-to-apply rule, include at least one task where it should not engage and add false-positive cost to the rubric. Write what a user would type. Never name the behavioral tell the rubric grades, and never name or path a skill (if you measure dated headings, do not say "dated note" in the prompt). No other leakage of what is measured. If the task prompt itself is the target, write matched current and proposed versions here; otherwise every arm gets the same prompt.
+2. **Author an organic prompt set.** Include at least one task where the behavior should apply. For skill-presence or skill-content variants, and for any change to a description, routing, sticky behavior, or when-to-apply rule, include at least one task where it should not engage and add false-positive cost to the rubric. Write what a user would type. Never name the behavioral tell the rubric grades, and never name or path a skill (if you measure dated headings, do not say "dated note" in the prompt). No other leakage of what is measured. If the task prompt itself is the target, write matched current and proposed versions here; otherwise every arm gets the same prompt.
 3. **Build comparison arms.** Before editing, snapshot any prior skill contents the control will need. Variant gets the proposed skill, structure, or prompt. Control gets the current version. For skill presence or content changes, run both a prior-version control and a skill-absent arm unless absence is impossible. Never plant the ablation-target skill into a skill-absent control. Hold the project skeleton, model mix, and every non-target input constant. Controls are other sanitized labels. Promote only when the variant beats the prior control on the rubric without looking worse than absent on false positives.
 4. **Set up isolated trials.** Fresh per-trial workspace with only that arm's variant and organic-task context. Identical project skeleton across arms. A fresh workspace does not clear skills from workspace `.cursor/skills/`, user `~/.cursor/skills/`, or plugin installs: for a skill-absent arm, use a workspace-local isolation (or disable that is restored before any other arm runs). Never apply a shared user/plugin disable that also strips the skill from the variant arm. Preflight resolved sources and fail setup if the skill remains visible on a skill-absent arm or missing on a variant arm. For sticky, mode, description, or other always-on triggers, preflight that the variant reaches the candidate the way production does (reminder in context, description always loaded, and so on). If the harness cannot inject it that way, stop: the bakeoff is invalid for that variant class. Record each trial's workspace path and transcript ID as orchestrator-only metadata. Cheap deterministic preflights aid synthesis only; they never replace the blinded rubric.
 5. **Run 2-3 one-shot trials per prompt and arm.** Launch each runner directly in its recorded workspace with that arm's isolated context. Fan out in parallel with no shared grounding and no candidate-visible files across workspaces. Match model and trial pairings across arms. If the skill ships across models, use at least two model families; matched pairings on one family are not enough. Ask only for the organic task output, not a graft rationale. Missing output fails the trial. No retries, coaching, or repair. When budget binds, prefer 2 trials on fewer models over 1 on many.
 6. **Spawn one blinded judge** on a different model family after every trial finishes. In one pass, score every output by randomized sanitized label against the same rubric. Mark each criterion and output pass or fail. Programmatic checks may filter obvious fails before the judge; they do not decide promote or reject. Do not run the arena pick/graft workflow. This bakeoff ends at arm-level scoring.
 7. **Inspect transcripts after scoring to explain how, not to decide pass or fail.** Read only the recorded transcript for each trial from that workspace's transcript directory (normally `~/.cursor/projects/<trial-workspace-slug>/agent-transcripts/`), using the session or transcript ID from setup. Derive the slug from the recorded workspace path. Do not glob across `~/.cursor/projects/*/` or open unregistered workspaces. Transcripts verify isolation and explain the output. They are not a pass gate.
-8. **Read the outputs yourself.** At small N, read every output end to end. At large N, read every fail plus a stated random sample of passes; silent skim is not enough. Report pass rates by arm and prompt, then compare with the judge. Promote only when the variant beats the control overall without adding false positives. Otherwise reject. Explain disagreements as judge bias, contamination, or rubric ambiguity.
+8. **Read the outputs yourself.** At small N, read every output end to end. At large N, read every fail plus a stated random sample of passes; silent skim is not enough. Report pass rates by arm and prompt, then compare with the judge. Apply the Step 3 promote bar: for skill presence or content changes, the variant must beat the prior-version control overall without looking worse than the skill-absent arm on false positives; other variants must beat their control overall without adding false positives. Otherwise reject. Explain disagreements as judge bias, contamination, or rubric ambiguity.
 
 **Related:** Shipped skills may keep a separate standing regression pack of 5-20 cases. It is distinct from this bakeoff.
 
-**Reply:** variant and control, prompt set, rubric, trial pass rates, per-candidate notes, judge's verdict, your synthesis, and the promote-or-reject decision.
+**Reply:** variant and comparison arms (including the prior-version control and skill-absent arm when required), prompt set, rubric, trial pass and false-positive rates by arm and prompt, per-candidate notes, judge's verdict, your synthesis against the Step 3 promote bar, and the promote-or-reject decision.

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit e330ed5. Configure here.}

[cursor]

Final promote bar contradicts ablation

High Severity · Logic Bug

Step 3 now requires evaluating against both a prior-version control and a skill-absent arm, with promotion gated on beating the prior control without looking worse than absent on false positives. However, Step 8 and the 'Reply' section still refer to a singular 'control,' which could lead to the full evaluation criteria from Step 3 being overlooked in the final synthesis.

 

^{Reviewed by Cursor Bugbot for commit e330ed5. Configure here.}

[cursor]

FP gate lacks negative prompts

Medium Severity · Logic Bug

Step 3’s promote rule requires the variant not look worse than the skill-absent arm on false positives for skill presence or content changes, but step 2 only authors negative organic prompts and false-positive rubric cost when the variant changes description, routing, sticky, or when-to-apply behavior. Pure content edits therefore hit an FP-vs-absent gate with no prompt set or rubric criteria that measure it.

Additional Locations (1)

• pstack/skills/poteto-mode/playbooks/eval.md#L21-L22

 

^{Reviewed by Cursor Bugbot for commit e330ed5. Configure here.}