Repository navigation
Guard undetected rule slugs; cover cloud subagent profile gaps - #4
Merged
Merged
Conversation
- rule values were mapped to pstack-M / pstack-M-ro profiles on the assumption they are real slugs; a hand-edited or stale rule entry (e.g. a dotted grok-4.7-... or a suffix the family lacks) produced a junk profile plus a failed spawn before the rejected-slug fallback rescued it. treat undetected entries as rejected slugs up front and never provision a profile for them (arena x2, swarm, interrogate, how, why, reflect, poteto-mode). - poteto-mode, no-comments: plugin subagent profiles don't load in cloud Devin sessions, so pstack:poteto-agent and pstack:comment-sicko don't resolve there. name the subagent_general substitute instead of failing on a missing profile. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
caelaxie
commented
Oct 18, 2026
caelaxie
left a comment
caelaxie
left a comment
Contributor
Author
There was a problem hiding this comment.
verdict: changes requested (posted as comment, same-account pr).
blocker: no devin harness proof. all three test-plan boxes are unchecked and there's no output in the pr. need, on 5551cf2: devin plugins install + devin plugins list showing pstack loaded, plus the changed skills exercised: a bogus rule value (e.g. grok-4.7-high-fast) through /pstack:interrogate or /pstack:swarm showing the announced fallback and no ~/.config/devin/agents/pstack-grok-4.7-high-fast*.md written, and /pstack:no-comments taking the subagent_general + agents/comment-sicko.md path. paste it in the body or a comment.
minimum changes: this is port-only text, which is ok since the profile mapping only exists in the devin port. keep it to one identical sentence everywhere, see line comments.
parity/versions: no upstream sync here, plugin.json untouched, fine.
linkage: no issue to close, fine.
| | Reviewer B | `swe-2-high` | | ||
|
|
||
| Each reviewer is read-only and gets `profile` from its configured entry, or its table default when there is no configured line. A real slug `M` becomes `pstack-M-ro` (create it per `/pstack:setup-pstack` if missing). An `auto` or `inherit-parent` entry uses `subagent_explore`, so that reviewer runs on the parent-assigned default; use `subagent_general` with a read-only brief when the review needs `exec` or MCP. | ||
| Each reviewer is read-only and gets `profile` from its configured entry, or its table default when there is no configured line. A real slug `M` becomes `pstack-M-ro` (create it per `/pstack:setup-pstack` if missing). An entry `devin models list` doesn't report counts as a rejected slug; never create a profile for it. An `auto` or `inherit-parent` entry uses `subagent_explore`, so that reviewer runs on the parent-assigned default; use `subagent_general` with a read-only brief when the review needs `exec` or MCP. |
Contributor
Author
There was a problem hiding this comment.
conflicts with line 36: "never treat an alias entry as a rejected slug". devin models list won't report opus/sonnet/swe/codex/gpt/gemini, so read literally this sends every alias entry to the fallback. scope it to full slugs, e.g. "a full slug entry devin models list doesn't report counts as a rejected slug".
| 1. State the artifact each candidate is producing. | ||
| 2. Derive the rubric, 3-6 concrete gradeable criteria for what success looks like on *this* task. It is the picker's tool in Phase D. Candidates only see the task. | ||
| 3. Pick the runners. Use the `arena runners` line in `~/.devin/rules/pstack-models.md`. If the rule or that line is missing, default to one each on `claude-opus-5-5-xhigh` and `swe-2-high`. Map each entry to the spawn's `profile`: a real slug `M` becomes `pstack-M`, and `auto` or `inherit-parent` means the parent model, so use `subagent_general` for it. Create a missing `pstack-M` profile per `/pstack:setup-pstack`. If a pinned slug is rejected at spawn, run that seat on its family's default and say so. Families go by prefix: `claude-*`, `gpt-*`, `swe-*`, `gemini-*`, plus the short aliases (`opus`, `sonnet`, `swe`, `codex`, `gpt`, `gemini`). With no family match, use `claude-opus-5-5-xhigh`. If it rejects a default, use the closest valid slug of the same family `devin models list` reports. Spawn more runners when the arena covers multiple design directions. Use the same model N times when the work is generation-bound rather than judgment-sensitive. Pick each subagent's reasoning level and fast or standard speed for the job, per the Subagents section of the **poteto-mode** skill. | ||
| 3. Pick the runners. Use the `arena runners` line in `~/.devin/rules/pstack-models.md`. If the rule or that line is missing, default to one each on `claude-opus-5-5-xhigh` and `swe-2-high`. Map each entry to the spawn's `profile`: a real slug `M` becomes `pstack-M`, and `auto` or `inherit-parent` means the parent model, so use `subagent_general` for it. Create a missing `pstack-M` profile per `/pstack:setup-pstack`. An entry `devin models list` doesn't report counts as a rejected slug; never create a profile for it. If a pinned slug is rejected at spawn, run that seat on its family's default and say so. Families go by prefix: `claude-*`, `gpt-*`, `swe-*`, `gemini-*`, plus the short aliases (`opus`, `sonnet`, `swe`, `codex`, `gpt`, `gemini`). With no family match, use `claude-opus-5-5-xhigh`. If it rejects a default, use the closest valid slug of the same family `devin models list` reports. Spawn more runners when the arena covers multiple design directions. Use the same model N times when the work is generation-bound rather than judgment-sensitive. Pick each subagent's reasoning level and fast or standard speed for the job, per the Subagents section of the **poteto-mode** skill. |
Contributor
Author
There was a problem hiding this comment.
same alias problem as interrogate: this line lists the short aliases as valid families, and the new sentence would reject them. scope to full slugs.
|
|
||
| **Use `profile: "pstack:poteto-agent"` for any subagent you spawn inside a playbook step** (code-writing delegates, ad-hoc helpers). The chat that receives `/pstack:poteto-mode` runs the playbook. Never spawn a `poteto-agent` to run the playbook itself. Routed workflow skills (`how`, `why`, `interrogate`, `reflect`, `swarm`) set their own `profile` for diverse-model review. Respect what the skill prescribes, don't override to `poteto-agent`. | ||
| **Use `profile: "pstack:poteto-agent"` for any subagent you spawn inside a playbook step** (code-writing delegates, ad-hoc helpers). The chat that receives `/pstack:poteto-mode` runs the playbook. Never spawn a `poteto-agent` to run the playbook itself. Plugin subagent profiles don't load in cloud Devin sessions, so there spawn delegates on `subagent_general` and open the brief with the same read-the-`poteto-mode`-SKILL.md-first instruction the profile carries. Routed workflow skills (`how`, `why`, `interrogate`, `reflect`, `swarm`) set their own `profile` for diverse-model review. Respect what the skill prescribes, don't override to `poteto-agent`. | ||
|
|
Contributor
Author
There was a problem hiding this comment.
trim to the same sentence the other six skills use. the em dash list and the grok-4.7-… example are extra divergence and slop; the rule is "an entry devin models list doesn't report counts as a rejected slug: use the role's default, say so, never write a profile for it". same alias carve-out applies.
resolve skills/swarm/SKILL.md: upstream dropped the set-N step; keep their numbering and re-apply the undetected-slug guard to the worker-model step. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This was referenced Oct 19, 2026
caelaxie
added a commit
that referenced
this pull request
Oct 20, 2026
## Summary pstack 0.15.28 never mentions Fusion. On a Fusion chat the lead gets a `sidekick` tool, and Devin tells it "Do not use subagents other than `sidekick` unless the user explicitly asks you to." (string from the devin 3000.11.3 binary). Without guidance, a lead running `/pstack:poteto-mode` could split one unit of work between the sidekick and pstack subagents, run it twice, or skip pstack's panels. - **Single hands-on delegates go to the sidekick on Fusion.** That covers the code-writing delegate a playbook names, fix rounds, hillclimb attempts, a single investigation and a verification run. It runs on the sidekick's model, not the step's role line (`hardest tasks` included). The lead takes a change back when the sidekick can't land it. Each brief points the sidekick at `poteto-mode/SKILL.md`, the same way the `poteto-agent` profile does. Each brief also restates every directive still in force, because the sidekick keeps its context between handoffs instead of starting fresh. - **Panels, fan-out and second opinions stay on `run_subagent`** with their role profiles: arena runners and judges, swarm workers, reviewers, synthesizers. One sidekick can't run several agents in parallel or give an opinion from a different model. Invoking a pstack skill counts as the user's explicit ask for these subagents. The same unit of work never goes to both. - **`auto` / `inherit-parent` on Fusion.** A seat on `subagent_general` runs on the lead model at the lead's price, with no sidekick. The guide, `poteto-help` and `setup-pstack` now say so in their cost advice. - **Model-family rules on Fusion** (arena cross-judge, show-me-your-work reviewer) count both the lead's and the sidekick's families as taken. They prefer a family outside both, else outside the lead's. - **`fusion-*` slugs are no longer valid role values.** Each one names a lead and sidekick pair for the main chat, not one subagent model. `devin models list` prints 575 of them, so `setup-pstack` drops them from the detected set: its effort rule would otherwise rewrite the sidekick's effort. `poteto-mode` and every routed skill (`arena`, `how`, `interrogate`, `reflect`, `swarm`, `why`) treat one as a rejected slug, the same way #4 handles undetected slugs. **Chats without Fusion are unchanged.** Every behavior change only applies when the session has a `sidekick` tool. The `poteto-mode` paragraphs say so up front and end with "Sessions without a `sidekick` tool skip these two paragraphs". The only change that applies everywhere is rejecting `fusion-*` values, which a non-Fusion rule has no reason to contain. **How the `subagent_general` claim was checked.** I ran it on devin 3000.11.3 under `fusion-claude-opus-5-5-max-sidekick-swe-2-high`, spawning one `subagent_general` that was told to make no tool calls. Its stored thinking signature has the same format as the Claude lead's (`CAQS…`), not SWE-2's (`sealed.v…`). Its system prompt has no sidekick tool and no Fusion guidance. Not touched: the `plugin.json` version (a release decision, as in #4). #### Test plan - [x] CI's "Validate plugin" script, run locally with bun: `Validated 56 skills, 2 agents, and 23 playbooks.` - [x] Every new behavior sentence is gated on Fusion or the `sidekick` tool; checked against the word diff - [ ] Fusion chat, `/pstack:poteto-mode` on a small Feature task: the implementation goes to the sidekick, and `/pstack:arena` or `/pstack:interrogate` still spawns its panel with `run_subagent` - [ ] Non-Fusion chat, same task: the delegate still runs on the `pstack-<slug>` profile from `feature, refactoring` - [ ] `/pstack:setup-pstack` on a Fusion chat never offers a `fusion-*` slug and explains the sidekick split Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to the Devin-compat audit. Two remaining edges in the rule-value →
run_subagentprofile pipeline:pstack-models.mdrule valueMtopstack-M/pstack-M-roon the assumption it's a real slug, and says "create a missing profile per setup-pstack." A hand-edited or stale rule carrying a valuedevin models listdoesn't report (dotted family form likegrok-4.7-…, a-fast/-prioritysuffix the family lacks, a renamed model) would produce a profile pinning an invalidmodel:, then a failed spawn, before the rejected-slug fallback rescued it. Now an undetected entry counts as a rejected slug up front — family/default fallback, announced — and never gets a profile written. Applied uniformly inpoteto-mode(master mapping paragraph),arena(runners + cross-judge),swarm,interrogate,how,why,reflect.architectinherits via its deferral to arena's Phase A.pstack:poteto-agentandpstack:comment-sickodon't resolve in cloud Devin sessions (swarm/orchestrate already document this at routing level).poteto-modenow names thesubagent_generalsubstitute with the profile's read-the-SKILL.md-first brief;no-commentsfalls back tosubagent_generalfed theagents/comment-sicko.mdbody, or runs the pass inline.Deliberately not touched: stale-rule detection across other tools' rules dirs (dropped in afe207a) and the
plugin.jsonversion (release decision).Test plan
grok-4.7-high-fast):/pstack:interrogateor/pstack:swarmfalls back to a detected slug with a notice instead of writing~/.config/devin/agents/pstack-grok-4.7-high-fast*.mddevin plugins updatepicks up the new skill text;/pstack:poteto-helpstill routes correctlysubagent_general/subagent_exploreper skill text, not a missing-profile errorGenerated with Devin