Repository navigation
Studio Code: Drop the plan guardrail that repeats the hosting-plans-helper skill - #5190
Merged
Merged
Conversation
Collaborator
📊 Performance Test ResultsComparing e24b9af vs trunk app-size
site-editor
site-startup
Results are median values from multiple test runs. Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff) |
This was referenced Oct 20, 2026
chubes4
added a commit
that referenced
this pull request
Oct 20, 2026
…5193) ## Related issues - Supports the series simplifying the Studio Code system prompt (#5189, #5190, #5192): each change needs an eval, and some changes only show up in token cost or across several messages. ## How AI was used Claude (via the Pi coding agent in a Roadie session) wrote the change, updated the README, and used it to measure #5192. I reviewed the diff and the results. ## Proposed Changes Two additions to `apps/cli/ai/eval-runner.ts`, so prompt changes can be compared on cost and across a conversation, not just on one request's outcome: - **`usage` in the result:** input, output, cache read, cache write and total tokens, summed from every assistant message. Promptfoo assertions can report it next to pass/fail. - **`followUps` var:** a list of further user messages (`{ "prompt": "…" }` objects, since promptfoo expands an array of strings into one test case per element), each sent in the same session once the previous one finishes (including any `STUDIO_EVAL_AUTO_CONTINUE` resumes). The timeout covers the whole conversation. - **The transcript, every run:** the session JSONL is always written (to `<tmpdir>/studio-eval-sessions`, or `STUDIO_EVAL_SESSION_DIR`), and the result reports its path as `sessionFile`. Before, it was kept only when that variable was set. - **`stopReason` and `errorMessage`** from the last assistant message. A provider failure ends the turn without throwing, so the result used to say `success: false` with `error: null`. These supersede #3376's diagnostics. Its transcript, final text and last-message usage are covered by the above and by `textSegments`/`timedOut`, and its "no useful output" heuristic is a judgment that belongs in each case's assertions. The README documents all of it. ## Testing Instructions 1. `cd apps/cli && npx vite build --config ./vite.config.dev.ts` 2. `node dist/cli/eval-runner.mjs "What's this site's tagline?" "" '{"vars":{"followUps":[{"prompt":"Which theme is active?"}]}}'` 3. In the result file, expect a `usage` object with non-zero counts, `stopReason: "stop"`, both answers in `textSegments`, and a `sessionFile` that exists and holds both messages. I ran it as the measurement for #5192: a seeded site and eight follow-up messages per run (results on that PR). ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?
chubes4
added a commit
that referenced
this pull request
Oct 20, 2026
…epending it to every message (#5192) ## Related issues - Follows #5170, #5189, #5190: moving Studio Code context from prose into structure, one small step at a time. - First step toward composing site facts the way the system prompt already composes tool snippets: each fact is rendered by the code that owns it, from Studio's own data. ## How AI was used Claude (via the Pi coding agent in a Roadie session) traced how the active site reaches the model, wrote the change and tests, and ran the eval below. I reviewed the diff and the eval traces. ## Proposed Changes The active site was stated by prepending a `[Active site: …]` line to every user message. So the same fact piled up in the conversation history once per turn, and it was mixed into what the user typed. This PR states it once, in an `## Active site` section of the system prompt instead: - `buildSystemPrompt` takes `activeSite` and renders the section. The text is the same as before, minus the brackets: name, path, running/stopped, and the DESIGN.md path when there is one; or ID, URL and "WordPress.com" for a remote site. - The Pi runtime already rebuilds the session, and so the system prompt, on every turn. It now passes the `activeSite` it was already receiving into `buildSystemPrompt`, so the section always reflects the current site and running state, including after compaction. - `ai` and the eval runner stop prepending the line. The user message is exactly what the user typed. - The three skills and the one prompt line that referred to "the active-site line" now say "the Active site section". Trade-off: when the active site or its running state changes, the system prompt changes, so that one turn misses the prompt cache. Before, each turn added the line to the history instead. ## Testing Instructions Unit tests: `npx vitest run apps/cli/ai apps/cli/commands/ai` (245 pass). - `system-prompt.test.ts`: the section for a local site, a WordPress.com site, and a site with a DESIGN.md. It sits above the user's global instructions and is absent without an active site. - `commands/ai/tests/index.test.ts`: the dispatched message is the typed text, and the runtime gets the site with the daemon's running state. Eval: four cases that depend on the agent knowing its active site, 2 runs each, this branch against trunk (`76f5c92bf`), using the `fast` model: | Case | What passes | Branch | Trunk | |---|---|---|---| | Tagline | Sets `blogdescription` and `blog_public` on the active site | 2/2 | 2/2 | | Plugins | Lists every installed plugin of the active site | 2/2 | 2/2 | | Admin bar (core file) | Core checksums still verify, and the change lives in `wp-content/` | 1/2 | 1/2 | | Design system | Reads the seeded DESIGN.md and the new page uses its required CTA text | 2/2 | 2/2 | Neither admin-bar failure involves the active site: - On the branch, the agent left core alone and set the height through Additional CSS (`wp_update_custom_css_post`). That's valid, but the check only looks at files. - On trunk, the model emitted a malformed tool call and the turn ended. In all 16 runs the agent worked in the correct site directory. The result is parity: the same facts, no longer repeated in the history. Multi-message eval, using #5193's token usage and `followUps`: a seeded site and eight messages (tagline, theme, pages, create a draft page, plugins, permalinks, site title), 2 runs per arm. To match the app, the trunk arm adds the site line to every message. | | Trunk (line in every message) | This PR (system-prompt section) | |---|---|---| | Final state correct (tagline, title, draft page) | 2/2 | 2/2 | | Site lines saved in the session | 8 per run | 0 | | Copies of the line re-sent across the run's requests | 73–78 | 0 (the section is sent once per request) | | Total tokens per run | 218k, 244k | 225k, 214k | The line is about 25 tokens, so this saves roughly 1.5k tokens per eight-message run, under 1% and well inside the run-to-run noise. The gain is in the transcript (it holds what the user typed, and no outdated copies of the site's state), not in the token bill. Manual: 1. `studio ai` with a site selected, then ask "what site are you working on?" The agent answers from the system prompt, and your message in the history has no prefix. 2. Switch site with `/site`, ask again: the answer reflects the new site. ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?
1 task
chubes4
added a commit
that referenced
this pull request
Oct 20, 2026
… installed skills (#5195) ## Related issues - Follows #5192, #5190 and #5189: drop instructions that repeat something the agent already gets. ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) traced how skills reach a site, found the stale list, wrote the change, and drafted this description. I reviewed the change and the reasoning. ## Proposed Changes `STUDIO.md`'s "Available Skills" section lists six skills as installed in every site. Only the skills a user picks in **Settings → Skills** are installed (`installAiInstructionsToSite` copies just `selectedSkills`), and they can be added or removed later. So the list often names skills the site doesn't have, and it's the same for every site. Agents don't need it: Claude Code finds skills in `.claude/skills/` and Codex in `.agents/skills/`, each from the skill's own name and description, and they see exactly what's installed. This removes the section: - **The list** goes. Agents discover the installed skills themselves. - **The `studio wp` prefix note** goes. It repeats the file's first section, "WP-CLI in WordPress Studio". - **The `wp-project-triage` note** stays, moved under "Studio-Specific Notes", since nothing else says it. Net: 16 lines removed, 1 added. `STUDIO.md` is a managed file, so existing sites get the change the next time they start (`updateManagedInstructionFiles`). ## Testing Instructions 1. In Settings → Skills, install only **Studio CLI**, then create a site. 2. Open the site's `STUDIO.md`: no "Available Skills" section, and the triage note sits under "Studio-Specific Notes". 3. `ls <site>/.claude/skills <site>/.agents/skills` shows only `studio-cli`, which is what agents discover. On trunk, `STUDIO.md` also names the five WordPress skills that aren't there. `npx vitest run packages/common/lib/tests/agent-skills.test.ts` passes (17). No eval: this file isn't loaded by Studio Code. It only reaches external agents opened in a site folder. ## Pre-merge Checklist - [ ] Have you checked for TypeScript, React or other console errors?
chubes4
added a commit
that referenced
this pull request
Oct 21, 2026
….com site (#5230) ## Related issues - Supports the series simplifying the Studio Code system prompt (#5189, #5190, #5192): each change gets an eval, and changes that move a rule from prose into a tool often move it into a question to the user. ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) found the gap while evaluating a change to `site_push`, wrote both commits and the README text, and ran the checks below. I reviewed the diff and the results. ## Proposed Changes **The eval runner gave the agent no way to ask the user anything.** The app and `studio ai` always pass `onAskUser` to `runStudioAgentTurn`, so the agent has `AskUserQuestion` and tools can confirm with the user. The runner passed nothing, so every eval so far measured an agent that cannot ask. Anything that depends on a question (confirming a target, picking a design, clarifying a request) was evaluated on a different agent from the one users get. The README already listed a `questions` result field that the runner never returned. - **An `answers` var scripts the user's replies.** It is a list of `{ "label": "…" }` objects, not strings, because promptfoo turns a list of strings into one test per element. With it, the turn runs with an ask-the-user handler: each question takes the next label in order, and once the labels run out the user cancels. Without it, behaviour is unchanged. - **The result reports `questions`.** Each entry holds the question the agent asked, its option labels and the scripted answer. The README line describing it is now true. - **The seeded active site can have connected WordPress.com sites.** `seed.activeSite.connectedWpcomSites` takes the same fields as `seed.connectedWpcomSites` without `localSiteId`, which is the new site's id and only known once the site exists. This lets push and pull flows be evaluated on the active site. - **One of the user's WordPress.com sites can be the active site.** `seed.remoteActiveSite` (`name`, `url`, `wpcomSiteId`) makes an existing site active for the turn, as picking it in the app or CLI does, so the Live chat path can be evaluated. Nothing is created or deleted, so the case owns cleaning up what the agent changes on that site. ## Testing Instructions - `npx vitest run apps/cli/ai apps/cli/commands/ai`: 255 pass. - Build the runner (`cd apps/cli && npx vite build --config ./vite.config.dev.ts`), then run the same prompt with and without `answers`: ```sh node apps/cli/dist/cli/eval-runner.mjs x "" '{"vars":{"prompt":"Use AskUserQuestion to ask me which color I prefer, with options Red and Blue. Then tell me my answer in one sentence.","answers":[{"label":"Blue"}]}}' ``` - With `answers`: the agent calls `AskUserQuestion`. `questions` is `[{ "question": "Which color do you prefer?", "options": ["Red", "Blue"], "answer": "Blue" }]`, and the reply is "You prefer the color blue." - Without `answers` (same build): no tool call, and the reply starts "I don't have an `AskUserQuestion` tool available in this session". - Used for a 22-run push eval: one connected WordPress.com site, seeded with `activeSite.connectedWpcomSites`, and "Push my changes to WordPress.com.". On trunk the agent asked before pushing in 4 of 4 runs, and each question and answer appears in `questions`. ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?
chubes4
added a commit
that referenced
this pull request
Oct 21, 2026
…e site instead of a rule to load it (#5231) ## Related issues - Continues the series simplifying the Studio Code system prompt (#5189, #5190, #5192): moves a rule from prose into structure. - The eval uses `seed.remoteActiveSite` from #5230. ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) wrote the change and the tests, ran the eval below against a WordPress.com site I own, and cleaned up the posts it created. I reviewed the diff and the results. ## Proposed Changes The system prompt carried a general rule telling the agent to load the `edit-manage-remote-site` skill first for any WordPress.com site. That rule stood in for a fact Studio already knows: since #5192, the active site is stated in the prompt, including whether it is a WordPress.com site. - **When a WordPress.com site is active, its runbook comes with it.** The Active site section now includes the `edit-manage-remote-site` runbook. The agent starts out knowing that the site changes only through `wpcom_request` and how to use that tool, so it no longer spends a step loading the skill first. - **The rule is removed from the general rules.** For a WordPress.com site the user names while a local site is active, the skill list already covers it: the `Skill` tool and `studio_instructions` list `edit-manage-remote-site` with its description, "Edit and manage one of the user's live remote sites…". - Prompts for local sites and for no active site are unchanged, apart from one rule fewer. ## Testing Instructions - `npx vitest run apps/cli/ai apps/cli/commands/ai`: 256 pass. The new test checks that the runbook is in the prompt only when a WordPress.com site is active. - **Eval:** 24 runs, 12 per arm, on Franklin Works, a Business-plan WordPress.com site. Trunk and this branch both include #5230. - **read:** WordPress.com site active, "Which theme is active on this site, and does its plan let me install plugins?" - **write:** WordPress.com site active, "Add a draft post titled … with one short paragraph about field notes." - **named:** a local site active, the same draft request naming "my WordPress.com site Franklin Works (chubesmattic.wpcomstaging.com)". | Case | Arm | Correct outcome | Loaded the skill | Median tokens | |---|---|---|---|---| | read | trunk | 4/4 | 4/4 | 74k | | read | branch | 4/4 | 0/4 | 66k | | write | trunk | 4/4 | 4/4 | 49k | | write | branch | 4/4 | 1/4 | 38k | | named | trunk | 4/4 | 4/4 | 83k | | named | branch | 4/4 | 4/4 | 102k | - **Correct outcome:** - read: the answer names Assembler and says the Business plan allows plugins. - write and named: exactly one draft with that title on Franklin Works. - In every run, the agent changed the site only through `wpcom_request`; no local tool was called. - **named, branch:** the agent loaded the skill in 4 of 4 runs without the rule. One run took 752k tokens, because `GET !/me/sites` with `plan` in `fields` returns 2.3 MB, and `wpcom_request` passes it through whole. Trunk has the same path. Without that run, the branch's median is 84k. - All 16 drafts were trashed afterwards. ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?
This was referenced Oct 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issues
How AI was used in this PR
Claude (via the Pi coding agent in a Roadie session) found the duplicated rule, wrote the change and the test update, and ran the checks and the eval below.
Proposed Changes
hosting-plans-helperfor any plan, pricing or feature-tier question and never to answer from memory.description, andhosting-plans-helper's reads: "Load before answering ANY plan, pricing, or feature-gating question; never answer these from memory." The skill body repeats it as its first rule.wpcom/v2/plans/featuresdid not exist yet, so the paragraph was the main safeguard. The endpoint is live now (public, HTTP 200, all five plans with their features), and the skill fetches from it.Testing Instructions
npx vitest run apps/cli/ai/tests/system-prompt.test.ts: the test now checks that the skill's description carries the instruction and the prompt no longer repeats it.hosting-plans-helperand fetches the plan data before answering; the answers were then checked against the live endpoints.hosting-plans-helper, fetchedwpcom/v2/plans/features(andproductsfor prices), and answered from the fetched data. Both arms correctly said plugins need Personal or above, the case the guardrail called out. The answers match the live data: plugins and custom CSS from Personal, SSH from Business, Business at $300 a year.hosting-plans-helperalongside the task's own skill (plugin-recommendationsoredit-manage-site) and fetched the live plan data.Pre-merge Checklist