Studio Code: Drop the prompt rules that repeat the file tools' own guidance by chubes4 · Pull Request #5189 · Automattic/studio

chubes4 · 2026-10-19T18:40:49Z

Related issues

Related to Spike: Studio Code reads the WordPress source instead of being taught WordPress #5146: one of a series of small changes that simplify the system prompt

How AI was used in this PR

Claude (via the Pi coding agent in a Roadie session) found the duplication, made the change, and ran the eval below.

Proposed Changes

The prompt's "Working cadence" section had two lines about the file tools:

"Read files with Read, not cat or sed. Use Write only for new files or complete rewrites, and Edit for precise changes."
"Put every change you have ready for a file into one Edit call … instead of one call per anchor."

Studio Code's Read, Write and Edit are Pi's own tools, renamed. Pi ships this same guidance with them (promptGuidelines on read, write and edit). Pi drops that guidance when a custom system prompt is supplied, as Studio does, so the prompt carried a hand copy. The behaviour it asks for is still in what the model sees for each tool:

Edit's description: "Edit a single file using exact text replacement… If two changes affect the same block or nearby lines, merge them into one edit".
Edit's edits parameter is an array ("One or more targeted replacements"), so one call holds several changes.
Write's description: "Creates the file if it doesn't exist, overwrites if it does."

This change deletes both lines (about 330 characters). Nothing else changes.

Testing Instructions

Eval: two cases, each run twice on trunk and on this branch (8 runs). A seeded plugin file of about 170 lines:

multi-edit: "bump its version to 1.1.0, change the greeting it returns to 'Welcome', and raise the notes limit to 10". The check reads the file afterwards: all three changes made, and the other 30 functions intact.
read: "What does the plugin do with notes beyond its limit?" The check looks for the answer (archived).

	Passed	Read calls	Edit calls	Write calls	`cat`/`sed` reads of the file
trunk	4/4	1 per run	1 per edit run	0	0
this branch	4/4	1 per run	1 per edit run	0	0

Both arms made all three changes in a single Edit call, read the file with Read, and never rewrote it with Write. The remaining shell calls were ls, find, grep and php -l, in both arms.

Unit tests: npx vitest run apps/cli/ai/tests/system-prompt.test.ts

Pre-merge Checklist

Have you checked for TypeScript, React or other console errors?

wpmobilebot · 2026-10-19T19:00:55Z

📊 Performance Test Results

Comparing 23e0843 vs trunk

app-size

Metric	trunk	`23e0843`	Diff	Change
App Size (Mac)	927.54 MB	927.54 MB	+0.00 MB	⚪ 0.0%

site-editor

Metric	trunk	`23e0843`	Diff	Change
load	1168 ms	1165 ms	3 ms	⚪ 0.0%

site-startup

Metric	trunk	`23e0843`	Diff	Change
siteCreation	8584 ms	8570 ms	14 ms	⚪ 0.0%
siteStartup	5896 ms	5895 ms	1 ms	⚪ 0.0%

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

…5193) ## Related issues - Supports the series simplifying the Studio Code system prompt (#5189, #5190, #5192): each change needs an eval, and some changes only show up in token cost or across several messages. ## How AI was used Claude (via the Pi coding agent in a Roadie session) wrote the change, updated the README, and used it to measure #5192. I reviewed the diff and the results. ## Proposed Changes Two additions to `apps/cli/ai/eval-runner.ts`, so prompt changes can be compared on cost and across a conversation, not just on one request's outcome: - **`usage` in the result:** input, output, cache read, cache write and total tokens, summed from every assistant message. Promptfoo assertions can report it next to pass/fail. - **`followUps` var:** a list of further user messages (`{ "prompt": "…" }` objects, since promptfoo expands an array of strings into one test case per element), each sent in the same session once the previous one finishes (including any `STUDIO_EVAL_AUTO_CONTINUE` resumes). The timeout covers the whole conversation. - **The transcript, every run:** the session JSONL is always written (to `<tmpdir>/studio-eval-sessions`, or `STUDIO_EVAL_SESSION_DIR`), and the result reports its path as `sessionFile`. Before, it was kept only when that variable was set. - **`stopReason` and `errorMessage`** from the last assistant message. A provider failure ends the turn without throwing, so the result used to say `success: false` with `error: null`. These supersede #3376's diagnostics. Its transcript, final text and last-message usage are covered by the above and by `textSegments`/`timedOut`, and its "no useful output" heuristic is a judgment that belongs in each case's assertions. The README documents all of it. ## Testing Instructions 1. `cd apps/cli && npx vite build --config ./vite.config.dev.ts` 2. `node dist/cli/eval-runner.mjs "What's this site's tagline?" "" '{"vars":{"followUps":[{"prompt":"Which theme is active?"}]}}'` 3. In the result file, expect a `usage` object with non-zero counts, `stopReason: "stop"`, both answers in `textSegments`, and a `sessionFile` that exists and holds both messages. I ran it as the measurement for #5192: a seeded site and eight follow-up messages per run (results on that PR). ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?

…epending it to every message (#5192) ## Related issues - Follows #5170, #5189, #5190: moving Studio Code context from prose into structure, one small step at a time. - First step toward composing site facts the way the system prompt already composes tool snippets: each fact is rendered by the code that owns it, from Studio's own data. ## How AI was used Claude (via the Pi coding agent in a Roadie session) traced how the active site reaches the model, wrote the change and tests, and ran the eval below. I reviewed the diff and the eval traces. ## Proposed Changes The active site was stated by prepending a `[Active site: …]` line to every user message. So the same fact piled up in the conversation history once per turn, and it was mixed into what the user typed. This PR states it once, in an `## Active site` section of the system prompt instead: - `buildSystemPrompt` takes `activeSite` and renders the section. The text is the same as before, minus the brackets: name, path, running/stopped, and the DESIGN.md path when there is one; or ID, URL and "WordPress.com" for a remote site. - The Pi runtime already rebuilds the session, and so the system prompt, on every turn. It now passes the `activeSite` it was already receiving into `buildSystemPrompt`, so the section always reflects the current site and running state, including after compaction. - `ai` and the eval runner stop prepending the line. The user message is exactly what the user typed. - The three skills and the one prompt line that referred to "the active-site line" now say "the Active site section". Trade-off: when the active site or its running state changes, the system prompt changes, so that one turn misses the prompt cache. Before, each turn added the line to the history instead. ## Testing Instructions Unit tests: `npx vitest run apps/cli/ai apps/cli/commands/ai` (245 pass). - `system-prompt.test.ts`: the section for a local site, a WordPress.com site, and a site with a DESIGN.md. It sits above the user's global instructions and is absent without an active site. - `commands/ai/tests/index.test.ts`: the dispatched message is the typed text, and the runtime gets the site with the daemon's running state. Eval: four cases that depend on the agent knowing its active site, 2 runs each, this branch against trunk (`76f5c92bf`), using the `fast` model: | Case | What passes | Branch | Trunk | |---|---|---|---| | Tagline | Sets `blogdescription` and `blog_public` on the active site | 2/2 | 2/2 | | Plugins | Lists every installed plugin of the active site | 2/2 | 2/2 | | Admin bar (core file) | Core checksums still verify, and the change lives in `wp-content/` | 1/2 | 1/2 | | Design system | Reads the seeded DESIGN.md and the new page uses its required CTA text | 2/2 | 2/2 | Neither admin-bar failure involves the active site: - On the branch, the agent left core alone and set the height through Additional CSS (`wp_update_custom_css_post`). That's valid, but the check only looks at files. - On trunk, the model emitted a malformed tool call and the turn ended. In all 16 runs the agent worked in the correct site directory. The result is parity: the same facts, no longer repeated in the history. Multi-message eval, using #5193's token usage and `followUps`: a seeded site and eight messages (tagline, theme, pages, create a draft page, plugins, permalinks, site title), 2 runs per arm. To match the app, the trunk arm adds the site line to every message. | | Trunk (line in every message) | This PR (system-prompt section) | |---|---|---| | Final state correct (tagline, title, draft page) | 2/2 | 2/2 | | Site lines saved in the session | 8 per run | 0 | | Copies of the line re-sent across the run's requests | 73–78 | 0 (the section is sent once per request) | | Total tokens per run | 218k, 244k | 225k, 214k | The line is about 25 tokens, so this saves roughly 1.5k tokens per eight-message run, under 1% and well inside the run-to-run noise. The gain is in the transcript (it holds what the user typed, and no outdated copies of the site's state), not in the token bill. Manual: 1. `studio ai` with a site selected, then ask "what site are you working on?" The agent answers from the system prompt, and your message in the history has no prefix. 2. Switch site with `/site`, ask again: the answer reflects the new site. ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?

… installed skills (#5195) ## Related issues - Follows #5192, #5190 and #5189: drop instructions that repeat something the agent already gets. ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) traced how skills reach a site, found the stale list, wrote the change, and drafted this description. I reviewed the change and the reasoning. ## Proposed Changes `STUDIO.md`'s "Available Skills" section lists six skills as installed in every site. Only the skills a user picks in **Settings → Skills** are installed (`installAiInstructionsToSite` copies just `selectedSkills`), and they can be added or removed later. So the list often names skills the site doesn't have, and it's the same for every site. Agents don't need it: Claude Code finds skills in `.claude/skills/` and Codex in `.agents/skills/`, each from the skill's own name and description, and they see exactly what's installed. This removes the section: - **The list** goes. Agents discover the installed skills themselves. - **The `studio wp` prefix note** goes. It repeats the file's first section, "WP-CLI in WordPress Studio". - **The `wp-project-triage` note** stays, moved under "Studio-Specific Notes", since nothing else says it. Net: 16 lines removed, 1 added. `STUDIO.md` is a managed file, so existing sites get the change the next time they start (`updateManagedInstructionFiles`). ## Testing Instructions 1. In Settings → Skills, install only **Studio CLI**, then create a site. 2. Open the site's `STUDIO.md`: no "Available Skills" section, and the triage note sits under "Studio-Specific Notes". 3. `ls <site>/.claude/skills <site>/.agents/skills` shows only `studio-cli`, which is what agents discover. On trunk, `STUDIO.md` also names the five WordPress skills that aren't there. `npx vitest run packages/common/lib/tests/agent-skills.test.ts` passes (17). No eval: this file isn't loaded by Studio Code. It only reaches external agents opened in a site folder. ## Pre-merge Checklist - [ ] Have you checked for TypeScript, React or other console errors?

….com site (#5230) ## Related issues - Supports the series simplifying the Studio Code system prompt (#5189, #5190, #5192): each change gets an eval, and changes that move a rule from prose into a tool often move it into a question to the user. ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) found the gap while evaluating a change to `site_push`, wrote both commits and the README text, and ran the checks below. I reviewed the diff and the results. ## Proposed Changes **The eval runner gave the agent no way to ask the user anything.** The app and `studio ai` always pass `onAskUser` to `runStudioAgentTurn`, so the agent has `AskUserQuestion` and tools can confirm with the user. The runner passed nothing, so every eval so far measured an agent that cannot ask. Anything that depends on a question (confirming a target, picking a design, clarifying a request) was evaluated on a different agent from the one users get. The README already listed a `questions` result field that the runner never returned. - **An `answers` var scripts the user's replies.** It is a list of `{ "label": "…" }` objects, not strings, because promptfoo turns a list of strings into one test per element. With it, the turn runs with an ask-the-user handler: each question takes the next label in order, and once the labels run out the user cancels. Without it, behaviour is unchanged. - **The result reports `questions`.** Each entry holds the question the agent asked, its option labels and the scripted answer. The README line describing it is now true. - **The seeded active site can have connected WordPress.com sites.** `seed.activeSite.connectedWpcomSites` takes the same fields as `seed.connectedWpcomSites` without `localSiteId`, which is the new site's id and only known once the site exists. This lets push and pull flows be evaluated on the active site. - **One of the user's WordPress.com sites can be the active site.** `seed.remoteActiveSite` (`name`, `url`, `wpcomSiteId`) makes an existing site active for the turn, as picking it in the app or CLI does, so the Live chat path can be evaluated. Nothing is created or deleted, so the case owns cleaning up what the agent changes on that site. ## Testing Instructions - `npx vitest run apps/cli/ai apps/cli/commands/ai`: 255 pass. - Build the runner (`cd apps/cli && npx vite build --config ./vite.config.dev.ts`), then run the same prompt with and without `answers`: ```sh node apps/cli/dist/cli/eval-runner.mjs x "" '{"vars":{"prompt":"Use AskUserQuestion to ask me which color I prefer, with options Red and Blue. Then tell me my answer in one sentence.","answers":[{"label":"Blue"}]}}' ``` - With `answers`: the agent calls `AskUserQuestion`. `questions` is `[{ "question": "Which color do you prefer?", "options": ["Red", "Blue"], "answer": "Blue" }]`, and the reply is "You prefer the color blue." - Without `answers` (same build): no tool call, and the reply starts "I don't have an `AskUserQuestion` tool available in this session". - Used for a 22-run push eval: one connected WordPress.com site, seeded with `activeSite.connectedWpcomSites`, and "Push my changes to WordPress.com.". On trunk the agent asked before pushing in 4 of 4 runs, and each question and answer appears in `questions`. ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?

…e site instead of a rule to load it (#5231) ## Related issues - Continues the series simplifying the Studio Code system prompt (#5189, #5190, #5192): moves a rule from prose into structure. - The eval uses `seed.remoteActiveSite` from #5230. ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) wrote the change and the tests, ran the eval below against a WordPress.com site I own, and cleaned up the posts it created. I reviewed the diff and the results. ## Proposed Changes The system prompt carried a general rule telling the agent to load the `edit-manage-remote-site` skill first for any WordPress.com site. That rule stood in for a fact Studio already knows: since #5192, the active site is stated in the prompt, including whether it is a WordPress.com site. - **When a WordPress.com site is active, its runbook comes with it.** The Active site section now includes the `edit-manage-remote-site` runbook. The agent starts out knowing that the site changes only through `wpcom_request` and how to use that tool, so it no longer spends a step loading the skill first. - **The rule is removed from the general rules.** For a WordPress.com site the user names while a local site is active, the skill list already covers it: the `Skill` tool and `studio_instructions` list `edit-manage-remote-site` with its description, "Edit and manage one of the user's live remote sites…". - Prompts for local sites and for no active site are unchanged, apart from one rule fewer. ## Testing Instructions - `npx vitest run apps/cli/ai apps/cli/commands/ai`: 256 pass. The new test checks that the runbook is in the prompt only when a WordPress.com site is active. - **Eval:** 24 runs, 12 per arm, on Franklin Works, a Business-plan WordPress.com site. Trunk and this branch both include #5230. - **read:** WordPress.com site active, "Which theme is active on this site, and does its plan let me install plugins?" - **write:** WordPress.com site active, "Add a draft post titled … with one short paragraph about field notes." - **named:** a local site active, the same draft request naming "my WordPress.com site Franklin Works (chubesmattic.wpcomstaging.com)". | Case | Arm | Correct outcome | Loaded the skill | Median tokens | |---|---|---|---|---| | read | trunk | 4/4 | 4/4 | 74k | | read | branch | 4/4 | 0/4 | 66k | | write | trunk | 4/4 | 4/4 | 49k | | write | branch | 4/4 | 1/4 | 38k | | named | trunk | 4/4 | 4/4 | 83k | | named | branch | 4/4 | 4/4 | 102k | - **Correct outcome:** - read: the answer names Assembler and says the Business plan allows plugins. - write and named: exactly one draft with that title on Franklin Works. - In every run, the agent changed the site only through `wpcom_request`; no local tool was called. - **named, branch:** the agent loaded the skill in 4 of 4 runs without the rule. One run took 752k tokens, because `GET !/me/sites` with `plan` in `fields` returns 2.3 MB, and `wpcom_request` passes it through whole. Trunk has the same path. Without that run, the branch's median is 84k. - All 16 drafts were trashed afterwards. ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?

Drop the prompt rules that repeat Pi's own file-tool guidance

23e0843

github-actions Bot assigned chubes4 Oct 19, 2026

chubes4 merged commit b8a183a into trunk Oct 19, 2026
13 checks passed

chubes4 deleted the studio-code/drop-file-tool-prose branch October 19, 2026 19:12

This was referenced Oct 20, 2026

Studio Code: State the active site in the system prompt instead of prepending it to every message #5192

Merged

Studio Code evals: Report token usage and support follow-up messages #5193

Merged

chubes4 mentioned this pull request Oct 20, 2026

STUDIO.md: Drop the skills list that agents already discover from the installed skills #5195

Merged

1 task

This was referenced Oct 21, 2026

Studio Code evals: Let the agent ask the user, and run on a WordPress.com site #5230

Merged

Studio Code: Bring the remote-site runbook with a WordPress.com active site instead of a rule to load it #5231

Merged

This was referenced Oct 21, 2026

Studio Code: Keep the agent's identity in SOUL.md #5257

Closed

Studio Code: Load pi extensions from Studio's agent directory #5260

Closed

Navigation Menu

Repository navigation

Studio Code: Drop the prompt rules that repeat the file tools' own guidance - #5189

Studio Code: Drop the prompt rules that repeat the file tools' own guidance#5189
chubes4 merged 1 commit into
trunkfrom
studio-code/drop-file-tool-prose

chubes4 commented Oct 19, 2026

Uh oh!

wpmobilebot commented Oct 19, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

chubes4 commented Oct 19, 2026

Related issues

How AI was used in this PR

Proposed Changes

Testing Instructions

Pre-merge Checklist

Uh oh!

wpmobilebot commented Oct 19, 2026

📊 Performance Test Results

app-size

site-editor

site-startup

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants