Repository navigation
Studio Code: Give the agent its WordPress.com site's plan features instead of describing plan tiers - #5170
Merged
Conversation
…stead of describing plan tiers wpcom_request stripped the whole plan.features object from GET /, so the edit-manage-remote-site skill described from memory what each plan tier allows. It now keeps plan.features.active, the features this site's plan includes, and strips only features.available, the 65K catalogue of what other plans would add. The tier prose, and the plan check in the system prompt rule, go.
chubes4
changed the title
chubes4
force-pushed
the
studio-code/wpcom-plan-facts
branch
from
October 17, 2026 13:36
c25b686 to
7db0a89
Compare
chubes4
marked this pull request as ready for review
October 17, 2026 13:54
Collaborator
📊 Performance Test ResultsComparing 7db0a89 vs trunk app-size
site-editor
site-startup
Results are median values from multiple test runs. Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff) |
1 task done
chubes4
added a commit
that referenced
this pull request
Oct 20, 2026
…epending it to every message (#5192) ## Related issues - Follows #5170, #5189, #5190: moving Studio Code context from prose into structure, one small step at a time. - First step toward composing site facts the way the system prompt already composes tool snippets: each fact is rendered by the code that owns it, from Studio's own data. ## How AI was used Claude (via the Pi coding agent in a Roadie session) traced how the active site reaches the model, wrote the change and tests, and ran the eval below. I reviewed the diff and the eval traces. ## Proposed Changes The active site was stated by prepending a `[Active site: …]` line to every user message. So the same fact piled up in the conversation history once per turn, and it was mixed into what the user typed. This PR states it once, in an `## Active site` section of the system prompt instead: - `buildSystemPrompt` takes `activeSite` and renders the section. The text is the same as before, minus the brackets: name, path, running/stopped, and the DESIGN.md path when there is one; or ID, URL and "WordPress.com" for a remote site. - The Pi runtime already rebuilds the session, and so the system prompt, on every turn. It now passes the `activeSite` it was already receiving into `buildSystemPrompt`, so the section always reflects the current site and running state, including after compaction. - `ai` and the eval runner stop prepending the line. The user message is exactly what the user typed. - The three skills and the one prompt line that referred to "the active-site line" now say "the Active site section". Trade-off: when the active site or its running state changes, the system prompt changes, so that one turn misses the prompt cache. Before, each turn added the line to the history instead. ## Testing Instructions Unit tests: `npx vitest run apps/cli/ai apps/cli/commands/ai` (245 pass). - `system-prompt.test.ts`: the section for a local site, a WordPress.com site, and a site with a DESIGN.md. It sits above the user's global instructions and is absent without an active site. - `commands/ai/tests/index.test.ts`: the dispatched message is the typed text, and the runtime gets the site with the daemon's running state. Eval: four cases that depend on the agent knowing its active site, 2 runs each, this branch against trunk (`76f5c92bf`), using the `fast` model: | Case | What passes | Branch | Trunk | |---|---|---|---| | Tagline | Sets `blogdescription` and `blog_public` on the active site | 2/2 | 2/2 | | Plugins | Lists every installed plugin of the active site | 2/2 | 2/2 | | Admin bar (core file) | Core checksums still verify, and the change lives in `wp-content/` | 1/2 | 1/2 | | Design system | Reads the seeded DESIGN.md and the new page uses its required CTA text | 2/2 | 2/2 | Neither admin-bar failure involves the active site: - On the branch, the agent left core alone and set the height through Additional CSS (`wp_update_custom_css_post`). That's valid, but the check only looks at files. - On trunk, the model emitted a malformed tool call and the turn ended. In all 16 runs the agent worked in the correct site directory. The result is parity: the same facts, no longer repeated in the history. Multi-message eval, using #5193's token usage and `followUps`: a seeded site and eight messages (tagline, theme, pages, create a draft page, plugins, permalinks, site title), 2 runs per arm. To match the app, the trunk arm adds the site line to every message. | | Trunk (line in every message) | This PR (system-prompt section) | |---|---|---| | Final state correct (tagline, title, draft page) | 2/2 | 2/2 | | Site lines saved in the session | 8 per run | 0 | | Copies of the line re-sent across the run's requests | 73–78 | 0 (the section is sent once per request) | | Total tokens per run | 218k, 244k | 225k, 214k | The line is about 25 tokens, so this saves roughly 1.5k tokens per eight-message run, under 1% and well inside the run-to-run noise. The gain is in the transcript (it holds what the user typed, and no outdated copies of the site's state), not in the token bill. Manual: 1. `studio ai` with a site selected, then ask "what site are you working on?" The agent answers from the system prompt, and your message in the history has no prefix. 2. Switch site with `/site`, ask again: the answer reflects the new site. ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?
This was referenced Oct 21, 2026
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issues
How AI was used in this PR
Claude (via the Pi coding agent in a Roadie session) wrote the change and its test, and ran the read-only check against live WordPress.com sites below. Opened as a draft for human review.
Proposed Changes
For WordPress.com sites, the system prompt says the
edit-manage-remote-siteskill "starts with a plan check". The skill then has the agent readplan.product_slugand decide what the plan allows from a description of the tiers: "Free plans allow content only…; Paid plans progressively add custom CSS, global styles, plugin management…". That description lives in Studio and drifts from what WordPress.com actually sells.WordPress.com already says what a site's plan includes:
GET /sites/{id}returnsplan.features.active, slugs such asglobal-styles,custom-designandinstall-plugins. The agent never saw them, becausewpcom_requestdeleted the wholeplan.featuresobject to keep the result small. Nearly all of that object isfeatures.available, the catalogue of what other plans would add (about 65K characters).features.activeis 12 slugs on a Free site and 76 on Premium.wpcom_requestkeepsplan.features.activeand still stripsfeatures.available.plan.features.activelists what the site's plan includes; a change needing a feature it lacks needs an upgrade (loadhosting-plans-helperfor which plan adds it); inline styles and style attributes are no way around it. The tier descriptions go.Studio holds no plan knowledge after this; WordPress.com stays the source of truth.
Testing Instructions
npx vitest run apps/cli/ai/tests/wpcom-request.test.ts. AGET /result keepsplan.features.activeand dropsfeatures.available.GET /throughwpcom_requestfor two of my sites.custom-design.custom-design.studio code, pick a Free WordPress.com site with/site, and ask for a global style change. The agent reads the plan's features and says the change needs an upgrade.Pre-merge Checklist