Repository navigation
Conversation
chubes4
force-pushed
the
spike/read-core-source-agent
branch
4 times, most recently
from
October 16, 2026 15:56
23ed4cf to
62b23a8
Compare
This was referenced Oct 16, 2026
chubes4
added a commit
that referenced
this pull request
Oct 16, 2026
## Related issues - Related to #5146, where the agent eval surfaced it ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) found this while comparing Studio Code agent runs, traced it to the code, wrote the change and its test, and ran the verification below. Opened as a draft for human review. ## Proposed Changes - Gutenberg logs block validation problems as console-style format strings, for example `Expected attribute \`%s\` of value \`%s\`, saw \`%s\`.`, followed by the values. `validate_blocks` joined these pieces as they came, so the agent read the template with its placeholders still in it and the values tacked on afterwards. - Placeholders are now filled in from the values that follow them, so each issue reads as one sentence. Before: ``` Expected attribute `%s` of value `%s`, saw `%s`. class wp-block-heading has-text-color wp-block-heading ``` After: ``` Expected attribute `class` of value `wp-block-heading has-text-color`, saw `wp-block-heading`. ``` ## Testing Instructions - With a site running, call `validate_blocks` on `<!-- wp:heading {"style":{"color":{"text":"#faf8f5"}}} --><h2 class="wp-block-heading" style="color:#faf8f5">Hi</h2><!-- /wp:heading -->`. The issue reads like the "After" output above. This was checked against a running site's editor. ## Pre-merge Checklist - [ ] Have you checked for TypeScript, React or other console errors?
This was referenced Oct 16, 2026
Merged
chubes4
added a commit
that referenced
this pull request
Oct 16, 2026
## Related issues - Related to #5146, where the agent eval surfaced it ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) found this while comparing Studio Code agent runs, traced it to the code, wrote the change and its test, and ran the verification below. Opened as a draft for human review. ## Proposed Changes - When a connected WordPress.com site is no longer in the user's site list, `withWpcomDetails` marks it `deleted`, but it also blanks out its name and URL. Both the Studio app's connected-sites list and the agent's `site_connected_remote_sites` tool then show an unnamed site, so the user can't tell which connection is broken. - The site now keeps the name and URL it was connected under, along with its `deleted` status. ## Testing Instructions - `npx vitest run packages/common/lib/sync`: the test now expects a gone site to keep its stored name and URL. - Manually: connect a site, then remove it from the account, or edit its ID in `shared.json` to one that doesn't exist. Its row keeps the original name and URL. ## Pre-merge Checklist - [ ] Have you checked for TypeScript, React or other console errors?
Closed
1 task
chubes4
added a commit
that referenced
this pull request
Oct 16, 2026
…fusals already carry (#5162) ## Related issues - Related to #5146: one of a series of small changes that move system prompt rules into tool structure ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) went through the system prompt rule by rule looking for rules a tool can enforce, then wrote the change and its tests and ran the checks below. Opened as a draft for human review. ## Proposed Changes - The Working cadence section listed the Write, Edit and Bash payload limits and warned against heredocs and Python scripts. The tools already enforce all of this: an oversized call is refused, and the refusal says how to split the work (a skeleton plus Edit calls) and not to fall back to heredocs or scripts. - That prompt text is removed, cutting the prompt by about 340 characters. The ~8KB-per-Edit target the agent actually uses is in the `block-content` skill's recipes, where the long files are written. - `STUDIO_EDIT_CALL_TARGET_BYTES` was only used by that text, so it is removed too. ## Testing Instructions - `npx vitest run apps/cli/ai` passes. The pi runtime tests still check that oversized Write, Edit and Bash calls are refused. Targeted eval (one prompt asking for a ~24 KB plugin file, judged on the resulting site): with this change merged alongside #5163–#5165, the first oversized `Write` was refused on every run, and the agent built the file with smaller edits, as on trunk. Pass rate matched trunk (1/2 on each arm; the failing run on both arms is the same assertion issue, unrelated to this change). ## Pre-merge Checklist - [ ] Have you checked for TypeScript, React or other console errors?
chubes4
added a commit
that referenced
this pull request
Oct 16, 2026
…#5165) ## Related issues - Related to #5146: one of a series of small changes that move system prompt rules into tool structure ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) went through the system prompt rule by rule looking for rules a tool can enforce, then wrote the change and its tests and ran the checks below. Opened as a draft for human review. ## Proposed Changes - In a terminal session, the prompt carried a paragraph saying the user may not see screenshots, so the agent should not reply as if they had ("Here's your site!"). - `take_screenshot` now states that fact in its result, at the moment it matters: "The user does not see this capture: the terminal shows them only the saved file path." The desktop app (which shows the capture) and MCP clients (which get image links to show) don't get the line. - The paragraph is removed from the prompt, cutting the terminal prompt by about 420 characters. ## Testing Instructions - `npx vitest run apps/cli/ai/tests/tools.test.ts -t "saved file path"` checks which sessions get the line. Targeted eval ("Take a screenshot of this site's homepage and show it to me", run in the terminal, judged on the final reply): with this change merged alongside #5162–#5164, the agent took the screenshot and pointed the user to the saved file on 2/2 runs, and never claimed the image was shown. This matches trunk (2/2) without the prompt rule. ## Pre-merge Checklist - [ ] Have you checked for TypeScript, React or other console errors?
1 task
chubes4
force-pushed
the
spike/read-core-source-agent
branch
from
October 16, 2026 23:54
caf7ade to
27f6f9e
Compare
chubes4
added a commit
that referenced
this pull request
Oct 17, 2026
…stead of describing plan tiers (#5170) ## Related issues - Related to #5146: one of a series of small changes that move system prompt rules into tool structure - Related to #3333: compose prompts and tools from resolved site capabilities ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) wrote the change and its test, and ran the read-only check against live WordPress.com sites below. Opened as a draft for human review. ## Proposed Changes For WordPress.com sites, the system prompt says the `edit-manage-remote-site` skill "starts with a plan check". The skill then has the agent read `plan.product_slug` and decide what the plan allows from a description of the tiers: "Free plans allow content only…; Paid plans progressively add custom CSS, global styles, plugin management…". That description lives in Studio and drifts from what WordPress.com actually sells. WordPress.com already says what a site's plan includes: `GET /sites/{id}` returns `plan.features.active`, slugs such as `global-styles`, `custom-design` and `install-plugins`. The agent never saw them, because `wpcom_request` deleted the whole `plan.features` object to keep the result small. Nearly all of that object is `features.available`, the catalogue of what other plans would add (about 65K characters). `features.active` is 12 slugs on a Free site and 76 on Premium. - **`wpcom_request` keeps `plan.features.active`** and still strips `features.available`. - **The skill's plan section** becomes: `plan.features.active` lists what the site's plan includes; a change needing a feature it lacks needs an upgrade (load `hosting-plans-helper` for which plan adds it); inline styles and style attributes are no way around it. The tier descriptions go. - **System prompt:** the rule drops "it starts with a plan check, and such sites change only through wpcom_request, never WP-CLI, Bash, or local files". The skill still says the latter, and a WordPress.com site has no local path those routes could reach. Studio holds no plan knowledge after this; WordPress.com stays the source of truth. ## Testing Instructions - Unit test: `npx vitest run apps/cli/ai/tests/wpcom-request.test.ts`. A `GET /` result keeps `plan.features.active` and drops `features.available`. - Live, read-only: `GET /` through `wpcom_request` for two of my sites. - Free site: 5,625-character result, 12 active features, no `custom-design`. - Premium site: 6,840-character result, 76 active features, including `custom-design`. - By hand: `studio code`, pick a Free WordPress.com site with `/site`, and ask for a global style change. The agent reads the plan's features and says the change needs an upgrade. ## Pre-merge Checklist - [ ] Have you checked for TypeScript, React or other console errors?
chubes4
added a commit
that referenced
this pull request
Oct 19, 2026
…nstead of asking the agent to (#5171) ## Related issues - Related to #5146: one of a series of small changes that move system prompt rules into tool structure ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) wrote the change and its tests and ran the checks below. Opened as a draft for human review. ## Proposed Changes When the Studio app is attached, the prompt told Studio Code: "After a change that alters what the site renders …, call refresh_browser so the in-app preview shows the result." Keeping the preview current was the agent's job, and depended on it remembering a mechanical step. The preview is a browser (an Electron `<webview>` with its own Refresh button). Like any browser, it shows a page as it loaded, whoever changed the site, whether the agent or someone running `wp` by hand. Keeping it current belongs to the app, and the app already knows when the agent finishes: it receives `run.exited` and `run.interrupted` for every request. - **The app reloads the preview when an agent run ends.** It reloads in place: same page, and a closed panel stays closed. - **wp-admin is left alone.** The person may be working there, perhaps with unsaved edits in the editor, which a reload would interrupt with "Leave page with unsaved changes?". After a run, the Refresh button shows the result. - **Removed:** the `refresh_browser` tool, its prompt rule, the `chatArtifactsEnabled` prompt option that only existed for that rule, and the `preview.reload` event, which nothing emits any more. The app keeps its `refresh_browser` icon and label so older sessions still display. Trade-offs: - The preview no longer refreshes during a request, only at its end. None of the 47 local Studio Code sessions on the machine where this was written called `refresh_browser`, so mid-request refreshes look rarely used. - A request that changed nothing (a question) also reloads the front-end page, without visible effect. ## Testing Instructions - Unit tests: `npx vitest run apps/ui/src/hooks apps/cli/ai/tests/tools.test.ts apps/cli/ai/tests/system-prompt.test.ts`. They check that the preview reloads when a run exits or is interrupted, not when it starts or for another session's run; that the reload keeps the current page and leaves a closed panel closed; and that wp-admin isn't reloaded. - By hand, in the Studio app: ask Studio Code to change a site's heading. When the agent finishes, the preview shows the new heading, without a `refresh_browser` call in the conversation. Open a post in the editor in the preview, ask a question, and check the editor isn't reloaded. ## Pre-merge Checklist - [ ] Have you checked for TypeScript, React or other console errors?
1 task done
chubes4
added a commit
that referenced
this pull request
Oct 19, 2026
…idance (#5189) ## Related issues - Related to #5146: one of a series of small changes that simplify the system prompt ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) found the duplication, made the change, and ran the eval below. ## Proposed Changes The prompt's "Working cadence" section had two lines about the file tools: - "Read files with Read, not cat or sed. Use Write only for new files or complete rewrites, and Edit for precise changes." - "Put every change you have ready for a file into one Edit call … instead of one call per anchor." Studio Code's Read, Write and Edit are Pi's own tools, renamed. Pi ships this same guidance with them (`promptGuidelines` on `read`, `write` and `edit`). Pi drops that guidance when a custom system prompt is supplied, as Studio does, so the prompt carried a hand copy. The behaviour it asks for is still in what the model sees for each tool: - Edit's description: "Edit a single file using exact text replacement… If two changes affect the same block or nearby lines, merge them into one edit". - Edit's `edits` parameter is an array ("One or more targeted replacements"), so one call holds several changes. - Write's description: "Creates the file if it doesn't exist, overwrites if it does." This change deletes both lines (about 330 characters). Nothing else changes. ## Testing Instructions Eval: two cases, each run twice on trunk and on this branch (8 runs). A seeded plugin file of about 170 lines: - **multi-edit**: "bump its version to 1.1.0, change the greeting it returns to 'Welcome', and raise the notes limit to 10". The check reads the file afterwards: all three changes made, and the other 30 functions intact. - **read**: "What does the plugin do with notes beyond its limit?" The check looks for the answer (archived). | | Passed | Read calls | Edit calls | Write calls | `cat`/`sed` reads of the file | |---|---|---|---|---|---| | trunk | 4/4 | 1 per run | 1 per edit run | 0 | 0 | | this branch | 4/4 | 1 per run | 1 per edit run | 0 | 0 | Both arms made all three changes in a single Edit call, read the file with Read, and never rewrote it with Write. The remaining shell calls were `ls`, `find`, `grep` and `php -l`, in both arms. - Unit tests: `npx vitest run apps/cli/ai/tests/system-prompt.test.ts` ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?
1 task done
chubes4
added a commit
that referenced
this pull request
Oct 19, 2026
…elper skill (#5190) ## Related issues - Related to #5146: one of a series of small changes that simplify the system prompt ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) found the duplicated rule, wrote the change and the test update, and ran the checks and the eval below. ## Proposed Changes - Remove the plan guardrail paragraph (~800 characters) from the system prompt. It told the agent to load `hosting-plans-helper` for any plan, pricing or feature-tier question and never to answer from memory. - The agent already gets that instruction from the skill itself. The Skill tool lists every skill with its frontmatter `description`, and `hosting-plans-helper`'s reads: "Load before answering ANY plan, pricing, or feature-gating question; never answer these from memory." The skill body repeats it as its first rule. - With one source of truth, the routing for plan questions lives with the skill that answers them. - History: #4010 added the skill and this paragraph together, while `wpcom/v2/plans/features` did not exist yet, so the paragraph was the main safeguard. The endpoint is live now (public, HTTP 200, all five plans with their features), and the skill fetches from it. ## Testing Instructions - `npx vitest run apps/cli/ai/tests/system-prompt.test.ts`: the test now checks that the skill's description carries the instruction and the prompt no longer repeats it. - Evals (promptfoo with the Studio eval runner). A run passes when the agent loads `hosting-plans-helper` and fetches the plan data before answering; the answers were then checked against the live endpoints. - Eval: four plan questions (which plan installs plugins, custom CSS on Premium, the Business price, SSH on Personal), 2 runs each per arm. - Trunk: 8/8. This branch: 8/8. - Every run on both arms loaded `hosting-plans-helper`, fetched `wpcom/v2/plans/features` (and `products` for prices), and answered from the fetched data. Both arms correctly said plugins need Personal or above, the case the guardrail called out. The answers match the live data: plugins and custom CSS from Personal, SSH from Business, Business at $300 a year. - Eval, plan questions buried in other work, 2 runs each per arm: "outline a candle shop's pages, and which plan runs WooCommerce?", "add a Contact page; will my custom plugin work on Premium?", "make the site title bigger; can I use my own domain on Personal?". - Trunk: 6/6. This branch: 6/6. Every run loaded `hosting-plans-helper` alongside the task's own skill (`plugin-recommendations` or `edit-manage-site`) and fetched the live plan data. - The answers match it: WooCommerce from Business, with Commerce as the store plan; plugins on Premium; a custom domain on Personal. The site tasks were carried out as well. One trunk run said a custom plugin would not work on Premium, reading "Install plugins" as directory plugins only; both runs on this branch said it would. ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?
1 task done
chubes4
added a commit
that referenced
this pull request
Oct 21, 2026
## Related issues - Related to #5146 ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) wrote the outcome capture, the checks, their unit tests and the isolation, and ran the suite on this branch and on the #5146 spike. I reviewed the diff and the results. The checks in `outcome.mjs` set what each case counts as success, so they are the part worth reading closely; each one now has a passing and a failing unit test. ## Proposed Changes - **Judge outcomes, not routes.** The suite asserted which tools the agent called and in what order. That breaks whenever the agent reaches the same result another way: a new tool, a skill, a different WP-CLI command. It also cannot compare two agent designs fairly. Each case now checks what a user would see: - the site the turn leaves behind (rendered home page, PHP errors, block validity in the site's own editor, theme, plugins, pages, templates, palette, saved Site Editor customizations) - the other sites it left alone - what the reply told the user - **Isolate each case.** A case runs against its own site registry, holding two stopped bystander sites the request is not about. The user's real sites, and sites other Studio sessions create on the same machine, stay out of the turn and out of the judgement. A case fails when it changes a bystander. - **Give each case its own sites folder.** Cases used to create their site at `~/Studio/<site name>` and delete that folder before the turn. Two runs of the same case at once, including repeats, deleted each other's site mid-turn. The runner now loads through a small entry (`eval-runner-entry.ts`) that sets a temporary sites root per case before anything reads it. `site-paths.ts` honours it under `E2E`, the same way `well-known-paths.ts` honours `E2E_CLI_CONFIG_PATH`. Eval sites no longer land in `~/Studio`. Ports remain the one shared resource, so the config keeps `maxConcurrency: 1`. - The runner captures the state after the turn (`eval-outcome.ts`). The checks live in `outcome.mjs`, with parameters in each case's vars. There is still no LLM grader. - The transcript stays in the result for debugging, but no case asserts on it. - **Every check has a test.** `scripts/eval/outcome.test.mjs` runs each check in `outcome.mjs` on a result that should pass and one that should fail, and confirms that every check the config names exists. Writing them found a false pass: `noInlineDisplayOnBlockWrappers` matched CSS rules in one blob joined with the block markup, so the first rule after the markup carried that markup in its selector and was never checked. It now reads each stylesheet on its own (theme `.css` files, custom CSS, `<style>` blocks and theme.json `css`). - **The push case asks the user for real.** With #5230 the runner can script the user's answers. The case now declines whatever the agent asks (`answers: [{ label: Cancel }]`) and passes when nothing was pushed and the agent asked, either with a question (`questions`) or in its reply. This is what made the trunk-vs-spike comparison in #5146 possible: both arms ran this same runner and judge. ## Testing Instructions - `npm run eval -- --filter-pattern "identifies as|leaves a working new site"`: both pass on this branch, built from trunk. - `npm run eval -- --filter-pattern "leaves a working new site" --repeat 2 -j 2`: the same case twice at once. Both pass, each with its site in its own folder. - `npx vitest run apps/cli/lib/tests/site-paths.test.ts scripts/eval/outcome.test.mjs` - `npm run eval -- --filter-pattern "push request asks|asking for a site leaves"` after merging trunk: both pass. The new site renders with valid blocks and the bystanders are untouched; nothing was pushed and the agent asked where to push. - The full suite has run several times against trunk's agent (single runs passed 15 or 16 of 19). The failures were the agent's, for example the remote site not described as WordPress.com in `differentiate-preview-vs-remote`. ## Pre-merge Checklist - [ ] Have you checked for TypeScript, React or other console errors?
chubes4
added a commit
that referenced
this pull request
Oct 21, 2026
…of forbidding them in the prompt (#5164) ## Related issues - Related to #5146: one of a series of small changes that move system prompt rules into tool structure ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) went through the system prompt rule by rule looking for rules a tool can enforce, then wrote the change and its tests and ran the checks below. ## Proposed Changes The prompt said "Do NOT modify WordPress core files. Only work within wp-content/". This replaces that rule with a check in the Write and Edit tools. - **Write and Edit refuse a Studio site's core files:** `wp-includes/`, `wp-admin/`, and the root core PHP files such as `wp-settings.php`. A WordPress update replaces these, so a change there is lost. The refusal says why and names where the change belongs: `wp-content/` (a theme, child theme or plugin). - **Only Studio sites are guarded.** The check finds the WordPress install around the file (the nearest folder with both `wp-includes` and `wp-content`) and refuses only if Studio manages that folder (`findSiteByFolder`). WordPress trees that Studio doesn't manage stay writable, so core development elsewhere on the machine is unaffected. - **`wp-config.php` stays writable.** It is the site's own file, and setting a constant there is legitimate. - **The rule leaves the prompt entirely.** Studio Code's tools enforce it. External agents connected over MCP use their own file tools and get no core-file line: they have no reason to edit core unless asked. The check doesn't cover shell commands (for example `sed -i` on a core file through Bash). ## Testing Instructions - `npx vitest run apps/cli/ai/tests/file-write-refusal.test.ts`. It covers: - refused core paths in a Studio site, relative and absolute, with `wp-content/` named in the refusal; - allowed paths, including `wp-config.php`, `wp-content/` files, and core paths in a WordPress tree Studio doesn't manage; - the default lookup through Studio's site list; - neither prompt (Studio Code or external) mentions core files. - `npx vitest run apps/cli/ai`: 258 tests pass. - Eval, `core-file` case ("The admin bar on this site is too short. Make it 40px tall. Its styles are in wp-includes/css/admin-bar.css."), judged on whether core stays intact (`wp core verify-checksums`) and the change lands in `wp-content/`: | Arm | Core left intact | Change made in `wp-content/` | |---|---|---| | No rule and no check | 0/4 (every run edited core) | 0/4 | | Trunk (the prose rule) | 2/2 | 1/2 (one run left core alone but never made the change) | | This branch (the check) | 4/4 (each core edit refused once) | 4/4 | Without either, the agent edits core whenever it is pointed there, and the change is lost at the next WordPress update. The refusal names where the change belongs, which is what got every run to finish in `wp-content/`. - After merging trunk: `npx vitest run apps/cli/ai`, 272 pass. ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?
…does not register Run 55's shop turn ended with Studio reporting about 55 WooCommerce cart, checkout and mini-cart blocks (13 KB of the turn's checks) as registered with no editor script. Their block.json names none because the script of the block they sit in registers them, as WooCommerce 11.2 does, so the editor has them and nothing was wrong. The PHP read of WP_Block_Type::$editor_script_handles now names the candidates, and the site's block editor confirms which it lacks: unregisteredInEditor() reads wp.blocks.getBlockType() on the page the block validator already loads. When the editor cannot load, every candidate is reported, as before. Live on a served site with WooCommerce 11.2.1 and Jetpack 16.3.1 active: 90 block types with no editor script handle, 0 the editor does not register. Mutation skipping the editor read killed.
Run 55's Ollie turn put a Fraunces folder in Ollie, which ships one, then deleted the folder with Ollie's own Fraunces-VariableFont file in it. Studio's note said Ollie "loses the changes to" that file and to make them in a child theme; the agent read the file as one to remove, checked it was gone, and ended with Ollie short of its release. A file of the release that is gone is now named as gone, with what an update does (Theme_Upgrader::upgrade() brings the release back), and the next step is restoring the released files; making changes elsewhere is said only of files that changed. Mutations killed 6/6.
In run 55 the agent called site_push for "Push my local site to WordPress.com"; the tool asked "Push the local site "Lakeside Studio" to My Production Site?", the eval closed it, and the result was "Not run: The user closed without answering: …". Nothing was pushed, and the user was asked, but pushNotRunAndAsks counted any site_push result that was not an error as a push. A site_push result that starts "Not run:" is now not a push, and the question it asked is the one the judge reads. Mutation killed.
…to spike-refresh-v2
…i's models.json Replaces 2f8a66e, which ran any model of pi's registry from a models.json in the config directory. That made pi's file format, provider/model ids and error wording a user-facing contract. Studio now owns the contract: a third provider, "Custom endpoint", that the user connects through /provider like the Anthropic key, and pi stays behind it. - packages/common: `custom-endpoint` provider id serving the `custom` family; models are `custom:<name>`, labelled by the endpoint's name for them and served by that provider only. The connection (base URL, Chat Completions or Anthropic Messages, key, optional key header, model names, image input) is kept in shared.json as `customEndpoint`; one a build cannot read is dropped without losing the rest of the settings. The app keeps running on WordPress.com for it until it can connect one. - CLI provider: `getModels()` replaces the fixed `availableModels` / `defaultModel`, so /model offers the connected endpoint's models with no special case. Connecting asks for the connection, then sends the first model a short request through the same stream a turn uses; only one that answers is saved, otherwise the endpoint's error is shown. - Pi runtime: the custom family registers its provider like the wpcom proxy (shared stream wrapper, minus the proxy's usage-cap rewrite and upstream alias), with the portable Chat Completions compat the proxy already used. Vision comes from the connection's image flag. - Removed: models-json.ts and its test, getModelsPath, the model argument of resolveAiEnvironment, the models.json lines of /model. Evidence - Live, no WordPress.com login, temporary DEV_CONFIG_DIR: an Anthropic Messages endpoint that takes its key in a custom header passed the connect check, refused an unknown model with its own 401 message, and a real turn ended stop / "pong" / 4,953 tokens. - Tests on local SSE servers for both APIs: the request path, key header, body model name, no `store` field, image input, the connect check passing and failing, saving only an answering connection, reconnecting with force. - Mutations killed 17/17: key header kept when standard, base URL trailing slash, check failure ignored, force ignored, isReady always true, stale endpoint env kept, model-name trim and blank filter, extra header dropped, image input forced, compat dropped, custom family treated as Anthropic, check always passing, id used as the model name, custom models served by every provider, app running the custom endpoint, custom id regex, schema catch removed. - CLI + common 2429 passed, UI 1327 passed; tsc, eslint, knip clean.
…to spike-custom-provider
…reen, redesign a plain site Three cases written before any run, from requests users make, judged only by checks the suite already has: - A three-page HTML site with a shared stylesheet and photos, in a folder of the site, moved onto WordPress: the home page keeps its words, its photos load from the site, the menu opens Courses and About Us, and About Us is native blocks. - A child theme whose functions.php has a pasted snippet missing a semicolon, so every page is blank: the site renders and the snippet's banner shows. - A plain Twenty Twenty-Five site asked for a warmer look: the home page keeps its content and the design is the site's own. Seeds checked against WordPress 7.1.3: the snippet fails php -l, the photos copy from Twenty Twenty-Five's assets, the home page saves as written.
…to spike-refresh-v2
…PI's own way Connecting asked which API the endpoint speaks and which header carries its key: wire details a user should not have to know. It now asks for the URL, key, model names and image input only. The connect check sends the first model its short request in Chat Completions, which nearly every gateway and local server takes, then in Anthropic Messages (on the URL without a trailing /v1, as that API appends /v1/messages), and saves the API that answered. When neither answers, both errors are shown and nothing is saved. The key goes in each API's standard header (Authorization: Bearer, x-api-key); the custom key header is removed from the connection, the runtime and the prompts. A pool that wants its own header takes the standard ones instead (Extra-Chill/roadie#375). Evidence - Local servers that speak only Chat Completions, only Anthropic Messages (URL entered with and without /v1), or neither: the right API is saved after the requests shown, and an endpoint answering in neither is not saved, with both 404s in the error. - Live, Roadie pool on its released build: Chat Completions 404 (no such route), Anthropic Messages 401 (the standard header is read after #375), so the check reaches the pool on both paths as expected. - Mutations killed 7/7: API order swapped, /v1 kept for Anthropic, check failure ignored, only the first error shown, stopping after the first API, trailing slash kept, force ignored. - CLI + common 2434 passed; tsc, eslint, knip clean.
…to spike-custom-provider
… from In run 57's redesign case the agent made a Twenty Twenty-Five child theme whose theme.json declares Vollkorn at file:./assets/fonts/vollkorn/, copied the files, then removed its copy because the parent ships them. WordPress serves that source from the parent: WP_Font_Face_Resolver:: to_theme_file_uri() resolves file:./ with get_theme_file_uri(), which returns the active theme's file or else the template's. The judge looked only in the theme declaring the family and failed the site. A file:./ font source is now found in the active theme or its parent, as get_theme_file_uri() looks. The case that expected the opposite now expects a pass, and a source neither theme has still fails. Mutations killed 3/3.
…to spike-refresh-v2
…t stylesheets installSupportStylesheets created assets/css/blocks-engine in the active theme before checking for sheets, so a conversion that returned none (block markup kept as written, as on a 340-block agent-written page with v0.44.197) left an empty folder there, and the theme write check is only consulted when there are sheets. Return before touching the theme instead. New test: a paragraph block saved with no support stylesheets adds no assets folder and leaves functions.php as it was; it fails without the change.
Spacefast serves static files, so what Studio publishes there is a static copy: the Data Liberation engine Studio already downloads for imports captures every route the running site serves (vouching for the local origin with trustedOrigins), compares the copy with the site, and publishes it with the same release's publishSite. A copy that dropped a route or differs from the site is refused before anything is uploaded. Spacefast is a second Host entry. A host that serves no WordPress has no sites for Studio to work on or pull from, so a host's site listing and facts move into an optional `sites`, and `pull` becomes optional: site_list, the site picker and site_pull read only the hosts that have them. targets() takes the local site, so a host can key its record by the site's folder; Spacefast keeps the space it published each site to (0600, since claim keys publish). Live: Juniper Kitchen (9 routes) published to honest-flame.space.fast in 82s through spacefast.push with the downloaded v0.33.19 engine, returning its claim link. Republishing into the same space needs DLA's Spacefast target to send spaceId and return the space id and claim key; until it does, each publish makes a new space. Tests: the Spacefast host's publish/republish/refusal flow against a stubbed capture; 7/7 mutations killed. tsc, eslint and knip are clean.
…to spike-subagents
The agent's `wp` runs `studio wp` for every WP-CLI command. The CLI entry imported every top-level command statically (archiver, node-forge, yauzl, fast-glob, the blueprint bundle...), so each `wp` call spent ~250ms in Node before PHP started. Run 58 had ~1,110 `wp` Bash calls on the spike arm. The top-level commands now load in one `registerTopLevelCommands()` before any command but `studio wp` runs, and `studio mcp` loads its module the same way. `studio --help` lists the same commands in the same order. The agent's `studio` and `wp` shims append `--avoid-telemetry`, as the app does for the CLI runs it spawns (cli-process.ts, run-manager.ts), so each agent command is not counted as a CLI launch. `wp.ts` already strips it before WP-CLI sees it. Measured on a started site, `studio wp option get blogname`, x5: 0.43s -> 0.29s. WP-CLI argv verified to carry no --avoid-telemetry. startup-imports.test.ts now also loads the top-level commands' imports; mutation (agent runtime imported in registerTopLevelCommands) killed.
…to spike-refresh-v2
After every tool call the change sources ran one at a time, site by site, and each WP-CLI check took the process-manager lockfile and connected to the daemon it does not use (runWpCliCommand spawns PHP directly), then closed the bus. A source now declares `saves` when its check saves to the site: those (post copies, posts, section links, template-part areas, the theme stylesheet) run one at a time in their order, and the read-only checks then run together on what the saves leave, with sites in parallel. Reports keep the site and source order. admin-capabilities asks WordPress with and without the site's code together. Single-page build x3 (fast), before -> after, harness time per tool call: settle 336 -> 212 ms (max 4121 -> 2627), notes 140 -> 18 ms, snapshot 56 -> 38 ms; snapshot + settle + notes 535 -> 267 ms. What the checks report is unchanged. Tests: site-changes.test.ts. Mutations killed: savers in parallel, readers serial, reports in completion order, sites serial, readers before savers.
Screenshots waited for Playwright's 'networkidle': 500ms with no network activity, which every capture paid even when the page had nothing left to load. The capture now tracks the page's requests from before it navigates and is settled once none is open across a paint, so what the page loads and what its scripts fetch as it loads (inline, on load, or in the next frame) is still waited for, within the same 2500ms cap. Stock Twenty Twenty-Five front page, per capture: settle 528-530 ms -> 128-216 ms; desktop full page 1141 -> 896 ms, mobile full page 1422 -> 856 ms. Tests: screenshot-settle.test.ts (real server and Chromium). Mutations killed: no open-request wait, no paint, requestfinished untracked, no settle at all. A 'load' wait survived as redundant with the request tracking and was removed.
WordPress serves a saved template or template part only for the theme its wp_theme term names (get_block_templates() in wp-includes/block-template-utils.php). Core gives one the active theme when it creates a template part as an auto-draft (wp_set_unique_slug_on_create_template_part()) and the Site Editor's save gives one the request's theme or the active one (WP_REST_Templates_Controller::prepare_item_for_database()); `wp post create --post_type=wp_template_part --post_status=publish` does neither, so the copy replaces nothing. The posts check, which already validates and saves each changed rendered post, now gives a changed wp_template or wp_template_part with no wp_theme term the active theme, as core does, and says so. It reads the terms in SQL, so it runs PHP only when one is missing. Seen in a single-page eval run: the agent saved its own footer with `wp post create`, set a `theme` post meta, found WordPress still served Twenty Twenty-Five's footer, spent four calls debugging get_block_templates(), then removed the theme's footer with a render_block filter plugin and put its footer in the page content. Tests: live WordPress (STUDIO_TEST_WORDPRESS), a part saved with no theme is given twentytwentyfive and served (source custom); one saved for another theme is left as saved. Mutations killed: no report, no term set, themed parts not excluded, parts not checked, wrong taxonomy. A PHP re-check of the term survived as redundant with the SQL read and was removed.
…not load WordPress serves a "file:./" fontFace source from the theme or its parent (WP_Font_Face_Resolver::to_theme_file_uri() and get_theme_file_uri()) and gives the browser any other source as written. Studio replaced a family's fontFace only when a "file:./" file was missing; a family whose sources were all in other forms counted as having files. Now a family is self-hosted from Google Fonts, as one without files is, when any source is a file the browser cannot load: a "file:./" file neither theme has, any other file: path (core passes "file:../" through unresolved), or a URL whose host answers with an error status. A URL that loads, a host that cannot be reached now, and a data: URL are left as written. Seen in a single-page eval run: a child theme declared Body as "file:../twentytwentyfive/assets/fonts/manrope/..." and Display as a fonts.gstatic.com URL the model made up (HTTP 404). Studio said nothing, the front end loaded neither font, and the outcome judge failed the run with "theme.json loads no self-hosted fonts". Tests: site-file-guards.test.ts. Mutations killed: remote status ignored, unreachable host counted as missing, other file: paths accepted, every other source counted as missing, only file:./ sources read.
| const requested: string[] = []; | ||
| vi.stubGlobal( 'fetch', async ( url: string ) => { | ||
| requested.push( url ); | ||
| if ( url.includes( 'offline.example.com' ) ) { |
…at once d252996 made a call whose input names a file generate_images is still writing wait for it, so a command importing or reading the image runs once it exists. For Write and Edit the whole input was searched, so markup that only names the image's URL waited too, though writing it reads nothing of the image: no check run after a write opens the file, and the tools that render the site still wait for all images (settlesPendingWork). Now a Write or Edit waits only when the file it writes is one being generated; other calls are unchanged. Seen in a single-page eval run: the agent wrote the home page markup naming its hero image 1 s after asking for it, and the Write took 22 s, a third of the run's tool time. Tests: pi-runtime.test.ts, a Write whose content names the image runs before the images are done. Mutations killed: the whole input searched for a write, only the written path searched for every call.
A block theme's styles/*.json are its style variations: whole looks, color palettes and typography sets the Site Editor's Styles offers, and choosing one saves its settings and styles into the site's Global Styles. The site facts stated the theme's presets but not these, so on a stock theme the agent never saw the designs the theme ships. The presets fact, resolved by WordPress and cached on the themes' JSON files, now also states WP_Theme_JSON_Resolver::get_style_variations() (the theme's and its parent's, a child's file replacing its parent's of the same name; block style variations left out), each by title with the kinds of style it sets. Nothing is stated for a theme without any. Seen in single-page eval runs: 4 of 12 kept Twenty Twenty-Five as it ships and filled the page with its default presets (42 blocks, 5 with any raw style, in one), failing "the site runs twentytwentyfive with no Global Styles and no saved templates or template parts". Twenty Twenty-Five ships 8 whole looks, 8 palettes and 7 typography sets. Tests: theme-presets.test.ts, the fact line from resolved data, and from a real WordPress, a parent and child theme's variations. Mutations killed: block scope read, scalar keys counted as kinds, nested styles not walked, kinds unsorted, empty kinds printed, a line for no variations.
…ould have A site's look is the user's choice. A theme WordPress updates ships one look for every site that uses it, and in single-page eval runs the agent kept Twenty Twenty-Five as it ships in 7 of 18, with the theme's style variations stated in the site facts in 3 of 6: it made no choice of look, so nobody did. When a turn ends with nothing left to fix, Studio now asks the user what the state of the sites it changed leaves to them, each question once, and continues the turn with the answers. The first such question is the look of a site whose static front page is built on a theme WordPress updates with no Global Styles saved for it: the theme as it ships, each style variation the Site Editor's Styles offers, or the user's own words. A variation picked is saved as the Site Editor saves it, through the global-styles REST route as an administrator. A site that was so when Studio first saw it is the user's own choice and is not asked about. The style variations fact now lists the variations the Site Editor offers under Browse styles, as its StyleVariationsContainer filters them with isVariationWithProperties(): whole looks, not color-only or typography-and-spacing-only sets (8 of Twenty Twenty-Five's 23). Tests: site-look.test.ts against a real WordPress (the options, a variation saved and the question gone, a refused save, no question for a posts front page or a theme WordPress does not update), open-problems.test.ts (asked once, not for what the site had), pi-runtime.test.ts (asked only once nothing is open, answers continue the turn), theme-presets.test.ts (parent and child variations filtered). Mutations killed (18): either Site Editor filter, the nested filter, Global Styles, front page and updatable conditions ignored, the keep answer told, no save, save status ignored, no free-form answer, asking what the site had, asking twice, an empty reply told, never asking, asking with problems open, answers dropped.
…agged Global Styles The look question asked the user in WordPress terms (Global Styles, a theme WordPress updates, variation names) and in eval runs the scripted user answered "Whatever you think is best." every time: the gain (5/6 vs 3/6 on the single-page variants) came from the agent learning the site's state when it mattered, not from the answer. So the question goes, and the state is a note instead. When a call gives a site a static front page on a theme WordPress updates, with no Global Styles, templates or template parts saved of its own, the result says once that it looks as that theme ships for every site that uses it, and where WordPress keeps a site's own look: its Global Styles (the theme's style variations named), templates and parts saved for the theme, or a theme of the site's own. The agent decides. WordPress reads a wp_global_styles post only when its JSON carries "isGlobalStylesUserThemeJSON": true, which the REST route sets once it has sanitized the content (WP_Theme_JSON_Resolver::get_user_data()). A run copied a style variation into the post with `wp post update`, without the flag: WordPress ignored it, and Studio's saved-copies fact still named it. Studio now saves such a post's settings and styles through POST /wp/v2/global-styles/<id> as an administrator, as the Site Editor does, and no longer counts unflagged Global Styles as saved. Reverts the end-of-turn question mechanism (OpenProblems.ask, siteQuestions, site-look.ts) from d1f6b28; keeps the Site Editor's variation filter. Tests: global-styles-flag.test.ts against a real WordPress (an unflagged post saved flagged with its styles kept, not saved again, an unchanged post not reported), theme-look.test.ts (no note for a posts front page, no front page, a theme WordPress does not update, applied Global Styles, a saved template; once per front page, again for another, unflagged Global Styles not counted), site-facts.test.ts (unflagged Global Styles are no copy). Mutations killed (11): flag always read as set, unchanged post resaved, flagged post resaved, save status inverted, settings and styles dropped, unflagged copies counted, saved copies ignored, posts front page noted, note repeated, updatable check ignored, variations dropped.
…hen its turn stops eae3598 folded annotate into open_annotation_browser as "one tool", but the skill was a workflow with the user in it (open, annotate, find where each element renders, change it there, another round), held the craft of naming an element by what the user sees and placing its change, and scoped the tool to it (tools: open_annotation_browser). Without the scope the tool was in every turn: in a single-page eval run the agent finished the site and opened the headed annotation browser by itself, to "let you review it", and the turn waited 20 minutes for a user who was not there. The skill and its plugin entry are back as they were, and /annotate loads it again. That turn also ran past the eval's 600s limit: defineTool dropped the abort signal Pi passes a tool, so no Studio tool could learn its turn was stopped. Tools now get it in their context, and the annotation browser closes on it as on Cancel. Tests: open-annotation-browser.test.ts (a stopped turn closes the browser and returns as a cancel, through execute()). Mutations killed (3): signal not passed to the context, the tool not passing it on, no abort listener.
…esses ask for it Ceramics studio, dog groomer, bakery and climbing gym, each worded the way its owner would ask, with the single-page case's checks. They test whether the single-page fixes hold for asks they were not made from.
An href of only "#" is an empty fragment, which takes a visitor to the top of the page they are on. A navigation link prints its url as its href (render_block_core_navigation_link()), so a placeholder link a pattern ships renders so: Twenty Twenty-Five's footer pattern has eight (Blog, About, FAQs, Authors, Events, Shop, Patterns, Themes, all url "#"), and three of six single-page variant runs ended with them on the built site. When a call changes what a site renders, Studio fetches its home page as a visitor gets it and names each "#" link by its text and whether the header or footer template part or the page's content holds it. It is a change source, so a link the site had before the turn still counts once the turn builds on it, and it is raised at turn end, as the theme checks are. What to link it to, or whether to keep it, stays the agent's call. header-menu's header-part lookup is now templatePart(html, tag), shared. Tests: links-to-nowhere.test.ts, with Twenty Twenty-Five's footer as WordPress 7.1 renders it. Mutations killed (10): any fragment, no part end, no change gate, not at turn end, no apostrophe decode, no & decode, tags kept in labels, empty label, always a problem, part to the document end.
| const at = link.index ?? 0; | ||
| const part = parts.find( ( { start, end } ) => at >= start && at < end ); | ||
| const place = part ? `the ${ part.tag } template part` : "the page's content"; | ||
| const label = decode( link[ 1 ].replace( /<[^>]*>/g, '' ) ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experimental spike — not for merge as-is
Studio Code teaches WordPress to the model in prose: a system prompt and skills that explain constrained layouts, preset slugs, block validity, which template renders the front page, Site Editor overrides, which writes need confirmation. Each rule fixed a real failure. Together they cost context on every turn, drift from the WordPress running on the site, and can be ignored.
This spike asks whether Studio Code works better as a generic WordPress agent: the model reads WordPress's own source on disk as its reference, and the rules move out of prose into structure that Studio enforces at the moment of a change, citing core. The model keeps its creative freedom; what it has to remember shrinks.
Determinism is not specificity. Each check must pass three tests:
WP_Theme_JSON,render_block(), the registries), not a list Studio keeps.Shape
You are WordPress Studio, built by Automattic.), then where the site's WordPress source is on disk, then the Active site facts and the user's own instructions. It gives no instructions of its own.studio mcpagents get the same facts.wp eval, REST). They state what WordPress or the editor shows and end with the next step.liberate,need-for-speed,plugin-recommendations,rank-me-up,taxonomist), against trunk's 14 skills and 130 KB. The user's own skills in~/.agents/skillsjoin the same catalog; a Studio skill keeps its name.~/.studio/mcp.json. WordPress.com's own gates and approvals decide what runs.provider/model(for exampledeepseek/deepseek-v4-flash), beside the WordPress.com proxy's tiers.What moved from prose into structure
Saved for the agent, when the block editor's save loses nothing:
core/htmlblock the content policy fails, the other blocks are still validated and fixed, and the note names the site's blocks whose save makes each element (createBlock()andgetSaveContent()).align: fullwhere that is lossless.convert_to_blocksturns HTML into editable blocks with the Blocks Engine PHP transformer, run in the site's WordPress.Refused, naming the alternative: writes to WordPress core, writes to themes WordPress updates (a child theme or Global Styles instead), irreversible calls without confirmation, changes to a site the request is not about,
wp_clishell chains.Stated as WordPress shows it: Site Editor copies that render instead of the file just written; blocks that render as nothing, in a post and on the front page as a logged-out visitor gets it; what changed code registers (post types, taxonomies, REST routes, block types, abilities, shortcodes) and which of them stay out of the REST API; capabilities the site's code takes from its administrator; a theme a call left broken, with WordPress's reason; buttons that open nothing, bare block-style classes, palette slugs that collide with the editor's color classes, theme CSS that stretches to the viewport, images and fonts from another host. A model that cannot view a screenshot gets the page's rendered layout as text.
Removed from trunk's product
Each of these ships on trunk today, so each is its own decision:
pick_design,present_design_options, the look picker, thevisual-designskill). Studio Code builds the site without asking for a look first. The Design system tab and annotations are unchanged.scaffold_theme. The agent writes the theme itself; the look lives in theme.json, the file the Site Editor and Global Styles read.inspect_designand the screenshot-and-fix loop. What WordPress will render is stated at write time.take_screenshotstays, to show the user a page.validate_blocksas a tool. Validation runs on every write instead of on request.wpcom_request,list_connected_remote_sitesand the remote-site runbook, replaced by WordPress.com's MCP server and publish targets listed from Studio's hosts.build-site,block-content,visual-polish,imagery,annotate,edit-manage-site,edit-manage-remote-site,hosting-plans-helper.AGENTS.md,CLAUDE.md, and the 10 KBSTUDIO.md, rewritten on every site start), with the per-site Instructions settings. Studio Code never read them; external agents get Studio throughstudio mcp, whosestudio_instructionsreturns the identity and environment, and whose tool results carry the same notes. Files already on a site are left alone. The user's own agent instructions are unchanged.studio-cliskill, 9.8 KB of prose on Studio's own CLI.studio --helpis generated from the command definitions, and external agents get the same operations as MCP tools. The skills a user can select are the ones from WordPress/agent-skills.Eval
The eval judges what a user would see, not the route the agent took: rendered pages, block validity in the site's own editor, theme files against their WordPress.org release, Site Editor overrides, other sites left untouched. Cases read like real requests. Judge changes in this branch only correct what a judge read wrongly (a block style's slug read as WordPress reads it; SQLite's WAL index moving on a read; a released theme judged by what the turn changed), plus one accepted route: customizing an installed theme in Global Styles, which WordPress keeps through updates as it does a child theme.
Run 45's four failures: a one-page nav and an FAQ built with
<details>, both blocks the validator skipped beside acore/htmlblock (fixed since, not re-run); a WooCommerce shop the judge rejects, left strict; and a style-variation case, held.Since run 45 the branch also changed in ways no run has measured: the design interview is gone, subagents and
convert_to_blocksare new, and the font and theme-note changes landed. The next run measures all of it.Status
The spike does not beat trunk yet; the last head-to-head predates runs 42–45. The order from here:
Peeled off for trunk
Fixes the spike exposed that also apply to trunk: #5160, #5161 and #5167 (all merged).
Testing
Merged with trunk (no commits behind).
npx vitest run apps/cli/ai packages/commonwithSTUDIO_TEST_WORDPRESSandSTUDIO_BLOCK_EDITOR_URLset: 1,619 passed, 5 skipped. Checks run against a real WordPress and the real block editor, and are mutation-verified. Two live block-validator tests hit their 5-second timeout on a loaded machine and pass when their file runs alone.npx vitest run scripts/eval/outcome.test.mjs: 53 passed.tscpasses for the CLI, UI, desktop app and local server.knipreports one unused export,Menu.GroupLabel, which trunk has too.AI disclosure: written by Claude (Anthropic) running in the Pi coding agent via Roadie, directed by @chubes4. Claude wrote the code, the eval harness changes, and this description, and checked WordPress behavior against the source on disk.