Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
584d517
feat: add no setup test recording
hurali97 Nov 16, 2026
67097e7
fix: issues
hurali97 Nov 16, 2026
c34b26f
fix: ui bugs
hurali97 Nov 16, 2026
58fc61b
feat: allow non-project recordings
hurali97 Nov 16, 2026
c759edf
fix: bug with version conflict
hurali97 Nov 17, 2026
94b550c
feat: use latest e2e
hurali97 Nov 17, 2026
93ec6dc
feat(testing): run a test on several devices from a lane registry
hurali97 Nov 17, 2026
5adf64d
feat(testing): matrix device checklist and per-device lanes in the Te…
hurali97 Nov 17, 2026
98080a1
docs(testing): document the test matrix
hurali97 Nov 17, 2026
10dc289
test(testing): cover the matrix UI and keep the Devices section from …
hurali97 Nov 17, 2026
9d48d48
fix: device hang
hurali97 Nov 17, 2026
1a0ba88
fix: no project mode bugs
hurali97 Nov 18, 2026
c2fd888
fix: only show ios devices if android test was recorded
hurali97 Nov 18, 2026
541f5ad
feat: add platform differences fix
hurali97 Nov 18, 2026
9584ee9
fix: bugs
hurali97 Nov 18, 2026
be455ca
feat: wire agents
hurali97 Nov 18, 2026
b691e4d
fix: input requiring with agent session
hurali97 Nov 18, 2026
d709102
fix: agent driven usage
hurali97 Nov 19, 2026
341264a
fix: all bugs in agent driven device runs
hurali97 Nov 19, 2026
0b63ac8
feat: project mode
hurali97 Nov 20, 2026
1796954
feat: rename to testing studio
hurali97 Nov 20, 2026
50ac9c7
fix: bugs
hurali97 Nov 20, 2026
ec622b1
fix: code reviews
hurali97 Nov 20, 2026
bc02ccb
Merge branch 'main' of https://github.com/callstackincubator/codex-mo…
hurali97 Nov 20, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,14 @@ jobs:
cache-dependency-path: |
package-lock.json
packages/runtimes/agent-device/package-lock.json
packages/runtimes/e2e/package-lock.json
packages/runtimes/serve-emu/package-lock.json

- name: Install dependencies and device runtimes
run: |
npm ci --no-audit --no-fund
npm run vendor:agent-device
npm run vendor:e2e
npm run vendor:serve-emu

# macOS-only paths skip themselves here; the release build runs the full suite on macOS.
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/dependency-updates.yml
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ jobs:
cache-dependency-path: |
package-lock.json
packages/runtimes/agent-device/package-lock.json
packages/runtimes/e2e/package-lock.json
packages/runtimes/serve-emu/package-lock.json

# Same toolchain as the release build, so rebuilt helpers match what ships.
Expand All @@ -94,6 +95,7 @@ jobs:
run: |
npm ci --no-audit --no-fund
npm run vendor:agent-device
npm run vendor:e2e
npm run vendor:serve-emu

# Applies, prepares and commits each lane on the update branch, then runs the
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ jobs:
cache-dependency-path: |
package-lock.json
packages/runtimes/agent-device/package-lock.json
packages/runtimes/e2e/package-lock.json
packages/runtimes/serve-emu/package-lock.json

- name: Validate release version and tag
Expand Down Expand Up @@ -82,6 +83,7 @@ jobs:
npm ci --no-audit --no-fund
npm run lint:commits -- --last --verbose
npm run vendor:agent-device
npm run vendor:e2e
npm run vendor:serve-emu

- name: Restore or build native helpers with matching symbols
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ relevant checks, and packaging steps again.

`vendor:agent-device` installs the exact npm packages in `packages/runtimes/agent-device/package-lock.json` with package scripts disabled. `build` copies this full runtime into `dist/agent-device`, records its integrity and lockfile hash in `release.json`, and copies the commands the plugin servers serve, `packages/runtimes/agent-device/tools.json`. After changing the pin, run `npm run catalog:agent-device` to rewrite that catalog and the agent-device skill's workflow guide (`packages/skills/agent-device/references/workflow.md`, the runtime's `help workflow`); unit tests check both against the pinned runtime, and the weekly dependency update rewrites them. The installed plugin needs no npm install or runtime download. Xcode compiles the shipped Apple runner source on its first use.

agent-device is held at 0.21.33 in the weekly dependency updater. Versions 0.21.34 and 0.21.35 reject iOS input when an obsolete Xcode selection link points at a deleted installation, even when `xcode-select -p` and `xcrun` resolve the installed Xcode. See [upstream #3995](https://github.com/callstack/agent-device/pull/3995). Verify that case before raising the pin and removing the updater hold.
agent-device is pinned by hand at the version `@e2e-dev/mobile` installs in `packages/runtimes/e2e` (0.21.37 now). The bundled e2e runner and Mobile Dev's own client share one agent-device state directory, so a newer runner daemon makes the older client refuse with "Daemon … is newer than this client". `agent-device-runtime-match.test.ts` fails when the two drift; the weekly dependency updater reports agent-device updates but does not apply them. Versions from 0.21.34 carry [upstream #3995](https://github.com/callstack/agent-device/pull/3995), which rejected iOS input when an obsolete Xcode selection link pointed at a deleted installation; check that case when raising the pin.

`build` also bundles JavaScript and CSS with esbuild and copies Baguette into `dist/baguette`. It does not run TypeScript, lint, Biome, visual checks, or React Doctor.

Expand Down
2 changes: 1 addition & 1 deletion THIRD_PARTY_NOTICES.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This plugin includes [Baguette 0.2.1](https://github.com/tddworks/baguette/tree/

Native helpers statically link [Sentry Native 0.17.1](https://github.com/getsentry/sentry-native/tree/0.17.1), licensed under MIT. Its license and the Android libunwindstack and Linux syscall support licenses ship in each helper's `SENTRY-LICENSE.txt` or `third-party-licenses.txt`. Both Android collectors include the statically linked LLVM libc++ runtime's Apache-2.0 license with LLVM exceptions. Shared collector telemetry source ships in `dist/telemetry` beside the physical iOS log reader's source and build scripts.

This plugin also includes [agent-device 0.21.33](https://github.com/callstack/agent-device/tree/v0.21.33) by Callstack under the MIT License. Its license sits at `dist/agent-device/node_modules/agent-device/LICENSE`. The package includes the official npm release, its Apple runner source, and its bundled JavaScript dependencies. `runtimes/agent-device/package-lock.json` pins the package and its integrity hash. `dist/agent-device/release.json` records the npm archive, integrity hash, and lockfile SHA-256. The agent-device skill's workflow guide, `skills/agent-device/references/workflow.md`, is that CLI version's `help workflow` output.
This plugin also includes [agent-device 0.21.37](https://github.com/callstack/agent-device/tree/v0.21.37) by Callstack under the MIT License. Its license sits at `dist/agent-device/node_modules/agent-device/LICENSE`. The package includes the official npm release, its Apple runner source, and its bundled JavaScript dependencies. `runtimes/agent-device/package-lock.json` pins the package and its integrity hash. `dist/agent-device/release.json` records the npm archive, integrity hash, and lockfile SHA-256. The agent-device skill's workflow guide, `skills/agent-device/references/workflow.md`, is that CLI version's `help workflow` output.

The Baguette MCP server and panel bundle the Model Context Protocol SDK, MCP Apps SDK, OpenAI MCP Extensions, Zod, ws, React, Radix UI, Legend List, Sentry's React and Node SDKs, and their required JavaScript code. The panel uses shadcn/ui components under the MIT License and Tailwind CSS styles. It bundles Lucide icons under the ISC License, with Feather icons under the MIT License and Inter under the SIL Open Font License 1.1. `dist/third-party-licenses.txt` contains their licenses. The build preserves bundled license comments too.

Expand Down
4 changes: 4 additions & 0 deletions apps/codex-plugin/tooling/smoke-package.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,10 @@ try {
for (const name of ["mobile_list_ios_devices", "mobile_list_android_devices", "mobile_boot_android_emulator", "mobile_android_stream_session", "mobile_ios_mirror_session", "mobile_ios_mirror_reset", "mobile_ios_mirror_close"]) assert.ok(tools.tools.some(tool => tool.name === name));
for (const name of ["device_open", "device_snapshot", "device_press", "device_screenshot"]) assert.ok(tools.tools.some(tool => tool.name === name), name);
await access(join(plugin, "dist/agent-device/tools.json"));
await access(join(plugin, "dist/e2e/node_modules/e2e/dist/cli/bin.js"));
await access(join(plugin, "dist/e2e/node_modules/@e2e-dev/mobile/dist/nodes.js"));
await access(join(plugin, "dist/e2e/release.json"));
await access(join(plugin, "dist/e2e/THIRD_PARTY_NOTICES.txt"));
const physicalMirror = tools.tools.find(tool => tool.name === "mobile_ios_mirror_session");
assert.deepEqual(physicalMirror._meta.ui.visibility, ["app"]);
await access(join(plugin, "dist/serve-emu/src/cli.mjs"));
Expand Down
40 changes: 31 additions & 9 deletions docs/agent-recording.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
# Agent-driven recording in the Testing panel
# Agent-driven recording in the Testing Studio panel

Status: implemented. The five model tools, `mobile_testing_agent_state` and
`mobile_testing_set_context` are in `packages/core/src/testing-agent.ts`; the panel polls, adopts
and locks the mirror; the skill is `packages/skills/mobile-dev-testing`. Android emulators are supported
through the context's `deviceId` and `platform`; physical devices, Maestro, saving and running are
still out, as below.

The Testing panel records an e2e test from the user's own taps on the mirror
The Testing Studio panel records an e2e test from the user's own taps on the mirror
([docs/testing-panel.md](testing-panel.md)). This adds the other driver: the user tells their
coding agent *"use Mobile Dev and record the test for the important areas on the Explore Details
screen"*, the Testing tab opens, and the steps appear in it one at a time as the agent works. When
screen"*, the Testing Studio tab opens, and the steps appear in it one at a time as the agent works. When
the agent stops, the flow is sitting in the panel unsaved and the user adjusts it, saves it and
runs it exactly as if they had tapped it themselves.

Expand Down Expand Up @@ -56,10 +56,10 @@ in `packages/core/src/app-flow-tools.ts`):

| Tool | Input | Does |
|---|---|---|
| `mobile_testing_agent_begin` | `{ title, goal }` | Opens the Testing tab, starts one session against the panel's current project, device and bundle id. Refuses, with what is missing, when any of the three is unset. |
| `mobile_testing_agent_begin` | `{ title, goal }` | Opens the Testing Studio tab, starts one session against the panel's current project, device and bundle id. Refuses, with what is missing, when any of the three is unset. |
| `mobile_testing_agent_screen` | `{}` | The runner's element tree as the agent's eyes: one compact row per element — `ref`, `role`, `name`, `text`, `placeholder`, `value`, `testId`, `enabled`, `selected`, `bounds` — **and the selector the recorder would emit for it**, with its rank and whether it is unique. The agent never writes a locator; it names a `ref`. |
| `mobile_testing_agent_step` | `{ ref?, kind, value?, direction?, instruction? }` | Records one step and performs it. `kind` is a `PinnedStepKind`; `instruction` with no `ref` records an `AgentStep`. Assertions record without touching the device, mirroring the element menu. An acting step answers with `navigated: true` when it replaced the screen — see "Where a flow ends". |
| `mobile_testing_agent_finish` | `{ note? }` | Ends the session with the steps left dirty in the panel. Saves nothing, runs nothing, and keeps the device session for the panel's recorder. |
| `mobile_testing_agent_finish` | `{ note? }` | Ends the session. In a project the steps are left dirty in the panel and nothing is saved; in Testing Studio the test is saved, pinned to the recorded app, and its `relativePath` returned. Runs nothing either way. |
| `mobile_testing_agent_abort` | `{ reason }` | Drops the session; whatever was recorded stays in the panel. |

App-only:
Expand All @@ -69,9 +69,31 @@ App-only:
| `mobile_testing_agent_state` | `{}` | `{ active, status, revision, title, goal, steps, note, error }`. The panel polls it. |
| `mobile_testing_set_context` | `{ projectPath, udid, bundleId }` | The panel reports its selection whenever it changes. This is the only thing that knows the selected device, and `agent_begin` reads it. |

The eight existing `mobile_testing_*` tools stay app-only. A narrow, intention-shaped surface is
what keeps `write_file` and `run_test` out of the model's hands, which is what
`packages/core/src/testing-tools.ts:17` already says.
## Testing Studio mode: the agent saves, runs and repairs

In No project (Testing Studio) mode the agent drives the whole loop, so more of `testing-tools.ts` is
model-visible (`sharedVisibility`): `run_test`, `run_matrix`, `run_progress`, `cancel`, `set_target`,
`read_file`, `tests`, plus two new tools. The rest (`write_file`, `stage_file`, `save_staged_file`,
`delete_file`, drafts, setup) stay app-only.

- **Gate.** Each model-visible tool refuses unless the Testing Studio tab has reported a project
(`mobile_testing_set_context`), and `projectPath` defaults to it. The agent never needs to know
where Testing Studio lives.
- **`agent_finish` saves** in bundled mode: `tests/<slug>.e2e.ts`, never replacing a name that is
taken, pinned to the recorded platform and bundle id. It returns `relativePath`; no draft is kept.
In a project it still hands over unsaved.
- **`mobile_testing_wait_for_app({ platform, timeoutMs })`** polls the booted devices of a
platform for a foreground app (home screens and launchers excluded) and returns its `bundleId`,
or `found: false` with a message to relay.
- **`mobile_testing_branch_line({ relativePath, line, platform, locator })`** applies
`branchStoredStepLocator` (now `packages/shared/src/stored-test-steps.ts`, one copy for core and
the panel) and writes the result. Only the locator changes, so an assertion cannot be weakened
through it.
- **Failure context.** A failed lane gets `failure: { line, source, error, elements }` before it
settles: the failing line and the device's tree as `role "name" → screen.getBy…` rows (at most 60).
`run_test` and `run_progress` both carry it.
- **Panel.** The panel polls `run_progress`; a run it did not start is adopted as live lanes
(`agentRun`) and locks the same controls as `agent.active`. A saved recording refreshes the test list.

One session at a time. A second `agent_begin` refuses rather than stomping on the first.

Expand Down Expand Up @@ -153,7 +175,7 @@ describes: all three plugins for Codex and Cursor, and `apps/claude/skills` for

It covers:
- **When**: the user asks to record, write or capture an e2e test for a screen or a flow.
- **Prerequisites**: the Testing tab, a chosen project, a booted iOS simulator with the app
- **Prerequisites**: the Testing Studio tab, a chosen project, a booted iOS simulator with the app
launched. Missing any, say which and ask — never open the folder picker on the user's behalf.
- **The loop**: `agent_begin` → `agent_screen` → choose a `ref` → `agent_step` → re-read → … →
`agent_finish`.
Expand Down
4 changes: 4 additions & 0 deletions docs/mcp-tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,10 @@
| `mobile_appflow_next_decision` | Long-poll for the next active-agent sweep decision (model only) |
| `mobile_appflow_submit_decision` | Submit one role-valid JSON decision and wait briefly for the next (model only) |
| `mobile_appflow_screenshot` | Read an authorized App Flow PNG (app only) |
| `mobile_testing_run_test` | Run one saved Testing Studio test on the single device the runner picks and wait for its steps (app only). It is a matrix of one |
| `mobile_testing_run_matrix` | Run one test on several devices at once, one lane per device; returns a `matrixId` immediately. `devices: {deviceId, platform}[]`, `apps: {ios?, android?}`, `limit` (default 3, max 8, the rest queue). Boots devices that are off. Bundled mode copies the app from another booted device when missing (app only). A project with its own e2e config runs its target for each device's platform (`targets: {ios?, android?}` picks by name) on that device; a missing app is copied the same way and the target's engine must be a direct `mobile(...)` call |
| `mobile_testing_run_progress` | Read a matrix's lanes: state (`queued`, `booting`, `installing`, `running`, `passed`, `failed`, `cancelled`, `error`), live steps and result per device. `matrixId` optional; without it the latest run (app only) |
| `mobile_testing_cancel` | Cancel one device's lane (`deviceId`) or every unfinished lane of a `matrixId`; a queued lane never starts (app only) |

The plugin servers for Codex, Claude and Cursor add the pinned agent-device runtime's commands as `device_*` tools with compact, validated schemas, including `device_open`, `device_snapshot`, `device_screenshot`, `device_press`, `device_fill`, `device_type`, `device_scroll`, `device_wait`, `device_find`, `device_get`, `device_is`, `device_close`, and debugging tools. The agent reads screens and sends input with them; `device_screenshot` also returns the saved image. Their annotations let hosts run reads, device input and app control without asking, as with the panel's tools, while installs, scripts, app-defined events, file outputs, the clipboard and development servers still ask (see [Architecture](architecture.md#runtime)). Their input schemas describe each command; `logs` and `perf` are left out for Mobile Dev's own tools. The agent-device skill explains the control loop and ships the runtime's own workflow guide.

Expand Down
16 changes: 14 additions & 2 deletions docs/telemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,13 @@ The panel UI sends nothing to Mixpanel, and its CSP is unchanged.

## Volume policy

Testing setup adds `testing.setup.operation.samples`. Its attributes are limited to `component`,
`surface`, `operation` (`plan`, `apply`, `login`, `preference`), bounded `outcome`, and a supported
provider id (unknown values become `other`). It never includes project paths, bundle identifiers,
model ids, endpoints, prompts, test/source text, OAuth codes, API keys or tokens. The existing UI
Testing timings and outcome counters likewise carry no user content. Cancellation and partial failure
are outcomes, not error payloads, and telemetry opt-out applies unchanged.

Since 0.1.164, application metrics in the UI, Node servers and native helpers use
independent 0.5% point sampling before SDK buffering and network transmission.
This reduces metric point volume by 99.5% on average, including automatic Node
Expand Down Expand Up @@ -539,7 +546,7 @@ it does not confirm user submission or a model response. Notes remain until the
user clears them. Message/context construction timings, surface attribution, and
content scrubbing are preserved; no note text or selection evidence enters telemetry.

The Testing panel's recorder measures `ui.testing.tree_read`, the read that resolves a tap into a
The Testing Studio panel's recorder measures `ui.testing.tree_read`, the read that resolves a tap into a
selector, and counts `ui.testing.record_start`/`record_stop`, `ui.testing.step_tap`,
`ui.testing.step_assert`, `ui.testing.step_unresolved`, `ui.testing.hover_walk`,
`ui.testing.menu_open`. Editing a recorded step counts `ui.testing.step_duplicate`,
Expand Down Expand Up @@ -603,6 +610,11 @@ as well, so a flow tried a step at a time is not read as a flow run end to end;
beside the verdict, not instead of it. The test's title, its path, the project path, the steps and
the runner's output are the user's own content and are **not** sent.

A run on several devices measures `ui.testing.matrix_run`, the panel's whole call from starting the
matrix to its last lane ending, and counts `ui.testing.matrix_run` at the start and
`ui.testing.matrix_passed` or `ui.testing.matrix_failed` at the end (passed only when every lane
passed). Device names and ids, app ids and the number of devices are not sent.

Unsaved recordings — the flows the panel keeps so one survives the panel document being rebuilt —
measure `ui.testing.draft_save`, the autosave write, and count `ui.testing.draft_restore` when one
is reopened into the step list and `ui.testing.draft_delete` when the user throws one away. The
Expand Down Expand Up @@ -633,7 +645,7 @@ while this one answers "how long is Record unusable for" — the number the over
of. A read that fails still records the timing, because the wait happened; it carries no selector,
element name, bundle id or project path.

The Testing panel records on Android emulators as well as iOS simulators, so `ui.testing.tree_read`,
The Testing Studio panel records on Android emulators as well as iOS simulators, so `ui.testing.tree_read`,
`ui.testing.record_prepare` and the `testing.agent.screen.*` metrics now carry a `platform`
attribute (`ios` or `android`; two values, no device ids). One series hid that waiting on an
emulator is a different number from waiting on a simulator. Names, units and boundaries are
Expand Down
Loading
Loading