Skip to content

feat: e2e no setup and multi device testing - #131

Open
hurali97 wants to merge 24 commits into
mainfrom
hur/e2e-auto-setup
Open

hurali97 wants to merge 24 commits into
mainfrom
hur/e2e-auto-setup

Conversation

@hurali97

Copy link
Copy Markdown
Collaborator

Description

This PR enables the user to have an e2e test recorded without needing to configure e2e. Just open the app on simulator and start recording. Users can now also run the same test on different devices.

hurali97 and others added 22 commits November 16, 2026 13:58
Replace the single-run globals with a registry of matrices and lanes, a
scheduler (default 3 at once, 1-8), per-lane device-pinned bundled config
and output dir, and mobile_testing_run_matrix / run_progress / cancel.
Lanes boot an off device and copy the app from another matrix device.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…sting panel

Tick devices and a parallel limit in E2E setup (saved per project), run on
N devices, and watch one lane per device with cancel and a per-lane step
view. Disabled with a note when a project config is in use.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…reusing a sibling's state

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
mobile-dev Error Error Nov 20, 2026 11:58am UTC

Request Review

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The packaging test fails against the pinned runtime version, while unresolved defects can break rewritten tests, misidentify running devices, prematurely mark cancelled lanes free, and omit nested dependency notices.

5 open issues
2 files not reviewed
  • packages/runtimes/agent-device/package-lock.json: Generated file
  • packages/runtimes/e2e/package-lock.json: Generated file
What changed in this PR

This PR expands Mobile Dev’s testing workspace into Testing Studio, enabling recording without an existing e2e setup and running the same test across multiple iOS simulators and Android emulators.

Changes:

  • Bundles a pinned e2e runtime for zero-setup recording and execution.
  • Adds multi-device scheduling, per-device progress and cancellation, mirror navigation, and platform-specific locator repair.
  • Updates packaging, automated coverage, agent guidance, and user documentation.

Telemetry adds bounded setup counters and matrix-run measurements while preserving existing recorder metrics.

File Description
website/​src/​pages/​index.astro Renames the Testing navigation link.
website/​src/​content/​docs/​docs/​testing.md Documents zero-setup and multi-device workflows.
website/​src/​content/​docs/​docs/​privacy.md Updates Testing Studio terminology.
website/​src/​content/​docs/​docs/​introduction.md Renames the testing feature link.
website/​src/​content/​docs/​docs/​getting-started.md Updates the workspace tab name.
website/​astro.config.mjs Renames the documentation sidebar entry.
tooling/​vendor-e2e.mjs Installs and indexes the bundled runtime.
tooling/​update-dependencies.mjs Includes e2e updates and holds agent-device alignment.
tooling/​test/​e2e-runtime-package.test.ts Checks runtime metadata and notices.
tooling/​host-runtime.mjs Requires bundled e2e files in host packages.
tooling/​dependency-lanes.mjs Adds e2e to dependency-update preparation.
tooling/​build.mjs Validates and copies the e2e runtime.
THIRD_PARTY_NOTICES.md Updates the agent-device version notice.
packages/​ui/​test/​testing-panel-project-state.test.ts Covers project persistence and lane selection.
packages/​ui/​test/​testing-matrix.test.ts Covers matrix state, execution, and preferences.
packages/​ui/​test/​testing-drafts-panel.test.ts Makes the project-mode preview fixture explicit.
packages/​ui/​test/​testing-agent-run-render.test.ts Covers rendering and adoption of agent runs.
packages/​ui/​test/​test-recorder.test.ts Covers locator picking, errors, and overlay isolation.
packages/​ui/​src/​testing-matrix.ts Adds matrix parsing and preference helpers.
packages/​ui/​src/​test-recorder.ts Adds locator picking and preserves runner errors.
packages/​ui/​src/​simulator-panel.ts Supports switching mirrors to a selected device.
packages/​ui/​src/​popover-placement.ts Updates Testing Studio wording in a comment.
packages/​ui/​src/​host-bridge.ts Exposes the host’s link-opening capability.
packages/​ui/​src/​device-workspace.ts Adds platform focus and device switching.
packages/​ui/​src/​components/​testing-note.tsx Adds reusable testing information and warning notes.
packages/​ui/​src/​components/​testing-matrix.tsx Adds device selection and run-lane controls.
packages/​ui/​src/​components/​test-recorder-overlay.tsx Adds locator insertion and platform-specific overlays.
packages/​ui/​src/​components/​simulator-view.tsx Passes platform identity to the recorder overlay.
packages/​ui/​src/​components/​cascading-menu.tsx Updates Testing Studio wording in a comment.
packages/​skills/​mobile-dev-testing/​SKILL.md Describes agent recording, running, and repair workflows.
packages/​skills/​agent-device/​references/​workflow.md Refreshes the pinned runtime’s workflow guidance.
packages/​shared/​test/​stored-test-steps.test.ts Tests platform-specific locator branching.
packages/​shared/​src/​test-recording.ts Renames the comment emitted in recorded tests.
packages/​shared/​src/​stored-test-steps.ts Adds platform-specific locator rewriting.
packages/​shared/​src/​panel-target.ts Adds testing intent, project context, and heartbeat timing.
packages/​runtimes/​e2e/​release.json Records pinned runtime versions and integrity metadata.
packages/​runtimes/​e2e/​package.json Declares the bundled e2e dependencies.
packages/​runtimes/​agent-device/​package.json Aligns the agent-device runtime version.
packages/​runtimes/​agent-device/​package-lock.json Locks the updated agent-device package.
packages/​mcp-app/​src/​workspace-app.tsx Connects testing lanes to mirror controls.
packages/​mcp-app/​src/​components/​workspace.tsx Renames the tab and follows lane platform selections.
packages/​core/​test/​testing-target-discovery.test.ts Tests Expo and foreground-app target discovery.
packages/​core/​test/​testing-setup.test.ts Covers setup planning, application, and sign-in state.
packages/​core/​test/​testing-setup-telemetry.test.ts Checks bounded, content-free setup metrics.
packages/​core/​test/​testing-prepare.test.ts Covers device booting, app copying, and preparation.
packages/​core/​test/​testing-matrix.test.ts Covers scheduling, cancellation, and registry behavior.
packages/​core/​test/​testing-matrix-project.test.ts Covers project targets and device-specific lanes.
packages/​core/​test/​testing-foreground-target.test.ts Covers foreground app resolution for recording.
packages/​core/​test/​testing-device.test.ts Updates coverage for bundled projection fallback.
packages/​core/​test/​testing-chat-project.test.ts Covers chat project selection and panel-opening behavior.
packages/​core/​test/​testing-agent.test.ts Adapts recording tests to asynchronous startup.
packages/​core/​test/​plugin.test.ts Tests opening the Testing Studio tab.
packages/​core/​test/​mobile-dev-skill.test.ts Checks the expanded testing workflow guidance.
packages/​core/​test/​cpu-sessions.test.ts Tests optional inclusion of Apple system apps.
packages/​core/​test/​browser-panel.test.ts Accounts for repeated-panel warnings.
packages/​core/​test/​agent-device-tools.test.ts Updates tests for the revised press arguments.
packages/​core/​test/​agent-device-runtime-match.test.ts Checks alignment between bundled agent-device versions.
packages/​core/​test/​agent-device-catalog.test.ts Updates catalog checks for the runtime upgrade.
packages/​core/​src/​testing-target-discovery.ts Discovers app identifiers from Expo or foreground state.
packages/​core/​src/​testing-prepare.ts Prepares devices and copies missing apps.
packages/​core/​src/​testing-matrix.ts Adds lane scheduling, state, and cancellation.
packages/​core/​src/​testing-failure.ts Builds failure context for locator repair.
packages/​core/​src/​testing-drafts.ts Updates Testing Studio wording in a comment.
packages/​core/​src/​testing-device.ts Adds projection fallback and selective device release.
packages/​core/​src/​telemetry.ts Adds bounded setup-operation counters.
packages/​core/​src/​plugin.ts Wires shared testing services and chat project context.
packages/​core/​src/​panel-tools.ts Adds testing-opening guidance and repeat-open warnings.
packages/​core/​src/​e2e-runtime.ts Locates and validates project or bundled runtimes.
packages/​core/​src/​device-apps/​apps.ts Resolves foreground simulator apps, including Apple apps.
package.json Adds the e2e vendoring command.
package-lock.json Updates dependency development-only classifications.
docs/​test-matrix.md Documents matrix architecture and implementation status.
docs/​telemetry.md Documents setup counters and matrix measurements.
docs/​mcp-tools.md Lists matrix execution and cancellation tools.
docs/​agent-recording.md Describes bundled-mode saving, running, and repair.
CONTRIBUTING.md Explains agent-device version alignment requirements.
apps/​codex-plugin/​tooling/​smoke-package.mjs Checks packaged e2e runtime assets.
.github/​workflows/​release.yml Installs and caches e2e for releases.
.github/​workflows/​dependency-updates.yml Prepares e2e during dependency updates.
.github/​workflows/​ci.yml Installs and caches e2e in CI.

Review effort: Balanced

Comment thread packages/core/src/testing-matrix.ts Resolved
Comment thread packages/ui/src/testing-matrix.ts Resolved
Comment thread tooling/vendor-e2e.mjs Resolved

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants