Contrib/2384 by miguelg719 · Pull Request #2385 · browserbase/stagehand

miguelg719 · 2026-07-23T08:05:22Z

why

The CUA and hybrid agents dispatch every pointer action as mouse input. Mobile layouts commonly gate their handlers on touch/pointer events, so a synthesized mouse click never registers — e.g. on adidas.co.il (Browserbase os: "mobile") a size selector highlights but keeps reporting "please choose a size", blocking add-to-cart. A trusted touch (Input.dispatchTouchEvent) works.

Originates from #2384 by @alonle.

what changed

New useTouch option (explicit opt-in, default false). Forwarded to the server on the API execution path.
page.tap(x, y) and locator.tap(): trusted touch taps mirroring their click counterparts.
When useTouch is set: CUA routes single left clicks to tap (right/middle/multi-click, hover, and drag stay on mouse); hybrid click/type/fillFormVision tap for coordinate focus/activation.
Replay: taps record as tap steps and replay as touch. tap is not exposed to the LLM during live inference.

test plan

Unit: touch-actuation (7), touch-tool-routing (9), touch-cua-routing (7) — resolution, tool routing, recording.
E2E: perform-understudy-method (8) — asserts pointerdown:touch/touchstart/touchend delivery for both tap paths (event-level, since Chromium synthesizes a click from a tap and activation-only assertions can't tell touch from mouse).
Server: start.test.ts covers useTouch on /sessions/start.
Manual: with useTouch: true on a Browserbase os: "mobile" session, the agent completes the adidas.co.il size + add-to-cart flow.

The computer-use agent actuates every click via `Input.dispatchMouseEvent` (page.click). On a mobile session the browser renders the site's touch-gated mobile layout, whose handlers only respond to touch/pointer events — so a synthesized mouse click does not register (e.g. an SFCC size selector keeps showing "please choose a size" and add-to-cart fails). Add a coordinate `page.tap(x, y)` (trusted `Input.dispatchTouchEvent`, mirroring `page.click`) and route the CUA `click` action to it when the session presents as mobile (`navigator.userAgentData.mobile`, detected once per run). Desktop sessions are unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…or replay - Route only a single LEFT click to tap; right/middle/multi-click keep the mouse path (preserves context-menu and double-click behavior). - On a recording (replay) mobile run, record the tap as a deterministic "tap" step (page.tap gains returnXpath) and add a replayable tap method: locator.tap() + METHOD_HANDLER_MAP.tap. Not added to SupportedUnderstudyAction — replay dispatches by map lookup, so the LLM is not offered tap during live inference. - page.tap now updates the cursor overlay like other coordinate actions. - Add an integration test for tap actuation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

changeset-bot · 2026-07-23T08:05:25Z

🦋 Changeset detected

Latest commit: acf6c0c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@browserbasehq/stagehand	Patch
@browserbasehq/stagehand-evals	Patch
@browserbasehq/stagehand-server-v3	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

cubic-dev-ai

1 issue found and verified against the latest diff

Confidence score: 5/5

In packages/core/lib/v3/handlers/v3CuaAgentHandler.ts, the new CUA dispatch branch lacks handler-level regression tests for mobile/non-mobile action selection and recording, so a subtle mapping mistake could route tap/click incorrectly and silently skew interaction playback or analytics; add focused mocks that assert both page.tap/page.click calls and recorded tap/click outputs across both paths.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/core/lib/v3/handlers/v3CuaAgentHandler.ts">

<violation number="1" location="packages/core/lib/v3/handlers/v3CuaAgentHandler.ts:358">
P3: This new CUA dispatch branch has no handler-level regression coverage for mobile/non-mobile selection or recording. Add focused mocks asserting `page.tap`/`page.click` and recorded `tap`/`click` behavior for both paths.

(Based on your team's feedback about unit tests for new behavior.)</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

Replace the lazy per-run probe in the CUA click handler (a page.evaluate of navigator.userAgentData.mobile, cached on first click) with a `usesTouch` getter resolved from configuration: - explicit `useTouch` wins, so callers can force either mode; - otherwise derived from Browserbase `browserSettings.os` ("mobile"/"tablet") or a local session's `localBrowserLaunchOptions.hasTouch`. The probe had three problems the config signal does not: it could not run before the first click, `userAgentData` is absent on insecure origins and on about:blank (so a first click there cached a false negative for the whole run), and `userAgentData.mobile` is false for tablets. It also cost a round trip per run and made behavior depend on which page happened to be loaded first. Note `os` is unavailable when resuming via browserbaseSessionID; `useTouch` is the explicit opt-in for that case. Adds unit coverage for the resolution matrix, which needs no browser.

The CUA handler was the only path actuating touch, but hybrid mode goes through V3AgentHandler and its own tools, so it still sent mouse clicks. Hybrid is the auto-selected mode for Claude / GPT-5.4 / Gemini 3, i.e. the default most callers get -- and "CUA hybrid agent (mouse): blocked" was a row in the original bug report, so the reported failure survived the first fix. Route on v3.usesTouch in: - clickTool, and record the step as "tap" so a cached mobile run does not replay as a mouse click; - typeTool and fillFormVisionTool, whose click-to-focus has the same problem -- an unregistered focus click means the text goes nowhere. Their recorded step stays a "type", which replays via the selector rather than a pointer event. Deliberately unchanged: double/triple click, right/middle click, drag and hover. Real touch drag needs touchMove interpolation rather than a routing switch. Adds unit coverage asserting each tool picks tap vs click and records the matching method.

Format with prettier and drop the three `any`s, which were failing the Lint job and -- because lint failure cancels the rest of the workflow -- taking every other check on the branch down with it. Inherited from #2373 on main; the eslint errors were latent there because `format:check` runs first and exited before eslint ever saw the file. Types the CDP payload the reducer actually reads instead of indexing an `any` record. Behavior is unchanged: the added `?? ""` only feeds `includes()`, where `undefined` and `""` are both misses, and the new optional chains sit behind guards that already established the object exists.

page.tap and locator.tap awaited touchStart before writing touchEnd, so every tap cost two serial round trips where click batches press/release into one burst. CDP preserves per-session command order, so the pair can be written together. Halves tap latency locally (16.7ms -> 6.8ms) and saves a full network RTT per tap against a remote browser. Soaked 500 iterations with no failure -- 300 local, 200 on a Browserbase os:"mobile" session -- asserting per iteration both that the element activated and that the page observed touchstart before touchend, since reordering is the risk pipelining introduces.

cubic-dev-ai

All reported issues were addressed across 14 files

Architecture diagram

sequenceDiagram
    participant Agent as Agent/LLM
    participant Tool as Hybrid Tool (click/type/fill)
    participant V3 as V3 Instance
    participant Page as Page (understudy)
    participant Session as CDP Session
    participant CUA as CUA Handler
    participant Locator as Locator (replay)

    Note over Agent,Locator: === TOOL EXECUTION (first run or non‑replay) ===
    Agent->>Tool: Execute tool with coordinates
    Tool->>V3: usesTouch (getter)
    Note over V3: Checks explicit useTouch,<br/>then Browserbase os, then local hasTouch
    alt usesTouch == true (mobile/tablet session)
        Tool->>Page: tap(x,y,{returnXpath})
        Note over Page: Pipelined touchStart + touchEnd
        Page->>Session: Input.dispatchTouchEvent (type:"touchStart", touchPoints:[{x,y}])
        Page->>Session: Input.dispatchTouchEvent (type:"touchEnd", touchPoints:[])
        opt recording (isAgentReplayActive)
            Page->>Page: resolveXpathForLocation(x,y)
            Page-->>Tool: xpath
            alt tool is clickTool
                Tool->>V3: recordAgentReplayStep(method:"tap")
            else tool is typeTool or fillFormVisionTool
                Note over Tool,V3: Actuation changes but recorded step stays "type"
            end
        end
    else usesTouch == false (desktop session)
        Tool->>Page: click(x,y,{returnXpath})
        Page->>Session: Input.dispatchMouseEvent (mousePressed / mouseReleased)
        opt recording
            Page->>Page: resolveXpathForLocation(x,y)
            Page-->>Tool: xpath
            Tool->>V3: recordAgentReplayStep(method:"click")
        end
    end
    Tool-->>Agent: execution result

    Note over Agent,Locator: === CUA EXECUTION ===
    Agent->>CUA: action (click, x, y, button, clickCount)
    CUA->>CUA: isPrimarySingleClick = button=="left" && clickCount==1
    alt isPrimarySingleClick && usesTouch
        CUA->>Page: tap(x,y,{returnXpath})
        opt recording
            CUA->>V3: recordCuaActStep(method:"tap")
        end
        CUA-->>Agent: { success: true }
    else not primary single click or !usesTouch
        CUA->>Page: click(x,y,options)
        CUA-->>Agent: result (mouse path unchanged)
    end

    Note over Agent,Locator: === REPLAY (cached step with method:"tap") ===
    Agent->>V3: performUnderstudyMethod (method:"tap", xpath)
    V3->>V3: Lookup METHOD_HANDLER_MAP["tap"] → tapElement
    V3->>Locator: tapElement(locator)
    Locator->>Locator: resolveNode() → objectId
    Locator->>Session: DOM.scrollIntoViewIfNeeded (objectId)
    Locator->>Session: DOM.getBoxModel (objectId)
    Locator->>Locator: compute center (cx,cy)
    Locator->>Session: Input.dispatchTouchEvent (touchStart, [{cx,cy}])
    Locator->>Session: Input.dispatchTouchEvent (touchEnd, [])
    Locator->>Session: Runtime.releaseObject (objectId)
    Locator-->>V3: success (or StagehandClickError)

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

Resolve reduce-logs.ts conflict by taking main's version, which supersedes our lint-only fix and already validates untrusted CDP payload shapes (Array.isArray on args, typeof checks) — addressing the review comment.

The existing tap test asserted only that the target activated, which Chromium synthesizes a click for on both paths — a regression to mouse still passed it (verified by temporarily routing locator.tap through click: the old test passed, the new ones fail on pointerdown:mouse). - integration: assert pointerdown:touch / touchstart / touchend delivery and ordering for both locator.tap and page.tap, and that pointerdown:mouse is absent. - unit: CUA handler click routing — tap on touch sessions, click on desktop, right/middle/multi-click stay on the mouse path, and tap-vs-click recording.

V3Options gained a public field; references/stagehand.mdx mirrors that interface, so add it to both the interface block and the ParamField list.

mintlify · 2026-08-03T22:22:35Z

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
stagehand	🟢 Ready	View Preview	Aug 3, 2026, 10:24 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

Both tap docstrings claimed a touch-capable session is required. It isn't — CDP delivers a trusted touch sequence on a session reporting maxTouchPoints: 0, which the new integration tests demonstrate by asserting pointerdown:touch on a plain desktop local session. The claim would mislead callers into configuring touch emulation they don't need.

cubic-dev-ai

All reported issues were addressed across 5 files (changes from recent commits).

_{Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic}

usesTouch was resolved purely from creation config, which left two session kinds silently stuck on mouse: resumed by browserbaseSessionID (the API does not echo browserSettings back) and attached via cdpUrl. Both now probe Browser.getVersion once during init and derive touch from the UA string. Why Browser.getVersion and not a DOM signal, all verified on live sessions: - session-level, so no page / secure-context dependency — userAgentData is absent on about:blank and insecure origins, and stays false when a UA is set via launch flag; maxTouchPoints / pointer:coarse report 0 / false even on Browserbase os:"mobile" sessions - reflects the launch-level UA, which Browserbase mobile sessions carry and a local --user-agent override sets - one round trip (~0.2ms local / ~35ms remote), once per init Config-created sessions are untouched: config stays authoritative and the probe never runs (a hasTouch launch changes nothing UA-visible, so config catches what the probe provably cannot). Explicit useTouch still beats both. Verified end-to-end on live Browserbase: a verified os:"mobile" session resumed by id with zero touch config now resolves usesTouch=true; a resumed desktop session stays false.

cubic-dev-ai

All reported issues were addressed across 4 files (changes from recent commits).

_{Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic}

akeimach

lgtm

Drop both automatic resolutions — the config-derived signal (Browserbase browserSettings.os / local hasTouch) and the init-time Browser.getVersion UA probe for resumed/attached sessions. usesTouch is now exactly `opts.useTouch === true`. Simpler contract: touch actuation never turns on without the caller asking, there is no heuristic to misfire on custom user agents, and the option's behavior is identical across created, resumed, and cdpUrl-attached sessions.

cubic-dev-ai

All reported issues were addressed across 5 files (changes from recent commits).

_{Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic}

In non-experimental Browserbase mode, agent execution runs server-side via apiClient.agentExecute — but useTouch was resolved only on the client, so the server's V3 saw it undefined and actuated with mouse. That silently broke the headline scenario: a CUA agent on a mobile session without experimental: true. Wire it through the whole chain: SessionStartRequest schema → client init payload → server start route → session store → server-side V3Options. The schema is not strict, so a new client talking to an older server degrades exactly to today's behavior.

cubic-dev-ai

All reported issues were addressed across 6 files (changes from recent commits).

_{Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic}

- Add a start-route integration test passing useTouch, following the extended-options pattern (repo rule: REST API surface changes need a test under the server integration suite). - Replace em dashes in the useTouch docs prose per the docs prose guide.

@miguelg719

This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to v3, this PR will be updated. # Releases ## @browserbasehq/stagehand@3.7.2 ### Patch Changes - [#2775](#2775) [`b954a46`](b954a46) Thanks [@miguelg719](https://github.com/miguelg719)! - Fix Anthropic structured output options for provider-prefixed models - [#2767](#2767) [`98fa7a4`](98fa7a4) Thanks [@miguelg719](https://github.com/miguelg719)! - Fix: Normalize CUA coordinates to actual viewport - [#2385](#2385) [`7566804`](7566804) Thanks [@miguelg719](https://github.com/miguelg719)! - Add `useTouch` option to actuate agent clicks as trusted touch ## @browserbasehq/stagehand-evals@2.1.1 ### Patch Changes - Updated dependencies \[[`b954a46`](b954a46), [`98fa7a4`](98fa7a4), [`7566804`](7566804)]: - @browserbasehq/stagehand@3.7.2 ## @browserbasehq/stagehand-server-v3@3.7.4 ### Patch Changes - Updated dependencies \[[`b954a46`](b954a46), [`98fa7a4`](98fa7a4), [`7566804`](7566804)]: - @browserbasehq/stagehand@3.7.2 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

alonle and others added 3 commits July 23, 2026 10:34

merge: PR #2384 — actuate CUA clicks as touch on mobile sessions

c668d44

miguelg719 marked this pull request as draft July 23, 2026 08:05

miguelg719 commented Jul 23, 2026

View reviewed changes

Comment thread .changeset/touch-tap-actuation.md Outdated

Apply suggestion from @miguelg719

ae0cb94

cubic-dev-ai Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread packages/core/lib/v3/handlers/v3CuaAgentHandler.ts

Comment thread packages/core/lib/v3/understudy/page.ts

Comment thread packages/core/lib/v3/handlers/v3CuaAgentHandler.ts

miguelg719 added 6 commits July 23, 2026 16:32

chore: retrigger CI

c77ad1e

docs: restore terse changeset

c8c8fa3

miguelg719 marked this pull request as ready for review July 28, 2026 22:34

cubic-dev-ai Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread packages/core/lib/v3/understudy/locator.ts

Comment thread packages/cli/src/lib/cloud/reduce-logs.ts Outdated

Comment thread packages/core/lib/v3/understudy/locator.ts

miguelg719 added 3 commits August 3, 2026 15:15

merge: main into contrib/2384

670a1fc

Resolve reduce-logs.ts conflict by taking main's version, which supersedes our lint-only fix and already validates untrusted CDP payload shapes (Array.isArray on args, typeof checks) — addressing the review comment.

docs: document the useTouch option

2766a14

V3Options gained a public field; references/stagehand.mdx mirrors that interface, so add it to both the interface block and the ParamField list.

miguelg719 requested a review from a team as a code owner August 3, 2026 22:22

mintlify Bot deployed to staging - packages/docs August 3, 2026 22:24 View deployment

cubic-dev-ai Bot reviewed Aug 3, 2026

View reviewed changes

Comment thread packages/core/tests/integration/perform-understudy-method.spec.ts

mintlify Bot deployed to staging - packages/docs August 3, 2026 23:30 View deployment

cubic-dev-ai Bot reviewed Aug 3, 2026

View reviewed changes

Comment thread packages/core/lib/v3/v3.ts Outdated

akeimach approved these changes Aug 4, 2026

View reviewed changes

mintlify Bot deployed to staging - packages/docs August 4, 2026 16:44 View deployment

miguelg719 commented Aug 4, 2026

View reviewed changes

Comment thread .changeset/touch-tap-actuation.md Outdated

Apply suggestion from @miguelg719

c4eb908

cubic-dev-ai Bot reviewed Aug 4, 2026

View reviewed changes

Comment thread packages/docs/v3/references/stagehand.mdx Outdated

cubic-dev-ai Bot reviewed Aug 4, 2026

View reviewed changes

Comment thread packages/server-v3/src/routes/v1/sessions/start.ts

mintlify Bot deployed to staging - packages/docs August 4, 2026 17:09 View deployment

miguelg719 merged commit 7566804 into main Aug 4, 2026
251 checks passed

This was referenced Aug 4, 2026

Release Stagehand packages #2576

Merged

Release Stagehand packages CloudEngineHub/stagehand#1

Open

Release Stagehand packages edisplay/stagehand#5

Open

Version Packages MaxiBracamonte/stagehand#1

Open

Version Packages #2662

Merged

miguelg719 mentioned this pull request Aug 9, 2026

chore: reconcile main into v4-spike ahead of the squash-merge #2664

Merged

miguelg719 mentioned this pull request Aug 31, 2026

feat(agent): actuate CUA clicks as touch on mobile sessions #2384

Closed

Uh oh!

Conversation

miguelg719 commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

why

what changed

test plan

Uh oh!

changeset-bot Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mintlify Bot commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

akeimach left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

miguelg719 commented Jul 23, 2026 •

edited

Loading

changeset-bot Bot commented Jul 23, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

mintlify Bot commented Aug 3, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading