feat(agent): actuate CUA clicks as touch on mobile sessions - #2384
Conversation
|
|
This PR is from an external contributor and must be approved by a stagehand team member with write access before CI can run. |
github-actions
Bot
added
external-contributor
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
1 issue found across 3 files
Confidence score: 4/5
- In
packages/core/lib/v3/understudy/page.ts, the newPage.tap()plus mobile detection/routing inV3CuaAgentHandlerintroduce complex CDP touch dispatch and conditional action paths, so mismatches in capability detection could route interactions incorrectly and cause flaky or broken mobile actions; add focused coverage for touch dispatch and both routing branches to de-risk regressions.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/core/lib/v3/understudy/page.ts">
<violation number="1" location="packages/core/lib/v3/understudy/page.ts:1908">
P2: The new `tap()` method on `Page` and the mobile detection/routing logic in `V3CuaAgentHandler` add non-trivial behavior (CDP touch event dispatching, per-run capability detection, conditional action routing) without accompanying unit tests. Consider adding focused tests that cover: the `tap()` method's CDP touch event dispatch (happy path), the `usesTouch` detection with `userAgentData.mobile` (true/false), and the graceful catch path when `userAgentData` is unavailable.</violation>
</file>
Architecture diagram
sequenceDiagram
participant CUA as CUA Agent
participant Handler as V3CuaAgentHandler
participant Page as Understudy Page
participant Browser as CDP Browser
participant Site as Touch-Gated Site
Note over CUA,Site: Click Action Flow
CUA->>Handler: click(x, y)
Handler->>Handler: Check isMobile cache
alt usesTouch === undefined (first click)
Handler->>Page: evaluate(navigator.userAgentData.mobile)
Page->>Browser: Runtime.evaluate
Browser-->>Page: boolean result
Page-->>Handler: mobile flag
Handler->>Handler: Cache in this.usesTouch
end
alt usesTouch === true (mobile session)
Handler->>Page: tap(x, y)
Page->>Browser: Input.dispatchTouchEvent(type: touchStart, touchPoints: [{x, y}])
Page->>Browser: Input.dispatchTouchEvent(type: touchEnd, touchPoints: [])
Browser-->>Page: Success
Page-->>Handler: void
Handler-->>CUA: { success: true }
Note over Browser,Site: Site registers touch/pointer events<br/>Size selector responds correctly
else usesTouch === false (desktop session)
Handler->>Page: click(x, y, {button, clickCount})
alt recording enabled
Page->>Page: Resolve xpath
Page->>Browser: Input.dispatchMouseEvent
Browser-->>Page: Success + xpath
Page-->>Handler: xpath
else recording disabled
Page->>Browser: Input.dispatchMouseEvent
Browser-->>Page: Success
Page-->>Handler: void
end
Handler-->>CUA: { success: true }
end
Note over Handler: Detection fallback<br/>If evaluate fails → usesTouch = false<br/>(safe default for desktop)
Tip: cubic used a learning from your PR history. Let your coding agent read cubic learnings directly with the cubic MCP.
Re-trigger cubic
| * never registers — respond correctly. Coordinates are relative to the viewport | ||
| * origin (top-left). Does not scroll. Requires a touch-capable (e.g. mobile) session. | ||
| */ | ||
| @FlowLogger.wrapWithLogging({ eventType: "PageTap" }) |
There was a problem hiding this comment.
P2: The new tap() method on Page and the mobile detection/routing logic in V3CuaAgentHandler add non-trivial behavior (CDP touch event dispatching, per-run capability detection, conditional action routing) without accompanying unit tests. Consider adding focused tests that cover: the tap() method's CDP touch event dispatch (happy path), the usesTouch detection with userAgentData.mobile (true/false), and the graceful catch path when userAgentData is unavailable.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/core/lib/v3/understudy/page.ts, line 1908:
<comment>The new `tap()` method on `Page` and the mobile detection/routing logic in `V3CuaAgentHandler` add non-trivial behavior (CDP touch event dispatching, per-run capability detection, conditional action routing) without accompanying unit tests. Consider adding focused tests that cover: the `tap()` method's CDP touch event dispatch (happy path), the `usesTouch` detection with `userAgentData.mobile` (true/false), and the graceful catch path when `userAgentData` is unavailable.</comment>
<file context>
@@ -1897,6 +1897,26 @@ export class Page {
+ * never registers — respond correctly. Coordinates are relative to the viewport
+ * origin (top-left). Does not scroll. Requires a touch-capable (e.g. mobile) session.
+ */
+ @FlowLogger.wrapWithLogging({ eventType: "PageTap" })
+ async tap(x: number, y: number): Promise<void> {
+ await this.mainSession.send<never>("Input.dispatchTouchEvent", {
</file context>
There was a problem hiding this comment.
Added an integration test for the tap actuation path in perform-understudy-method.spec.ts (drives performUnderstudyMethod("tap", …) → locator.tap end to end), following the repo's existing pattern for pointer actions — click/fill/drag are covered the same way (browser-backed integration, run against the built dist). There's no in-repo unit harness that mocks a CDP session to assert Input.* payloads, so I didn't introduce net-new mocking infra for this.
|
Thanks for the review — all four points addressed in the latest commit. 1. Right/middle/multi-click routed to tap (v3CuaAgentHandler) — fixed. Only a single left click now routes to 2. Recording flow bypassed on mobile (v3CuaAgentHandler) — fixed. On a replay-recording run the tap is now recorded as a deterministic step: 3. Cursor overlay stale after tap (page.ts) — fixed. 4. Missing tests (page.ts) — added an integration test for One transparency note on #4: I couldn't run the browser-backed integration suite in my environment (Playwright browsers not installed; the suite runs against a built Open questions from the PR description still stand if you'd prefer a different shape: routing |
c20830d to
ca253d4
Compare
There was a problem hiding this comment.
1 issue found across 6 files (changes from recent commits).
Confidence score: 5/5
- In
packages/core/lib/v3/understudy/page.ts, mobile replay now relies ontapreturning a hit XPath, and without focused coverage this contract could drift and cause touch replays to target the wrong element or fail silently on mobile flows — add tap tests that validate touch dispatch and both explicit and default XPath return paths.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/core/lib/v3/understudy/page.ts">
<violation number="1" location="packages/core/lib/v3/understudy/page.ts:1912">
P3: Mobile replay now depends on `tap` returning the hit XPath, but this new contract has no coverage. Add focused tap tests for touch dispatch plus both requested and default XPath return values to catch replay regressions.
(Based on your team's feedback about unit tests for new behavior.) [FEEDBACK_USED]</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| async tap( | ||
| x: number, | ||
| y: number, | ||
| options?: { returnXpath?: boolean }, |
There was a problem hiding this comment.
P3: Mobile replay now depends on tap returning the hit XPath, but this new contract has no coverage. Add focused tap tests for touch dispatch plus both requested and default XPath return values to catch replay regressions.
(Based on your team's feedback about unit tests for new behavior.)
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/core/lib/v3/understudy/page.ts, line 1912:
<comment>Mobile replay now depends on `tap` returning the hit XPath, but this new contract has no coverage. Add focused tap tests for touch dispatch plus both requested and default XPath return values to catch replay regressions.
(Based on your team's feedback about unit tests for new behavior.) </comment>
<file context>
@@ -1906,7 +1906,28 @@ export class Page {
+ async tap(
+ x: number,
+ y: number,
+ options?: { returnXpath?: boolean },
+ ): Promise<string> {
+ let xpathResult: string | undefined;
</file context>
There was a problem hiding this comment.
The tap actuation is covered by the new integration test in perform-understudy-method.spec.ts. The returnXpath contract on page.tap mirrors page.click's existing (also integration-tested) returnXpath path. Happy to add a focused case asserting the returned-vs-empty XPath if you'd prefer it called out explicitly.
1762ac3 to
fb58929
Compare
- locator.tap / page.tap: dispatch touchStart+touchEnd via Promise.all (like click) so a slow round trip on a remote session can't stretch a tap into a long press. - tapElement: throw UnderstudyCommandException (like doubleClick/dragAndDrop) instead of StagehandClickError, so a tap failure isn't mislabeled as a click. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
8a552b1 to
8d8f463
Compare
# why The CUA and hybrid agents dispatch every pointer action as mouse input. Mobile layouts commonly gate their handlers on touch/pointer events, so a synthesized mouse click never registers — e.g. on `adidas.co.il` (Browserbase `os: "mobile"`) a size selector highlights but keeps reporting "please choose a size", blocking add-to-cart. A trusted touch (`Input.dispatchTouchEvent`) works. Originates from #2384 by @alonle. # what changed - New `useTouch` option (explicit opt-in, default `false`). Forwarded to the server on the API execution path. - `page.tap(x, y)` and `locator.tap()`: trusted touch taps mirroring their click counterparts. - When `useTouch` is set: CUA routes single left clicks to tap (right/middle/multi-click, hover, and drag stay on mouse); hybrid `click`/`type`/`fillFormVision` tap for coordinate focus/activation. - Replay: taps record as `tap` steps and replay as touch. `tap` is not exposed to the LLM during live inference. # test plan - Unit: `touch-actuation` (7), `touch-tool-routing` (9), `touch-cua-routing` (7) — resolution, tool routing, recording. - E2E: `perform-understudy-method` (8) — asserts `pointerdown:touch`/`touchstart`/`touchend` delivery for both tap paths (event-level, since Chromium synthesizes a click from a tap and activation-only assertions can't tell touch from mouse). - Server: `start.test.ts` covers `useTouch` on `/sessions/start`. - Manual: with `useTouch: true` on a Browserbase `os: "mobile"` session, the agent completes the adidas.co.il size + add-to-cart flow. --------- Co-authored-by: alonl <alonl@wix.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
merged and released per #2385 |
2 participants
Problem
The computer-use agent actuates every pointer action through
Input.dispatchMouseEvent(viapage.click(x, y)inV3CuaAgentHandler). On a mobile session the browser renders the site's touch-gated mobile layout, whose handlers listen for touch/pointer events (pointerType: "touch") — a synthesized mouse click doesn't register there.Concrete repro on
adidas.co.il(BrowserbasebrowserSettings.verified: true, os: "mobile"), same product/size, empty cart:Input.dispatchTouchEvent)Input.dispatchMouseEvent)The size visually highlights but is never registered, so add-to-cart fails. This blocks mobile e-commerce flows (size selectors, add-to-cart) on any touch-gated layout.
Fix
page.tap(x, y)to the understudyPage— a trusted touch tap viaInput.dispatchTouchEvent, mirroring the existing coordinatepage.click.V3CuaAgentHandler, route theclickaction topage.tapwhen the session presents as mobile, detected once per run vianavigator.userAgentData.mobile. Desktop sessions are unchanged (stillpage.click).Detection note: Browserbase
os: "mobile"reportsmaxTouchPoints: 0andpointer: fine, butnavigator.userAgentData.mobile === true— souserAgentData.mobileis the reliable signal (verified on live sessions). The trusted CDP touch works even though the context doesn't advertise touch.Scope / open questions for maintainers
clickaction is routed.double_click(and drag) could get the same treatment if desired.userAgentData.mobile). Happy to switch to an explicit config flag (e.g.agentConfigoption) if you'd prefer opt-in over auto-detect.page.click(returnXpath)), so a tapped step isn't captured for replay. Can extendpage.tapto resolve+return the xpath if that matters.Test plan
pnpm --filter @browserbasehq/stagehand typecheck— cleanprettier --check/eslinton changed files — cleanverified + os:"mobile"Browserbase session, the CUA agent selects a size and adds to cart on adidas.co.il (previously blocked with mouse).🤖 Generated with Claude Code
Summary by cubic
Actuate CUA clicks as real touch on mobile sessions so touch-gated UIs register correctly. Routes a single left
clickto a trusted tap and records it for replay; other click types are unchanged.New Features
page.tap(x, y, { returnXpath })andlocator.tap(); registeredtapin the act method map.tapupdates the cursor overlay and returns XPath for deterministic replay.navigator.userAgentData.mobile, cached), only a single left click (count = 1) becomestap; right/middle/multi-clicks stay mouse. Recording stores atapstep with XPath.Bug Fixes
touchStart+touchEndviaPromise.allto avoid long-press on slow sessions.taperrors now throwUnderstudyCommandException(notStagehandClickError) for correct classification.Written for commit 8d8f463. Summary will update on new commits.