Skip to content

fix(find): enforce offscreen and non-hittable guarantees on focus - #4954

Merged
thymikee merged 2 commits into
callstack:mainfrom
samanyudas:fix/3734-find-focus-interaction-guarantees
Nov 22, 2026
Merged

thymikee merged 2 commits into
callstack:mainfrom
samanyudas:fix/3734-find-focus-interaction-guarantees

Conversation

@samanyudas

@samanyudas samanyudas commented Nov 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Close the offscreen and nonHittable waivers on find-focus-dispatch in packages/contracts/src/interaction-guarantees.ts. Refs #3734.

  • Move throwIfOffscreenTarget and describeNonHittableTarget into @agent-device/selectors so command-runtime and daemon share one rule.
  • Refuse off-screen find focus and find type targets; limit iOS rescue to one exact id or label probe match and mark rescued targets hittable: true.
  • Expose targetHittable: false and hint on find responses, FindCommandResponseData, CLI stderr warnings, and MCP text.
  • Keep targetStability waived on find-focus-dispatch pending an ADR 0039 decision.

Before / after

Before: find focus and find type dispatched off-screen points and reported silent success on non-hittable targets.

After: off-screen targets fail with a find-specific retry hint, non-hittable targets include targetHittable: false and a warning hint, and iOS rescue verifies exact target identity.

Validation

  • Head c97e5b99a: pnpm check:affected --run passed 22/23 gates (full unit, format, lint, typecheck, layering, fallow, build, package, vitest-related) plus all 20 other local gates; depgraph exceeded its 5 s CPU budget locally on both branch and unchanged base under host load.
  • Red/green proof: 3 scenarios fail on origin/main and pass on c97e5b99a.

Reviewed with sol-loop: 4 rounds with gpt-6.2-sol. Sol verdict: approved.
Created with Claude Opus 5.5 and Gemini 4 Neon in T3 Code.


Start Proof
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

View guided diff

samanyudas and others added 2 commits November 22, 2026 19:12
…ation

Move the off-screen refusal core and its selector and @ref refusal
shapes into @agent-device/selectors/interaction-offscreen, and move
describeNonHittableTarget into interaction-targeting. The daemon cannot
import command-runtime, so this lets the find focus seam apply the same
rules as the acting paths. throwIfOffscreenInteractionTarget stays the
runtime paths' enforcement point and supplies the backend rescue hook.

The thin assertVisible*Target wrappers fold into their three call sites,
which keeps command-runtime's eager closure at its merge-base weight.

Refs callstack#3734

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ones

find focus and the focus leg of find type dispatch their own point, so
they skipped two rules the acting paths run. A closed-drawer match was
focused at coordinates outside the viewport and reported success, and a
target still reporting hittable: false after promotion answered without
the targetHittable/hint annotation press, click and fill give.

The seam now runs the shared off-screen rule on the node it would focus,
with the offscreen_ref refusal shape and the iOS live-read rescue (the
find-focus and find-type uses declare querySelector as conditional, like
the captured acting uses). find's ranking still prefers an on-screen
twin first. Both responses carry the shared non-hittable annotation.

This closes the offscreen and nonHittable cells of the
find-focus-dispatch row. targetStability stays waived: ADR 0039 keeps one
admission loop in command runtime and keeps find's carried targets out
of it, so closing it needs an ADR decision, which the waiver now states.

Refs callstack#3734

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thymikee

Copy link
Copy Markdown
Member

Thanks for the PR.

The code change looks right at c97e5b9, but one live check is still missing before merge. The find focus/type path now refuses before dispatch and, on iOS, sends a new ios.runner.querySelector hittability probe (find.ts:466). The PR body cites only check:affected and the contract harness, which uses scripted runner transcripts. Not verified here: the real runner's reply shape for a label probe (matchCount, rect, hittable), and the refusal on a real closed drawer. A mismatch would make every stale-tree rescue fail closed or refuse visible targets. Please record a live iOS simulator run on this head, per docs/agents/device-verification.md, with node bin/agent-device.mjs. It should show three things. First, find label <on-screen field> focus and find label <field> type hello are unchanged: Focused (x, y), text typed, no Warning line. Second, a target in a closed drawer or off-screen list, run with --raw, returns COMMAND_FAILED with details.reason offscreen_ref, dispatched no, the hint 're-run find to focus|type', and no focus change. Third, a target whose node reports hittable:false gives response JSON with targetHittable:false and a hint, and the CLI prints the Warning line on stderr. Please include the device, the SHA and the command output.

Could the rescue be simpler? Keep the selectors move and the throwIfOffscreenTarget/describeNonHittableTarget calls in find.ts. Then give find the same rescue the acting route builds: confirmIosOffscreenTargetVisible over deriveDirectIosNodeSelector(node) (direct-ios-selector.ts:36) with the lean hittability probe and matchCount === 1 for all callers. That deletes directIdentityFindSelector, the exactSelector branch and the readSimpleSelectorTarget import. It also gives role and text locators the same rescue that press gets. This needs agreement first that the tap-route rescue should also require a unique live match, which tightens #1542 for press, click and fill.

Not blocking, and fine to take or leave: find.ts:513 keeps a second rescue policy for #1542 beside the tap rescue. collectResponseWarnings (output-common.ts:82) is shared, so press/click/fill now print a Warning line for targetHittable:false and duplicate warnings collapse for every command, but the PR body and tests cover only find. The shared rule in interaction-offscreen.ts:46 now patches hittable: true onto every rescued node, so the tap route no longer annotates a rescued node that the bulk tree marked non-hittable. dispatchFocusForFindMatch (find.ts:468) builds both refusal and annotation with 'focus', so find type says 'not safe to focus', and passing action to both helpers would fix that.

The two Body rules runs that are not passing were cancelled as superseded, and that job checks PR body text, which this diff does not touch, so the checks look unrelated to the change. Tests and check:affected were not run here, so red/green was judged from the pre-change seam code. Whether the stale-tree false refusal can reach text and role locators depends on which capture modes keep off-screen nodes, and that was not traced exhaustively. No conflicts.

@samanyudas

Copy link
Copy Markdown
Contributor Author

Note

🤖 Gemini 4 Neon responding on behalf of samanyudas

Recorded the live iOS Simulator verification on c97e5b99ab4e79cd49a148a978bb4890938c5e38 using node bin/agent-device.mjs with an isolated --state-dir on iPhone 17 (iOS 27.0 (24A5423a), UDID A07FBEDE-5C0D-401A-9574-CCFE8679F93B):

1. On-screen field (focus + type) — com.apple.mobilesafari (TextField label: "Address")

$ node bin/agent-device.mjs find label "Address" focus --session verify-3734 --platform ios --udid A07FBEDE-5C0D-401A-9574-CCFE8679F93B --state-dir "$STATE_DIR"
# exit 0, stderr empty (no Warning line)
Focused (203, 816)

$ node bin/agent-device.mjs find label "Address" type hello --session verify-3734 --platform ios --udid A07FBEDE-5C0D-401A-9574-CCFE8679F93B --state-dir "$STATE_DIR"
# exit 0, stderr empty (no Warning line)
Typed 5 chars

Post-type snapshot --json confirmed "label": "Address", "value": "hello", "focused": true.

2. Off-screen target (--raw) + live querySelector probe — com.apple.MobileAddressBook (New Contact, Button label: "Insert add birthday", y: 1051.67 below viewport y: 62, height: 812)

$ node bin/agent-device.mjs find label "Insert add birthday" focus --raw --json --session verify-3734 --platform ios --udid A07FBEDE-5C0D-401A-9574-CCFE8679F93B --state-dir "$STATE_DIR"
# exit 1, stderr empty
{
  "success": false,
  "error": {
    "code": "COMMAND_FAILED",
    "message": "Ref @e80 is off-screen and not safe to focus",
    "hint": "Scroll down toward it or open its container, then re-run find to focus.",
    "details": {
      "reason": "offscreen_ref",
      "ref": "e80",
      "rect": { "x": 39.166666666666664, "y": 1051.6666647593179, "width": 25.666666666666664, "height": 25.666666666666742 },
      "viewport": { "x": 0, "y": 62, "width": 402, "height": 812 },
      "scrollDirection": "down",
      "dispatched": "no"
    }
  }
}

Focus remained on @e27 (First name) before and after. Request diagnostics and runner.log confirmed the real runner accepted and completed command=querySelector (Elements matching predicate 'label ==[c] "Insert add birthday"', ok=1, 310 ms) and confirmIosOffscreenTargetVisible declined rescue.

3. Non-hittable on-screen target (hittable: false) — com.apple.MobileAddressBook (Button label: "Done", hittable: false)

$ node bin/agent-device.mjs find label "Done" focus --json --session verify-3734 --platform ios --udid A07FBEDE-5C0D-401A-9574-CCFE8679F93B --state-dir "$STATE_DIR"
# exit 0, stderr empty
{
  "success": true,
  "data": {
    "x": 364,
    "y": 100,
    "message": "Focused (364, 100)",
    "targetHittable": false,
    "hint": "The resolved element reports hittable: false, so this focus may have had no visible effect. Verify with a snapshot, or prefer a @ref or a longer/more specific selector to target the intended element."
  }
}

$ node bin/agent-device.mjs find label "Done" focus --session verify-3734 --platform ios --udid A07FBEDE-5C0D-401A-9574-CCFE8679F93B --state-dir "$STATE_DIR"
# exit 0
# stdout:
Focused (364, 100)
# stderr:
Warning: The resolved element reports hittable: false, so this focus may have had no visible effect. Verify with a snapshot, or prefer a @ref or a longer/more specific selector to target the intended element.

(Independently reproduced on com.apple.DocumentsApp Files Button "Forward" with identical targetHittable: false, hint, and stderr Warning: behavior. Session closed with --shutdown and daemon stopped with --clean.)

Rescue simplification question

Happy to fold find onto the shared confirmIosOffscreenTargetVisible over deriveDirectIosNodeSelector(node) with the lean hittability probe + matchCount === 1 (and delete directIdentityFindSelector / exactSelector / readSimpleSelectorTarget) if you want to tighten #1542 for press/click/fill in this PR; otherwise leaving the separate exactSelector path intact preserves existing tap-route behavior as-is. Also note InteractionAction does not currently include 'type', which is why dispatchFocusForFindMatch passed 'focus' — happy to adjust if you want 'type' added or prefer keeping 'focus'.

@thymikee

Copy link
Copy Markdown
Member

The live iOS run on c97e5b9 covers what the review asked for: on-screen focus and type are unchanged with no Warning line; the off-screen target is refused with offscreen_ref, dispatched: no and the re-run hint, and focus does not move; the hittable: false target returns targetHittable: false with the hint and prints the stderr Warning. The runner log confirms the real querySelector probe. The simpler rescue would also tighten the tap route for press, click and fill, so keeping the current path in this PR is reasonable; that change can be decided separately. Nothing else blocks this head.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 22, 2026
@thymikee
thymikee merged commit 050feba into callstack:main Nov 22, 2026
19 of 21 checks passed
thymikee added a commit that referenced this pull request Nov 22, 2026
* origin/main:
  fix(auth): send a stored CLI session only to the cloud that issued it (#4921)
  refactor(reasons): declare the daemon reasons in ReasonTable rows (#4936)
  fix(cli): drop --remote-config from auth and skip reading it for unsupported commands (#4957)
  refactor(snapshot): deduplicate system-surface projections and warnings (#4959)
  fix(find): enforce offscreen and non-hittable guarantees on focus (#4954)
  feat(auth): store a rotated CLI session credential and show the login code (#4833)
  fix(daemon): report Windows SIGTERM daemon exits as forced (#4659)

# Conflicts:
#	packages/capture-kit/src/snapshot-quality-verdict.ts
#	packages/command-runtime/src/commands/interaction/runtime/target-visibility-stages.ts
#	packages/contracts/src/device-inventory.ts
#	packages/daemon/src/execution/interaction/internal/__tests__/find.test.ts
#	packages/daemon/src/execution/interaction/internal/find-target-capture.ts
#	packages/replay-port/src/daemon-port/session-replay-runtime-failure-response.ts
#	website/docs/docs/error-reasons.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants