Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions skills/agent-device/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ agent-device help remote
agent-device help macos
agent-device help dogfood
agent-device help tv
agent-device help ios-system-ui # iOS SpringBoard, widgets, and system-UI surfaces
```

Default loop: `open -> snapshot/-i -> get/is/find or press/fill/scroll/wait -> verify -> close`. When target-specific help says capture or selectors are unsupported, use its control-only loop and the device display as visual truth.
Expand Down
15 changes: 15 additions & 0 deletions src/cli/parser/__tests__/cli-help-topics.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -502,6 +502,21 @@ test('usageForCommand resolves physical-device help topic', async () => {
);
});

test('usageForCommand resolves ios-system-ui help topic', async () => {
const help = await usageForCommand('ios-system-ui');
if (help === null) throw new Error('Expected ios-system-ui help text');
assert.match(help, /agent-device help ios-system-ui/);
assert.match(help, /agent-device open com\.apple\.springboard --platform ios/);
assert.match(help, /longpress <x> <y> on an empty area of the home screen/);
assert.match(help, /discover them from the current snapshot/);
assert.match(
help,
/verified on iOS simulator; physical-iPhone SpringBoard support is not yet verified/,
);
assert.match(help, /Do not hard-code Edit\/Done\/Add Widget or other SpringBoard label text/);
assert.match(help, /Reopen the app bundle under test/);
});

test('usageForCommand resolves manual QA help topic', async () => {
const help = await usageForCommand('manual-qa');
if (help === null) throw new Error('Expected manual QA help text');
Expand Down
37 changes: 35 additions & 2 deletions src/cli/parser/cli-help.ts
Original file line number Diff line number Diff line change
Expand Up @@ -438,7 +438,8 @@ Escalate:
help react-devtools React Native performance, profiling, props/state/hooks, slow renders, rerenders
help react-native React Native app automation hazards, overlays, Metro/Re.Pack, and routing
help remote remote/cloud config, tenant, lease, local service tunnels
help macos desktop, frontmost-app, menu bar surfaces`,
help macos desktop, frontmost-app, menu bar surfaces
help ios-system-ui iOS SpringBoard, widget add/edit/remove, system-UI surfaces`,
},
tv: {
summary: 'Android TV, tvOS, and Vega VVD focus-first remote navigation',
Expand Down Expand Up @@ -842,7 +843,39 @@ iOS physical-device prerequisites:

Android physical-device prerequisites:
Enable USB debugging and confirm the device appears in agent-device devices --platform android.
Android does not need the iOS runner signing setup. For React Native/Expo Metro reachability, read help react-native.`,
Android does not need the iOS runner signing setup. For React Native/Expo Metro reachability, read help react-native.

For iOS SpringBoard, widget, or other system-UI surfaces, read agent-device help ios-system-ui.`,
},
'ios-system-ui': {
summary: 'iOS SpringBoard, widget, and system-surface workflow',
body: `agent-device help ios-system-ui

Use this when a task needs iOS SpringBoard (home screen), widget add/edit/remove, or other system-UI surfaces instead of the app under test.

This works today by opening SpringBoard as the session app; there is no separate widget/system command. System labels vary by iOS version and locale, so discover them from the current snapshot instead of relying on the literal strings shown below. This workflow is verified on iOS simulator; physical-iPhone SpringBoard support is not yet verified.

Core loop:
1. Reach the app state you want to prepare (for example, arrange the widget/Live Activity data the app should show) with normal app automation, then agent-device open com.apple.springboard --platform ios. From an existing app session, agent-device home first also lands on the home screen, but open com.apple.springboard is what actually binds the session to SpringBoard for selector-driven commands.
2. agent-device snapshot -i to read the current localized SpringBoard controls.
3. agent-device longpress <x> <y> on an empty area of the home screen to enter edit mode. This is the one deliberate coordinate step; there is no reliable non-coordinate way to trigger it.
4. Re-snapshot and use selectors from the fresh tree to drive the Edit menu -> widget gallery -> search -> size picker -> Add Widget, for example:
agent-device snapshot -i
agent-device press 'label="Add Widget"'
agent-device fill 'label="Search"' "Calendar"
5. The widget-gallery search-result rows currently fall back to unlabeled nodes (a known capture gap), so tap the result by coordinates read from a screenshot until that is fixed:
agent-device screenshot
agent-device press <x> <y>
6. Continue with the semantic size picker and agent-device press 'label="Add Widget"' to place it.
7. To edit or remove an installed widget, longpress it, then re-snapshot and use the fresh context-menu selectors (Edit Widget / Remove Widget).
8. Use screenshots for visual assertions, and as the fallback wherever a system surface exposes sparse accessibility, not only in the gallery step.
9. Reopen the app bundle under test (agent-device open <app-id> --platform ios) to return to normal app automation; leaving SpringBoard bound does not resume the app session on its own.

Rules:
Do not hard-code Edit/Done/Add Widget or other SpringBoard label text into a plan as a fixed assumption; take them from the latest snapshot -i so the plan survives iOS version/locale differences.
A real system permission alert can appear mid-flow; it composes with this workflow normally, so handle it with alert wait/accept/dismiss or by pressing the visible label like any other step.
Prefer refs/selectors from the fresh snapshot for every step except the two documented coordinate fallbacks (empty-space long-press to enter edit mode, and the gallery search-result tap).
This topic covers what already works by opening SpringBoard as the session app. It does not yet cover keeping an app session open while alternating individual commands against SpringBoard, or Live Activity/Dynamic Island semantics; those land separately.`,
},
remote: {
summary: 'Direct proxy, cloud profiles, and remote config',
Expand Down
10 changes: 10 additions & 0 deletions website/docs/docs/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ agent-device help remote
agent-device help web
agent-device help macos
agent-device help dogfood
agent-device help ios-system-ui
```

Skills are recommended for auto-routing when your agent runtime supports them, but they are not required. The CLI help topics are the version-matched operating contract.
Expand Down Expand Up @@ -1037,3 +1038,12 @@ For CLI-discoverable setup guidance, run `agent-device help physical-device`.
- If you override the iOS runner derived-data path and also force cleanup, keep `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH` under the project `.tmp/` directory. Other cleanup override paths are rejected with a recovery hint.
- For daemon startup troubleshooting:
- follow stale metadata hints for `<state-dir>/daemon.json` and `<state-dir>/daemon.lock` (`state-dir` defaults to `~/.agent-device` for packaged installs, or a worktree-scoped dir under `~/.agent-device/dev/` from source)

## iOS SpringBoard, widgets, and system-UI surfaces

For CLI-discoverable workflow guidance, run `agent-device help ios-system-ui`.

- `agent-device open com.apple.springboard --platform ios` binds the session to SpringBoard today; this is verified on iOS simulator only. Physical-iPhone SpringBoard support is not yet verified — see [#1296](https://github.com/callstack/agent-device/issues/1296).
- The full widget add/edit/remove flow is selector-driven from a fresh `snapshot -i`, except two coordinate-based steps: the empty-space long-press that enters edit mode, and (until fixed) the widget-gallery search-result rows, which currently return unlabeled accessibility nodes.
- SpringBoard labels vary by iOS version and locale; discover them from the current snapshot rather than hard-coding strings like `Edit` or `Add Widget`.
- Reopen the app bundle under test to return to normal app automation after a SpringBoard step.
Loading