test(ios): expand simulator e2e coverage

[thymikee]

Summary

• add a registry-derived, exhaustive coverage manifest for all 53 public commands and map them to meaningful simulator scenarios
• add smoke and full iOS simulator fixture journeys covering lifecycle, interaction, replay, diagnostics, permissions, media, and state restoration
• reuse the fingerprinted native test-app artifact and unchanged XCUITest runner cache, with bounded producer waiting, safe inline fallback, and no queued-producer stall
• make simulator orientation verification converge within a bounded 2s window, and tolerate a bounded 15s recordVideo destination startup under CI load
• replace viewport-tuned replay swipes with edge-aware traversal while retaining direct swipe evidence; reduce redundant deep-link snapshots and full-tier assertions
• avoid duplicate iOS native builds on fork PRs while preserving Android producer behavior
• fix trace activation without --debug, tighten scroll hint scope, and enforce real .app install-source coverage
• document the smoke/nightly split and native reuse contract

Scope: 63 touched files across the fixture app, CI workflows/actions, simulator E2E suites, docs, provider tests, recording/diagnostics, scroll hints, and Apple runner paths. This deliberately expands beyond the original E2E suite to make its live artifact and cache behavior executable.

Validation

• pnpm check
• iOS and macOS XCTest runner builds
• focused recording, orientation, and runner-provider regression suites (15 tests)
• static iOS simulator coverage contract (10 tests)
• fixture-cache fallback scenarios, including failed, absent, and queued producers
• workflow validation with actionlint
• latest full PR CI green before this follow-up

Tracks #320: the remaining live system permission-prompt accept/dismiss/recovery journey stays explicitly owned there; this PR does not claim to close it.
Tracks #1407

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://callstack.github.io/agent-device/pr-preview/pr-1408/
Built to branch gh-pages at 2026-07-28 11:14 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[github-actions]

Size Report

Metric	Base	Current	Diff
JS raw	1.87 MB	1.87 MB	+1.4 kB
JS gzip	598.7 kB	599.1 kB	+418 B
npm tarball	714.4 kB	715.1 kB	+715 B
npm unpacked	2.50 MB	2.50 MB	+3.1 kB

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	27.5 ms	26.3 ms	-1.2 ms
CLI --help	56.7 ms	55.6 ms	-1.1 ms

Top changed chunks:

Chunk	Raw diff	Gzip diff
dist/src/internal/daemon.js	+811 B	+237 B
dist/src/recording-export-quality.js	+207 B	+50 B
dist/src/sdk-android-adb.js	+35 B	+10 B
dist/src/runner-disposal.js	+17 B	+7 B
dist/src/runner-client.js	+4 B	+6 B

[thymikee]

Code review is clean at 00632a5: no actionable implementation or regression-test defect found. #320’s fixture-coverage intent is met without deleting the system-app coverage, #1407 remains explicitly owned as a live typed viewport known gap, and the catalog-derived command/behavior manifests point to executable scenario evidence. The production trace/diagnostics changes and fixture artifact reuse flow are coherent.

P2 owner action: the PR body must disclose the required touched-file count and scope expansion. This is a 38-file (+2,751/-51) change spanning fixture app, workflows, docs, E2E suites, and two production diagnostics/daemon paths; add that statement before merge.

Current-head required checks observed so far are passing, with iOS smoke, Coverage, FreeRange, Linux, and Android still in progress and no failures. The body records a full local simulator run (1/1, 186s); independent live iOS CI evidence is pending. Code review is clean, so ready-for-human may be applied while CI completes; if CI fails, remove it and address the failure.

[thymikee]

Re-reviewed 2cd3b7a after the latest CI-stabilization delta. No actionable code or test finding remains. The Android helper cache keys include the helper sources, packaging scripts, version, and the exact android/snapshot-helper/debug.keystore used to sign the IME helper. The iOS permission-reset restart/assertion is sound. All 27 checks pass, including the changed fixture E2E smoke and host-focus assertion, so the existing ready-for-human label remains justified.

P2 owner action remains and the numbers have expanded: update the PR body with the required scope disclosure for the current 41-file (+2,836/-78) diff spanning fixture app, workflows/actions, docs, E2E suites, and production diagnostics/daemon paths. This is required before merge.

[thymikee]

Re-reviewed 86759f1. P1 test-route failure: resetMicrophonePermissionAndRestart() now immediately reads id="automation-microphone-permission" after restoring the Automation screen, but that control is below the initial viewport. The existing clickMicrophonePermission() path deliberately handles offscreen_selector by scrolling; the new precondition assertion bypasses that recovery.

The full simulator run at this exact head fails in Replay Nightly run 30231730509, Run full fixture-backed iOS simulator E2E, with COMMAND_FAILED: Selector did not match: id="automation-microphone-permission" at live-full-scenarios.ts:170. This is a deterministic test-route defect, not infrastructure. Make the assertion after the control is brought into view, or give the reset helper equivalent bounded scroll/retry behavior, then rerun the full tier.

The 27 PR checks are green because they exercise the smoke tier; they do not cover this affected full:lifecycle-system path. ready-for-human must be removed until the full-tier failure is fixed. The PR-body scope disclosure also remains missing and now covers 41 files (+2,836/-78).

[thymikee]

Exact-head full-tier validation is still red on 0c937df (Replay Nightly run 30234556565). The run now fails earlier in smoke:automation-input: after the relaunch/deep-link step, the app remains on the Home/Gesture Lab surface and wait text "Automation lab" 10000 times out. Because the run never reaches full:lifecycle-system, the new inactive-app permission reset and scroll-aware assertion do not yet have passing live evidence, so the prior P1 cannot be confirmed closed. Please diagnose/rerun the full tier and require it to reach and pass the permission scenario. The 27 PR checks are green but cover the smoke tier only.

Also update the PR body with the required large-scope disclosure (currently 42 files, +2,876/-78).

[thymikee]

Re-reviewed 1f40a9a. The latest delta has no new code finding: re-delivering the trusted cold deep link directly addresses the earlier Home/Gesture Lab failure, and orientation now validates native XCUIDevice readback instead of assuming the app window must resize.

Readiness is still blocked on live evidence. There is no exact-head Replay Nightly run for 1f40a9a, so the full iOS route has not reached and passed the previously failing permission lifecycle. The ordinary PR checks cover smoke/static gates and cannot close that P1. Run the full Replay Nightly at this head and require full:lifecycle-system to pass before applying ready-for-human.

The PR body also still needs the required scope disclosure: 49 files, +2,973/-92, expanding from E2E coverage into Apple runner/core orientation contract behavior.

[thymikee]

P1 remains on exact head 1f40a9a. Replay Nightly run 30238489078 now gets past the deep-link failure and reaches full:lifecycle-system, so that earlier blocker is fixed. The required permission route still fails deterministically: state clear, microphone TCC reset, reopen, route restore, control discovery, and click id="automation-request-microphone" all succeed; then alert wait 15000 times out with no prompt. Runner/session diagnostics are healthy—no restart, transport, simulator-boot, or infrastructure failure—so increasing the timeout did not fix the route; the prompt is absent, not late.

Diagnose why the fixture request does not produce a system permission prompt, assert the app-visible permission result on the no-prompt path, and rerun the exact-head full tier through a passing permission denial/reset/recovery cycle. Keep ready-for-human withheld.

[thymikee]

P1: 4ca3c900 avoids the failing permission-prompt route instead of fixing the behavior required by linked issue #320. The delta deletes prompt wait/dismiss/accept, renames permission-prompt-recovery to permission-state-recovery, and substitutes direct settings permission reset/grant/deny/reset assertions. That is useful deterministic settings coverage, but the separate app-native alert scenario does not prove the requested system permission prompt accept/dismiss/recovery journey. Keep the state assertions, but restore executable permission-prompt recovery evidence or stop claiming this PR closes #320 and move that requirement to an explicitly owned follow-up.

No exact-head Replay Nightly or iOS smoke/full evidence exists yet for the replacement route. GitHub also reports the branch conflicting/dirty against merged main; update the branch and rerun the required checks/full tier. The PR body scope disclosure remains missing and is now 51 files, +3,157/-96.

[thymikee]

Two P1 blockers remain on the exact head:

1. Closes #320 is still unsupported. The linked issue explicitly requires a real permission-prompt accept/dismiss/recovery journey, but full:lifecycle-system now terminates the app and injects settings permission reset/grant/deny/reset, explicitly avoiding the transient system prompt. That is useful permission-state coverage, but it is not evidence for prompt interaction or recovery. Restore a live prompt accept/dismiss/recovery path, or stop closing test: move replay coverage toward stable mobile app patterns #320 and link a concrete owner for the omitted journey.

2. Exact-head Replay Nightly run 30245882004 fails deterministically in the full iOS E2E. events 100 only returns entries 0–99, while the first required press event has moved past that page after the added Settings wait events; the assertion at live-full-scenarios.ts:280 therefore fails. Make the assertion pagination-aware (or otherwise query the relevant range) and provide a green exact-head full nightly.

The PR body also needs the required scope disclosure: 51 files, +3,158/−96.

[thymikee]

Re-review of a7f2ce32: not ready.

• P1 remains — test: move replay coverage toward stable mobile app patterns #320 is still not satisfied. full:lifecycle-system still uses direct reset/grant/deny/reset state injection and contains no real permission-prompt accept/dismiss/recovery journey, while the body still says Closes #320. Restore that live prompt flow or stop closing test: move replay coverage toward stable mobile app patterns #320 and link a concrete owner for it; fix(ios): align simulator viewport capability with backend support #1407 is unrelated viewport work.
• The new pagination implementation coherently fixes the prior fixed-page assertion in code and has focused tests, but it is not live-proven yet. Exact-head Replay Nightly 30248536233 is red: the full iOS tier fails earlier at record stop (failed to stop recording: Note: No display specified…), so it never reaches the paginated-events assertion. This failure is outside the three-file pagination delta, but a green exact-head full tier is still required before readiness.
• The body still needs the required scope disclosure: 52 files, +3,321/−96.

Ordinary checks are green and GitHub reports CLEAN/MERGEABLE; ready-for-human remains withheld because of the confirmed #320 gap and red required validation.

[thymikee]

Re-reviewed exact head 6769e4ea. The recorder-lifetime delta is sound and non-vacuously tested, and ordinary checks are green. Two P1 blockers remain: (1) Closes #320 is still inaccurate—the full lifecycle scenario uses direct permission reset/grant/deny/reset and explicitly avoids the real system permission prompt, so it does not cover prompt accept/dismiss/recovery; either restore that live journey or stop claiming #320. (2) exact-head Replay Nightly run 30255398120 is still in progress and has not reached the full fixture-backed iOS E2E, so the prior record-stop/full-route failure is not yet evidenced resolved. P2: the PR body still omits required scope disclosure (58 files, +3,743/−191 across CI, docs, E2E, provider tests, and daemon recording behavior). Keep ready-for-human withheld.

[thymikee]

Re-reviewed exact head 8fde9e46. Not ready.

• P1 — dependency-cache key silently drops Node major. .github/actions/setup-test-app-dependencies/action.yml runs node -p 'process.versions.node.split(\\".\\")[0]'; the backslashes reach Node and throw SyntaxError: Expected unicode escape. Exact-head Test App Build Cache logs then save test-app-dependencies-macOS-ARM64-node-pnpm11.1.2-…, with an empty Node major, because the surrounding echo masks the substitution failure. This can reuse native node_modules across Node majors. Remove the escaping (for example split(".") as actual JS), then rerun producer/consumer evidence.\n- P1 — Closes #320 remains unsupported. The full lifecycle scenario injects reset/grant/deny/reset state and explicitly avoids the transient system prompt; it never demonstrates prompt accept/dismiss/recovery. Restore that live path or stop closing test: move replay coverage toward stable mobile app patterns #320 and create/own the follow-up.\n- P2 — scope disclosure remains missing. Current diff is 59 files, +3,789/−198 across CI/actions, fixture app, docs, E2E/provider tests, and recording/diagnostics paths.\n\nReplay Nightly 30255398120 passed at prior code-relevant head 6769e4e, including full fixture E2E and record stop, so that prior blocker is resolved. Ordinary exact-head checks are green, but the cache-key failure is masked; no exact-head full nightly exists for the cache-only delta.

[thymikee]

Re-reviewed exact head 274e6da. The cache P1 is not fixed. .github/actions/setup-test-app-dependencies/action.yml still executes node -p 'process.versions.node.split(\\".\\")[0]'; the backslashes that caused SyntaxError: Expected unicode escape remain. Renaming outputs to node_major / pnpm_version does not make Node produce a value, so the key will still collapse to …-node-pnpm… and can reuse native dependencies across Node majors. Remove the JS escaping and require exact-head producer/consumer logs showing a populated node<major> key.\n\nThe other blockers also remain: Closes #320 still lacks a real permission-prompt accept/dismiss/recovery journey, and the required 59-file (+3,789/−198) scope disclosure is absent. Prior full/record-stop evidence at 6769e4e remains valid for that route but does not validate this cache delta. Keep ready-for-human withheld.

[thymikee]

P1: setup-fixture-app restores the newest non-expired repository artifact by fingerprinted name alone, then unpacks and executes it. The producer workflow’s fork guard does not establish consumer trust: the repository artifact query does not verify that the selected artifact came from the designated Test App Build Cache workflow/run or same-repository provenance. Filter candidates to a trusted producer/run and repository identity before download; otherwise take the safe inline fallback.

The latest commit correctly removes the superseded ios-system-ui waiver, so that prior deterministic Coverage failure is addressed, but exact-head Coverage and iOS/device evidence are still pending; the successful full live run cited in the PR is from an older head. Also refresh the stale scope/validation disclosure (current base-to-head scope exceeds the stated 59 files).

The registry-derived coverage manifest itself is structurally sound and honestly owns the #1407/#320 gaps.

[thymikee]

Re-reviewed exact head ee94efa. P1 remains: untrusted fixture artifact selection.

setup-fixture-app still selects the first non-expired repository artifact by fingerprinted name and immediately downloads/unpacks it. The new producer inputs and queued-run handling only govern a cache miss/wait; they never establish that a cache hit came from the designated Test App Build Cache iOS Release producer, its expected run/head, or trusted repository provenance. The safe inline fallback is therefore bypassed for a colliding artifact.

Filter/select candidates through the trusted producer run (and reject ambiguity/untrusted provenance), then fall back inline when no trusted artifact is available. Add a collision regression proving an untrusted same-name artifact cannot be restored, plus exact-head producer/consumer evidence.

The queued-producer fallback itself is correctly added and covered. Current PR checks, including Coverage and the macOS fixture E2E smoke, are green, but they do not validate this provenance boundary.

[thymikee]

Updated review at bf578bdf — two merge blockers remain.

1. Artifact provenance: fixture cache hits still do not prove trusted provenance. query_artifact accepts the first non-expired name match, while producer/head checks run only after a miss. Require the selected artifact to be tied to an approved Test App Build Cache producer run and trusted producer repository (with the applicable producer/head relationship validated), then add a collision regression proving an untrusted same-name artifact is rejected in favor of inline build or a trusted candidate. The current consumer run at bf578bdf restored a fingerprint artifact produced at an earlier head; native-fingerprint reuse can be valid, but the action does not establish its provenance.

2. Current iOS smoke: the fixture fetch succeeded; the failure is later. The first home snapshot was sparse with reasonCode: budget after an 8.96s XCTest capture, but the E2E immediately requires the title. Add bounded retry/recovery for that typed condition, or provide fresh passing live evidence after a targeted fix. This is runtime/test reliability under CI load, not GitHub infrastructure or an artifact-download failure.

[thymikee]

P1 — The latest wait/scoped snapshot fixes the prior sparse home observation, but the next navigation assertion is vacuous: after clicking id="automation-continue-catalog", assertWaitText(context, 'Catalog') can pass on the unchanged Automation screen because that screen already contains the button text ‘Continue to catalog.’ Exact-head iOS smoke demonstrates this: it records the navigation evidence, then times out waiting for Settings while the current surface is still Automation Lab. Wait for a unique catalog landmark such as id="catalog-title" and diagnose/fix the click route if it does not appear. This is a deterministic test-route/runtime failure, not an infrastructure flake.

[thymikee]

Re-reviewed 7756367: the navigation false positive is fixed—the flow now waits for unique id="catalog-title" before recording evidence, and the focused test pins that route. PR is still not ready: the previously reported artifact-provenance P1 is unchanged, GitHub reports DIRTY/CONFLICTING, and only CodeQL ran at this head, so the route that failed on the previous head has no exact-head iOS smoke evidence. Resolve/recompute the conflict and require a passing exact-head iOS smoke after the provenance fix.

[thymikee]

Re-reviewed current head 44b51cf after the rebase. The exact-destination navigation fix is good: targeting id="catalog-title" removes the ambiguous Catalog-text match and the regression test pins that behavior. I found no new issue in this delta.

The PR is still not ready because the previously confirmed P1 fixture-artifact provenance blocker remains unchanged: setup-fixture-app may accept the first non-expired same-name repository artifact without proving it came from the trusted Test App Build Cache workflow/run/repository. Validate trusted producer identity and run provenance before consuming the artifact, cover name-collision/untrusted-producer cases, and fall back safely otherwise.

GitHub now reports the branch mergeable; fresh exact-head CI is queued/running, but this review is not waiting for CI and the provenance finding independently blocks readiness.