Skip to content

refactor(macos): ship macOS desktop automation as the managed platform plugin - #4933

Merged
thymikee merged 15 commits into
0.22from
refactor/0.22-macos-platform-plugin
Nov 22, 2026
Merged

thymikee merged 15 commits into
0.22from
refactor/0.22-macos-platform-plugin

Conversation

@thymikee

@thymikee thymikee commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Summary

Part of #4675. macOS desktop automation and its Swift helper leave the core for the managed plugin @agent-device/macos (still "private": true).

  • Design A: the plugin holds the leaf claim { kind: 'platform', family: 'apple', appleOs: 'macos' }. The bundled Apple owner keeps apple and calls the installed leaf at its macOS branches.
  • Runner option (a): the plugin owns both app backends' macOS code, the desktop surfaces and macos-helper. Core keeps the XCTest runner and gives the leaf a device-bound runner capability.
  • Without the plugin, --platform macos answers plugin-not-installed with the install hint, and unfiltered discovery omits the Mac. audio probe needs the plugin: the sampler is the helper's.

Size: 163 files, +3462 −1472. It exceeds the cap because the claim key, the leaf port, the move and its gates cannot merge separately, as in the Linux precedent.

Value

Clean builds, npm pack --ignore-scripts, base 80c44fe45e against 4dc93c9ff8:

  • Packed core: 1,638,382 B to 1,616,982 B.
  • Unpacked core: 5,275,885 B to 5,188,827 B.
  • macos-helper files in core: 11 to 0. Plugin tarball: 70,157 B.

Validation

At 68d339fd2d, passing: pnpm lint, pnpm typecheck, pnpm format, pnpm check:layering, fallow audit --base origin/0.22, pnpm run check:package (13 smokes), pnpm check:command-docs, pnpm check:daemon-wire-compat, unit-core (21,099 tests), provider-integration (318 tests), Swift helper tests.

Live check on a Mac (transcript in the first comment): the native backend flow passed. The default XCTest backend is blocked there: XCUITest Automation Mode is not authorized.

CI must still prove the macOS smoke lane, which now installs the packed plugin.


Show it working
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

View guided diff Turn on auto-fix

A platform claim key gains an optional leaf part, so a managed package can hold
`{ kind: 'platform', family: 'apple', appleOs: 'macos' }` while the bundled Apple
owner keeps the `apple` family claim for iOS and tvOS. Only `apple` narrows, and
only to macOS. The registry keeps a leaf setup apart from the family setups, and
the platform composition answers for the leaf through its own selection: an
absent package raises `plugin-not-installed` with the `@agent-device/macos`
install command.

Part of #4675
…rface

`@agent-device/macos` is a managed platform plugin that contributes the leaf
`{ kind: 'platform', family: 'apple', appleOs: 'macos' }`. Its `setup(host)`
registers `bind(tools)`, which returns every macOS operation the bundled Apple
owner delegates.

The Apple owner keeps the `apple` claim and gains a leaf port. The composition
binds the process's leaf selection to it, and the owner asks the port at its
macOS branches. An absent package answers typed `plugin-not-installed` with the
`@agent-device/macos` install command.

The owner hands the leaf two capabilities per call: the Apple tool routes of the
calling request, so a scoped request reaches the leaf and no command starts from
the plugin's bundled host kit, and the XCTest runner bound to one macOS device
for the runner's macOS wire cases.

Part of #4675
…nto the plugin

The published core no longer contains the macOS desktop implementation. The
`os/macos` sources (both app backends' macOS code, the desktop surfaces, the
LaunchServices probe, the host routes and the helper client) and the
`apple/macos-helper` Swift package move to `packages/platform-macos`, and the
`@agent-device/platform-apple/macos` facade is removed.

Every macOS branch of the bundled Apple owner now calls the installed leaf. The
owner still assembles the native backend's interactor and still owns the one
XCTest runner; the leaf maps a desktop scroll and an alternate click onto the
runner's `desktopScroll` and `mouseClick` wire cases through the device-bound
capability the owner hands it. The runner scroll normalization both share moves
beside the scroll plan in contracts.

The plugin runs each command through the Apple tools the owner resolved for the
request, and takes the screen-capture lock, machine facts and diagnostics from
the host its setup received. `PlatformHost.processes.withLock` gains a `signal`
so a cancelled request stops waiting for a held lock, as it did in core.

Unfiltered discovery lists this Mac only when the package is installed, and a
macOS request without it answers `plugin-not-installed`. Host system audio
sampling is the helper's, so the audio probe needs the package for every family.

Part of #4675
The layering policy learns a managed leaf package: `packages/platform-macos`
holds `{ platform: 'apple', appleOs: 'macos' }` and obeys every rule a managed
family package obeys. The packed-core check refuses any macOS desktop path or
helper marker, and `check:package` gains the macOS package smoke.

The helper pack rules leave the core size report for a plugin-owned check, so
the core guard now refuses a helper path outright and exempts no test path.
Lane ownership, the native-dupes scan root, the eager-closure facade list and
the Fallow facade entry follow the files. The macOS smoke lane installs the
packed plugin before the replay gate, and the provider scenarios compose it.

Part of #4675
…described

The `help macos` topic, the website command and installation pages, CONTRIBUTING
and the device verification guide state that macOS desktop automation is the
`@agent-device/macos` plugin and how a request without it answers. ADR 0034
records the leaf claim, the owner delegation and the two rejected alternatives.

Part of #4675
… the extraction measurements

`plugins list` prints `(apple macos)` for a package that holds the macOS leaf,
so the listing does not read as a claim on the whole Apple family. The evidence
note records the packed-core size against the 0.22 base and the live check.

Part of #4675
…election

Every platform composition bound the Apple owner's macOS leaf port, including
the selection narrowed to the web family. That selection names no leaf, so it
unbound an installed macOS leaf in the same process. The daemon composition and
an unnarrowed installed-platform selection now bind explicitly, and composing a
selection has no side effect on the owner.

Part of #4675
@thymikee

Copy link
Copy Markdown
Member Author

Live check transcript (macOS host, Xcode 27.1, head 4dc93c9ff8)

Packed core (npm pack --ignore-scripts) installed into an empty directory under the repository .tmp/, with AGENT_DEVICE_HOME and AGENT_DEVICE_STATE_DIR inside it. The plugin is the packed @agent-device/macos tarball. The host name and local paths are redacted.

Result

  • Without the plugin, devices --platform macos answers UNSUPPORTED_PLATFORM / plugin-not-installed with the install hint.
  • After plugins add ./macos.tgz: devices and open TextEdit pass.
  • Blocker, default XCTest backend: snapshot -i and prepare ios-runner fail. Typed error: COMMAND_FAILED: Runner did not accept connection (xcodebuild exited early). The runner log states Failed to initialize for UI testing: Error Domain=com.apple.dt.XCTest.XCTFuture Code=1000 "Timed out while enabling automation mode.". XCUITest Automation Mode is not authorized on this host. I did not grant it and did not work around it. The runner is the shared core runner, which this PR does not change.
  • Opt-in native backend (AGENT_DEVICE_MACOS_APP_BACKEND=native): open, snapshot -i (45 nodes), screenshot (1172x976), press @e33 and close pass. The plugin built its helper from the packed macos-helper source.
  • The daemon was stopped. No runner, daemon or TextEdit process remained.
Transcript
$ agent-device --version
0.22.0-dev
[exit 0]

$ agent-device devices --platform macos
Error (UNSUPPORTED_PLATFORM): macOS desktop automation requires the @agent-device/macos plugin. Next: Install: agent-device plugins add @agent-device/macos@nightly-0.22 Then retry your original command. If a local daemon is already running, close its sessions and run agent-device daemon stop with the same --state-dir before retrying.
Log: <live>/state/sessions/cwd_a1ec01a0ee7d7cd5_macos/requests/3987810fae37e33c.ndjson
[exit 1]

$ agent-device plugins add ./macos.tgz
@agent-device/macos@0.22.0-dev: installed; compatibility unchecked. Run agent-device plugins list.
Restart the local daemon after closing active sessions to use the changed plugin set.
[exit 0]

$ agent-device plugins list
@agent-device/macos@0.22.0-dev (apple)
[exit 0]

$ agent-device daemon stop
Daemon stopped (graceful).
[exit 0]

$ agent-device devices --platform macos
<host-mac> (macos device target=desktop) booted=true
[exit 0]

$ agent-device open TextEdit --platform macos
Opened: TextEdit
Session state: <live>/state/sessions/cwd_a1ec01a0ee7d7cd5_macos
[exit 0]

$ agent-device snapshot -i --platform macos
Building Apple runner (one-time per agent-device version)...
Starting XCTest runner...
Error (COMMAND_FAILED): Daemon request timed out. Not dispatched. Next: The timed-out snapshot request was canceled; the daemon was kept alive so the session can still be closed or inspected. Read the request log for the timed-out step. If this was the first Apple-platform snapshot on the device, run agent-device prepare ios-runner with the same --platform before snapshot/test so runner startup is handled explicitly.
Log: ~/.agent-device/logs/default/2026-11-21/2026-11-21T20-17-37-128Z-mwqulpru-db6a07d2.ndjson
[exit 1]

$ agent-device prepare ios-runner --platform macos
Error (COMMAND_FAILED): Runner did not accept connection (xcodebuild exited early). Not dispatched. Next: Retry runner startup, inspect xcodebuild logs, and verify simulator responsiveness before command execution. If runner build products look stale or corrupted, run `pnpm clean:xcuitest` in a local checkout, or remove ~/.agent-device/apple-runner/derived, then retry.
Log: <live>/state/sessions/cwd_a1ec01a0ee7d7cd5_macos/runner.log
[exit 1]

$ agent-device snapshot -i --platform macos
Error (COMMAND_FAILED): Daemon request timed out. Not dispatched. Next: The timed-out snapshot request was canceled; the daemon was kept alive so the session can still be closed or inspected. Read the request log for the timed-out step. If this was the first Apple-platform snapshot on the device, run agent-device prepare ios-runner with the same --platform before snapshot/test so runner startup is handled explicitly.
Log: ~/.agent-device/logs/default/2026-11-21/2026-11-21T20-20-18-895Z-mwqup6lh-5739a5b2.ndjson
[exit 1]

$ agent-device close --platform macos
Closed: default
[exit 0]

$ agent-device daemon stop
Daemon stopped (graceful).
[exit 0]

# The same flow on the opt-in native app backend (AGENT_DEVICE_MACOS_APP_BACKEND=native)

$ agent-device open TextEdit --platform macos
Opened: TextEdit
Session state: <live>/state/sessions/cwd_a1ec01a0ee7d7cd5_macos
[exit 0]

$ agent-device snapshot -i --platform macos
Page: TextEdit
App: com.apple.TextEdit
Snapshot: 45 nodes
@e1 [application] "TextEdit"
@e2 [window] "Untitled"
@e3 [scroll-area] "_NS:8" [scrollable]
@e4 [text-view] "First Text View" [editable]
@e5 [axscrollbar] "_NS:24" [disabled]
@e6 [axscrollbar] "_NS:27" [disabled]
@e7 [axruler]
@e8 [axrulermarker]
@e9 [axrulermarker]
@e10 [axrulermarker]
@e11 [axrulermarker]
@e12 [axrulermarker]
@e13 [axrulermarker]
@e14 [axrulermarker]
@e15 [axrulermarker]
@e16 [axrulermarker]
@e17 [axrulermarker]
@e18 [axrulermarker]
@e19 [axrulermarker]
@e20 [axrulermarker]
@e21 [axrulermarker]
@e22 [axrulermarker]
@e23 [axsegment] "bold"
@e24 [axsegment] "italic"
@e25 [axsegment] "underline"
@e26 [axsegment] "strikethrough"
@e27 [axcolorwell] "text color"
@e28 [axcolorwell] "text background color"
@e29 [axpopupbutton] "typeface"
@e30 [axpopupbutton] "style"
@e31 [axcombobox] "font size"
@e32 [button] "font size"
@e33 [axsegment] "align left"
@e34 [axsegment] "align center"
@e35 [axsegment] "align right"
@e36 [axsegment] "fully justify"
@e37 [axmenubutton] "list style"
@e38 [axpopupbutton] "line spacing"
@e39 [text] "—" [disabled]
@e40 [button]
@e41 [button]
@e42 [button]
@e43 [axmenubutton] "Edited" [disabled]
@e44 [aximage] "Untitled.rtf"
@e45 [text] "Untitled"
[exit 0]

$ agent-device screenshot <live>/textedit.png --platform macos
<live>/textedit.png (1172x976)
[exit 0]

$ agent-device press @e33 --platform macos
Tapped @e33 (1279, 124)
[exit 0]

$ agent-device close --platform macos
Closed: default
[exit 0]

$ agent-device daemon stop
Daemon stopped (graceful).
[exit 0]

@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 6.16 MB 6.07 MB -86.6 kB
Package (unpacked) 6.16 MB 6.07 MB -86.6 kB
Package (download) 1.88 MB 1.86 MB -22.2 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 19.8 ms 20.0 ms +0.2 ms
CLI --help 42.7 ms 42.1 ms -0.6 ms

@apex-sentinel-staging

This comment has been minimized.

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 4 new findings on 68d339f. The verdict and the full list are in the summary comment.

Comment thread packages/platform-apple/src/runtime.ts Resolved
Comment thread packages/plugins/src/validate.ts Resolved
@thymikee

Copy link
Copy Markdown
Member Author

Review of 68d339f: two problems must be fixed before merge.

The release install-hint guard does not list @agent-device/macos. scripts/release-install-hints.ts:33 reads only OPTIONAL_PLATFORMS, provider declarations and replay formats. MACOS_PLATFORM_PLUGIN_DECLARATION has no family-catalog row, and the package is "private": true, so it is not published and the guard passes. The 0.22 branch publishes nightly-0.22 daily. After merge, the next nightly drops macOS desktop from core, while the help, docs and plugin-not-installed hint tell users to run agent-device plugins add @agent-device/macos@nightly-0.22, which cannot resolve. The rule: every package an install hint names must be a package the release publishes. Please make the guard enumerate the macOS declaration too, ideally from one exported catalog of plugin declarations that both the hints and this script read. Then drop "private": true from packages/platform-macos/package.json, or hold the merge until the package can be published.

Host audio capture for every family now goes through the macOS plugin (src/platform-runtime-audio-probe-host.ts:50). On a darwin host, audio probe start --platform ios|android without the plugin now throws plugin-not-installed, where the base sampled through core. This is a regression for iOS simulator and Android emulator callers, and the issue says core keeps macOS host support. Please either keep the ScreenCaptureKit sampler as a core-owned host capability for the audio-probe host, or get explicit maintainer sign-off that iOS and Android audio probe require the macOS plugin and record it in the issue.

The Ownership and context budgets check fails: docs/agents is 40143 bytes against the 40000 limit. It was 39940 bytes at the merge base, and the only change is the two lines added in docs/agents/device-verification.md:62-63 (+203 bytes). Trimming at least 143 bytes makes it pass. The two Body rules jobs show as cancelled by newer runs, not as failures. No conflicts are known.

Not blocking, and fine to take or leave: the versioned help Audio block (cli-help-workflows.ts:110) and website/docs/docs/debugging-profiling.md:250 do not say the macOS plugin is now required, and devices --platform apple without a target now omits the Mac, which commands.md does not mention. In validate.ts:62 the leaf is generalized for one value (ClaimLeaf alias, optional leafField/readLeaf hooks), so reading appleOs directly through one requireClaimLeaf would be simpler. In macos-leaf.ts:17 the leaf selection is a module-level binding that registry close() does not unbind, so passing it through the Apple owner's construction hosts would be safer as a follow-up.

The inline thread on the audio probe (discussion_r4654911495) still stands. Two lower-priority threads still apply. One thread does not apply and can be resolved: runtime.ts:236 throws only for a macOS device, and without the plugin no macOS device is discoverable, so the typed plugin-not-installed outcome is intended (discussion_r4654911484).

Not verified here: packing of core or the plugin (the size numbers in the PR body were measured at 4dc93c9, not at this head), a local test run, a file-by-file comparison of the moved and deleted tests against their replacements, and the desktop, frontmost-app and menubar helper surfaces in a live run. The XCTest backend rests on the passing macOS smoke job.

Under AGENT_DEVICE_MACOS_APP_BACKEND=native the admission-facts spread asked the macOS leaf for
its native refusals, so a capability read raised plugin-not-installed for a Mac the process cannot
drive. The backend is a daemon setting and listing facts is not a macOS call: ask the leaf only
when it is installed, and keep the typed install outcome with the commands that need it.
The two readers that describe a slot whose load failed read the declared `appleOs` with the
leaf-tolerant reader, so `{ kind: 'platform', family: 'apple', appleOs: 'ios' }` came back as the
whole `apple` claim. The registry reserved that widened claim, and the next package that really
holds the family lost it as a duplicate. Both readers now refuse an un-narrowable leaf through the
one helper the validating path uses, and a refused slot records the refusal instead of a claim.
The ScreenCaptureKit sampler moved with the Swift helper, so a macOS session, iOS simulator, or
Android emulator probe answers plugin-not-installed with no package installed. Say so in the
audio help topic, the audio-probe and client API docs, and the capabilities generator, and pin the
typed outcome with a host test.
@thymikee

Copy link
Copy Markdown
Member Author

The maintainer sign-off recorded on #4675 settles the audio probe item: the sampler stays in @agent-device/macos, so that finding no longer blocks. At 68d339f two items remain: the release install-hint guard does not list @agent-device/macos while the package is still "private": true, and the Ownership and context budgets check fails on docs/agents.

…dget

The plugin note took docs/agents past its 40 000-byte total. State the instruction once and leave
the install command to the user docs that own it.
@thymikee

Copy link
Copy Markdown
Member Author

Reply to #4933 (comment), head a938263:

@apex-sentinel-staging

Copy link
Copy Markdown

Apex Sentinel · ✅ Approve · review of a938263

Since the review of 68d339f: 0 new · 0 still open · 4 no longer raised (4 marked fixed)

No open findings.

Not in the reviewers' prompt (38 files)
  • .fallowrc.json: left out as an asset or for its size; reviewers could still search it
  • .github/actions/smoke-macos/action.yml: left out as an asset or for its size; reviewers could still search it
  • .jscpd-native.json: left out as an asset or for its size; reviewers could still search it
  • CONTRIBUTING.md: left out as an asset or for its size; reviewers could still search it
  • contracts/fixtures/runner-requests.json: left out as an asset or for its size; reviewers could still search it
  • docs/adr/0034-provider-plugin-registry.md: left out as an asset or for its size; reviewers could still search it
  • docs/agents/device-verification.md: left out as an asset or for its size; reviewers could still search it
  • docs/evidence/macos-platform-plugin-extraction-2026-11-21.md: left out as an asset or for its size; reviewers could still search it
  • examples/sdk/tsconfig.json: left out as an asset or for its size; reviewers could still search it
  • package.json: left out as an asset or for its size; reviewers could still search it
  • packages/contracts/package.json: left out as an asset or for its size; reviewers could still search it
  • packages/platform-apple/package.json: left out as an asset or for its size; reviewers could still search it
  • packages/platform-apple/src/core/__tests__/macos-helper-test-utils.ts: left out as an asset or for its size; reviewers could still search it
  • packages/platform-apple/src/macos-facade.ts: left out as an asset or for its size; reviewers could still search it
  • packages/platform-apple/src/os/macos/app-backend.test.ts: left out as an asset or for its size; reviewers could still search it
  • packages/platform-apple/src/os/macos/app-backend.ts: left out as an asset or for its size; reviewers could still search it
  • packages/platform-apple/src/os/macos/desktop-scroll.ts: left out as an asset or for its size; reviewers could still search it
  • packages/platform-apple/src/os/macos/native-backend-facts.test.ts: left out as an asset or for its size; reviewers could still search it
  • packages/platform-apple/src/os/macos/native-backend-facts.ts: left out as an asset or for its size; reviewers could still search it
  • packages/platform-macos/package.json: left out as an asset or for its size; reviewers could still search it
  • packages/platform-macos/src/apps.test.ts: left out as an asset or for its size; reviewers could still search it
  • packages/platform-macos/src/helper-build.test.ts: left out as an asset or for its size; reviewers could still search it
  • packages/platform-macos/src/helper.test.ts: left out as an asset or for its size; reviewers could still search it
  • packages/platform-macos/src/native-app-actions.test.ts: left out as an asset or for its size; reviewers could still search it
  • packages/platform-macos/test/package-smoke.mjs: left out as an asset or for its size; reviewers could still search it
  • packages/platform-macos/tsconfig.json: left out as an asset or for its size; reviewers could still search it
  • packages/plugins/src/platform.test.ts: left out as an asset or for its size; reviewers could still search it
  • pnpm-lock.yaml: generated or a lockfile, so its lines were left out
  • scripts/__tests__/fixtures/size-report-npm-pack.json: left out as an asset or for its size; reviewers could still search it
  • scripts/__tests__/size-report-package.test.ts: left out as an asset or for its size; reviewers could still search it
  • scripts/layering/contracts-exports.snapshot.json: left out as an asset or for its size; reviewers could still search it
  • test/repo-guards/macos-helper-package-contents.test.ts: left out as an asset or for its size; reviewers could still search it
  • tsconfig.typecheck.json: left out as an asset or for its size; reviewers could still search it
  • website/docs/docs/capabilities.md: left out as an asset or for its size; reviewers could still search it
  • website/docs/docs/client-api.md: left out as an asset or for its size; reviewers could still search it
  • website/docs/docs/commands.md: left out as an asset or for its size; reviewers could still search it
  • website/docs/docs/debugging-profiling.md: left out as an asset or for its size; reviewers could still search it
  • website/docs/docs/installation.md: left out as an asset or for its size; reviewers could still search it

1 finding below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (d6767e0910fc).

💬 Talking to Sentinel

In a finding's thread, reply with:

Command What it does
@apex-sentinel-staging note <context> Adds context; Sentinel re-checks the cited code
@apex-sentinel-staging pivot <why> Says the finding is wrong; Sentinel re-checks the cited code
@apex-sentinel-staging like Marks the finding as useful
@apex-sentinel-staging dislike <why> Marks it as not useful, so it is not raised again

On this conversation, comment @apex-sentinel-staging review to run a full review once again.

@thymikee

Copy link
Copy Markdown
Member Author

This PR is ready at a938263. The install-hint guard now covers the macOS declaration with a publishable package, and the docs/agents size check passes at exactly 40000 bytes. The earlier findings are fixed, and all 38 checks pass at this commit. No conflicts.

Not blocking: the bare catch around readableClaimsOfValue in registry.ts drops every error and every readable claim of that value, so a failed plugin with a valid provider claim loses it in the inventory. It could catch only the typed invalid_plugin error, or drop only the refused claim. Take it or leave it.

Not verified here: the release pack or pnpm pack of @agent-device/macos, and the npm placeholder and trusted-publisher setup for that package. Tests were not run locally; CI results are taken from the checks at this commit.

The inline threads on the absent-leaf guard (#4933 (comment)), the readable-reader leaf check (#4933 (comment)) and the plugin requirement docs (#4933 (comment)) are fixed at this commit and can be resolved. The audio probe thread (#4933 (comment)) does not apply: the maintainer signed off that the sampler stays in the plugin, so it can be resolved too.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 21, 2026
@thymikee
thymikee merged commit 1b34f29 into 0.22 Nov 22, 2026
38 checks passed
@thymikee
thymikee deleted the refactor/0.22-macos-platform-plugin branch November 22, 2026 06:59
@thymikee

Copy link
Copy Markdown
Member Author

Cleanup after merge: the worktree and the local branches refactor/0.22-macos-platform-plugin and backup/macos-plugin-pre-rebase are removed, and the remote head branch is gone. The validation log is removed. No T3 device, daemon or session was opened for this PR. #4675 stays open because the live XCTest run was blocked by Automation Mode on this host.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant