Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
cdecce1
Extract AWS Device Farm provider plugin
thymikee Nov 16, 2026
f0e642e
refactor(aws-device-farm): consolidate WebDriver imports
thymikee Nov 16, 2026
4d007df
test(aws-device-farm): colocate connection coverage
thymikee Nov 16, 2026
0007ab2
Test packed AWS provider error boundary
thymikee Nov 16, 2026
ebb2759
refactor(provider-webdriver): remove unused extraction scaffolding
thymikee Nov 16, 2026
00c580d
chore(gates): register the AWS packed smoke entry
thymikee Nov 16, 2026
319ef2a
chore(gates): verify the explicit WebDriver plugin subpath
thymikee Nov 16, 2026
2964f1d
test(layering): update WebDriver export boundary
thymikee Nov 16, 2026
29e2a70
chore(gates): separate root dependency ownership from workspace refer…
thymikee Nov 16, 2026
c534bb9
chore(gates): check workspace declarations in script test imports
thymikee Nov 16, 2026
adb8ec5
fix(plugins): defer host command execution imports
thymikee Nov 16, 2026
5348c3e
refactor(webdriver): remove unused provider exports
thymikee Nov 16, 2026
8272e3f
fix(plugins): preserve supplied command host context
thymikee Nov 16, 2026
9a5dc22
fix: address AWS plugin review findings
thymikee Nov 16, 2026
d2c0f9f
fix(aws-device-farm): publish typed plugin entrypoints
thymikee Nov 16, 2026
6182cfe
test(docs): resolve installed workspace plugins
thymikee Nov 16, 2026
2579246
fix(ci): select moved snapshot differential test
thymikee Nov 16, 2026
d138084
refactor(aws): import host capability types from the plugin SDK
thymikee Nov 17, 2026
ce76aa0
test(aws): cover routed plugin admission
thymikee Nov 17, 2026
22c9a79
test(aws): register provider for stored lease fixtures
thymikee Nov 17, 2026
8c5fabc
chore(aws): enable package publication
thymikee Nov 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/aws-device-farm-plugin.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'agent-device': minor
---

AWS Device Farm moves to a private, unpublished plugin. After publication, CLI hosts can install it with `agent-device plugins add @agent-device/aws-device-farm`; finish sessions, stop the detached daemon for its matching state directory, and retry. Embedded hosts import the factory, pass it to `startAgentDevice({ plugins: [awsDeviceFarm()] })`, and close and recreate the runtime after changes.
1 change: 1 addition & 0 deletions .fallowrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
"tools/depgraph-view/app.js",
"scripts/check-provider-plugin.mjs",
"scripts/check-affected/command.ts",
"packages/provider-aws-device-farm/test/package-smoke.mjs",
"packages/provider-browserstack/test/package-smoke.mjs",
"packages/provider-browserstack/tsdown.config.ts",
"packages/provider-testmu/test/package-smoke.mjs",
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ The same session and evidence model works at every step: the agent explores the
| --- | --- | --- |
| Local | Trying commands and debugging apps on simulators, emulators, physical devices, macOS, and Linux. | Follow the Quick Start. |
| CI/CD | Automated pull request and merge validation with replay scripts and captured artifacts. | Try the [EAS workflow template](https://github.com/callstackincubator/eas-agent-device/blob/main/.eas/workflows/agent-qa-mobile.yml). |
| Cloud / remote | Linux runners, managed devices, and remote jobs. | Set up a [remote proxy](https://agent-device.dev/docs/remote-proxy), connect a [device cloud](https://agent-device.dev/docs/device-clouds) (BrowserStack and TestMu AI through `agent-device plugins add`, AWS Device Farm, Limrun), or [contact Callstack](mailto:hello@callstack.com) for team QA. |
| Cloud / remote | Linux runners, managed devices, and remote jobs. | Set up a [remote proxy](https://agent-device.dev/docs/remote-proxy), connect a [device cloud](https://agent-device.dev/docs/device-clouds) (BrowserStack and TestMu AI through `agent-device plugins add`, AWS Device Farm after its plugin is published, or Limrun), or [contact Callstack](mailto:hello@callstack.com) for team QA. |

## How it works

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import Foundation
import XCTest

/// Replays `contracts/fixtures/snapshot-actionability-policy.json` against the Swift predicate. The
/// TypeScript twin replays the same rows in `scripts/ios-snapshot-differential.test.ts`.
/// TypeScript twin replays the same rows in `packages/capture-kit/test/ios-snapshot-differential.test.ts`.
final class ActionabilityPolicyTests: XCTestCase {
private struct Table: Decodable {
let cases: [PolicyCase]
Expand Down
2 changes: 1 addition & 1 deletion contracts/fixtures/snapshot-actionability-policy.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"description": "Golden vector table for the shared `hittable` predicate (#1933, #2891), for input shapes the fixed 320x240 fold fixture in ios-snapshot-engine-conformance.json cannot reach. RULE: a node is actionable when it is enabled, its own rect passes the box guard below, and its center falls inside the viewport box, half-open on the right and bottom edges. BOX GUARD: a box is usable when all four components are finite, both extents (x+width, y+height) are finite, the size is positive, and it is not CGRect.infinite — whose components and extents are all finite, so only identity refuses it. SnapshotGeometry.isPositiveFinite (Swift) and isPositiveFiniteRect in packages/kernel/src/rect.ts are the twins that enforce exactly this, one per language. Implementations of the predicate: SnapshotGeometry.isGeometricallyActionable (Swift, replayed by ActionabilityPolicyTests) and isGeometricallyActionable in packages/kernel/src/rect.ts (replayed by scripts/ios-snapshot-differential.test.ts). The viewport is the declared fact `reported`, `derived`, or `missing` (IosViewportEvidence); a box the guard refuses never becomes `reported`. `hittable: null` is the absent bit: with no viewport box, containment is undecided, while a disabled or degenerate node is still a declared `false` because that half of the rule needs no box. Every row declares `swift` and `typescript` and at least one must run it; a row one side skips carries `asymmetry`.",
"description": "Golden vector table for the shared `hittable` predicate (#1933, #2891), for input shapes the fixed 320x240 fold fixture in ios-snapshot-engine-conformance.json cannot reach. RULE: a node is actionable when it is enabled, its own rect passes the box guard below, and its center falls inside the viewport box, half-open on the right and bottom edges. BOX GUARD: a box is usable when all four components are finite, both extents (x+width, y+height) are finite, the size is positive, and it is not CGRect.infinite — whose components and extents are all finite, so only identity refuses it. SnapshotGeometry.isPositiveFinite (Swift) and isPositiveFiniteRect in packages/kernel/src/rect.ts are the twins that enforce exactly this, one per language. Implementations of the predicate: SnapshotGeometry.isGeometricallyActionable (Swift, replayed by ActionabilityPolicyTests) and isGeometricallyActionable in packages/kernel/src/rect.ts (replayed by packages/capture-kit/test/ios-snapshot-differential.test.ts). The viewport is the declared fact `reported`, `derived`, or `missing` (IosViewportEvidence); a box the guard refuses never becomes `reported`. `hittable: null` is the absent bit: with no viewport box, containment is undecided, while a disabled or degenerate node is still a declared `false` because that half of the rule needs no box. Every row declares `swift` and `typescript` and at least one must run it; a row one side skips carries `asymmetry`.",
"cases": [
{
"name": "a center strictly inside the reported box is actionable",
Expand Down
1 change: 1 addition & 0 deletions examples/sdk/tsconfig.json
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@
{ "path": "../../packages/provider-limrun" },
{ "path": "../../packages/provider-revyl" },
{ "path": "../../packages/provider-testmu" },
{ "path": "../../packages/provider-aws-device-farm" },
{ "path": "../../packages/provider-browserstack" },
{ "path": "../../packages/provider-webos" },
{ "path": "../../packages/provider-tizen" },
Expand Down
6 changes: 3 additions & 3 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@
"maestro:conformance": "node --experimental-strip-types scripts/node-test-tmpdir.ts --experimental-strip-types --test scripts/maestro-conformance/format-generated-json.test.mjs scripts/maestro-conformance/verify-cli-version.test.mjs packages/maestro/test/conformance/verify.test.ts packages/maestro/test/conformance/differential/engine-process.test.ts packages/maestro/test/conformance/differential/report-output.test.ts packages/maestro/test/conformance/differential/run.test.ts packages/maestro/test/conformance/differential/invariants.test.ts",
"maestro:conformance:regenerate": "node --experimental-strip-types scripts/maestro-conformance/regenerate.mjs",
"maestro:conformance:differential": "node --experimental-strip-types packages/maestro/test/conformance/differential/run.ts",
"test:ios-snapshot-differential": "node --experimental-strip-types scripts/swift-toolchain-tmpdir.ts swift test --package-path apple/snapshot-presentation && node --experimental-strip-types scripts/node-test-tmpdir.ts --experimental-strip-types --test scripts/ios-snapshot-differential.test.ts",
"test:ios-snapshot-differential": "node --experimental-strip-types scripts/swift-toolchain-tmpdir.ts swift test --package-path apple/snapshot-presentation && node --experimental-strip-types scripts/node-test-tmpdir.ts --experimental-strip-types --test packages/capture-kit/test/ios-snapshot-differential.test.ts",
"size": "node scripts/size-report.mjs",
"perf": "node --experimental-strip-types scripts/perf/run.ts",
"perf:compare": "node --experimental-strip-types scripts/perf/compare-cli.ts",
Expand Down Expand Up @@ -376,10 +376,10 @@
"yaml": "^2.9.0",
"yauzl": "^3.4.0",
"@agent-device/testmu": "workspace:*",
"@agent-device/aws-device-farm": "workspace:*",
"@agent-device/browserstack": "workspace:*",
"@agent-device/revyl": "workspace:*",
"@agent-device/webos": "workspace:*",
"@agent-device/tizen": "workspace:*",
"@agent-device/build-config": "workspace:*"
"@agent-device/tizen": "workspace:*"
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -8,20 +8,20 @@ import type { Rect } from '@agent-device/kernel/snapshot';
import {
compareDifferentialCases,
SWIFT_RUN_TIMEOUT_MS,
} from '../packages/capture-kit/src/ios-snapshot-engine/conformance-harness.ts';
} from '../src/ios-snapshot-engine/conformance-harness.ts';
import {
swiftToolchainAvailable,
writeDifferentialFailureArtifact,
} from '../packages/capture-kit/src/ios-snapshot-engine/conformance-harness.fixtures.ts';
import { differentialBatchArbitrary } from '../packages/capture-kit/src/ios-snapshot-engine/conformance-generator.ts';
import { readIosSnapshotEngineFixture } from '../packages/capture-kit/src/ios-snapshot-engine/conformance-fixture.ts';
} from '../src/ios-snapshot-engine/conformance-harness.fixtures.ts';
import { differentialBatchArbitrary } from '../src/ios-snapshot-engine/conformance-generator.ts';
import { readIosSnapshotEngineFixture } from '../src/ios-snapshot-engine/conformance-fixture.ts';
import {
hostRowNodes,
runnerNodes,
safariWebViewNodes,
threadNodes,
viewport as capturedViewport,
} from '../packages/capture-kit/src/ios-snapshot-engine/runner-presentation-fixtures.ts';
} from '../src/ios-snapshot-engine/runner-presentation-fixtures.ts';

type DifferentialCase = Parameters<typeof compareDifferentialCases>[0][number];

Expand Down Expand Up @@ -269,6 +269,8 @@ type ActionabilityVector = Readonly<{
const ACTIONABILITY_POLICY_PATH = path.resolve(
import.meta.dirname,
'..',
'..',
'..',
'contracts',
'fixtures',
'snapshot-actionability-policy.json',
Expand Down
2 changes: 1 addition & 1 deletion packages/command-surface/src/replay/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -319,7 +319,7 @@ export const testDaemonWriter: AsyncDaemonWriter = async (input, host, deps) =>
await import('@agent-device/replay-port/replay-backend-registry')
).withReplayBackend(
'maestro',
host.env,
host,
(backend) => collectSources(async () => backend),
deps?.resolveReplayBackend,
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,7 @@ export async function resourceAdmissionWorld(
]
: []),
]),
input.host.env,
input.host,
{
adaptWebDriver: async () => {
throw new Error('Resource fixture cannot create providers');
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ async function fixture(storeEnv?: NodeJS.ProcessEnv) {
const discover = vi.fn(async () => []);
const registry = await startProviderRegistry(
await resolveProviderPlugins([]),
{},
{ env: {}, cwd: '/' },
{
assertReplaySelectionCurrent: storeEnv ? createPluginSelectionGuard([], storeEnv) : undefined,
adaptWebDriver: async () => {
Expand Down
3 changes: 3 additions & 0 deletions packages/maestro/src/plugin.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ const setupHost = {
env: {},
options: {},
clientVersion: '0.22.0-dev',
runHostCommand: async () => {
throw new Error('host command unused');
},
createError: () => {
throw new Error('setup must not use host services');
},
Expand Down
2 changes: 1 addition & 1 deletion packages/maestro/test/packed-runtime.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ const { default: maestroPlugin } = await import(
async function registry() {
const registry = await startProviderRegistry(
await resolveProviderPlugins([{ source: 'embedder', load: () => maestroPlugin() }]),
{},
{ env: {}, cwd: '/' },
{
adaptWebDriver: async () => {
throw new Error('not a provider');
Expand Down
2 changes: 1 addition & 1 deletion packages/plugins/src/connection.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import { storePluginCandidates } from './store.ts';
export type PluginConnectionCallbacks = Readonly<{
resolve(context: Omit<PluginConnectionContext, 'host'>): ReturnType<PluginConnection['resolve']>;
verify(
context: Omit<Pick<PluginConnectionContext, 'flags' | 'env' | 'host'>, 'host'>,
context: Omit<Pick<PluginConnectionContext, 'flags' | 'env' | 'cwd' | 'host'>, 'host'>,
): ReturnType<PluginConnection['verify']>;
}>;

Expand Down
147 changes: 147 additions & 0 deletions packages/plugins/src/host-command.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
import { expect, test } from 'vitest';
import { withCommandExecutorOverride } from '@agent-device/host-kit/command';
import { createPluginHost } from './host.ts';

test('plugin host runs commands with the AWS command bounds', async () => {
await withCommandExecutorOverride(
async (command, args, options) => {
expect(command).toBe('aws');
expect(args).toEqual(['devicefarm', 'get-project']);
expect(options.maxBuffer).toBe(10 * 1024 * 1024);
expect(options.timeoutMs).toBe(30_000);
return { stdout: '{"project":{}}', stderr: '', exitCode: 0 };
},
async () => {
await expect(
createPluginHost({ env: {}, cwd: '/' }, undefined).runHostCommand('aws', [
'devicefarm',
'get-project',
]),
).resolves.toEqual({ stdout: '{"project":{}}' });
},
);
});

test('commands use the supplied host and overlay request environment without ambient values', async () => {
const { mkdtempForTestSync } = await import('@agent-device/test-kit');
const { realpathSync } = await import('node:fs');
const { withCommandHost } = await import('@agent-device/host-kit/command');
const cwd = realpathSync(mkdtempForTestSync('plugin-command-host-'));
const host = { cwd, env: { PLUGIN_VALUE: 'host', HOST_ONLY: 'present' } };
const pluginHost = createPluginHost(host, undefined);
const args = [
'-e',
'process.stdout.write(JSON.stringify({cwd:process.cwd(), value:process.env.PLUGIN_VALUE, hostOnly:process.env.HOST_ONLY, ambient:process.env.PLUGIN_AMBIENT}))',
];
const previous = process.env.PLUGIN_AMBIENT;
process.env.PLUGIN_AMBIENT = 'ambient';
try {
const plain = await pluginHost.runHostCommand(process.execPath, args);
expect(JSON.parse(plain.stdout)).toEqual({ cwd, value: 'host', hostOnly: 'present' });
const scoped = await withCommandHost({ host, requestEnv: { PLUGIN_VALUE: 'request' } }, () =>
pluginHost.runHostCommand(process.execPath, args),
);
expect(JSON.parse(scoped.stdout)).toEqual({ cwd, value: 'request', hostOnly: 'present' });
} finally {
if (previous === undefined) delete process.env.PLUGIN_AMBIENT;
else process.env.PLUGIN_AMBIENT = previous;
}
});

test('connection resolve and verify use each call host independently', async () => {
const { mkdtempForTestSync } = await import('@agent-device/test-kit');
const { realpathSync } = await import('node:fs');
const { resolveProviderPlugins } = await import('./registry.ts');
const dirs = [
realpathSync(mkdtempForTestSync('plugin-resolve-')),
realpathSync(mkdtempForTestSync('plugin-verify-')),
];
const observed: unknown[] = [];
const run = async (host: import('./host.ts').ProviderPluginHost) => {
const result = await host.runHostCommand(process.execPath, [
'-e',
'process.stdout.write(JSON.stringify([process.cwd(), process.env.CONNECTION_VALUE]))',
]);
observed.push(JSON.parse(result.stdout));
};
const { connections } = await resolveProviderPlugins([
{
source: 'embedder',
load: () => ({
apiVersion: 2,
provider: 'example',
credentials: { variables: [] },
setup: () => ({}),
connection: {
capabilities: {
leaseKind: 'direct-device-provider',
requiresAppAttachment: false,
requiresRemoteDaemon: false,
supportsArtifacts: true,
supportsDeferredAppSelection: false,
supportsDirectPortReverse: false,
usesCloudWebDriverLease: true,
},
resolve: async ({ host }: { host: import('./host.ts').ProviderPluginHost }) => {
await run(host);
return { profile: { leaseProvider: 'example', platform: 'ios' } };
},
verify: async ({ host }: { host: import('./host.ts').ProviderPluginHost }) => {
await run(host);
return { provider: 'example', service: 'example', verificationMessage: 'ok' };
},
},
}),
},
]);
const connection = connections.get('example')!;
const flags = { json: false, help: false, version: false };
await connection.resolve({
flags,
stateDir: dirs[0]!,
cwd: dirs[0]!,
env: { CONNECTION_VALUE: 'resolve' },
});
await connection.verify({ flags, cwd: dirs[1]!, env: { CONNECTION_VALUE: 'verify' } });
expect(observed).toEqual([
[dirs[0], 'resolve'],
[dirs[1], 'verify'],
]);
});

test('provider registry setup receives its supplied command host', async () => {
const { mkdtempForTestSync } = await import('@agent-device/test-kit');
const { realpathSync } = await import('node:fs');
const { resolveProviderPlugins, startProviderRegistry } = await import('./registry.ts');
const cwd = realpathSync(mkdtempForTestSync('plugin-setup-host-'));
let observed: unknown;
const resolved = await resolveProviderPlugins([
{
source: 'embedder',
load: () => ({
apiVersion: 2,
provider: 'example',
credentials: { variables: [] },
setup: async (host: import('./host.ts').ProviderPluginHost) => {
const result = await host.runHostCommand(process.execPath, [
'-e',
'process.stdout.write(JSON.stringify([process.cwd(), process.env.SETUP_VALUE]))',
]);
observed = JSON.parse(result.stdout);
return { webDriver: { provider: 'example' } };
},
}),
},
]);
const registry = await startProviderRegistry(
resolved,
{ cwd, env: { SETUP_VALUE: 'supplied' } },
{
adaptWebDriver: async () => {
throw new Error('not needed for host assertion');
},
},
);
expect(observed).toEqual([cwd, 'supplied']);
await registry.close();
});
25 changes: 23 additions & 2 deletions packages/plugins/src/host.ts
Original file line number Diff line number Diff line change
@@ -1,21 +1,42 @@
import type { HostEnvironment } from '@agent-device/kernel/source-value';
import { AppError, type AppErrorCode, type AppErrorDetails } from '@agent-device/kernel/errors';
import { readVersion } from '@agent-device/host-kit/version';

export type HostCommandResult = Readonly<{ stdout: string }>;

export type RunHostCommand = (
command: string,
args: readonly string[],
) => Promise<HostCommandResult>;

export type ProviderPluginHost = Readonly<{
/** Host-supplied command environment. Include PATH, HOME and provider inputs when needed. */
env: Readonly<Record<string, string | undefined>>;
options: Readonly<Record<string, unknown>>;
clientVersion: string;
createError(code: AppErrorCode, message: string, details?: AppErrorDetails): AppError;
/** Runs in the supplied host cwd/env; ambient process values are not added. */
runHostCommand: RunHostCommand;
}>;

export function createPluginHost(
env: NodeJS.ProcessEnv,
host: HostEnvironment,
options: Record<string, unknown> | undefined,
): ProviderPluginHost {
return Object.freeze({
env: Object.freeze({ ...env }),
env: Object.freeze({ ...host.env }),
options: Object.freeze({ ...options }),
clientVersion: readVersion(),
createError: (code, message, details) => new AppError(code, message, details),
runHostCommand: async (command, args) => {
const { runCmd } = await import('@agent-device/host-kit/command');
const { stdout } = await runCmd(command, [...args], {
cwd: host.cwd,
env: host.env,
maxBuffer: 10 * 1024 * 1024,
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved
timeoutMs: 30_000,
});
Comment thread
thymikee marked this conversation as resolved.
Resolved
return { stdout };
},
} satisfies ProviderPluginHost);
}
2 changes: 1 addition & 1 deletion packages/plugins/src/load.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ async function startStore(env: NodeJS.ProcessEnv, builtin: readonly string[] = [
...claimProviderIds(builtin),
...(await storePluginCandidates(env)),
]);
return await startProviderRegistry(resolved, env, FAKE_POLICY);
return await startProviderRegistry(resolved, { env, cwd: '/' }, FAKE_POLICY);
}

test('a store plugin loads its factory with the selection options and the host error constructor', async () => {
Expand Down
Loading
Loading