Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/adr/0007-remote-device-leases.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ Runner and process locks remain separate backend guards.
The proxy token authenticates the host, not the client: every client of one proxy can hold the same token.
The lease id is the credential that proves ownership of a leased session.
A request on a session that holds a lease must send that lease's id. The daemon refuses a request that omits it with `LEASE_SESSION_MISMATCH` and never fills it from the session.
Output that a leased request produced keeps that lease as its owner: a downloadable artifact, and a request diagnostics record. A request reads such output only when it sends the lease id, for as long as the daemon keeps the output, also after the lease ends. The daemon refuses another reader with `LEASE_OUTPUT_MISMATCH`. Ownership stays with the output, not with the session, because the output can outlive its session and its lease. Stored output keeps a SHA-256 of the lease id, and diagnostics name a lease by a fingerprint of it, because a served or archived record must not hold a credential.
Tenant, run id, lease backend, client id, device key, and lease provider are scope labels. The daemon checks a label when the request sends it. A label never grants access.
A lease id that a client chooses on allocate needs at least 128 random bits. The daemon returns a lease id only to the caller that allocated the lease.
A daemon that enforces this rule reports the `session-lease-id` capability.
Expand Down
3 changes: 2 additions & 1 deletion docs/adr/0024-screen-recording-provable-signal.md
Original file line number Diff line number Diff line change
Expand Up @@ -287,7 +287,8 @@ Artifact cleanup is a separate, explicit decision (rule 6).
Session teardown and failed-start rollback dispose through the backend under the fence.
They do this only after the manifest records that no recoverable export is lost.

A leased session's teardown that commits an export at a remote temp path registers it, and its telemetry, as downloadable artifacts of the lease's tenant (rule 1, #2281).
A leased session's teardown that commits an export at a remote temp path registers it, and its telemetry, as downloadable artifacts of that lease (rule 1, #2281).
Only a request that sends the id of the ended lease lists or downloads them (ADR 0007).
The inventory names the session, and the usual delete-after-download and 15-minute expiry apply.
A later `record stop` replay of that manifest serves the same artifact ids.
If registration fails, teardown deletes the files.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
import assert from 'node:assert/strict';
import fs from 'node:fs';
import path from 'node:path';
import { beforeEach, test, vi } from 'vitest';
import { beforeEach, onTestFinished, test, vi } from 'vitest';
import { loadRealHostKitCapabilities } from '@agent-device/test-kit';
import { IOS_SIMULATOR } from './device-fixtures.ts';
import { makeBackgroundRunner } from './runner-session-fixtures.ts';
import { mkdtempForTestSync } from './tmp-dir.ts';
Expand Down Expand Up @@ -124,6 +125,71 @@ test('the listener-ready marker is read back from the runner log file', async ()
await waitFor(() => assert.equal(launched.startupRetryWake.aborted, true));
});

test('runner trace output is scoped to the trace and the request that started the runner', async () => {
const { withDiagnosticsScope } = await loadRealHostKitCapabilities();
const logPath = runnerLogPath();
const traceLogPath = path.join(path.dirname(logPath), 'trace.log');
let finish = () => {};
const wait = new Promise<{ stdout: string; stderr: string; exitCode: number }>((resolve) => {
finish = () => resolve({ stdout: '', stderr: '', exitCode: 0 });
});
onTestFinished(finish);
mockRunCmdBackground.mockReturnValue({ ...makeBackgroundRunner(4242), wait });
let traceActive = true;
const launched = await withDiagnosticsScope(
{ traceLogPath, traceLogWritable: () => traceActive },
async () => {
const runner = launchRunnerProcess({
device: IOS_SIMULATOR,
xctestrunPath: '/tmp/runner.xctestrun',
derivedPath: '/tmp/runner-derived',
logPath,
});
fs.appendFileSync(logPath, 'runner output\n');
await waitFor(() => assert.equal(fs.readFileSync(traceLogPath, 'utf8'), 'runner output\n'));
traceActive = false;
fs.appendFileSync(logPath, 'AGENT_DEVICE_RUNNER_LISTENER_READY\n');
await waitFor(() => assert.equal(runner.startupRetryWake.aborted, true));
return runner;
},
);
traceActive = true;
fs.appendFileSync(logPath, 'after producer request\n');
finish();
await launched.output;
assert.equal(fs.readFileSync(traceLogPath, 'utf8'), 'runner output\n');
});

test('runner output stays bound to an active caller-owned trace after the request ends', async () => {
const { withDiagnosticsScope } = await loadRealHostKitCapabilities();
const logPath = runnerLogPath();
const traceLogPath = path.join(path.dirname(logPath), 'caller.trace');
let finish = () => {};
const wait = new Promise<{ stdout: string; stderr: string; exitCode: number }>((resolve) => {
finish = () => resolve({ stdout: '', stderr: '', exitCode: 0 });
});
onTestFinished(finish);
mockRunCmdBackground.mockReturnValue({ ...makeBackgroundRunner(4242), wait });
let traceActive = true;
const launched = await withDiagnosticsScope(
{ traceLogPath, runnerTraceLogWritable: () => traceActive },
() =>
launchRunnerProcess({
device: IOS_SIMULATOR,
xctestrunPath: '/tmp/runner.xctestrun',
derivedPath: '/tmp/runner-derived',
logPath,
}),
);
fs.appendFileSync(logPath, 'after the request\n');
await waitFor(() => assert.equal(fs.readFileSync(traceLogPath, 'utf8'), 'after the request\n'));
traceActive = false;
fs.appendFileSync(logPath, 'after trace stop\n');
finish();
await launched.output;
assert.equal(fs.readFileSync(traceLogPath, 'utf8'), 'after the request\n');
});

test('output an older runner generation left in the same log file does not wake startup', async () => {
const logPath = runnerLogPath();
fs.writeFileSync(logPath, 'AGENT_DEVICE_RUNNER_LISTENER_READY\n');
Expand Down
6 changes: 5 additions & 1 deletion packages/apple-runner/src/host.ts
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,10 @@ export type AppleRunnerHost = Pick<
| 'isCommandTimeoutError'
| 'commandDeveloperDir'
> &
Pick<typeof HostDiagnostics, 'emitDiagnostic' | 'withDiagnosticTimer'> &
Pick<
typeof HostDiagnostics,
'captureDiagnosticTraceWriter' | 'emitDiagnostic' | 'withDiagnosticTimer'
> &
Pick<typeof HostRetry, 'retryWithPolicy' | 'isEnvTruthy'> &
Pick<
typeof HostProcess,
Expand Down Expand Up @@ -196,6 +199,7 @@ export const isCommandTimeoutError = delegate('isCommandTimeoutError');
export const commandDeveloperDir = delegate('commandDeveloperDir');
export const hostEnvironment = delegate('hostEnvironment');
export const shellQuote = delegate('shellQuote');
export const captureDiagnosticTraceWriter = delegate('captureDiagnosticTraceWriter');
export const emitDiagnostic = delegate('emitDiagnostic');
export const withDiagnosticTimer = delegate('withDiagnosticTimer');
export const retryWithPolicy = delegate('retryWithPolicy');
Expand Down
7 changes: 4 additions & 3 deletions packages/apple-runner/src/runner-artifact.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ import {
import type { ExecBackgroundResult } from '@agent-device/host-kit/command';
import type { DeviceInfo } from '@agent-device/kernel/device';
import { classifyRunnerStartupFailure } from './runner-error-classification.ts';
import { logChunk } from './runner-io.ts';
import { bindRunnerTraceOutput, logChunk } from './runner-io.ts';
import {
runnerLaunchLogClassificationDetails,
runnerSimulatorSetFailureDetails,
Expand Down Expand Up @@ -900,6 +900,7 @@ async function buildRunnerXctestrun(
const performanceBuildSettings = resolveRunnerPerformanceBuildSettings();
const archBuildSettings = resolveRunnerArchBuildSettings(env);
const sandboxBuildArgs = resolveRunnerSandboxBuildArgs(env);
const writeTrace = bindRunnerTraceOutput(options.traceLogPath);
try {
await runCmdStreaming(
'xcodebuild',
Expand Down Expand Up @@ -932,10 +933,10 @@ async function buildRunnerXctestrun(
registerRunnerPrepProcess(device.id, child, options.startAdmission);
},
onStdoutChunk: (chunk) => {
logChunk(chunk, options.logPath, options.traceLogPath, options.verbose);
logChunk(chunk, options.logPath, writeTrace, options.verbose);
},
onStderrChunk: (chunk) => {
logChunk(chunk, options.logPath, options.traceLogPath, options.verbose);
logChunk(chunk, options.logPath, writeTrace, options.verbose);
},
},
);
Expand Down
19 changes: 17 additions & 2 deletions packages/apple-runner/src/runner-io.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { captureDiagnosticTraceWriter } from './host.ts';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
Expand All @@ -20,14 +21,28 @@ export async function getFreePort(): Promise<number> {
});
}

export function bindRunnerTraceOutput(
path: string | undefined,
): ((chunk: string) => void) | undefined {
const write = captureDiagnosticTraceWriter('runner');
if (!write) return path ? (chunk) => appendLogChunk(path, chunk) : undefined;
return (chunk) => {
try {
write(chunk);
} catch {
// A trace write must not interrupt runner startup or output.
}
};
}

export function logChunk(
chunk: string,
logPath?: string,
traceLogPath?: string,
writeTrace?: (chunk: string) => void,
verbose?: boolean,
): void {
if (logPath) appendLogChunk(logPath, chunk);
if (traceLogPath) appendLogChunk(traceLogPath, chunk);
writeTrace?.(chunk);
if (verbose) {
process.stderr.write(chunk);
}
Expand Down
10 changes: 8 additions & 2 deletions packages/apple-runner/src/runner-launched-session.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,12 @@ import type { DeviceInfo } from '@agent-device/kernel/device';
import { Deadline, emitDiagnostic } from './host.ts';
import { disposeRunnerSession, waitForRunnerLaunchExit } from './runner-disposal.ts';
import { mintRunnerToken } from './runner-contract.ts';
import { logChunk, resolveRunnerLaunchLogPath, flushRunnerLogAppends } from './runner-io.ts';
import {
bindRunnerTraceOutput,
logChunk,
resolveRunnerLaunchLogPath,
flushRunnerLogAppends,
} from './runner-io.ts';
import type { RunnerLauncher, RunnerLaunchObservation } from './runner-launcher.ts';
import type { AppleRunnerLifecycleOptions } from './runner-provider.ts';
import {
Expand Down Expand Up @@ -111,10 +116,11 @@ export async function launchRunnerSession(
session.retention?.cancel();
};
void launch.exited.then(() => settled(), settled);
const writeTrace = bindRunnerTraceOutput(options.traceLogPath);
observation.output = (async () => {
for await (const text of launch.output) {
output = (output + text).slice(-16_384);
logChunk(text, logPath, options.traceLogPath, options.verbose);
logChunk(text, logPath, writeTrace, options.verbose);
}
})().catch(async (error: unknown) => {
observation.failure = error;
Expand Down
6 changes: 4 additions & 2 deletions packages/apple-runner/src/runner-process-launch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import { resolveRunnerMaxConcurrentDestinationsFlag } from './runner-cache-metad
import { xcodebuildDestinationArgs } from './runner-device-set.ts';
import {
createRunnerLogFile,
bindRunnerTraceOutput,
logChunk,
readRunnerLogTail,
tailRunnerLogFile,
Expand Down Expand Up @@ -97,12 +98,13 @@ export function launchRunnerProcess(input: LaunchRunnerProcessInput): LaunchedRu
captureOutput: false,
},
);
const writeTrace = bindRunnerTraceOutput(input.traceLogPath);
const logTail = tailRunnerLogFile({
file: logFile,
onOutput: (chunk) => {
listenerReady.observe(chunk);
if (input.traceLogPath || input.verbose) {
logChunk(chunk, undefined, input.traceLogPath, input.verbose);
if (writeTrace || input.verbose) {
logChunk(chunk, undefined, writeTrace, input.verbose);
}
},
});
Expand Down
5 changes: 4 additions & 1 deletion packages/command-registry/src/daemon-command-descriptor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@ export type DaemonRefFrameEffect<TRequest = DispatchedCommand> =

export type SessionlessLeaseAdmissionExemption =
| Readonly<{ kind: 'unconditional' }>
| Readonly<{ kind: 'provider-app-catalog'; provider: string }>;
| Readonly<{ kind: 'provider-app-catalog'; provider: string }>
// The request names a lease that ended, to read output that the lease left. Its handler checks
// that lease id against the owner of the output.
| Readonly<{ kind: 'ended-lease-output' }>;

/**
* Daemon route + request-policy traits for one command. Generic over the request
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,10 @@ export const DEVICE_OPERATION_COMMAND_DESCRIPTORS = {
refFrameEffect: 'preserve',
allowInvalidRecording: true,
allowSessionlessDefaultDevice: isRecordingStartRequest,
sessionlessLeaseAdmissionExemption: (req) =>
req.positionals?.[0]?.toLowerCase() === 'stop'
? { kind: 'ended-lease-output' }
: undefined,
},
platformExecution: { kind: 'device-runtime', uses: screenRecordingRuntimePlanUses },
// A `record stop` export can outlast the request envelope. Resetting the daemon mid-export
Expand Down
4 changes: 3 additions & 1 deletion packages/command-surface/src/perf/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,9 @@ export const perfCommandMetadata = defineFieldCommandMetadata(
kind: enumField(PERF_KIND_VALUES),
template: stringField('xctrace template name, for example Time Profiler.'),
out: stringField('Output artifact path.'),
tracePath: stringField('Existing .trace path to report, defaults to the latest session trace.'),
tracePath: stringField(
'Existing local .trace path to report. For a remote report, omit it to use the session capture.',
),
},
);

Expand Down
2 changes: 1 addition & 1 deletion packages/contracts/src/replay-backend.ts
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ export type ReplaySessionState = Readonly<{
device: DeviceInfo;
appBundleId?: string;
actions: SessionAction[];
trace?: Readonly<{ outPath: string }>;
trace?: Readonly<{ outPath?: string }>;
/** Read for presence only: an active recording means replay video must not start another. */
screenRecording?: object;
snapshotDiagnostics?: Pick<SnapshotDiagnosticsState, 'samples'>;
Expand Down
17 changes: 10 additions & 7 deletions packages/daemon-contracts/src/daemon-diagnostics-scope.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,14 @@ export async function withDaemonDiagnosticsScope<T>(
body: () => Promise<T> | T,
): Promise<T> {
const { logPath, command = 'daemon', debug = true } = options;
return await withDiagnosticsScope({ command, session: 'daemon', logPath, debug }, async () => {
try {
return await body();
} finally {
flushDiagnosticsToSessionFile({ force: true });
}
});
return await withDiagnosticsScope(
{ command, session: 'daemon', logPath, debug, operatorLog: true },
async () => {
try {
return await body();
} finally {
flushDiagnosticsToSessionFile({ force: true });
}
},
);
}
6 changes: 4 additions & 2 deletions packages/daemon-contracts/src/session-artifact-paths.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,10 @@ export function resolveSessionTestArtifactsRoot(sessionDir: string): string {
export function resolveSessionRequestLogPath(
sessionDir: string,
requestId: string | undefined,
directory: 'requests' | 'request-records' | 'requests-unleased' = 'requests',
): string {
const safeRequestId = safeSessionName(requestId && requestId.length > 0 ? requestId : 'unknown');
return path.join(sessionDir, 'requests', `${safeRequestId}.ndjson`);
return path.join(sessionDir, directory, `${safeRequestId}.ndjson`);
}

/**
Expand All @@ -32,9 +33,10 @@ export function resolveSessionRequestLog(params: {
sessionDir: string;
session: string;
requestId: string | undefined;
directory?: 'request-records';
}): { path: string; ref: DiagnosticsRecordRef } {
return {
path: resolveSessionRequestLogPath(params.sessionDir, params.requestId),
path: resolveSessionRequestLogPath(params.sessionDir, params.requestId, params.directory),
ref: {
session: params.session,
requestId: params.requestId && params.requestId.length > 0 ? params.requestId : 'unknown',
Expand Down
27 changes: 26 additions & 1 deletion packages/daemon/docs/remote-device-lease-contracts.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Idle reap then shuts the daemon down within one more idle window.

A session cannot outlive the lease under which it was opened.
When the lease is released, expires, or its provider reports the instance ended, the daemon tears the session down under its execution locks and clears its device claim.
Any teardown of a leased session, including `close`, finishes an active recording. A finished export at a remote temp path then belongs to the lease's tenant as a downloadable artifact, like the artifact of any `record stop` (ADR 0024 2.5).
Any teardown of a leased session, including `close`, finishes an active recording. A daemon-owned export stays downloadable under its producing owner, as with `record stop` (ADR 0024 2.5).
This applies even when the device no longer answers.
A release is a tenant `lease_release` or a host `DELETE /admin/leases/<lease-id>`.
The daemon sweeps for expired leases every second.
Expand Down Expand Up @@ -55,6 +55,7 @@ Nested requests record on the ledger of the outermost request, so a step reply n
If a close finds its lease already inactive, it reports that lease as ended.
A client never sends a release after such a request.
If a client stores the lease id, it forgets that id when the reply names it.
The CLI keeps the id of an ended proxy lease apart, and sends it only on `artifacts` while the connection holds no lease.
A gateway that relays the reply reads the lease end from it.

Close and lease-end teardown take the same execution locks.
Expand Down Expand Up @@ -85,6 +86,30 @@ An allocate that repeats the labels of a live lease without that lease's id is r
A caller-chosen lease id on allocate must have at least 32 hex digits (128 bits). A shorter id fails with reason `LEASE_ID_TOO_SHORT`.
Commands that ignore the session lease, such as `devices`, `session_list`, and the lease RPCs, keep their own scope.

Output of a request that acted under a lease belongs to that lease.
This applies to a downloadable artifact and to a request diagnostics record.
The daemon gives such output only to a request that sends the lease id.
The `artifacts` RPC reads the lease id from the request. `GET /artifacts`, `GET /artifacts/<id>`, and the diagnostics route read the `x-agent-device-lease-id` header, which the proxy forwards.
This rule applies for as long as the daemon keeps the output, also after the lease ends. The lifetime of the output does not change.
An inventory omits an artifact of another lease.
A download or a diagnostics read without the lease id fails with `UNAUTHORIZED`, reason `LEASE_OUTPUT_MISMATCH`, and `field: 'leaseId'`.
A downloadable screenshot or xctrace result has its own daemon-owned path. A write to the caller's destination does not change an earlier download.
The daemon names each request diagnostics record. The name carries the owner of the record: the lease id that the request sent, or that it sent none.
The request id of the caller names no record. A nested request uses the parent's record only when both requests send the same lease id or both send no lease id.
A caller that sent a lease id reads only its own records, also the record of a request that a leased session refused.
New records are stored in the session's `request-records` directory. The diagnostics route serves legacy records only from the host-verified directory.
A recording manifest keeps the lease that started the recording, or that no lease started it.
`record stop` returns a completed export to a request that sends that lease id, also after the lease ended. Such a request needs no active lease.
A recording that still runs needs a request that is admitted under that lease.
`record stop` does not return a completed remote export whose manifest names neither.
The daemon keeps an owner as the SHA-256 of the lease id, never as the id.
A request diagnostics record names a lease by the first 8 hex digits of that SHA-256, never by the lease id.
An event that the daemon writes outside a request goes to the daemon log, which only the host operator reads. It names the lease id, because the operator needs it to end a rented device.
Output of a request that acted under no lease keeps the tenant rule.

Before you restore access to a legacy diagnostics record, verify its full contents came from unleased requests and contain no lease credentials. Move the file from the session's `requests` directory to its `requests-unleased` directory. Keep its filename.
Before you restore access to a legacy recording in the daemon session directory, verify that the producing request was unleased. The daemon refuses replay from legacy remote temp paths. Set `metadata.remoteCaptureLease` to `"unleased"` in its recording manifest.

## Admitted request work

A lease renews when a request is admitted, and never again while that request runs.
Expand Down
Loading
Loading