Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions packages/daemon/docs/remote-device-lease-contracts.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,20 @@ A request on that session, including `close`, `artifacts`, and other admission-e
Lease heartbeat and release name the lease id too.
If a request omits the lease id, the daemon refuses it with `UNAUTHORIZED`, reason `LEASE_SESSION_MISMATCH`, and `field: 'leaseId'`.
The daemon never fills a missing lease id from the session.
The daemon checks the lease id before the request reads, changes, or ends the session.
A request writes request events to the session journal only after that check. Thus, a request that is refused for the session lease id writes none, and it does not stamp session activity.
If a command writes request events and the request has access to an existing session, the daemon records its start before it runs the command.
For a lock-exempt `doctor`, `capabilities`, or `artifacts` command, the daemon records the start after the handler checks the session lease.
A request that sends the lease id of the session and is refused later, for example a `batch` after the lease deadline, can write its events.
The daemon checks the lease again when a lock-exempt command or `apps` reads the session.
If the lock-exempt command has access to the session and can take the session lock immediately, it can refresh the recording.
A `test` suite reads, records, and closes an attempt session only with the lease id that the suite sent.
After the lease deadline, a request that holds the session lock and sends the lease id of the session ends that session.
A request that sends a different active lease, and that lease admission accepts for a new session, also ends that expired session under its locks. It then runs as the new owner. Its reply and its diagnostics do not name the expired lease.
The successor starts admitted work before it ends the expired session.
A `batch`, a `replay`, and a bounded step do not end the session, and the lease sweep ends it. With the expired lease id, they are refused with `LEASE_NOT_FOUND`. With a different active lease, they are refused with `LEASE_SESSION_MISMATCH`.
A request without an active lease is refused and ends nothing. The lease sweep ends the session.
A command that holds no session lock ends no session. It leaves the expired session to the lease-end teardown.
Tenant, run id, lease backend, client id, device key, and lease provider are labels.
The daemon refuses a label that differs from the session lease. A missing label is not a refusal and grants nothing.
The daemon compares each value that a request sends for a label: the value in `meta`, the value in `flags`, and each alias of a param.
Expand All @@ -84,6 +98,7 @@ When an auth hook attests the tenant, the attested tenant routes the request. Th
An allocate that repeats the labels of a live lease without that lease's id is refused as contention, with `DEVICE_IN_USE` and reason `DEVICE_LEASE_BUSY`. It returns no lease id. If the live lease has no device key, a label that contradicts it stays a scope mismatch, with `UNAUTHORIZED` and reason `LEASE_SCOPE_MISMATCH`. A repeat of a live lease with a device key is `DEVICE_LEASE_BUSY` whatever its client id says. Only an allocate that sends the id of its live lease gets that lease back.
A caller-chosen lease id on allocate must have at least 32 hex digits (128 bits). A shorter id fails with reason `LEASE_ID_TOO_SHORT`.
Commands that ignore the session lease, such as `devices`, `session_list`, and the lease RPCs, keep their own scope.
On a leased session, these commands do not bind the session, refresh its recording, deliver its script, renew its lease, end it, or write request events to it.

## Admitted request work

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ export function createHumanControlHarness() {
deviceRuntimeGateway: lifecycleDeviceRuntimeGateway,
androidObservation: clearAndroidObservationFixture,
platformResourceCleanup: platformResourceCleanupFixture(),
settleExpiredSession: async () => {},
trackDownloadableArtifact: () => 'artifact-1',
});
return { registry, lease, sessionStore, sessionName, handleRequest };
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ import {
} from '../../server/request-router.ts';
import type { BindDeviceRuntime } from '../../core/request-runtime-binding.ts';
import type { DeviceInfo } from '@agent-device/kernel/device';
import type { SessionRef } from '../../core/session-state.ts';
import {
applicationLifecycleRuntimeFixture,
type LifecycleOperationsLoader,
Expand All @@ -33,6 +34,7 @@ import {
daemonPlatformServicesFixture,
} from './platform-services-fixture.ts';
import { platformResourceCleanupFixture } from './platform-resource-cleanup-fixture.ts';
import { teardownSessionResources } from '../../sessions/session-teardown.ts';
import { providerCredentialsFixture } from './provider-credentials-fixture.ts';
import {
unavailableBinding,
Expand Down Expand Up @@ -218,6 +220,7 @@ export function createRequestHandler(
| 'claimRecoveryGateway'
| 'androidObservation'
| 'platformResourceCleanup'
| 'settleExpiredSession'
> &
Partial<
Pick<
Expand All @@ -228,6 +231,7 @@ export function createRequestHandler(
| 'claimRecoveryGateway'
| 'androidObservation'
| 'platformResourceCleanup'
| 'settleExpiredSession'
>
>,
) {
Expand All @@ -238,10 +242,19 @@ export function createRequestHandler(
claimRecoveryGateway = daemonClaimRecoveryGatewayFixture(),
androidObservation = clearAndroidObservationFixture,
platformResourceCleanup = platformResourceCleanupFixture(),
// The resource half of the daemon-owned settle, over this handler's cleanup fixture.
settleExpiredSession = async (ref: SessionRef) =>
await teardownSessionResources({
appLog: 'run',
ref,
sessionStore: deps.sessionStore,
platformCleanup: platformResourceCleanup,
}),
...rest
} = deps;
const handle = createProductionRequestHandler({
...rest,
settleExpiredSession,
androidObservation,
platformResourceCleanup,
platformServices,
Expand Down
31 changes: 24 additions & 7 deletions packages/daemon/src/core/lease-context.ts
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,8 @@ export type SessionLease = Readonly<

type LeaseClaimField = keyof LeaseClaims;

type LeaseClaimRefusal = { kind: 'missing-id' } | { kind: 'mismatch'; field: LeaseClaimField };

type LeaseClaimRequest = Pick<DaemonRequest, 'flags' | 'meta' | 'internal'>;

/** The flags that carry each lease field. `meta` carries each field under the field's own name. */
Expand Down Expand Up @@ -237,21 +239,36 @@ export function assertRequestHoldsSessionLease(
* sends, and so does an empty label or a value that is not a string.
*/
export function assertRequestPresentsLease(req: LeaseClaimRequest, lease: SessionLease): void {
const claims = readLeaseClaims(req);
if (claims.leaseId.every((claim) => typeof claim === 'string' && !claim.trim())) {
const refusal = compareRequestLeaseClaims(req, lease);
if (!refusal) return;
if (refusal.kind === 'missing-id') {
throw new AppError('UNAUTHORIZED', 'This session holds a device lease. Send its lease id.', {
reason: LEASE_ADMISSION_REASONS.sessionMismatch.reason,
field: 'leaseId',
hint: 'Send the leaseId that the allocation returned (meta.leaseId or the leaseId param). From the CLI, open the session through connect proxy or connect --remote-config and keep the same --session; raw --tenant and --session flags do not send it.',
});
}
throw new AppError('UNAUTHORIZED', `Lease does not match session owner (${refusal.field})`, {
reason: LEASE_ADMISSION_REASONS.sessionMismatch.reason,
field: refusal.field,
});
}

/** Reports the first refusal without constructing an error. */
export function compareRequestLeaseClaims(
req: LeaseClaimRequest,
lease: SessionLease,
): LeaseClaimRefusal | undefined {
const claims = readLeaseClaims(req);
if (claims.leaseId.every((claim) => typeof claim === 'string' && !claim.trim())) {
return { kind: 'missing-id' };
}
for (const field of LEASE_CLAIM_FIELDS) {
if (claims[field].every((claim) => leaseClaimNamesLease(field, claim, lease))) continue;
throw new AppError('UNAUTHORIZED', `Lease does not match session owner (${field})`, {
reason: LEASE_ADMISSION_REASONS.sessionMismatch.reason,
field,
});
if (!claims[field].every((claim) => leaseClaimNamesLease(field, claim, lease))) {
return { kind: 'mismatch', field };
}
}
return undefined;
}

function readLeaseClaims(req: LeaseClaimRequest): Record<LeaseClaimField, unknown[]> {
Expand Down
8 changes: 7 additions & 1 deletion packages/daemon/src/resources/lease-in-flight-work.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ export type LeaseWorkWanted = () => boolean;
/** A claim on one leased device for the duration of one admitted request's work. */
export type LeaseWorkPass = Readonly<{
leaseId: string;
/** Whether this pass is still registered for its original lease. */
isRetained(): boolean;
/**
* Ends the pass. Reports whether the work was still wanted when it ended, which
* is what entitles it to renew the lease it just worked on.
Expand All @@ -39,7 +41,11 @@ export class LeaseInFlightWorkRegistry {
const entries = this.entriesByLeaseId.get(leaseId) ?? new Set<LeaseWorkEntry>();
entries.add(entry);
this.entriesByLeaseId.set(leaseId, entries);
return { leaseId, release: () => this.releasePass(leaseId, entries, entry) };
return {
leaseId,
isRetained: () => !entry.released && entries.has(entry),
release: () => this.releasePass(leaseId, entries, entry),
};
}

/**
Expand Down
54 changes: 45 additions & 9 deletions packages/daemon/src/resources/lease-registry.ts
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,18 @@ const LEASE_REGISTRY_REASONS = {

type OwnedHumanControlHold = { hold: HumanControlHold; ownerLeaseId?: string };

export type RetainedLeaseWork = Pick<LeaseWorkPass, 'leaseId' | 'release'> & {
/** Refuses a released pass, even if the host reused its lease id. */
assertRetained(): void;
};

function requireActiveLease(lease: DeviceLease | undefined): DeviceLease {
if (lease) return lease;
throw new AppError('UNAUTHORIZED', 'Lease is not active', {
reason: LEASE_REASONS.notFound.reason,
});
}

export class LeaseRegistry {
private readonly holdsByDevice = new Map<string, Map<string, OwnedHumanControlHold>>();
private readonly mutations = new DeviceMutationDrain();
Expand Down Expand Up @@ -230,11 +242,15 @@ export class LeaseRegistry {
const leaseId = normalizeRequiredLeaseId(request.leaseId);
this.cleanupExpiredLeases();
const lease = this.getActiveLease(leaseId);
return this.refreshLease(lease, this.resolveHeartbeatTtlMs(lease, request));
}

private resolveHeartbeatTtlMs(lease: DeviceLease, request: HeartbeatLeaseRequest): number {
assertLeaseOwnerScope(lease, request);
assertLeaseScopeMatch(lease, request);
const requestedTtlMs =
request.ttlMs === undefined ? leaseOwnTtlMs(lease) : this.resolveLeaseTtlMs(request.ttlMs);
return this.refreshLease(lease, requestedTtlMs);
return request.ttlMs === undefined
? leaseOwnTtlMs(lease)
: this.resolveLeaseTtlMs(request.ttlMs);
}

/**
Expand All @@ -246,10 +262,13 @@ export class LeaseRegistry {
* human-control hold does; a pass whose request was cancelled protects nothing.
* `wanted` is how the caller reports that its client is still waiting.
*/
retainLeaseWork(lease: Pick<DeviceLease, 'leaseId'>, wanted: LeaseWorkWanted): LeaseWorkPass {
retainLeaseWork(lease: Pick<DeviceLease, 'leaseId'>, wanted: LeaseWorkWanted): RetainedLeaseWork {
const pass = this.inFlightWork.retain(lease.leaseId, wanted);
return {
leaseId: pass.leaseId,
assertRetained: () => {
requireActiveLease(pass.isRetained() ? this.leases.get(pass.leaseId) : undefined);
},
release: () => {
const renewed = pass.release();
if (renewed) this.refreshProtectedLease(lease.leaseId, this.now());
Expand Down Expand Up @@ -311,6 +330,27 @@ export class LeaseRegistry {
assertLeaseScopeMatch(this.getActiveLease(scope.leaseId), scope);
}

/** Checks request admission and renewal without sweeping or refreshing any lease. */
inspectLeaseAdmission(request: HeartbeatLeaseRequest): DeviceLease {
return this.checkLeaseAdmission(request).lease;
}

/** Renews a successor before its request tears down the incumbent session. */
renewInspectedLeaseAdmission(request: HeartbeatLeaseRequest): DeviceLease {
const { lease, ttlMs } = this.checkLeaseAdmission(request);
return this.refreshLease(lease, ttlMs);
}

private checkLeaseAdmission(request: HeartbeatLeaseRequest): {
lease: DeviceLease;
ttlMs: number;
} {
const scope = normalizeLeaseAdmissionRequest(request);
const lease = requireActiveLease(this.inspectActiveLease(scope.leaseId));
assertLeaseScopeMatch(lease, scope);
return { lease, ttlMs: this.resolveHeartbeatTtlMs(lease, request) };
}

/** Whether this daemon ever allocated `tenantId` a `macos-app` lease, released or not. */
hasHeldMacOsAppLease(tenantId: string): boolean {
return this.macOsAppTenantIds.has(tenantId);
Expand Down Expand Up @@ -666,11 +706,7 @@ export class LeaseRegistry {
}

private getActiveLease(leaseId: string): DeviceLease {
const lease = this.leases.get(leaseId);
if (lease) return lease;
throw new AppError('UNAUTHORIZED', 'Lease is not active', {
reason: LEASE_REASONS.notFound.reason,
});
return requireActiveLease(this.leases.get(leaseId));
}

/** Every renewal route ends here, so a host-pinned lease never outlives the host's window. */
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import {
import { makeSessionStore } from '../../__tests__/support/store-factory.ts';
import { LeaseRegistry } from '../../resources/lease-registry.ts';
import { createRequestExecutionScope } from '../request-execution-scope.ts';
import { teardownSessionResources } from '../../sessions/session-teardown.ts';
import type { DaemonRequest } from '../../core/daemon-request.ts';
import {
platformRequestScopeFixture,
Expand Down Expand Up @@ -63,7 +64,7 @@ test('expired leases remove owned sessions before the next command and free capa
claimRecoveryGateway: daemonClaimRecoveryGatewayFixture(),
deviceRuntimeGateway: unavailableDeviceRuntimeGateway,
platformRequestScope: platformRequestScopeFixture(),
req: makeRequest({ command: 'snapshot' }),
req: makeRequest({ command: 'snapshot', meta: { leaseId: lease.leaseId } }),
sessionStore,
leaseRegistry,
});
Expand Down Expand Up @@ -121,15 +122,25 @@ test.each(['rebuild', 'retire'] as const)(
claimRecoveryGateway: daemonClaimRecoveryGatewayFixture(),
deviceRuntimeGateway: unavailableDeviceRuntimeGateway,
platformRequestScope: platformRequestScopeFixture(),
req: makeRequest({ session: address, flags: { session: address } }),
req: makeRequest({
session: address,
flags: { session: address },
meta: { leaseId: lease.leaseId },
}),
sessionStore: store,
leaseRegistry: leases,
platformResourceCleanup: {
stopSnapshotHelper,
closeManagedBrowser: async () => {},
cleanupSessionlessExecutionHost: async () => {},
retainExecutionHostAfterClose: () => false,
},
settleExpiredSession: async (ref) =>
await teardownSessionResources({
appLog: 'run',
ref,
sessionStore: store,
platformCleanup: {
stopSnapshotHelper,
closeManagedBrowser: async () => {},
cleanupSessionlessExecutionHost: async () => {},
retainExecutionHostAfterClose: () => false,
},
}),
});
expect(scope.sessionName).toBe(address);
const running = scope.runLocked(async () => 'ran');
Expand Down
14 changes: 0 additions & 14 deletions packages/daemon/src/server/application-lifecycle-recovery.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,6 @@ type SessionFinalization = Readonly<{
daemonShutdown?: boolean;
}>;

/**
* Both finalization entry points admit the same facts and run the same effects; they differ only
* in how the selected owner is bound and disposed, which each entry point supplies as `bind`.
*/
async function finalizeSessionApplicationLifecycle(params: SessionFinalization): Promise<void> {
const { inspectFacts, bind, session, stateDir, runtimeHints } = params;
const input = {
Expand All @@ -57,16 +53,6 @@ async function finalizeSessionApplicationLifecycle(params: SessionFinalization):
});
}

export async function finalizeBoundSessionApplicationLifecycle(params: {
inspectFacts: InspectDeviceRuntimeFacts;
bindDevice: BindDeviceRuntime;
session: SessionState;
stateDir: string;
runtimeHints: RuntimeHintValues;
}): Promise<void> {
await finalizeSessionApplicationLifecycle({ ...params, bind: params.bindDevice });
}

/**
* Finish a persisted session's lifecycle-owned resources when the daemon, not a request, owns the
* teardown. The selected owner is admitted from eager facts, then bound through the provider-first
Expand Down
36 changes: 22 additions & 14 deletions packages/daemon/src/server/http/daemon-runtime.ts
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ import {
getLeaseRegistryExecutionLocks,
} from '../request-execution-scope.ts';
import type { LeaseEndReason } from '../../sessions/lease-lifecycle.ts';
import { teardownSessionResources } from '../../sessions/session-teardown.ts';
import { settleExpiredSession } from '../../sessions/session-teardown.ts';
import { resolveDaemonSessionTeardownTimeoutMs } from '../../sessions/session-teardown-budget.ts';
import { finalizeDaemonSessionApplicationLifecycle } from '../application-lifecycle-recovery.ts';
import { runtimeHintValues } from '../../sessions/session-runtime.ts';
Expand Down Expand Up @@ -328,6 +328,9 @@ async function startDaemonRuntimeOnHost(
leaseRegistry,
endSessionsForLease: async (leaseId, reason) =>
await endSessionsForLease(leaseId, reason ?? 'LEASE_RELEASED'),
// A request that ends an expired session retires it, so the application is always finalized.
settleExpiredSession: async (ref: SessionRef) =>
await settleIdleExpiredSession(ref, 'finalize'),
appLogAdmissionLedger,
audioProbeAdmissionLedger,
perfCaptureAdmissionLedger,
Expand Down Expand Up @@ -402,26 +405,31 @@ async function startDaemonRuntimeOnHost(
// survive so the next pass can retry rather than leave a claim owned by a process that no longer
// knows what it holds. So: resources, then the platform finalization that stops the execution host
// and releases its lease, then the claim, cleared last, by the reaper.
const settleIdleExpiredSession = async (ref: SessionRef): Promise<void> => {
const settleIdleExpiredSession = async (
ref: SessionRef,
afterResourceFailure: 'stop' | 'finalize' = 'stop',
): Promise<void> => {
const session = sessionStore.resolveCurrent(ref) ?? ref.session;
const sessionName = ref.address;
const runtimeHints = runtimeHintValues(sessionStore.getRuntimeHints(sessionName));
await teardownSessionResources({
await settleExpiredSession({
appLog: 'run',
ref,
sessionStore,
platformCleanup: deps.platformResourceCleanup,
});
await finalizeDaemonSessionApplicationLifecycle({
gateway: deviceRuntimeGateway,
scope: createRecoveryPlatformScope(),
session,
stateDir: baseDir,
runtimeHints,
// The one caller that must say so: this daemon is staying alive, so there is no shutdown
// phase a healthy runner could be deferred to. Taking the ordinary-close path stops the
// runner and releases its lease instead of parking it until process exit.
daemonLeaving: false,
afterResourceFailure,
finalizeApplication: async () =>
await finalizeDaemonSessionApplicationLifecycle({
gateway: deviceRuntimeGateway,
scope: createRecoveryPlatformScope(),
session,
stateDir: baseDir,
runtimeHints,
// The one caller that must say so: this daemon is staying alive, so there is no shutdown
// phase a healthy runner could be deferred to. Taking the ordinary-close path stops the
// runner and releases its lease instead of parking it until process exit.
daemonLeaving: false,
}),
});
// ADR 0012 R7 binds this teardown too — the healed `.ad` is committed by
// `SessionStore.finalizeRepairTeardown` — but NOT from in here. That call publishes the script
Expand Down
Loading
Loading