From 27b74bae596a64216c30b9052a42b2449d34ff9f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Pierzcha=C5=82a?= Date: Mon, 27 Jul 2026 13:30:01 +0000 Subject: [PATCH 1/8] test: frozen replay-compat corpus with expected verdicts (#1417) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- docs/agents/testing.md | 1 + scripts/check-affected/model.test.ts | 6 + scripts/check-affected/model.ts | 16 + test/replay-compat/README.md | 49 ++ test/replay-compat/corpus.test.ts | 88 +++ test/replay-compat/manifest.ts | 528 ++++++++++++++++++ ...ext-header-conflicting-platform.v0.15.1.ad | 5 + .../docs/context-header-full.v0.15.1.ad | 5 + .../docs/gesture-rotate-velocity.v0.16.8.ad | 6 + .../gesture-swipe-preset-duration.v0.16.8.ad | 5 + .../record-as-parameterized-fill.v0.15.1.ad | 7 + .../docs/target-v1-annotated-click.v0.20.0.ad | 6 + .../target-v1-unbound-annotation.v0.20.0.ad | 7 + .../docs/vars-env-after-action.v0.15.1.ad | 5 + .../docs/vars-parameterized.v0.15.1.ad | 7 + .../docs/vars-quoting-fallback.v0.15.1.ad | 6 + .../docs/vars-reserved-env-key.v0.15.1.ad | 4 + .../docs/wait-landmark-forms.v0.20.0.ad | 8 + .../examples/checkout-form-android.v0.15.1.ad | 21 + .../examples/checkout-form-android.v0.16.8.ad | 23 + .../scripts/examples/checkout-form.v0.15.1.ad | 21 + .../scripts/examples/checkout-form.v0.16.8.ad | 22 + .../examples/gesture-lab-android.v0.20.0.ad | 48 ++ .../scripts/examples/gesture-lab.v0.16.0.ad | 39 ++ .../scripts/examples/gesture-lab.v0.16.8.ad | 40 ++ .../scripts/examples/gesture-lab.v0.20.0.ad | 47 ++ .../android-01-settings.v0.11.0.ad | 14 + .../android-01-settings.v0.11.2.ad | 15 + .../android-02-deep-navigation.v0.11.2.ad | 18 + .../android-02-deep-navigation.v0.12.5.ad | 18 + .../android-03-scroll-discovery.v0.11.2.ad | 15 + .../android-04-text-input-keyboard.v0.11.2.ad | 25 + .../android-05-app-lifecycle.v0.11.2.ad | 18 + .../android-06-swipe-gestures.v0.11.2.ad | 25 + ...os-device-01-physical-lifecycle.v0.11.0.ad | 8 + ...os-device-01-physical-lifecycle.v0.11.2.ad | 8 + ...os-device-01-physical-lifecycle.v0.15.0.ad | 8 + ...os-device-01-physical-lifecycle.v0.17.0.ad | 8 + .../ios-simulator-01-settings.v0.11.0.ad | 12 + .../ios-simulator-01-settings.v0.11.2.ad | 12 + .../ios-simulator-01-settings.v0.15.0.ad | 12 + .../ios-simulator-01-settings.v0.17.0.ad | 12 + ...os-simulator-02-deep-navigation.v0.11.2.ad | 11 + ...os-simulator-02-deep-navigation.v0.15.0.ad | 11 + ...os-simulator-02-deep-navigation.v0.17.0.ad | 11 + ...s-simulator-03-scroll-discovery.v0.11.2.ad | 11 + ...s-simulator-03-scroll-discovery.v0.15.0.ad | 11 + ...imulator-04-text-input-keyboard.v0.11.2.ad | 11 + ...imulator-04-text-input-keyboard.v0.15.0.ad | 11 + .../ios-simulator-05-app-lifecycle.v0.11.2.ad | 13 + .../ios-simulator-05-app-lifecycle.v0.15.0.ad | 13 + ...ios-simulator-06-swipe-gestures.v0.11.2.ad | 16 + ...ios-simulator-06-swipe-gestures.v0.15.0.ad | 16 + .../linux-01-desktop-smoke.v0.11.8.ad | 12 + .../linux-01-desktop-smoke.v0.15.0.ad | 12 + .../macos-01-system-settings.v0.11.0.ad | 16 + .../macos-01-system-settings.v0.11.2.ad | 16 + .../macos-01-system-settings.v0.17.0.ad | 15 + vitest.config.ts | 4 + 59 files changed, 1458 insertions(+) create mode 100644 test/replay-compat/README.md create mode 100644 test/replay-compat/corpus.test.ts create mode 100644 test/replay-compat/manifest.ts create mode 100644 test/replay-compat/scripts/docs/context-header-conflicting-platform.v0.15.1.ad create mode 100644 test/replay-compat/scripts/docs/context-header-full.v0.15.1.ad create mode 100644 test/replay-compat/scripts/docs/gesture-rotate-velocity.v0.16.8.ad create mode 100644 test/replay-compat/scripts/docs/gesture-swipe-preset-duration.v0.16.8.ad create mode 100644 test/replay-compat/scripts/docs/record-as-parameterized-fill.v0.15.1.ad create mode 100644 test/replay-compat/scripts/docs/target-v1-annotated-click.v0.20.0.ad create mode 100644 test/replay-compat/scripts/docs/target-v1-unbound-annotation.v0.20.0.ad create mode 100644 test/replay-compat/scripts/docs/vars-env-after-action.v0.15.1.ad create mode 100644 test/replay-compat/scripts/docs/vars-parameterized.v0.15.1.ad create mode 100644 test/replay-compat/scripts/docs/vars-quoting-fallback.v0.15.1.ad create mode 100644 test/replay-compat/scripts/docs/vars-reserved-env-key.v0.15.1.ad create mode 100644 test/replay-compat/scripts/docs/wait-landmark-forms.v0.20.0.ad create mode 100644 test/replay-compat/scripts/examples/checkout-form-android.v0.15.1.ad create mode 100644 test/replay-compat/scripts/examples/checkout-form-android.v0.16.8.ad create mode 100644 test/replay-compat/scripts/examples/checkout-form.v0.15.1.ad create mode 100644 test/replay-compat/scripts/examples/checkout-form.v0.16.8.ad create mode 100644 test/replay-compat/scripts/examples/gesture-lab-android.v0.20.0.ad create mode 100644 test/replay-compat/scripts/examples/gesture-lab.v0.16.0.ad create mode 100644 test/replay-compat/scripts/examples/gesture-lab.v0.16.8.ad create mode 100644 test/replay-compat/scripts/examples/gesture-lab.v0.20.0.ad create mode 100644 test/replay-compat/scripts/integration/android-01-settings.v0.11.0.ad create mode 100644 test/replay-compat/scripts/integration/android-01-settings.v0.11.2.ad create mode 100644 test/replay-compat/scripts/integration/android-02-deep-navigation.v0.11.2.ad create mode 100644 test/replay-compat/scripts/integration/android-02-deep-navigation.v0.12.5.ad create mode 100644 test/replay-compat/scripts/integration/android-03-scroll-discovery.v0.11.2.ad create mode 100644 test/replay-compat/scripts/integration/android-04-text-input-keyboard.v0.11.2.ad create mode 100644 test/replay-compat/scripts/integration/android-05-app-lifecycle.v0.11.2.ad create mode 100644 test/replay-compat/scripts/integration/android-06-swipe-gestures.v0.11.2.ad create mode 100644 test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.11.0.ad create mode 100644 test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.11.2.ad create mode 100644 test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.15.0.ad create mode 100644 test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.17.0.ad create mode 100644 test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.11.0.ad create mode 100644 test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.11.2.ad create mode 100644 test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.15.0.ad create mode 100644 test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.17.0.ad create mode 100644 test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.11.2.ad create mode 100644 test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.15.0.ad create mode 100644 test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.17.0.ad create mode 100644 test/replay-compat/scripts/integration/ios-simulator-03-scroll-discovery.v0.11.2.ad create mode 100644 test/replay-compat/scripts/integration/ios-simulator-03-scroll-discovery.v0.15.0.ad create mode 100644 test/replay-compat/scripts/integration/ios-simulator-04-text-input-keyboard.v0.11.2.ad create mode 100644 test/replay-compat/scripts/integration/ios-simulator-04-text-input-keyboard.v0.15.0.ad create mode 100644 test/replay-compat/scripts/integration/ios-simulator-05-app-lifecycle.v0.11.2.ad create mode 100644 test/replay-compat/scripts/integration/ios-simulator-05-app-lifecycle.v0.15.0.ad create mode 100644 test/replay-compat/scripts/integration/ios-simulator-06-swipe-gestures.v0.11.2.ad create mode 100644 test/replay-compat/scripts/integration/ios-simulator-06-swipe-gestures.v0.15.0.ad create mode 100644 test/replay-compat/scripts/integration/linux-01-desktop-smoke.v0.11.8.ad create mode 100644 test/replay-compat/scripts/integration/linux-01-desktop-smoke.v0.15.0.ad create mode 100644 test/replay-compat/scripts/integration/macos-01-system-settings.v0.11.0.ad create mode 100644 test/replay-compat/scripts/integration/macos-01-system-settings.v0.11.2.ad create mode 100644 test/replay-compat/scripts/integration/macos-01-system-settings.v0.17.0.ad diff --git a/docs/agents/testing.md b/docs/agents/testing.md index 6e778d1b2..3a1e091d2 100644 --- a/docs/agents/testing.md +++ b/docs/agents/testing.md @@ -19,6 +19,7 @@ The mapping it encodes, for when you need to run a gate directly or reason about | CLI help/guidance (`src/cli/parser/cli-help.ts`, `src/cli-schema/`) | `pnpm exec vitest run src/cli/parser/__tests__ src/cli-schema/command-schema-guards.test.ts scripts/__tests__` — the `scripts/__tests__` gates enforce help-topic benchmark coverage and pin the bench's quoted CLI samples to the real renderers | | Help benchmark cases (`scripts/help-conformance-*.mjs`) | `pnpm exec vitest run scripts/__tests__` (deterministic gates); model-backed: `pnpm bench:help-conformance` (paid LLM calls, local only) | | SkillGym prompts/assertions | `pnpm test:skillgym:case ` (broad: `pnpm test:skillgym`, filter with `-- --tag fixture-smoke` or `-- --tag skill-guidance`) — agentic routing + local-help-consumption proof only; command-planning knowledge checks belong in the help bench | +| `.ad` grammar (`src/replay/script.ts`, gesture arity, replay vars) | `pnpm exec vitest run --project unit-core test/replay-compat` — the frozen replay-compat corpus asserts which released script surfaces still parse; a flipped verdict is edited in `test/replay-compat/manifest.ts`, never in the script | | Anything in `src/`, `test/`, `skills/` | `pnpm format` | Two traps worth naming: diff --git a/scripts/check-affected/model.test.ts b/scripts/check-affected/model.test.ts index ed73554fe..16f51fb42 100644 --- a/scripts/check-affected/model.test.ts +++ b/scripts/check-affected/model.test.ts @@ -114,6 +114,12 @@ test('a non-.ts fixture under an owned root fails open (format alone is not owne assert.equal(result.failOpenReasons[0]?.rule, 'ambiguous-path'); }); +test('a frozen replay-compat corpus script selects the unit lane that asserts it', () => { + const result = plan(['test/replay-compat/scripts/examples/gesture-lab.v0.16.8.ad']); + assert.equal(result.failOpen, false); + assert.ok(result.checks.includes('unit')); +}); + test('skills guidance change selects format + skillgym, not docs-only', () => { const result = plan(['skills/agent-device/SKILL.md']); assert.equal(result.failOpen, false); diff --git a/scripts/check-affected/model.ts b/scripts/check-affected/model.ts index 71e71c135..86a810440 100644 --- a/scripts/check-affected/model.ts +++ b/scripts/check-affected/model.ts @@ -228,6 +228,21 @@ const nodeIntegrationOwnership: OwnershipRule = ({ file }) => ? [reason('integration-node', file, 'node-integration', 'node --test integration smoke')] : []; +// The frozen replay-compat corpus (#1417) is `.ad` fixture data whose only +// consumer is `test/replay-compat/corpus.test.ts` in the unit lane, so a corpus +// addition has a derivable owner instead of failing open on its extension. +const replayCompatOwnership: OwnershipRule = ({ file }) => + file.startsWith('test/replay-compat/') && file.endsWith('.ad') + ? [ + reason( + 'unit', + file, + 'own:replay-compat', + 'frozen replay-compat corpus is asserted by the unit-lane corpus test', + ), + ] + : []; + // SkillGym validates skill guidance (`skills/`) and owns its harness // (`test/skillgym/`); the Testing Matrix in docs/agents/testing.md routes // skill-prompt/assertion changes here. @@ -294,6 +309,7 @@ const OWNERSHIP_RULES: readonly OwnershipRule[] = [ srcProdGate, vitestRelatedOwnership, nodeIntegrationOwnership, + replayCompatOwnership, skillgymOwnership, buildOwnership, ]; diff --git a/test/replay-compat/README.md b/test/replay-compat/README.md new file mode 100644 index 000000000..82c4e7790 --- /dev/null +++ b/test/replay-compat/README.md @@ -0,0 +1,49 @@ +# Replay-compat corpus + +`.ad` scripts as **released** versions of agent-device wrote them, each paired with the verdict +today's parser owes it: `parses`, or `fails` with an error code and the migration hint substring. + +`corpus.test.ts` runs in `unit-core` and asserts every verdict, so a grammar change cannot quietly +alter which historical script surface still loads. #1417. + +## The freeze rule + +**Never edit a script under `scripts/` to make a parser change pass.** + +A corpus script is a recording that already exists in the wild; editing it deletes the evidence the +gate is here to keep. When a grammar change flips a verdict: + +1. Leave the script byte-identical. +2. Update its `verdict` in `manifest.ts` in the same PR. +3. Say in the PR description which historical surface broke, and why the new refusal is the intended + migration rather than a regression. + +A verdict that changes from `parses` to `fails` is a compatibility break by definition. That is +allowed — the corpus only insists it be deliberate, reviewed, and paired with a migration hint good +enough to put in front of whoever recorded the script. + +## What is in the corpus + +- `scripts/integration/` and `scripts/examples/` — mined from the git history of + `test/integration/replays` and `examples/test-app/replays`: one entry per distinct file content at + a release tag. +- `scripts/docs/` — surfaces the released grammar wrote that those two suites never exercised + (`${VAR}` parameterization, reserved/ordering `env` refusals, `wait stable` and landmark waits, + `target-v1` annotations, retired `gesture rotate` velocity and `gesture swipe` durationMs). + +Provenance lives in `manifest.ts`: `recordedBy` is the released tag whose grammar produced the +surface, and `source` is where it was mined from at that tag. + +**Released surfaces only.** A grammar state that only ever existed between commits is not compat +surface — `git tag --contains ` decides, and unreleased shapes stay out (AGENTS.md, +"Unreleased API surface dies free"). The corpus is baselined against tags up to v0.20.0. + +## Adding an entry + +1. Find the surface at a release tag (`git show :`, or the released grammar/docs that + emitted it) and copy it verbatim into `scripts//..ad`. +2. Add a `manifest.ts` entry with `recordedBy`, `source`, `covers`, and the observed verdict. +3. Run `pnpm exec vitest run --project unit-core test/replay-compat`. + +Add an entry when a change retires, renames, or narrows a `.ad` form — the corpus is the record of +what that costs someone with a saved recording. diff --git a/test/replay-compat/corpus.test.ts b/test/replay-compat/corpus.test.ts new file mode 100644 index 000000000..4cdf8bc37 --- /dev/null +++ b/test/replay-compat/corpus.test.ts @@ -0,0 +1,88 @@ +import { describe, expect, test } from 'vitest'; +import { readdirSync, readFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { AppError } from '../../src/kernel/errors.ts'; +import { parseReplayScriptDetailed, readReplayScriptMetadata } from '../../src/replay/script.ts'; +import { + REPLAY_COMPAT_CORPUS, + type ReplayCompatCoverage, + type ReplayCompatEntry, +} from './manifest.ts'; + +const CORPUS_DIR = fileURLToPath(new URL('.', import.meta.url)); + +/** + * Every historical surface the corpus exists to hold still. A coverage tag with + * no entry means the corpus stopped covering a compat surface #1417 named. + */ +const REQUIRED_COVERAGE: ReplayCompatCoverage[] = [ + 'context-header', + 'env-vars', + 'quoting', + 'retired-gesture', + 'target-annotation', + 'wait-landmark', +]; + +function readCorpusScript(entry: ReplayCompatEntry): string { + return readFileSync(join(CORPUS_DIR, entry.file), 'utf8'); +} + +/** The whole parse surface a `.ad` file meets before replay executes anything. */ +function parseCorpusScript(script: string): void { + readReplayScriptMetadata(script); + parseReplayScriptDetailed(script); +} + +function listCorpusScriptFiles(): string[] { + const areas = readdirSync(join(CORPUS_DIR, 'scripts'), { withFileTypes: true }); + return areas.flatMap((area) => + readdirSync(join(CORPUS_DIR, 'scripts', area.name)) + .filter((name) => name.endsWith('.ad')) + .map((name) => `scripts/${area.name}/${name}`), + ); +} + +describe('replay-compat corpus', () => { + test.each(REPLAY_COMPAT_CORPUS.map((entry) => [entry.id, entry] as const))('%s', (_id, entry) => { + const script = readCorpusScript(entry); + if (entry.verdict.kind === 'parses') { + expect(() => { + parseCorpusScript(script); + }).not.toThrow(); + return; + } + let thrown: unknown; + try { + parseCorpusScript(script); + } catch (error) { + thrown = error; + } + expect(thrown, `expected ${entry.id} to fail parsing`).toBeInstanceOf(AppError); + const error = thrown as AppError; + expect(error.code).toBe(entry.verdict.code); + expect(error.message).toContain(entry.verdict.hint); + }); + + test('every corpus script is claimed by exactly one manifest entry', () => { + const claimed = REPLAY_COMPAT_CORPUS.map((entry) => entry.file); + expect([...claimed].sort()).toEqual(listCorpusScriptFiles().sort()); + expect(new Set(claimed).size).toBe(claimed.length); + }); + + test('entry ids are unique and name the producing released version', () => { + const ids = REPLAY_COMPAT_CORPUS.map((entry) => entry.id); + expect(new Set(ids).size).toBe(ids.length); + for (const entry of REPLAY_COMPAT_CORPUS) { + expect(entry.recordedBy, entry.id).toMatch(/^v\d+\.\d+\.\d+$/); + expect(entry.source, entry.id).toContain(entry.recordedBy); + expect(entry.file, entry.id).toContain(`.${entry.recordedBy}.ad`); + } + }); + + test('the corpus covers every compat surface it is required to freeze', () => { + const covered = new Set(REPLAY_COMPAT_CORPUS.flatMap((entry) => entry.covers)); + expect([...covered].sort()).toEqual([...REQUIRED_COVERAGE].sort()); + }); +}); diff --git a/test/replay-compat/manifest.ts b/test/replay-compat/manifest.ts new file mode 100644 index 000000000..837ed4335 --- /dev/null +++ b/test/replay-compat/manifest.ts @@ -0,0 +1,528 @@ +/** + * The frozen `.ad` replay-compat corpus (#1417): one entry per historical script + * surface a RELEASED version could have written, paired with the verdict today's + * parser must return for it. + * + * The scripts under `scripts/` are FROZEN. A grammar change that flips a verdict + * edits the verdict here — never the script — and says why in the same PR. See + * `README.md`. + */ + +import type { AppErrorCode } from '../../src/kernel/errors.ts'; + +/** `parses`, or the migration refusal the parser owes a script of this vintage. */ +export type ReplayCompatVerdict = + | { kind: 'parses' } + | { kind: 'fails'; code: AppErrorCode; hint: string }; + +/** The historical surface an entry exists to hold still. */ +export type ReplayCompatCoverage = + | 'context-header' + | 'env-vars' + | 'quoting' + | 'retired-gesture' + | 'target-annotation' + | 'wait-landmark'; + +export type ReplayCompatEntry = { + id: string; + /** Corpus-relative path of the frozen script. */ + file: string; + /** The released tag whose grammar produced this surface. */ + recordedBy: string; + /** Where the surface was mined from, at `recordedBy`. */ + source: string; + covers: ReplayCompatCoverage[]; + verdict: ReplayCompatVerdict; + /** Why this entry exists, when the path and verdict do not say it. */ + note?: string; +}; + +/** + * Mined from the git history of `test/integration/replays` and + * `examples/test-app/replays` (one entry per distinct content at a release tag), + * plus surfaces the released grammar wrote that those suites never exercised. + * Git-history-only grammar states that never shipped are deliberately absent. + */ +export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ + { + id: 'integration/android-01-settings@v0.11.0', + file: 'scripts/integration/android-01-settings.v0.11.0.ad', + recordedBy: 'v0.11.0', + source: 'test/integration/replays/android/01-settings.ad@v0.11.0', + covers: ['context-header', 'quoting'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/android-01-settings@v0.11.2', + file: 'scripts/integration/android-01-settings.v0.11.2.ad', + recordedBy: 'v0.11.2', + source: 'test/integration/replays/android/01-settings.ad@v0.11.2', + covers: ['context-header', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/ios-device-01-physical-lifecycle@v0.11.0', + file: 'scripts/integration/ios-device-01-physical-lifecycle.v0.11.0.ad', + recordedBy: 'v0.11.0', + source: 'test/integration/replays/ios/device/01-physical-lifecycle.ad@v0.11.0', + covers: ['context-header', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/ios-device-01-physical-lifecycle@v0.11.2', + file: 'scripts/integration/ios-device-01-physical-lifecycle.v0.11.2.ad', + recordedBy: 'v0.11.2', + source: 'test/integration/replays/ios/device/01-physical-lifecycle.ad@v0.11.2', + covers: ['context-header', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/ios-device-01-physical-lifecycle@v0.15.0', + file: 'scripts/integration/ios-device-01-physical-lifecycle.v0.15.0.ad', + recordedBy: 'v0.15.0', + source: 'test/integration/replays/ios/device/01-physical-lifecycle.ad@v0.15.0', + covers: ['context-header', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/ios-device-01-physical-lifecycle@v0.17.0', + file: 'scripts/integration/ios-device-01-physical-lifecycle.v0.17.0.ad', + recordedBy: 'v0.17.0', + source: 'test/integration/replays/ios/device/01-physical-lifecycle.ad@v0.17.0', + covers: ['context-header', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/ios-simulator-01-settings@v0.11.0', + file: 'scripts/integration/ios-simulator-01-settings.v0.11.0.ad', + recordedBy: 'v0.11.0', + source: 'test/integration/replays/ios/simulator/01-settings.ad@v0.11.0', + covers: ['context-header', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/ios-simulator-01-settings@v0.11.2', + file: 'scripts/integration/ios-simulator-01-settings.v0.11.2.ad', + recordedBy: 'v0.11.2', + source: 'test/integration/replays/ios/simulator/01-settings.ad@v0.11.2', + covers: ['context-header', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/ios-simulator-01-settings@v0.15.0', + file: 'scripts/integration/ios-simulator-01-settings.v0.15.0.ad', + recordedBy: 'v0.15.0', + source: 'test/integration/replays/ios/simulator/01-settings.ad@v0.15.0', + covers: ['context-header', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/ios-simulator-01-settings@v0.17.0', + file: 'scripts/integration/ios-simulator-01-settings.v0.17.0.ad', + recordedBy: 'v0.17.0', + source: 'test/integration/replays/ios/simulator/01-settings.ad@v0.17.0', + covers: ['context-header', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/macos-01-system-settings@v0.11.0', + file: 'scripts/integration/macos-01-system-settings.v0.11.0.ad', + recordedBy: 'v0.11.0', + source: 'test/integration/replays/macos/01-system-settings.ad@v0.11.0', + covers: ['context-header', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/macos-01-system-settings@v0.11.2', + file: 'scripts/integration/macos-01-system-settings.v0.11.2.ad', + recordedBy: 'v0.11.2', + source: 'test/integration/replays/macos/01-system-settings.ad@v0.11.2', + covers: ['context-header', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/macos-01-system-settings@v0.17.0', + file: 'scripts/integration/macos-01-system-settings.v0.17.0.ad', + recordedBy: 'v0.17.0', + source: 'test/integration/replays/macos/01-system-settings.ad@v0.17.0', + covers: ['context-header', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/android-02-deep-navigation@v0.11.2', + file: 'scripts/integration/android-02-deep-navigation.v0.11.2.ad', + recordedBy: 'v0.11.2', + source: 'test/integration/replays/android/02-deep-navigation.ad@v0.11.2', + covers: ['context-header', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/android-02-deep-navigation@v0.12.5', + file: 'scripts/integration/android-02-deep-navigation.v0.12.5.ad', + recordedBy: 'v0.12.5', + source: 'test/integration/replays/android/02-deep-navigation.ad@v0.12.5', + covers: ['context-header', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/android-03-scroll-discovery@v0.11.2', + file: 'scripts/integration/android-03-scroll-discovery.v0.11.2.ad', + recordedBy: 'v0.11.2', + source: 'test/integration/replays/android/03-scroll-discovery.ad@v0.11.2', + covers: ['context-header', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/android-04-text-input-keyboard@v0.11.2', + file: 'scripts/integration/android-04-text-input-keyboard.v0.11.2.ad', + recordedBy: 'v0.11.2', + source: 'test/integration/replays/android/04-text-input-keyboard.ad@v0.11.2', + covers: ['context-header', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/android-05-app-lifecycle@v0.11.2', + file: 'scripts/integration/android-05-app-lifecycle.v0.11.2.ad', + recordedBy: 'v0.11.2', + source: 'test/integration/replays/android/05-app-lifecycle.ad@v0.11.2', + covers: ['context-header', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/android-06-swipe-gestures@v0.11.2', + file: 'scripts/integration/android-06-swipe-gestures.v0.11.2.ad', + recordedBy: 'v0.11.2', + source: 'test/integration/replays/android/06-swipe-gestures.ad@v0.11.2', + covers: ['context-header', 'retired-gesture', 'wait-landmark'], + verdict: { + kind: 'fails', + code: 'INVALID_ARGS', + hint: 'swipe accepts 4 arguments: x1 y1 x2 y2 (line 8). The trailing durationMs positional was removed: use "gesture pan 206 700 0 -450 300" for the same timed drag, or "swipe 206 700 206 250" for a default-duration swipe.', + }, + note: '#1393 retired the trailing swipe durationMs; a v0.11.2 Android recording carries it and must get the migration, not a silent default-duration swipe.', + }, + { + id: 'integration/ios-simulator-02-deep-navigation@v0.11.2', + file: 'scripts/integration/ios-simulator-02-deep-navigation.v0.11.2.ad', + recordedBy: 'v0.11.2', + source: 'test/integration/replays/ios/simulator/02-deep-navigation.ad@v0.11.2', + covers: ['context-header', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/ios-simulator-02-deep-navigation@v0.15.0', + file: 'scripts/integration/ios-simulator-02-deep-navigation.v0.15.0.ad', + recordedBy: 'v0.15.0', + source: 'test/integration/replays/ios/simulator/02-deep-navigation.ad@v0.15.0', + covers: ['context-header', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/ios-simulator-02-deep-navigation@v0.17.0', + file: 'scripts/integration/ios-simulator-02-deep-navigation.v0.17.0.ad', + recordedBy: 'v0.17.0', + source: 'test/integration/replays/ios/simulator/02-deep-navigation.ad@v0.17.0', + covers: ['context-header', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/ios-simulator-03-scroll-discovery@v0.11.2', + file: 'scripts/integration/ios-simulator-03-scroll-discovery.v0.11.2.ad', + recordedBy: 'v0.11.2', + source: 'test/integration/replays/ios/simulator/03-scroll-discovery.ad@v0.11.2', + covers: ['context-header', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/ios-simulator-03-scroll-discovery@v0.15.0', + file: 'scripts/integration/ios-simulator-03-scroll-discovery.v0.15.0.ad', + recordedBy: 'v0.15.0', + source: 'test/integration/replays/ios/simulator/03-scroll-discovery.ad@v0.15.0', + covers: ['context-header', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/ios-simulator-04-text-input-keyboard@v0.11.2', + file: 'scripts/integration/ios-simulator-04-text-input-keyboard.v0.11.2.ad', + recordedBy: 'v0.11.2', + source: 'test/integration/replays/ios/simulator/04-text-input-keyboard.ad@v0.11.2', + covers: ['context-header', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/ios-simulator-04-text-input-keyboard@v0.15.0', + file: 'scripts/integration/ios-simulator-04-text-input-keyboard.v0.15.0.ad', + recordedBy: 'v0.15.0', + source: 'test/integration/replays/ios/simulator/04-text-input-keyboard.ad@v0.15.0', + covers: ['context-header', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/ios-simulator-05-app-lifecycle@v0.11.2', + file: 'scripts/integration/ios-simulator-05-app-lifecycle.v0.11.2.ad', + recordedBy: 'v0.11.2', + source: 'test/integration/replays/ios/simulator/05-app-lifecycle.ad@v0.11.2', + covers: ['context-header', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/ios-simulator-05-app-lifecycle@v0.15.0', + file: 'scripts/integration/ios-simulator-05-app-lifecycle.v0.15.0.ad', + recordedBy: 'v0.15.0', + source: 'test/integration/replays/ios/simulator/05-app-lifecycle.ad@v0.15.0', + covers: ['context-header', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/ios-simulator-06-swipe-gestures@v0.11.2', + file: 'scripts/integration/ios-simulator-06-swipe-gestures.v0.11.2.ad', + recordedBy: 'v0.11.2', + source: 'test/integration/replays/ios/simulator/06-swipe-gestures.ad@v0.11.2', + covers: ['context-header', 'retired-gesture', 'wait-landmark'], + verdict: { + kind: 'fails', + code: 'INVALID_ARGS', + hint: 'swipe accepts 4 arguments: x1 y1 x2 y2 (line 6). The trailing durationMs positional was removed: use "gesture pan 197 650 0 -350 300" for the same timed drag, or "swipe 197 650 197 300" for a default-duration swipe.', + }, + }, + { + id: 'integration/ios-simulator-06-swipe-gestures@v0.15.0', + file: 'scripts/integration/ios-simulator-06-swipe-gestures.v0.15.0.ad', + recordedBy: 'v0.15.0', + source: 'test/integration/replays/ios/simulator/06-swipe-gestures.ad@v0.15.0', + covers: ['context-header', 'retired-gesture', 'wait-landmark'], + verdict: { + kind: 'fails', + code: 'INVALID_ARGS', + hint: 'swipe accepts 4 arguments: x1 y1 x2 y2 (line 6). The trailing durationMs positional was removed: use "gesture pan 197 650 0 -350 300" for the same timed drag, or "swipe 197 650 197 300" for a default-duration swipe.', + }, + }, + { + id: 'integration/linux-01-desktop-smoke@v0.11.8', + file: 'scripts/integration/linux-01-desktop-smoke.v0.11.8.ad', + recordedBy: 'v0.11.8', + source: 'test/integration/replays/linux/01-desktop-smoke.ad@v0.11.8', + covers: ['context-header', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'integration/linux-01-desktop-smoke@v0.15.0', + file: 'scripts/integration/linux-01-desktop-smoke.v0.15.0.ad', + recordedBy: 'v0.15.0', + source: 'test/integration/replays/linux/01-desktop-smoke.ad@v0.15.0', + covers: ['context-header', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'examples/checkout-form-android@v0.15.1', + file: 'scripts/examples/checkout-form-android.v0.15.1.ad', + recordedBy: 'v0.15.1', + source: 'examples/test-app/replays/checkout-form-android.ad@v0.15.1', + covers: ['context-header', 'env-vars', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'examples/checkout-form-android@v0.16.8', + file: 'scripts/examples/checkout-form-android.v0.16.8.ad', + recordedBy: 'v0.16.8', + source: 'examples/test-app/replays/checkout-form-android.ad@v0.16.8', + covers: ['context-header', 'env-vars', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'examples/checkout-form@v0.15.1', + file: 'scripts/examples/checkout-form.v0.15.1.ad', + recordedBy: 'v0.15.1', + source: 'examples/test-app/replays/checkout-form.ad@v0.15.1', + covers: ['context-header', 'env-vars', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'examples/checkout-form@v0.16.8', + file: 'scripts/examples/checkout-form.v0.16.8.ad', + recordedBy: 'v0.16.8', + source: 'examples/test-app/replays/checkout-form.ad@v0.16.8', + covers: ['context-header', 'env-vars', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'examples/gesture-lab@v0.16.0', + file: 'scripts/examples/gesture-lab.v0.16.0.ad', + recordedBy: 'v0.16.0', + source: 'examples/test-app/replays/gesture-lab.ad@v0.16.0', + covers: ['context-header', 'env-vars', 'quoting', 'retired-gesture', 'wait-landmark'], + verdict: { + kind: 'fails', + code: 'INVALID_ARGS', + hint: 'gesture fling accepts at most 4 arguments: direction x y [distance] (line 15). The trailing durationMs positional was removed: use "gesture fling up 195 443 80", or gesture pan for timed movement.', + }, + note: '#1393 retired the trailing fling durationMs; the shipped gesture-lab example wrote it until v0.20.0.', + }, + { + id: 'examples/gesture-lab@v0.16.8', + file: 'scripts/examples/gesture-lab.v0.16.8.ad', + recordedBy: 'v0.16.8', + source: 'examples/test-app/replays/gesture-lab.ad@v0.16.8', + covers: ['context-header', 'env-vars', 'quoting', 'retired-gesture', 'wait-landmark'], + verdict: { + kind: 'fails', + code: 'INVALID_ARGS', + hint: 'gesture fling accepts at most 4 arguments: direction x y [distance] (line 16). The trailing durationMs positional was removed: use "gesture fling up 195 443 80", or gesture pan for timed movement.', + }, + }, + { + id: 'examples/gesture-lab@v0.20.0', + file: 'scripts/examples/gesture-lab.v0.20.0.ad', + recordedBy: 'v0.20.0', + source: 'examples/test-app/replays/gesture-lab.ad@v0.20.0', + covers: ['context-header', 'env-vars', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'examples/gesture-lab-android@v0.20.0', + file: 'scripts/examples/gesture-lab-android.v0.20.0.ad', + recordedBy: 'v0.20.0', + source: 'examples/test-app/replays/gesture-lab-android.ad@v0.20.0', + covers: ['context-header', 'env-vars', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + }, + { + id: 'docs/vars-parameterized@v0.15.1', + file: 'scripts/docs/vars-parameterized.v0.15.1.ad', + recordedBy: 'v0.15.1', + source: 'website/docs/docs/replay-e2e.md@v0.15.1 ("Parametrise `.ad` scripts")', + covers: ['context-header', 'env-vars'], + verdict: { kind: 'parses' }, + note: 'The shipped `${VAR}` surface as documented by the release that introduced it.', + }, + { + id: 'docs/vars-quoting-fallback@v0.15.1', + file: 'scripts/docs/vars-quoting-fallback.v0.15.1.ad', + recordedBy: 'v0.15.1', + source: + 'website/docs/docs/replay-e2e.md@v0.15.1 ("Substitution happens inside parsed string values" / "Fallback and escape")', + covers: ['context-header', 'env-vars', 'quoting', 'wait-landmark'], + verdict: { kind: 'parses' }, + note: 'Quoted selector values, escaped inner quotes, and `${VAR:-default}` in one script.', + }, + { + id: 'docs/record-as-parameterized-fill@v0.15.1', + file: 'scripts/docs/record-as-parameterized-fill.v0.15.1.ad', + recordedBy: 'v0.15.1', + source: 'website/docs/docs/replay-e2e.md@v0.15.1 + src/replay/vars.ts@v0.15.1', + covers: ['context-header', 'env-vars', 'quoting'], + verdict: { kind: 'parses' }, + note: '`--record-as` projects a recorded value into exactly this shipped surface: an `env` declaration plus a `${VAR}` fill argument. The flag itself is not in any tag as of the v0.20.0 baseline, so the corpus freezes the released script surface, not the unreleased recorder flag.', + }, + { + id: 'docs/vars-reserved-env-key@v0.15.1', + file: 'scripts/docs/vars-reserved-env-key.v0.15.1.ad', + recordedBy: 'v0.15.1', + source: + 'website/docs/docs/replay-e2e.md@v0.15.1 ("User-defined keys starting with AD_ are rejected")', + covers: ['context-header', 'env-vars'], + verdict: { + kind: 'fails', + code: 'INVALID_ARGS', + hint: 'Invalid env key "AD_FOO" on line 2: the AD_* namespace is reserved for built-in variables. Rename AD_FOO to avoid the AD_ prefix.', + }, + note: 'Reserved-namespace refusal shipped with the variable surface itself.', + }, + { + id: 'docs/vars-env-after-action@v0.15.1', + file: 'scripts/docs/vars-env-after-action.v0.15.1.ad', + recordedBy: 'v0.15.1', + source: 'src/replay/script.ts@v0.15.1 (env ordering guard)', + covers: ['context-header', 'env-vars'], + verdict: { + kind: 'fails', + code: 'INVALID_ARGS', + hint: 'env directives must precede all actions (line 4).', + }, + note: 'Header-ordering guard for `env`, shipped with the variable surface.', + }, + { + id: 'docs/context-header-full@v0.15.1', + file: 'scripts/docs/context-header-full.v0.15.1.ad', + recordedBy: 'v0.15.1', + source: 'website/docs/docs/replay-e2e.md@v0.15.1 ("Run a lightweight `.ad` suite")', + covers: ['context-header', 'wait-landmark', 'quoting'], + verdict: { kind: 'parses' }, + note: 'Every documented context key on one header: platform, target, timeout, retries.', + }, + { + id: 'docs/context-header-conflicting-platform@v0.15.1', + file: 'scripts/docs/context-header-conflicting-platform.v0.15.1.ad', + recordedBy: 'v0.15.1', + source: + 'website/docs/docs/replay-e2e.md@v0.15.1 ("duplicate keys in the context header fail fast")', + covers: ['context-header'], + verdict: { + kind: 'fails', + code: 'INVALID_ARGS', + hint: 'Conflicting replay test metadata "platform" in context header: android vs ios.', + }, + note: 'Documented fail-fast for a conflicting header key.', + }, + { + id: 'docs/wait-landmark-forms@v0.20.0', + file: 'scripts/docs/wait-landmark-forms.v0.20.0.ad', + recordedBy: 'v0.20.0', + source: + 'src/commands/capture/wait-command-contract.ts@v0.20.0 + src/replay/script-utils.ts@v0.20.0', + covers: ['context-header', 'wait-landmark', 'quoting'], + verdict: { kind: 'parses' }, + note: 'All released wait kinds a recording can carry: duration, `stable`, selector landmark (#1349), and a generation-pinned `@ref`.', + }, + { + id: 'docs/target-v1-annotated-click@v0.20.0', + file: 'scripts/docs/target-v1-annotated-click.v0.20.0.ad', + recordedBy: 'v0.20.0', + source: 'docs/adr/0012-interactive-replay.md@v0.20.0 + src/replay/target-identity.ts@v0.20.0', + covers: ['context-header', 'target-annotation', 'quoting'], + verdict: { kind: 'parses' }, + note: 'ADR 0012 identity annotation bound to its action line.', + }, + { + id: 'docs/target-v1-unbound-annotation@v0.20.0', + file: 'scripts/docs/target-v1-unbound-annotation.v0.20.0.ad', + recordedBy: 'v0.20.0', + source: 'src/replay/script.ts@v0.20.0 (annotation binding guard)', + covers: ['context-header', 'target-annotation'], + verdict: { + kind: 'fails', + code: 'INVALID_ARGS', + hint: 'target-v1 annotation on line 4 must be immediately followed by its action line (line 5 is blank).', + }, + note: 'An annotation separated from its action is a hard parse refusal, not a comment.', + }, + { + id: 'docs/gesture-rotate-velocity@v0.16.8', + file: 'scripts/docs/gesture-rotate-velocity.v0.16.8.ad', + recordedBy: 'v0.16.8', + source: + 'src/commands/cli-grammar/gesture.ts@v0.16.8 (rotateGesturePositionals emitted a trailing velocity)', + covers: ['context-header', 'wait-landmark', 'quoting', 'retired-gesture'], + verdict: { + kind: 'fails', + code: 'INVALID_ARGS', + hint: 'gesture rotate accepts at most 3 arguments: degrees [x] [y] (line 5). The trailing velocity positional was removed: use "gesture rotate 35 195 443"; rotation pacing derives from degrees.', + }, + note: 'Pre-removal rotate velocity positional as the v0.16.8 recorder wrote it.', + }, + { + id: 'docs/gesture-swipe-preset-duration@v0.16.8', + file: 'scripts/docs/gesture-swipe-preset-duration.v0.16.8.ad', + recordedBy: 'v0.16.8', + source: + 'src/commands/cli-grammar/gesture.ts@v0.16.8 (swipePresetPositionals emitted a trailing durationMs)', + covers: ['context-header', 'retired-gesture'], + verdict: { + kind: 'fails', + code: 'INVALID_ARGS', + hint: 'gesture swipe accepts 1 argument: preset (line 4). The trailing durationMs positional was removed: use "gesture swipe up", or gesture pan for timed movement.', + }, + note: 'Pre-removal `gesture swipe` durationMs positional as the v0.16.8 recorder wrote it.', + }, +]; diff --git a/test/replay-compat/scripts/docs/context-header-conflicting-platform.v0.15.1.ad b/test/replay-compat/scripts/docs/context-header-conflicting-platform.v0.15.1.ad new file mode 100644 index 000000000..0a7911e34 --- /dev/null +++ b/test/replay-compat/scripts/docs/context-header-conflicting-platform.v0.15.1.ad @@ -0,0 +1,5 @@ +context platform=android +context platform=ios + +open settings --relaunch +close diff --git a/test/replay-compat/scripts/docs/context-header-full.v0.15.1.ad b/test/replay-compat/scripts/docs/context-header-full.v0.15.1.ad new file mode 100644 index 000000000..769213ff7 --- /dev/null +++ b/test/replay-compat/scripts/docs/context-header-full.v0.15.1.ad @@ -0,0 +1,5 @@ +context platform=android target=tv timeout=60000 retries=2 + +open settings --relaunch +wait "label=Apps" 5000 +close diff --git a/test/replay-compat/scripts/docs/gesture-rotate-velocity.v0.16.8.ad b/test/replay-compat/scripts/docs/gesture-rotate-velocity.v0.16.8.ad new file mode 100644 index 000000000..b70c8266e --- /dev/null +++ b/test/replay-compat/scripts/docs/gesture-rotate-velocity.v0.16.8.ad @@ -0,0 +1,6 @@ +context platform=ios + +open "Agent Device Tester" --relaunch +wait "Gesture lab" 30000 +gesture rotate 35 195 443 800 +close diff --git a/test/replay-compat/scripts/docs/gesture-swipe-preset-duration.v0.16.8.ad b/test/replay-compat/scripts/docs/gesture-swipe-preset-duration.v0.16.8.ad new file mode 100644 index 000000000..91109f932 --- /dev/null +++ b/test/replay-compat/scripts/docs/gesture-swipe-preset-duration.v0.16.8.ad @@ -0,0 +1,5 @@ +context platform=android + +open settings --relaunch +gesture swipe up 400 +close diff --git a/test/replay-compat/scripts/docs/record-as-parameterized-fill.v0.15.1.ad b/test/replay-compat/scripts/docs/record-as-parameterized-fill.v0.15.1.ad new file mode 100644 index 000000000..454b45a9b --- /dev/null +++ b/test/replay-compat/scripts/docs/record-as-parameterized-fill.v0.15.1.ad @@ -0,0 +1,7 @@ +context platform=ios +env PASSWORD="" + +open "Agent Device Tester" --relaunch +fill id="field-email" "ada@example.com" +fill id="password" "${PASSWORD}" +click id="submit-order" diff --git a/test/replay-compat/scripts/docs/target-v1-annotated-click.v0.20.0.ad b/test/replay-compat/scripts/docs/target-v1-annotated-click.v0.20.0.ad new file mode 100644 index 000000000..4470a1ebd --- /dev/null +++ b/test/replay-compat/scripts/docs/target-v1-annotated-click.v0.20.0.ad @@ -0,0 +1,6 @@ +context platform=macos target=desktop + +open "System Settings" +# agent-device:target-v1 {"id":"save","role":"button","label":"Save","ancestry":[{"role":"toolbar","label":"Editor"},{"role":"window"}],"sibling":0,"viewportOrder":0,"scrollRegion":{"role":"scrollview","id":"editor-scroll"},"verification":"verified"} +click @e12 "Save" +close diff --git a/test/replay-compat/scripts/docs/target-v1-unbound-annotation.v0.20.0.ad b/test/replay-compat/scripts/docs/target-v1-unbound-annotation.v0.20.0.ad new file mode 100644 index 000000000..ba0b5db80 --- /dev/null +++ b/test/replay-compat/scripts/docs/target-v1-unbound-annotation.v0.20.0.ad @@ -0,0 +1,7 @@ +context platform=macos target=desktop + +open "System Settings" +# agent-device:target-v1 {"role":"button","label":"Save","ancestry":[],"sibling":0,"viewportOrder":0,"verification":"verified"} + +click @e12 "Save" +close diff --git a/test/replay-compat/scripts/docs/vars-env-after-action.v0.15.1.ad b/test/replay-compat/scripts/docs/vars-env-after-action.v0.15.1.ad new file mode 100644 index 000000000..e821cbecd --- /dev/null +++ b/test/replay-compat/scripts/docs/vars-env-after-action.v0.15.1.ad @@ -0,0 +1,5 @@ +context platform=android + +open settings --relaunch +env APP_ID=settings +click "label=${APP_ID}" diff --git a/test/replay-compat/scripts/docs/vars-parameterized.v0.15.1.ad b/test/replay-compat/scripts/docs/vars-parameterized.v0.15.1.ad new file mode 100644 index 000000000..f0b749af1 --- /dev/null +++ b/test/replay-compat/scripts/docs/vars-parameterized.v0.15.1.ad @@ -0,0 +1,7 @@ +context platform=android +env APP_ID=settings +env WAIT_SHORT=500 + +open ${APP_ID} --relaunch +wait ${WAIT_SHORT} +click "label=${APP_ID}" diff --git a/test/replay-compat/scripts/docs/vars-quoting-fallback.v0.15.1.ad b/test/replay-compat/scripts/docs/vars-quoting-fallback.v0.15.1.ad new file mode 100644 index 000000000..fa4868c42 --- /dev/null +++ b/test/replay-compat/scripts/docs/vars-quoting-fallback.v0.15.1.ad @@ -0,0 +1,6 @@ +context platform=android +env SETTINGS="label=Account || label=Profile || label=User" + +click "${SETTINGS}" +wait "label=\"Software Update\" || label=About" 5000 +wait ${WAIT_MS:-500} diff --git a/test/replay-compat/scripts/docs/vars-reserved-env-key.v0.15.1.ad b/test/replay-compat/scripts/docs/vars-reserved-env-key.v0.15.1.ad new file mode 100644 index 000000000..d384eeb7f --- /dev/null +++ b/test/replay-compat/scripts/docs/vars-reserved-env-key.v0.15.1.ad @@ -0,0 +1,4 @@ +context platform=android +env AD_FOO=1 + +open settings --relaunch diff --git a/test/replay-compat/scripts/docs/wait-landmark-forms.v0.20.0.ad b/test/replay-compat/scripts/docs/wait-landmark-forms.v0.20.0.ad new file mode 100644 index 000000000..d533b269d --- /dev/null +++ b/test/replay-compat/scripts/docs/wait-landmark-forms.v0.20.0.ad @@ -0,0 +1,8 @@ +context platform=ios kind=simulator timeout=180000 + +open "Agent Device Tester" --relaunch +wait 500 +wait stable 500 10000 +wait "role=button label=About || label=\"Software Update\"" 30000 +wait @e7~s5 5000 +close diff --git a/test/replay-compat/scripts/examples/checkout-form-android.v0.15.1.ad b/test/replay-compat/scripts/examples/checkout-form-android.v0.15.1.ad new file mode 100644 index 000000000..53b62ff99 --- /dev/null +++ b/test/replay-compat/scripts/examples/checkout-form-android.v0.15.1.ad @@ -0,0 +1,21 @@ +context platform=android timeout=60000 +env APP_TARGET="Agent Device Tester" +env APP_URL="" +open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" +wait "label=\"Form\"" 30000 +click "label=\"Form\"" +wait "Checkout form" 5000 +fill id="field-name" "Ada Lovelace" +fill id="field-email" "ada@example.com" +keyboard dismiss +wait "Checkout form" 5000 +scroll down 0.6 +click id="shipping-pickup" +click id="payment-cash" +wait "Delivery choices" 5000 +scroll down 0.7 +click id="checkbox-agree" +click id="submit-order" +wait "Order summary" 5000 +wait "Ada Lovelace chose pickup with cash payment." 5000 +close diff --git a/test/replay-compat/scripts/examples/checkout-form-android.v0.16.8.ad b/test/replay-compat/scripts/examples/checkout-form-android.v0.16.8.ad new file mode 100644 index 000000000..8d1d0ada4 --- /dev/null +++ b/test/replay-compat/scripts/examples/checkout-form-android.v0.16.8.ad @@ -0,0 +1,23 @@ +context platform=android timeout=60000 +env APP_TARGET="Agent Device Tester" +env APP_URL="" +open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" +react-native dismiss-overlay +wait "label=\"Form\"" 30000 +click "label=\"Form\"" +wait "Checkout form" 5000 +fill id="field-name" "Ada Lovelace" +fill id="field-email" "ada@example.com" +keyboard dismiss +wait "Checkout form" 5000 +scroll down 0.6 +click id="shipping-pickup" +click id="payment-cash" +wait "Delivery" 5000 +scroll down 0.7 +click id="checkbox-agree" +click id="submit-order" +scroll up 0.6 +wait "Order summary" 5000 +wait "Ada Lovelace chose pickup with cash payment." 5000 +close diff --git a/test/replay-compat/scripts/examples/checkout-form.v0.15.1.ad b/test/replay-compat/scripts/examples/checkout-form.v0.15.1.ad new file mode 100644 index 000000000..8bd23afbd --- /dev/null +++ b/test/replay-compat/scripts/examples/checkout-form.v0.15.1.ad @@ -0,0 +1,21 @@ +context platform=ios timeout=60000 +env APP_TARGET="Agent Device Tester" +env APP_URL="" +open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" +wait "label=\"Form\"" 30000 +click "label=\"Form\"" +wait "Checkout form" 5000 +fill id="field-name" "Ada Lovelace" +fill id="field-email" "ada@example.com" +keyboard dismiss +wait "Checkout form" 5000 +scroll down 0.6 +click id="shipping-pickup" +click id="payment-cash" +wait "Delivery choices" 5000 +scroll down 0.7 +click id="checkbox-agree" +click id="submit-order" +wait "Order summary" 5000 +wait "Ada Lovelace chose pickup with cash payment." 5000 +close diff --git a/test/replay-compat/scripts/examples/checkout-form.v0.16.8.ad b/test/replay-compat/scripts/examples/checkout-form.v0.16.8.ad new file mode 100644 index 000000000..4c89212df --- /dev/null +++ b/test/replay-compat/scripts/examples/checkout-form.v0.16.8.ad @@ -0,0 +1,22 @@ +context platform=ios timeout=60000 +env APP_TARGET="Agent Device Tester" +env APP_URL="" +open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" +react-native dismiss-overlay +wait "label=\"Form\"" 30000 +click "label=\"Form\"" +wait "Checkout form" 5000 +fill id="field-name" "Ada Lovelace" +fill id="field-email" "ada@example.com" +keyboard dismiss +wait "Checkout form" 5000 +scroll down 0.6 +click id="shipping-pickup" +click id="payment-cash" +wait "Delivery choices" 5000 +scroll down 0.7 +click id="checkbox-agree" +click id="submit-order" +wait "Order summary" 5000 +wait "Ada Lovelace chose pickup with cash payment." 5000 +close diff --git a/test/replay-compat/scripts/examples/gesture-lab-android.v0.20.0.ad b/test/replay-compat/scripts/examples/gesture-lab-android.v0.20.0.ad new file mode 100644 index 000000000..cc626f1c0 --- /dev/null +++ b/test/replay-compat/scripts/examples/gesture-lab-android.v0.20.0.ad @@ -0,0 +1,48 @@ +context platform=android kind=emulator timeout=60000 + +env APP_TARGET="com.callstack.agentdevicelab" +env APP_URL="" + +open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" +react-native dismiss-overlay +wait "Gesture lab" 30000 +wait "gesture canary ready" 5000 +wait "two-pointer pan activations 0" 5000 + +gesture pan 220 700 180 0 500 +wait "two-pointer pan activations 0" 5000 + +gesture pan 220 700 180 0 500 --pointer-count 2 +wait "two-pointer pan activations 1" 5000 + +gesture fling right 500 700 300 +wait "fling 1" 5000 + +gesture fling left 500 700 300 +wait "fling 2" 5000 + +open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" +react-native dismiss-overlay +wait "gesture canary ready" 5000 +wait "pan changed no, pinch changed no, rotate changed no" 30000 + +gesture pinch 1.5 360 700 +wait "pan changed no, pinch changed yes, rotate changed no" 5000 + +open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" +react-native dismiss-overlay +wait "gesture canary ready" 5000 +wait "pan changed no, pinch changed no, rotate changed no" 30000 + +gesture rotate 35 360 700 +wait "pan changed no, pinch changed no, rotate changed yes" 5000 + +open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" +react-native dismiss-overlay +wait "gesture canary ready" 5000 +wait "pan changed no, pinch changed no, rotate changed no" 30000 + +gesture transform 360 700 60 0 1.75 30 600 +wait "pan changed yes, pinch changed yes, rotate changed yes" 5000 + +close diff --git a/test/replay-compat/scripts/examples/gesture-lab.v0.16.0.ad b/test/replay-compat/scripts/examples/gesture-lab.v0.16.0.ad new file mode 100644 index 000000000..a47066ee1 --- /dev/null +++ b/test/replay-compat/scripts/examples/gesture-lab.v0.16.0.ad @@ -0,0 +1,39 @@ +context platform=ios timeout=60000 + +env APP_TARGET="Agent Device Tester" +env APP_URL="" + +open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" +wait "Gesture lab" 30000 + +gesture fling left 195 443 180 +wait "fling 1" 5000 + +gesture fling right 195 443 180 +wait "fling 2" 5000 + +gesture fling up 195 443 80 80 +wait "fling 3" 5000 + +gesture fling down 195 443 80 80 +wait "fling 4" 5000 + +gesture pan 195 443 -80 0 +wait "x -" 5000 + +gesture pan 195 443 160 0 +wait "x 72" 5000 + +gesture pan 195 443 0 -80 +wait "y -" 5000 + +gesture pan 195 443 0 160 +wait "y 56" 5000 + +gesture pinch 1.25 195 443 +wait "pinch changed yes" 5000 + +gesture rotate 35 195 443 +wait "rotate changed yes" 5000 + +close diff --git a/test/replay-compat/scripts/examples/gesture-lab.v0.16.8.ad b/test/replay-compat/scripts/examples/gesture-lab.v0.16.8.ad new file mode 100644 index 000000000..2d6b3aee5 --- /dev/null +++ b/test/replay-compat/scripts/examples/gesture-lab.v0.16.8.ad @@ -0,0 +1,40 @@ +context platform=ios timeout=60000 + +env APP_TARGET="Agent Device Tester" +env APP_URL="" + +open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" +react-native dismiss-overlay +wait "Gesture lab" 30000 + +gesture fling left 195 443 180 +wait "fling 1" 5000 + +gesture fling right 195 443 180 +wait "fling 2" 5000 + +gesture fling up 195 443 80 80 +wait "fling 3" 5000 + +gesture fling down 195 443 80 80 +wait "fling 4" 5000 + +gesture pan 195 443 -80 0 +wait "x -" 5000 + +gesture pan 195 443 160 0 +wait "x 72" 5000 + +gesture pan 195 443 0 -80 +wait "y -" 5000 + +gesture pan 195 443 0 160 +wait "y 56" 5000 + +gesture pinch 1.25 195 443 +wait "pinch changed yes" 5000 + +gesture rotate 35 195 443 +wait "rotate changed yes" 5000 + +close diff --git a/test/replay-compat/scripts/examples/gesture-lab.v0.20.0.ad b/test/replay-compat/scripts/examples/gesture-lab.v0.20.0.ad new file mode 100644 index 000000000..ac3946e33 --- /dev/null +++ b/test/replay-compat/scripts/examples/gesture-lab.v0.20.0.ad @@ -0,0 +1,47 @@ +context platform=ios kind=simulator timeout=180000 + +env APP_TARGET="Agent Device Tester" +env APP_URL="" + +open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" +wait "Gesture lab" 30000 +wait "gesture canary ready" 5000 +wait "two-pointer pan activations 0" 5000 + +gesture pan 110 443 48 0 500 +wait "two-pointer pan activations 0" 5000 + +gesture pan 110 443 48 0 500 --pointer-count 2 +wait "two-pointer pan activations 1" 5000 + +gesture fling left 280 443 120 +wait "fling 1" 5000 + +gesture fling right 280 443 120 +wait "fling 2" 5000 + +open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" +wait "Gesture lab" 30000 +wait "gesture canary ready" 5000 +wait "pan changed no, pinch changed no, rotate changed no" 30000 + +gesture pinch 1.5 280 443 +wait "pan changed no, pinch changed yes, rotate changed no" 5000 + +open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" +wait "Gesture lab" 30000 +wait "gesture canary ready" 5000 +wait "pan changed no, pinch changed no, rotate changed no" 30000 + +gesture rotate 35 280 443 +wait "pan changed no, pinch changed no, rotate changed yes" 5000 + +open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" +wait "Gesture lab" 30000 +wait "gesture canary ready" 5000 +wait "pan changed no, pinch changed no, rotate changed no" 30000 + +gesture transform 280 443 24 -24 1.35 30 300 +wait "pan changed yes, pinch changed yes, rotate changed yes" 5000 + +close diff --git a/test/replay-compat/scripts/integration/android-01-settings.v0.11.0.ad b/test/replay-compat/scripts/integration/android-01-settings.v0.11.0.ad new file mode 100644 index 000000000..c91ca3f53 --- /dev/null +++ b/test/replay-compat/scripts/integration/android-01-settings.v0.11.0.ad @@ -0,0 +1,14 @@ +# Dogfood Android Settings flow through the replay suite runner. +context platform=android +open settings --relaunch +appstate +snapshot -i +is exists "label=Wait || label=\"Close app\" || label=Apps || label=\"Apps & notifications\" || label=\"Network & internet\" || label=\"Network and internet\" || label=\"Connected devices\" || label=Display || label=Battery || label=Notifications || label=Security || label=Privacy" +click "label=Wait || label=\"Close app\" || label=Apps || label=\"Apps & notifications\" || label=\"Network & internet\" || label=\"Network and internet\" || label=\"Connected devices\" || label=Display || label=Battery || label=Notifications || label=Security || label=Privacy" +open settings --relaunch +snapshot -i +is exists "label=Apps || label=\"Apps & notifications\" || label=\"Network & internet\" || label=\"Network and internet\" || label=\"Connected devices\" || label=Display || label=Battery || label=Notifications || label=Security || label=Privacy" +click "label=Apps || label=\"Apps & notifications\" || label=\"Network & internet\" || label=\"Network and internet\" || label=\"Connected devices\" || label=Display || label=Battery || label=Notifications || label=Security || label=Privacy" +snapshot -i +appstate +back diff --git a/test/replay-compat/scripts/integration/android-01-settings.v0.11.2.ad b/test/replay-compat/scripts/integration/android-01-settings.v0.11.2.ad new file mode 100644 index 000000000..47b9ff049 --- /dev/null +++ b/test/replay-compat/scripts/integration/android-01-settings.v0.11.2.ad @@ -0,0 +1,15 @@ +# Dogfood Android Settings flow through the replay suite runner. +context platform=android +open settings --relaunch +appstate +snapshot -i +is exists "label=Wait || label=\"Close app\" || label=Apps || label=\"Apps & notifications\" || label=\"Network & internet\" || label=\"Network and internet\" || label=\"Connected devices\" || label=Display || label=Battery || label=Notifications || label=Security || label=Privacy" +click "label=Wait || label=\"Close app\" || label=Apps || label=\"Apps & notifications\" || label=\"Network & internet\" || label=\"Network and internet\" || label=\"Connected devices\" || label=Display || label=Battery || label=Notifications || label=Security || label=Privacy" +back +wait 1000 +snapshot -i +is exists "label=Apps || label=\"Apps & notifications\" || label=\"Network & internet\" || label=\"Network and internet\" || label=\"Connected devices\" || label=Display || label=Battery || label=Notifications || label=Security || label=Privacy" +click "label=Apps || label=\"Apps & notifications\" || label=\"Network & internet\" || label=\"Network and internet\" || label=\"Connected devices\" || label=Display || label=Battery || label=Notifications || label=Security || label=Privacy" +snapshot -i +appstate +back diff --git a/test/replay-compat/scripts/integration/android-02-deep-navigation.v0.11.2.ad b/test/replay-compat/scripts/integration/android-02-deep-navigation.v0.11.2.ad new file mode 100644 index 000000000..483bccac1 --- /dev/null +++ b/test/replay-compat/scripts/integration/android-02-deep-navigation.v0.11.2.ad @@ -0,0 +1,18 @@ +# Deep navigation. +context platform=android + +open settings --relaunch +appstate +snapshot -i +screenshot "./test/screenshots/replays/android-deep-nav-root.png" + +click @e16 +wait 1000 +snapshot -i +find text "Notification history" exists +screenshot "./test/screenshots/replays/android-deep-nav-detail.png" + +back +wait 1000 +snapshot -i +find text "Search" exists diff --git a/test/replay-compat/scripts/integration/android-02-deep-navigation.v0.12.5.ad b/test/replay-compat/scripts/integration/android-02-deep-navigation.v0.12.5.ad new file mode 100644 index 000000000..2072595e8 --- /dev/null +++ b/test/replay-compat/scripts/integration/android-02-deep-navigation.v0.12.5.ad @@ -0,0 +1,18 @@ +# Deep navigation. +context platform=android + +open settings --relaunch +appstate +snapshot -i +screenshot "./test/screenshots/replays/android-deep-nav-root.png" + +click "label=Notifications" +wait 1000 +snapshot -i +find text "Notification history" exists +screenshot "./test/screenshots/replays/android-deep-nav-detail.png" + +back +wait 1000 +snapshot -i +find text "Search" exists diff --git a/test/replay-compat/scripts/integration/android-03-scroll-discovery.v0.11.2.ad b/test/replay-compat/scripts/integration/android-03-scroll-discovery.v0.11.2.ad new file mode 100644 index 000000000..f37427fae --- /dev/null +++ b/test/replay-compat/scripts/integration/android-03-scroll-discovery.v0.11.2.ad @@ -0,0 +1,15 @@ +# Scroll discovery. +context platform=android + +open settings --relaunch +snapshot -i +screenshot "./test/screenshots/replays/android-scroll-top.png" + +scroll down 3 +wait 500 +snapshot -i +screenshot "./test/screenshots/replays/android-scroll-found.png" + +scroll up 3 +wait 500 +snapshot -i diff --git a/test/replay-compat/scripts/integration/android-04-text-input-keyboard.v0.11.2.ad b/test/replay-compat/scripts/integration/android-04-text-input-keyboard.v0.11.2.ad new file mode 100644 index 000000000..4dbdc1c01 --- /dev/null +++ b/test/replay-compat/scripts/integration/android-04-text-input-keyboard.v0.11.2.ad @@ -0,0 +1,25 @@ +# Text input. +context platform=android + +open settings --relaunch +snapshot -i + +click @e2 +wait 1000 +snapshot -i + +keyboard status + +type "Display" +wait 1000 +snapshot -i + +screenshot "./test/screenshots/replays/android-search-results.png" + +keyboard status + +back +wait 1000 +keyboard status +snapshot -i +find text "Search" exists diff --git a/test/replay-compat/scripts/integration/android-05-app-lifecycle.v0.11.2.ad b/test/replay-compat/scripts/integration/android-05-app-lifecycle.v0.11.2.ad new file mode 100644 index 000000000..ee20d0724 --- /dev/null +++ b/test/replay-compat/scripts/integration/android-05-app-lifecycle.v0.11.2.ad @@ -0,0 +1,18 @@ +# App lifecycle. +context platform=android + +open settings --relaunch +appstate +snapshot -i +find text "Search" exists + +home +wait 2000 + +open settings --relaunch +wait 1000 +appstate +snapshot -i +screenshot "./test/screenshots/replays/android-lifecycle-reopen.png" + +find text "Search" exists diff --git a/test/replay-compat/scripts/integration/android-06-swipe-gestures.v0.11.2.ad b/test/replay-compat/scripts/integration/android-06-swipe-gestures.v0.11.2.ad new file mode 100644 index 000000000..686bace4c --- /dev/null +++ b/test/replay-compat/scripts/integration/android-06-swipe-gestures.v0.11.2.ad @@ -0,0 +1,25 @@ +# Swipe gestures. +context platform=android + +open settings --relaunch +snapshot -i +find text "Search" exists + +swipe 206 700 206 250 300 +wait 500 +snapshot -i + +swipe 206 650 206 300 300 --count 2 --pause-ms 200 --pattern one-way +wait 500 +snapshot -i + +scroll up 0.6 +wait 500 +snapshot -i +find text "Search" exists +screenshot "./test/screenshots/replays/android-swipe-before-back.png" + +scroll down 2 +wait 500 +snapshot -i +screenshot "./test/screenshots/replays/android-swipe-scroll.png" diff --git a/test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.11.0.ad b/test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.11.0.ad new file mode 100644 index 000000000..7f7f6cd97 --- /dev/null +++ b/test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.11.0.ad @@ -0,0 +1,8 @@ +# Dogfood iOS physical device lifecycle through the replay suite runner. +context platform=ios +open com.apple.Preferences +snapshot +click "role=cell label=General" +wait "label=About || label=\"Software Update\"" 5000 +back +is exists "role=cell label=General" diff --git a/test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.11.2.ad b/test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.11.2.ad new file mode 100644 index 000000000..797de36dc --- /dev/null +++ b/test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.11.2.ad @@ -0,0 +1,8 @@ +# Dogfood iOS physical device lifecycle through the replay suite runner. +context platform=ios +open com.apple.Preferences --relaunch +snapshot +click "role=cell label=General" +wait "label=About || label=\"Software Update\"" 5000 +back +is exists "role=cell label=General" diff --git a/test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.15.0.ad b/test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.15.0.ad new file mode 100644 index 000000000..7a5ea2f64 --- /dev/null +++ b/test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.15.0.ad @@ -0,0 +1,8 @@ +# Dogfood iOS physical device lifecycle through the replay suite runner. +context platform=ios target=mobile +open com.apple.Preferences --relaunch +snapshot +click "role=cell label=General" +wait "label=About || label=\"Software Update\"" 5000 +back +is exists "role=cell label=General" diff --git a/test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.17.0.ad b/test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.17.0.ad new file mode 100644 index 000000000..fb6e1c0b9 --- /dev/null +++ b/test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.17.0.ad @@ -0,0 +1,8 @@ +# Dogfood iOS physical device lifecycle through the replay suite runner. +context platform=ios target=mobile +open com.apple.Preferences --relaunch +snapshot +click "role=cell label=General || role=button label=General" +wait "label=About || label=\"Software Update\"" 5000 +back +is exists "role=cell label=General || role=button label=General" diff --git a/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.11.0.ad b/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.11.0.ad new file mode 100644 index 000000000..704f28b6d --- /dev/null +++ b/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.11.0.ad @@ -0,0 +1,12 @@ +# Dogfood iOS Settings flow through the replay suite runner. +context platform=ios +open com.apple.Preferences +screenshot "./test/screenshots/replays/ios-settings.png" +snapshot -i +appstate +click "role=cell label=General" +wait "label=About || label=\"Software Update\" || text=\"Manage your overall setup and preferences\"" 5000 +snapshot +is exists "label=About || label=\"Software Update\" || text=\"Manage your overall setup and preferences\"" +find text "Software Update" exists +back diff --git a/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.11.2.ad b/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.11.2.ad new file mode 100644 index 000000000..b9482558f --- /dev/null +++ b/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.11.2.ad @@ -0,0 +1,12 @@ +# Dogfood iOS Settings flow through the replay suite runner. +context platform=ios +open com.apple.Preferences --relaunch +screenshot "./test/screenshots/replays/ios-settings.png" +snapshot -i +appstate +click "role=cell label=General" +wait "label=About || label=\"Software Update\" || text=\"Manage your overall setup and preferences\"" 5000 +snapshot +is exists "label=About || label=\"Software Update\" || text=\"Manage your overall setup and preferences\"" +find text "Software Update" exists +back diff --git a/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.15.0.ad b/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.15.0.ad new file mode 100644 index 000000000..f34cb8170 --- /dev/null +++ b/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.15.0.ad @@ -0,0 +1,12 @@ +# Dogfood iOS Settings flow through the replay suite runner. +context platform=ios target=mobile +open com.apple.Preferences --relaunch +screenshot "./test/screenshots/replays/ios-settings.png" +snapshot -i +appstate +click "role=cell label=General" +wait "label=About || label=\"Software Update\" || text=\"Manage your overall setup and preferences\"" 5000 +snapshot +is exists "label=About || label=\"Software Update\" || text=\"Manage your overall setup and preferences\"" +find text "Software Update" exists +back diff --git a/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.17.0.ad b/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.17.0.ad new file mode 100644 index 000000000..418401db1 --- /dev/null +++ b/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.17.0.ad @@ -0,0 +1,12 @@ +# Dogfood iOS Settings flow through the replay suite runner. +context platform=ios target=mobile +open com.apple.Preferences --relaunch +screenshot "./test/screenshots/replays/ios-settings.png" +snapshot -i +appstate +click "role=cell label=General || role=button label=General" +wait "label=About || label=\"Software Update\" || text=\"Manage your overall setup and preferences\"" 5000 +snapshot +is exists "label=About || label=\"Software Update\" || text=\"Manage your overall setup and preferences\"" +find text "Software Update" exists +back diff --git a/test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.11.2.ad b/test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.11.2.ad new file mode 100644 index 000000000..977f082d9 --- /dev/null +++ b/test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.11.2.ad @@ -0,0 +1,11 @@ +# Deep navigation. +context platform=ios + +open com.apple.Preferences --relaunch +click "role=cell label=General" +wait 1000 +find text "About" exists + +back +wait 1000 +find text "General" exists diff --git a/test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.15.0.ad b/test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.15.0.ad new file mode 100644 index 000000000..edc97d285 --- /dev/null +++ b/test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.15.0.ad @@ -0,0 +1,11 @@ +# Deep navigation. +context platform=ios target=mobile + +open com.apple.Preferences --relaunch +click "role=cell label=General" +wait 1000 +find text "About" exists + +back +wait 1000 +find text "General" exists diff --git a/test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.17.0.ad b/test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.17.0.ad new file mode 100644 index 000000000..86c27387f --- /dev/null +++ b/test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.17.0.ad @@ -0,0 +1,11 @@ +# Deep navigation. +context platform=ios target=mobile + +open com.apple.Preferences --relaunch +click "role=cell label=General || role=button label=General" +wait 1000 +find text "About" exists + +back +wait 1000 +find text "General" exists diff --git a/test/replay-compat/scripts/integration/ios-simulator-03-scroll-discovery.v0.11.2.ad b/test/replay-compat/scripts/integration/ios-simulator-03-scroll-discovery.v0.11.2.ad new file mode 100644 index 000000000..898bf8423 --- /dev/null +++ b/test/replay-compat/scripts/integration/ios-simulator-03-scroll-discovery.v0.11.2.ad @@ -0,0 +1,11 @@ +# Scroll discovery. +context platform=ios + +open com.apple.Preferences --relaunch +scroll down 3 +wait 500 +find text "Accessibility" exists + +scroll up 3 +wait 500 +find text "General" exists diff --git a/test/replay-compat/scripts/integration/ios-simulator-03-scroll-discovery.v0.15.0.ad b/test/replay-compat/scripts/integration/ios-simulator-03-scroll-discovery.v0.15.0.ad new file mode 100644 index 000000000..2646c29e9 --- /dev/null +++ b/test/replay-compat/scripts/integration/ios-simulator-03-scroll-discovery.v0.15.0.ad @@ -0,0 +1,11 @@ +# Scroll discovery. +context platform=ios target=mobile + +open com.apple.Preferences --relaunch +scroll down 3 +wait 500 +find text "Accessibility" exists + +scroll up 3 +wait 500 +find text "General" exists diff --git a/test/replay-compat/scripts/integration/ios-simulator-04-text-input-keyboard.v0.11.2.ad b/test/replay-compat/scripts/integration/ios-simulator-04-text-input-keyboard.v0.11.2.ad new file mode 100644 index 000000000..0a8be87e3 --- /dev/null +++ b/test/replay-compat/scripts/integration/ios-simulator-04-text-input-keyboard.v0.11.2.ad @@ -0,0 +1,11 @@ +# Text input. +context platform=ios + +open com.apple.Preferences --relaunch +click "role=searchField || label=Search || label=\"Search\"" +wait 1000 + +type "General" +wait 1000 + +find text "General" exists diff --git a/test/replay-compat/scripts/integration/ios-simulator-04-text-input-keyboard.v0.15.0.ad b/test/replay-compat/scripts/integration/ios-simulator-04-text-input-keyboard.v0.15.0.ad new file mode 100644 index 000000000..a9f04c26d --- /dev/null +++ b/test/replay-compat/scripts/integration/ios-simulator-04-text-input-keyboard.v0.15.0.ad @@ -0,0 +1,11 @@ +# Text input. +context platform=ios target=mobile + +open com.apple.Preferences --relaunch +click "role=searchField || label=Search || label=\"Search\"" +wait 1000 + +type "General" +wait 1000 + +find text "General" exists diff --git a/test/replay-compat/scripts/integration/ios-simulator-05-app-lifecycle.v0.11.2.ad b/test/replay-compat/scripts/integration/ios-simulator-05-app-lifecycle.v0.11.2.ad new file mode 100644 index 000000000..75cb3ca57 --- /dev/null +++ b/test/replay-compat/scripts/integration/ios-simulator-05-app-lifecycle.v0.11.2.ad @@ -0,0 +1,13 @@ +# App lifecycle. +context platform=ios + +open com.apple.Preferences --relaunch +appstate + +home +wait 2000 + +open com.apple.Preferences --relaunch +wait 1000 +appstate +find text "General" exists diff --git a/test/replay-compat/scripts/integration/ios-simulator-05-app-lifecycle.v0.15.0.ad b/test/replay-compat/scripts/integration/ios-simulator-05-app-lifecycle.v0.15.0.ad new file mode 100644 index 000000000..32d2a15f5 --- /dev/null +++ b/test/replay-compat/scripts/integration/ios-simulator-05-app-lifecycle.v0.15.0.ad @@ -0,0 +1,13 @@ +# App lifecycle. +context platform=ios target=mobile + +open com.apple.Preferences --relaunch +appstate + +home +wait 2000 + +open com.apple.Preferences --relaunch +wait 1000 +appstate +find text "General" exists diff --git a/test/replay-compat/scripts/integration/ios-simulator-06-swipe-gestures.v0.11.2.ad b/test/replay-compat/scripts/integration/ios-simulator-06-swipe-gestures.v0.11.2.ad new file mode 100644 index 000000000..ca067860a --- /dev/null +++ b/test/replay-compat/scripts/integration/ios-simulator-06-swipe-gestures.v0.11.2.ad @@ -0,0 +1,16 @@ +# Swipe gestures. +context platform=ios + +open com.apple.Preferences --relaunch + +swipe 197 650 197 300 300 +wait 500 + +swipe 197 600 197 350 300 +wait 500 + +snapshot -i + +swipe 197 300 197 650 300 +wait 500 +find text "General" exists diff --git a/test/replay-compat/scripts/integration/ios-simulator-06-swipe-gestures.v0.15.0.ad b/test/replay-compat/scripts/integration/ios-simulator-06-swipe-gestures.v0.15.0.ad new file mode 100644 index 000000000..8f892f1c7 --- /dev/null +++ b/test/replay-compat/scripts/integration/ios-simulator-06-swipe-gestures.v0.15.0.ad @@ -0,0 +1,16 @@ +# Swipe gestures. +context platform=ios target=mobile + +open com.apple.Preferences --relaunch + +swipe 197 650 197 300 300 +wait 500 + +swipe 197 600 197 350 300 +wait 500 + +snapshot -i + +swipe 197 300 197 650 300 +wait 500 +find text "General" exists diff --git a/test/replay-compat/scripts/integration/linux-01-desktop-smoke.v0.11.8.ad b/test/replay-compat/scripts/integration/linux-01-desktop-smoke.v0.11.8.ad new file mode 100644 index 000000000..c707cfdb2 --- /dev/null +++ b/test/replay-compat/scripts/integration/linux-01-desktop-smoke.v0.11.8.ad @@ -0,0 +1,12 @@ +# Smoke test for Linux desktop automation on CI. +# Opens gnome-calculator, takes a snapshot, and validates the +# accessibility tree contains expected calculator UI elements. +context platform=linux +open gnome-calculator +wait 3000 +screenshot "./test/screenshots/replays/linux-calculator.png" +snapshot +# Verify calculator-specific elements: a window titled Calculator, +# or digit buttons that only a calculator app would expose. +is exists "label=Calculator || label=0 || label=1 || label=5" +snapshot -i diff --git a/test/replay-compat/scripts/integration/linux-01-desktop-smoke.v0.15.0.ad b/test/replay-compat/scripts/integration/linux-01-desktop-smoke.v0.15.0.ad new file mode 100644 index 000000000..1d88b91a8 --- /dev/null +++ b/test/replay-compat/scripts/integration/linux-01-desktop-smoke.v0.15.0.ad @@ -0,0 +1,12 @@ +# Smoke test for Linux desktop automation on CI. +# Opens gnome-calculator, takes a snapshot, and validates the +# accessibility tree contains expected calculator UI elements. +context platform=linux +open gnome-calculator +wait "appname=gnome-calculator || windowtitle=Calculator || label=Calculator || label=0 || label=1 || label=5" 15000 +screenshot "./test/screenshots/replays/linux-calculator.png" +snapshot +# Verify calculator-specific elements: a window titled Calculator, +# or digit buttons that only a calculator app would expose. +is exists "appname=gnome-calculator || windowtitle=Calculator || label=Calculator || label=0 || label=1 || label=5" +snapshot -i diff --git a/test/replay-compat/scripts/integration/macos-01-system-settings.v0.11.0.ad b/test/replay-compat/scripts/integration/macos-01-system-settings.v0.11.0.ad new file mode 100644 index 000000000..e3f7f39b5 --- /dev/null +++ b/test/replay-compat/scripts/integration/macos-01-system-settings.v0.11.0.ad @@ -0,0 +1,16 @@ +# Dogfood macOS System Settings flow through the replay suite runner. +context platform=macos +open "System Settings" +screenshot "./test/screenshots/replays/macos-system-settings.png" +appstate +snapshot -i +click "role=cell label=General" +wait "role=button label=About" 5000 +snapshot -i +is exists "role=button label=About" +click "role=button label=About" +wait "label=\"System Report...\" || label=\"Serial number\" || label=Chip || label=Processor || label=macOS" 5000 +snapshot -i +back +wait "role=button label=About" 5000 +is exists "role=button label=About" diff --git a/test/replay-compat/scripts/integration/macos-01-system-settings.v0.11.2.ad b/test/replay-compat/scripts/integration/macos-01-system-settings.v0.11.2.ad new file mode 100644 index 000000000..7df54a822 --- /dev/null +++ b/test/replay-compat/scripts/integration/macos-01-system-settings.v0.11.2.ad @@ -0,0 +1,16 @@ +# Dogfood macOS System Settings flow through the replay suite runner. +context platform=macos +open "System Settings" --relaunch +screenshot "./test/screenshots/replays/macos-system-settings.png" +appstate +snapshot -i +click "role=cell label=General" +wait "role=button label=About" 5000 +snapshot -i +is exists "role=button label=About" +click "role=button label=About" +wait "label=\"System Report...\" || label=\"Serial number\" || label=Chip || label=Processor || label=macOS" 5000 +snapshot -i +back +wait "role=button label=About" 5000 +is exists "role=button label=About" diff --git a/test/replay-compat/scripts/integration/macos-01-system-settings.v0.17.0.ad b/test/replay-compat/scripts/integration/macos-01-system-settings.v0.17.0.ad new file mode 100644 index 000000000..d43f79093 --- /dev/null +++ b/test/replay-compat/scripts/integration/macos-01-system-settings.v0.17.0.ad @@ -0,0 +1,15 @@ +# Dogfood macOS System Settings flow through the replay suite runner. +context platform=macos +open "System Settings" --relaunch +screenshot "./test/screenshots/replays/macos-system-settings.png" +appstate +snapshot -i +wait "role=button label=About" 5000 +snapshot -i +is exists "role=button label=About" +click "role=button label=About" +wait "label=\"System Report...\" || label=\"Serial number\" || label=Chip || label=Processor || label=macOS" 5000 +snapshot -i +back +wait "role=button label=About" 5000 +is exists "role=button label=About" diff --git a/vitest.config.ts b/vitest.config.ts index 14db5736b..3f95bd31f 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -41,6 +41,10 @@ export default defineConfig({ 'scripts/__tests__/help-conformance-sample-outputs.test.ts', 'scripts/__tests__/help-conformance-topic-coverage.test.ts', 'test/skillgym/suites/local-cli-help-policy.test.ts', + // The frozen replay-compat corpus (#1417): parse-only, no device or + // subprocess work, so it belongs in the fast lane next to the + // grammar it guards. + 'test/replay-compat/corpus.test.ts', // The Maestro conformance oracle runs via `node --test` in its own CI // job (scripts/maestro-conformance), like the layering guard. ], From 208e89cca89ebc8a4b17fed272c37fc7e9f9497e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Pierzcha=C5=82a?= Date: Mon, 27 Jul 2026 14:19:03 +0000 Subject: [PATCH 2/8] test: pin replay-compat corpus bytes to released blobs and assert via parseReplayInput Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- docs/agents/testing.md | 2 +- package.json | 1 + scripts/check-replay-compat-provenance.ts | 78 +++++ test/replay-compat/README.md | 31 +- test/replay-compat/corpus.test.ts | 46 ++- test/replay-compat/manifest.ts | 358 ++++++++++++++++++---- 6 files changed, 433 insertions(+), 83 deletions(-) create mode 100644 scripts/check-replay-compat-provenance.ts diff --git a/docs/agents/testing.md b/docs/agents/testing.md index 3a1e091d2..569f91e71 100644 --- a/docs/agents/testing.md +++ b/docs/agents/testing.md @@ -19,7 +19,7 @@ The mapping it encodes, for when you need to run a gate directly or reason about | CLI help/guidance (`src/cli/parser/cli-help.ts`, `src/cli-schema/`) | `pnpm exec vitest run src/cli/parser/__tests__ src/cli-schema/command-schema-guards.test.ts scripts/__tests__` — the `scripts/__tests__` gates enforce help-topic benchmark coverage and pin the bench's quoted CLI samples to the real renderers | | Help benchmark cases (`scripts/help-conformance-*.mjs`) | `pnpm exec vitest run scripts/__tests__` (deterministic gates); model-backed: `pnpm bench:help-conformance` (paid LLM calls, local only) | | SkillGym prompts/assertions | `pnpm test:skillgym:case ` (broad: `pnpm test:skillgym`, filter with `-- --tag fixture-smoke` or `-- --tag skill-guidance`) — agentic routing + local-help-consumption proof only; command-planning knowledge checks belong in the help bench | -| `.ad` grammar (`src/replay/script.ts`, gesture arity, replay vars) | `pnpm exec vitest run --project unit-core test/replay-compat` — the frozen replay-compat corpus asserts which released script surfaces still parse; a flipped verdict is edited in `test/replay-compat/manifest.ts`, never in the script | +| `.ad` grammar (`src/replay/script.ts`, gesture arity, replay vars) | `pnpm exec vitest run --project unit-core test/replay-compat` — the frozen replay-compat corpus asserts which released script surfaces still parse; a flipped verdict is edited in `test/replay-compat/manifest.ts`, never in the script. Adding or re-pinning a corpus entry also runs `pnpm check:replay-compat`, which re-derives each entry from its release tag in git history | | Anything in `src/`, `test/`, `skills/` | `pnpm format` | Two traps worth naming: diff --git a/package.json b/package.json index 27a3e9c46..11892ca07 100644 --- a/package.json +++ b/package.json @@ -122,6 +122,7 @@ "depgraph:test": "node --experimental-strip-types --test scripts/depgraph/model.test.ts", "check:production-exports": "fallow dead-code --config fallow-production-exports.json --production --unused-exports --fail-on-issues", "check:bundle-owner-files": "node --experimental-strip-types scripts/check-bundle-owner-files.ts", + "check:replay-compat": "node --experimental-strip-types scripts/check-replay-compat-provenance.ts", "check:freerange": "fr", "check:quick": "pnpm lint && pnpm typecheck", "sync:mcp-metadata": "node scripts/sync-mcp-metadata.mjs", diff --git a/scripts/check-replay-compat-provenance.ts b/scripts/check-replay-compat-provenance.ts new file mode 100644 index 000000000..1a4f7671d --- /dev/null +++ b/scripts/check-replay-compat-provenance.ts @@ -0,0 +1,78 @@ +/** + * Re-derives the frozen replay-compat corpus (#1417) from git history: every + * mined entry must still be the exact blob a RELEASED tag published at the path + * it claims, and every tag the corpus cites must be a tag that was actually cut. + * + * The unit gate (`test/replay-compat/corpus.test.ts`) hashes the checked-in + * bytes against the same object ids, which catches a rewritten script; only this + * check can say those object ids came from a release rather than from whoever + * wrote the manifest line. + */ + +import fs from 'node:fs'; +import path from 'node:path'; +import { REPLAY_COMPAT_CORPUS, REPLAY_COMPAT_RELEASED_TAGS } from '../test/replay-compat/manifest.ts'; +import { runCmdSync } from '../src/utils/exec.ts'; + +const repoRoot = path.resolve(import.meta.dirname, '..'); +const corpusRoot = path.join(repoRoot, 'test', 'replay-compat'); + +function git(args: string[]): string { + return runCmdSync('git', args, { cwd: repoRoot, allowFailure: true }).stdout.trim(); +} + +if (git(['rev-parse', '--is-shallow-repository']) === 'true') { + throw new Error( + 'Corpus provenance needs full history and tags. Run `git fetch --unshallow --tags` first.', + ); +} + +const failures: string[] = []; + +const existingTags = new Set(git(['tag', '--list']).split('\n').filter(Boolean)); +if (existingTags.size === 0) { + throw new Error('No release tags found. Run `git fetch --tags` first.'); +} +for (const tag of REPLAY_COMPAT_RELEASED_TAGS) { + if (!existingTags.has(tag)) { + failures.push(`${tag} is cited as a released version but is not a tag in this repository`); + } +} + +for (const entry of REPLAY_COMPAT_CORPUS) { + if (!REPLAY_COMPAT_RELEASED_TAGS.includes(entry.recordedBy)) { + failures.push(`${entry.id}: recordedBy ${entry.recordedBy} is not in REPLAY_COMPAT_RELEASED_TAGS`); + continue; + } + if (entry.provenance.kind !== 'mined') continue; + + const { path: sourcePath, blob } = entry.provenance; + const historical = git(['rev-parse', `${entry.recordedBy}:${sourcePath}`]); + if (historical !== blob) { + failures.push( + `${entry.id}: ${entry.recordedBy}:${sourcePath} is ${historical || ''}, manifest pins ${blob}`, + ); + continue; + } + const frozen = fs.readFileSync(path.join(corpusRoot, entry.file)); + const released = runCmdSync('git', ['cat-file', 'blob', blob], { + cwd: repoRoot, + binaryStdout: true, + }).stdoutBuffer; + if (!released?.equals(frozen)) { + failures.push(`${entry.id}: ${entry.file} no longer matches the released blob ${blob}`); + } +} + +if (failures.length > 0) { + throw new Error( + `Replay-compat corpus provenance is broken (see test/replay-compat/README.md):\n${failures + .map((failure) => `- ${failure}`) + .join('\n')}`, + ); +} + +const mined = REPLAY_COMPAT_CORPUS.filter((entry) => entry.provenance.kind === 'mined').length; +process.stdout.write( + `Verified ${mined} mined replay-compat scripts against their released blobs across ${REPLAY_COMPAT_RELEASED_TAGS.length} tags (${REPLAY_COMPAT_CORPUS.length - mined} derived entries are digest-pinned).\n`, +); diff --git a/test/replay-compat/README.md b/test/replay-compat/README.md index 82c4e7790..a8e87aa59 100644 --- a/test/replay-compat/README.md +++ b/test/replay-compat/README.md @@ -3,15 +3,17 @@ `.ad` scripts as **released** versions of agent-device wrote them, each paired with the verdict today's parser owes it: `parses`, or `fails` with an error code and the migration hint substring. -`corpus.test.ts` runs in `unit-core` and asserts every verdict, so a grammar change cannot quietly -alter which historical script surface still loads. #1417. +`corpus.test.ts` runs in `unit-core` and asserts every verdict through `parseReplayInput` — the same +composition `replay`/`test` use — so a grammar change cannot quietly alter which historical script +surface still loads. #1417. ## The freeze rule **Never edit a script under `scripts/` to make a parser change pass.** A corpus script is a recording that already exists in the wild; editing it deletes the evidence the -gate is here to keep. When a grammar change flips a verdict: +gate is here to keep, and the pinned identities below make an edited script fail rather than pass. +When a grammar change flips a verdict: 1. Leave the script byte-identical. 2. Update its `verdict` in `manifest.ts` in the same PR. @@ -31,8 +33,21 @@ enough to put in front of whoever recorded the script. (`${VAR}` parameterization, reserved/ordering `env` refusals, `wait stable` and landmark waits, `target-v1` annotations, retired `gesture rotate` velocity and `gesture swipe` durationMs). -Provenance lives in `manifest.ts`: `recordedBy` is the released tag whose grammar produced the -surface, and `source` is where it was mined from at that tag. +## How "frozen" and "released" are enforced, not asserted + +Provenance lives in `manifest.ts`. `recordedBy` is the released tag whose grammar produced the +surface, and `provenance` pins the bytes: + +- `mined` entries carry the **git object id of the historical blob**. `corpus.test.ts` hashes the + checked-in bytes and must reproduce that id, so a rewritten script cannot be made green by editing + the manifest to match — the id is only obtainable from the released content. +- `derived` entries (`scripts/docs/`) have no historical blob to point at, so their bytes are pinned + by SHA-256. + +`pnpm check:replay-compat` closes the other half: it re-derives every mined id from git history +(`git rev-parse :`) and checks every cited tag against `git tag --list`, so an +entry cannot claim a version that was never cut. It needs full history and tags, so it is a +maintainer/reviewer check rather than part of the shallow-clone-safe unit lane. **Released surfaces only.** A grammar state that only ever existed between commits is not compat surface — `git tag --contains ` decides, and unreleased shapes stay out (AGENTS.md, @@ -42,8 +57,10 @@ surface — `git tag --contains ` decides, and unreleased shapes stay ou 1. Find the surface at a release tag (`git show :`, or the released grammar/docs that emitted it) and copy it verbatim into `scripts//..ad`. -2. Add a `manifest.ts` entry with `recordedBy`, `source`, `covers`, and the observed verdict. -3. Run `pnpm exec vitest run --project unit-core test/replay-compat`. +2. Add a `manifest.ts` entry with `recordedBy`, `provenance`, `covers`, and the observed verdict. For + a mined entry the blob id is `git rev-parse :`; for a derived one it is + `shasum -a 256 `. Add the tag to `REPLAY_COMPAT_RELEASED_TAGS` if it is new. +3. Run `pnpm exec vitest run --project unit-core test/replay-compat` and `pnpm check:replay-compat`. Add an entry when a change retires, renames, or narrows a `.ad` form — the corpus is the record of what that costs someone with a saved recording. diff --git a/test/replay-compat/corpus.test.ts b/test/replay-compat/corpus.test.ts index 4cdf8bc37..48d7c34ad 100644 --- a/test/replay-compat/corpus.test.ts +++ b/test/replay-compat/corpus.test.ts @@ -1,11 +1,13 @@ import { describe, expect, test } from 'vitest'; +import { createHash } from 'node:crypto'; import { readdirSync, readFileSync } from 'node:fs'; import { join } from 'node:path'; import { fileURLToPath } from 'node:url'; +import { parseReplayInput } from '../../src/compat/replay-input.ts'; import { AppError } from '../../src/kernel/errors.ts'; -import { parseReplayScriptDetailed, readReplayScriptMetadata } from '../../src/replay/script.ts'; import { REPLAY_COMPAT_CORPUS, + REPLAY_COMPAT_RELEASED_TAGS, type ReplayCompatCoverage, type ReplayCompatEntry, } from './manifest.ts'; @@ -25,14 +27,13 @@ const REQUIRED_COVERAGE: ReplayCompatCoverage[] = [ 'wait-landmark', ]; -function readCorpusScript(entry: ReplayCompatEntry): string { - return readFileSync(join(CORPUS_DIR, entry.file), 'utf8'); +function readCorpusScript(entry: ReplayCompatEntry): Buffer { + return readFileSync(join(CORPUS_DIR, entry.file)); } -/** The whole parse surface a `.ad` file meets before replay executes anything. */ -function parseCorpusScript(script: string): void { - readReplayScriptMetadata(script); - parseReplayScriptDetailed(script); +/** Git's own object id for a blob, so a mined entry's bytes carry their history. */ +function gitBlobId(bytes: Buffer): string { + return createHash('sha1').update(`blob ${bytes.length}\0`).update(bytes).digest('hex'); } function listCorpusScriptFiles(): string[] { @@ -45,17 +46,21 @@ function listCorpusScriptFiles(): string[] { } describe('replay-compat corpus', () => { + // The verdict is asserted through `parseReplayInput` — the same composition + // (and the same ordering of the action parse and the metadata read) that + // `replay`/`test` hit — so a multi-fault script reports the code and hint + // production really surfaces, and moving the boundary breaks this gate. test.each(REPLAY_COMPAT_CORPUS.map((entry) => [entry.id, entry] as const))('%s', (_id, entry) => { - const script = readCorpusScript(entry); + const script = readCorpusScript(entry).toString('utf8'); if (entry.verdict.kind === 'parses') { expect(() => { - parseCorpusScript(script); + parseReplayInput(script, undefined); }).not.toThrow(); return; } let thrown: unknown; try { - parseCorpusScript(script); + parseReplayInput(script, undefined); } catch (error) { thrown = error; } @@ -65,18 +70,33 @@ describe('replay-compat corpus', () => { expect(error.message).toContain(entry.verdict.hint); }); + // The freeze rule needs teeth: a mined entry's bytes must still hash to the + // git object id of the blob it was mined from, so "update the script until the + // parser agrees" cannot pass as a corpus update. `pnpm check:replay-compat` + // re-derives the same ids from history, which this hash alone cannot do. + test.each(REPLAY_COMPAT_CORPUS.map((entry) => [entry.id, entry] as const))( + '%s is byte-identical to its pinned source', + (_id, entry) => { + const bytes = readCorpusScript(entry); + if (entry.provenance.kind === 'mined') { + expect(gitBlobId(bytes)).toBe(entry.provenance.blob); + return; + } + expect(createHash('sha256').update(bytes).digest('hex')).toBe(entry.provenance.sha256); + }, + ); + test('every corpus script is claimed by exactly one manifest entry', () => { const claimed = REPLAY_COMPAT_CORPUS.map((entry) => entry.file); expect([...claimed].sort()).toEqual(listCorpusScriptFiles().sort()); expect(new Set(claimed).size).toBe(claimed.length); }); - test('entry ids are unique and name the producing released version', () => { + test('entry ids are unique and name a released producing version', () => { const ids = REPLAY_COMPAT_CORPUS.map((entry) => entry.id); expect(new Set(ids).size).toBe(ids.length); for (const entry of REPLAY_COMPAT_CORPUS) { - expect(entry.recordedBy, entry.id).toMatch(/^v\d+\.\d+\.\d+$/); - expect(entry.source, entry.id).toContain(entry.recordedBy); + expect(REPLAY_COMPAT_RELEASED_TAGS, entry.id).toContain(entry.recordedBy); expect(entry.file, entry.id).toContain(`.${entry.recordedBy}.ad`); } }); diff --git a/test/replay-compat/manifest.ts b/test/replay-compat/manifest.ts index 837ed4335..34b94343f 100644 --- a/test/replay-compat/manifest.ts +++ b/test/replay-compat/manifest.ts @@ -1,7 +1,7 @@ /** * The frozen `.ad` replay-compat corpus (#1417): one entry per historical script - * surface a RELEASED version could have written, paired with the verdict today's - * parser must return for it. + * surface a RELEASED version wrote, paired with the verdict today's parser must + * return for it. * * The scripts under `scripts/` are FROZEN. A grammar change that flips a verdict * edits the verdict here — never the script — and says why in the same PR. See @@ -10,6 +10,39 @@ import type { AppErrorCode } from '../../src/kernel/errors.ts'; +/** + * Every release tag the corpus draws from. `pnpm check:replay-compat` verifies + * each one against `git tag --list`, so an entry cannot claim a version that was + * never cut. + */ +export const REPLAY_COMPAT_RELEASED_TAGS: readonly string[] = [ + 'v0.11.0', + 'v0.11.2', + 'v0.11.8', + 'v0.12.5', + 'v0.15.0', + 'v0.15.1', + 'v0.16.0', + 'v0.16.8', + 'v0.17.0', + 'v0.20.0', +]; + +/** + * Where the frozen bytes came from, and the identity that pins them there. + * + * - `mined`: the corpus file IS the historical blob. `blob` is that blob's git + * object id, so hashing the checked-in bytes reproduces it (the corpus test) + * and `git rev-parse :` reproduces it from history + * (`pnpm check:replay-compat`). Rewriting the script cannot stay green. + * - `derived`: a surface the released grammar wrote that no shipped `.ad` file + * in this repo carries, composed from the release's own grammar/docs. There is + * no historical blob to point at, so the bytes are pinned by digest. + */ +export type ReplayCompatProvenance = + | { kind: 'mined'; path: string; blob: string } + | { kind: 'derived'; from: string; sha256: string }; + /** `parses`, or the migration refusal the parser owes a script of this vintage. */ export type ReplayCompatVerdict = | { kind: 'parses' } @@ -30,8 +63,7 @@ export type ReplayCompatEntry = { file: string; /** The released tag whose grammar produced this surface. */ recordedBy: string; - /** Where the surface was mined from, at `recordedBy`. */ - source: string; + provenance: ReplayCompatProvenance; covers: ReplayCompatCoverage[]; verdict: ReplayCompatVerdict; /** Why this entry exists, when the path and verdict do not say it. */ @@ -49,7 +81,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/android-01-settings@v0.11.0', file: 'scripts/integration/android-01-settings.v0.11.0.ad', recordedBy: 'v0.11.0', - source: 'test/integration/replays/android/01-settings.ad@v0.11.0', + provenance: { + kind: 'mined', + path: 'test/integration/replays/android/01-settings.ad', + blob: 'c91ca3f53082fa1cabcd6df885dd14714d481177', + }, covers: ['context-header', 'quoting'], verdict: { kind: 'parses' }, }, @@ -57,7 +93,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/android-01-settings@v0.11.2', file: 'scripts/integration/android-01-settings.v0.11.2.ad', recordedBy: 'v0.11.2', - source: 'test/integration/replays/android/01-settings.ad@v0.11.2', + provenance: { + kind: 'mined', + path: 'test/integration/replays/android/01-settings.ad', + blob: '47b9ff049772143e1bd8b0cd2e64cb381a48ae2d', + }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -65,7 +105,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/ios-device-01-physical-lifecycle@v0.11.0', file: 'scripts/integration/ios-device-01-physical-lifecycle.v0.11.0.ad', recordedBy: 'v0.11.0', - source: 'test/integration/replays/ios/device/01-physical-lifecycle.ad@v0.11.0', + provenance: { + kind: 'mined', + path: 'test/integration/replays/ios/device/01-physical-lifecycle.ad', + blob: '7f7f6cd974b1358a0ac9e210ad172ab38132ec2f', + }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -73,7 +117,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/ios-device-01-physical-lifecycle@v0.11.2', file: 'scripts/integration/ios-device-01-physical-lifecycle.v0.11.2.ad', recordedBy: 'v0.11.2', - source: 'test/integration/replays/ios/device/01-physical-lifecycle.ad@v0.11.2', + provenance: { + kind: 'mined', + path: 'test/integration/replays/ios/device/01-physical-lifecycle.ad', + blob: '797de36dc0d5c5aa2287176de72e82a3ccb21bff', + }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -81,7 +129,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/ios-device-01-physical-lifecycle@v0.15.0', file: 'scripts/integration/ios-device-01-physical-lifecycle.v0.15.0.ad', recordedBy: 'v0.15.0', - source: 'test/integration/replays/ios/device/01-physical-lifecycle.ad@v0.15.0', + provenance: { + kind: 'mined', + path: 'test/integration/replays/ios/device/01-physical-lifecycle.ad', + blob: '7a5ea2f64756b5400f1e987a32d78b058734461c', + }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -89,7 +141,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/ios-device-01-physical-lifecycle@v0.17.0', file: 'scripts/integration/ios-device-01-physical-lifecycle.v0.17.0.ad', recordedBy: 'v0.17.0', - source: 'test/integration/replays/ios/device/01-physical-lifecycle.ad@v0.17.0', + provenance: { + kind: 'mined', + path: 'test/integration/replays/ios/device/01-physical-lifecycle.ad', + blob: 'fb6e1c0b90d331a5c7075f37f6690412d0c926f4', + }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -97,7 +153,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/ios-simulator-01-settings@v0.11.0', file: 'scripts/integration/ios-simulator-01-settings.v0.11.0.ad', recordedBy: 'v0.11.0', - source: 'test/integration/replays/ios/simulator/01-settings.ad@v0.11.0', + provenance: { + kind: 'mined', + path: 'test/integration/replays/ios/simulator/01-settings.ad', + blob: '704f28b6d38dadf820a658320dca4c611ea9684d', + }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -105,7 +165,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/ios-simulator-01-settings@v0.11.2', file: 'scripts/integration/ios-simulator-01-settings.v0.11.2.ad', recordedBy: 'v0.11.2', - source: 'test/integration/replays/ios/simulator/01-settings.ad@v0.11.2', + provenance: { + kind: 'mined', + path: 'test/integration/replays/ios/simulator/01-settings.ad', + blob: 'b9482558fcde1f424dfba55a5dd4dfaecddb7bb4', + }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -113,7 +177,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/ios-simulator-01-settings@v0.15.0', file: 'scripts/integration/ios-simulator-01-settings.v0.15.0.ad', recordedBy: 'v0.15.0', - source: 'test/integration/replays/ios/simulator/01-settings.ad@v0.15.0', + provenance: { + kind: 'mined', + path: 'test/integration/replays/ios/simulator/01-settings.ad', + blob: 'f34cb81707b2a8022d466e0ac2b45fcd43f4dcc7', + }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -121,7 +189,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/ios-simulator-01-settings@v0.17.0', file: 'scripts/integration/ios-simulator-01-settings.v0.17.0.ad', recordedBy: 'v0.17.0', - source: 'test/integration/replays/ios/simulator/01-settings.ad@v0.17.0', + provenance: { + kind: 'mined', + path: 'test/integration/replays/ios/simulator/01-settings.ad', + blob: '418401db198bdb8b96ed80a25d1474effa693d35', + }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -129,7 +201,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/macos-01-system-settings@v0.11.0', file: 'scripts/integration/macos-01-system-settings.v0.11.0.ad', recordedBy: 'v0.11.0', - source: 'test/integration/replays/macos/01-system-settings.ad@v0.11.0', + provenance: { + kind: 'mined', + path: 'test/integration/replays/macos/01-system-settings.ad', + blob: 'e3f7f39b53d9a56a14a233129b54f9f110640836', + }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -137,7 +213,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/macos-01-system-settings@v0.11.2', file: 'scripts/integration/macos-01-system-settings.v0.11.2.ad', recordedBy: 'v0.11.2', - source: 'test/integration/replays/macos/01-system-settings.ad@v0.11.2', + provenance: { + kind: 'mined', + path: 'test/integration/replays/macos/01-system-settings.ad', + blob: '7df54a82235767e42355b204406421d02bae122b', + }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -145,7 +225,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/macos-01-system-settings@v0.17.0', file: 'scripts/integration/macos-01-system-settings.v0.17.0.ad', recordedBy: 'v0.17.0', - source: 'test/integration/replays/macos/01-system-settings.ad@v0.17.0', + provenance: { + kind: 'mined', + path: 'test/integration/replays/macos/01-system-settings.ad', + blob: 'd43f790938cbde47f99e135737589002f014193f', + }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -153,7 +237,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/android-02-deep-navigation@v0.11.2', file: 'scripts/integration/android-02-deep-navigation.v0.11.2.ad', recordedBy: 'v0.11.2', - source: 'test/integration/replays/android/02-deep-navigation.ad@v0.11.2', + provenance: { + kind: 'mined', + path: 'test/integration/replays/android/02-deep-navigation.ad', + blob: '483bccac102023cf8116dab9eca04db37f218851', + }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -161,7 +249,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/android-02-deep-navigation@v0.12.5', file: 'scripts/integration/android-02-deep-navigation.v0.12.5.ad', recordedBy: 'v0.12.5', - source: 'test/integration/replays/android/02-deep-navigation.ad@v0.12.5', + provenance: { + kind: 'mined', + path: 'test/integration/replays/android/02-deep-navigation.ad', + blob: '2072595e8e03f76662af0f7dba34d8d7f6a0333f', + }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -169,7 +261,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/android-03-scroll-discovery@v0.11.2', file: 'scripts/integration/android-03-scroll-discovery.v0.11.2.ad', recordedBy: 'v0.11.2', - source: 'test/integration/replays/android/03-scroll-discovery.ad@v0.11.2', + provenance: { + kind: 'mined', + path: 'test/integration/replays/android/03-scroll-discovery.ad', + blob: 'f37427fae15ba0d905bad924585ea898d7ff5985', + }, covers: ['context-header', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -177,7 +273,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/android-04-text-input-keyboard@v0.11.2', file: 'scripts/integration/android-04-text-input-keyboard.v0.11.2.ad', recordedBy: 'v0.11.2', - source: 'test/integration/replays/android/04-text-input-keyboard.ad@v0.11.2', + provenance: { + kind: 'mined', + path: 'test/integration/replays/android/04-text-input-keyboard.ad', + blob: '4dbdc1c01cd4600a9d5cbdef7e5c62a07ce6848a', + }, covers: ['context-header', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -185,7 +285,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/android-05-app-lifecycle@v0.11.2', file: 'scripts/integration/android-05-app-lifecycle.v0.11.2.ad', recordedBy: 'v0.11.2', - source: 'test/integration/replays/android/05-app-lifecycle.ad@v0.11.2', + provenance: { + kind: 'mined', + path: 'test/integration/replays/android/05-app-lifecycle.ad', + blob: 'ee20d0724568ef8d91b96726b10a705d216b9c2e', + }, covers: ['context-header', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -193,7 +297,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/android-06-swipe-gestures@v0.11.2', file: 'scripts/integration/android-06-swipe-gestures.v0.11.2.ad', recordedBy: 'v0.11.2', - source: 'test/integration/replays/android/06-swipe-gestures.ad@v0.11.2', + provenance: { + kind: 'mined', + path: 'test/integration/replays/android/06-swipe-gestures.ad', + blob: '686bace4c6eed1ea1ee4c760276347fc3986c3cc', + }, covers: ['context-header', 'retired-gesture', 'wait-landmark'], verdict: { kind: 'fails', @@ -206,7 +314,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/ios-simulator-02-deep-navigation@v0.11.2', file: 'scripts/integration/ios-simulator-02-deep-navigation.v0.11.2.ad', recordedBy: 'v0.11.2', - source: 'test/integration/replays/ios/simulator/02-deep-navigation.ad@v0.11.2', + provenance: { + kind: 'mined', + path: 'test/integration/replays/ios/simulator/02-deep-navigation.ad', + blob: '977f082d918c5ec3999d6e71904331465c1ffa85', + }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -214,7 +326,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/ios-simulator-02-deep-navigation@v0.15.0', file: 'scripts/integration/ios-simulator-02-deep-navigation.v0.15.0.ad', recordedBy: 'v0.15.0', - source: 'test/integration/replays/ios/simulator/02-deep-navigation.ad@v0.15.0', + provenance: { + kind: 'mined', + path: 'test/integration/replays/ios/simulator/02-deep-navigation.ad', + blob: 'edc97d2854c813ed4230fe40533628d8ead897bc', + }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -222,7 +338,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/ios-simulator-02-deep-navigation@v0.17.0', file: 'scripts/integration/ios-simulator-02-deep-navigation.v0.17.0.ad', recordedBy: 'v0.17.0', - source: 'test/integration/replays/ios/simulator/02-deep-navigation.ad@v0.17.0', + provenance: { + kind: 'mined', + path: 'test/integration/replays/ios/simulator/02-deep-navigation.ad', + blob: '86c27387f4d91e6bfcc5c2a424de41159da68cf5', + }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -230,7 +350,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/ios-simulator-03-scroll-discovery@v0.11.2', file: 'scripts/integration/ios-simulator-03-scroll-discovery.v0.11.2.ad', recordedBy: 'v0.11.2', - source: 'test/integration/replays/ios/simulator/03-scroll-discovery.ad@v0.11.2', + provenance: { + kind: 'mined', + path: 'test/integration/replays/ios/simulator/03-scroll-discovery.ad', + blob: '898bf8423074fc3e8fb05c1dfef8f1c674eb0529', + }, covers: ['context-header', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -238,7 +362,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/ios-simulator-03-scroll-discovery@v0.15.0', file: 'scripts/integration/ios-simulator-03-scroll-discovery.v0.15.0.ad', recordedBy: 'v0.15.0', - source: 'test/integration/replays/ios/simulator/03-scroll-discovery.ad@v0.15.0', + provenance: { + kind: 'mined', + path: 'test/integration/replays/ios/simulator/03-scroll-discovery.ad', + blob: '2646c29e9b2323d4652a72c74ceeb1c88f49ce75', + }, covers: ['context-header', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -246,7 +374,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/ios-simulator-04-text-input-keyboard@v0.11.2', file: 'scripts/integration/ios-simulator-04-text-input-keyboard.v0.11.2.ad', recordedBy: 'v0.11.2', - source: 'test/integration/replays/ios/simulator/04-text-input-keyboard.ad@v0.11.2', + provenance: { + kind: 'mined', + path: 'test/integration/replays/ios/simulator/04-text-input-keyboard.ad', + blob: '0a8be87e3e2caacdf6b35820bb16cc3d492c8fea', + }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -254,7 +386,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/ios-simulator-04-text-input-keyboard@v0.15.0', file: 'scripts/integration/ios-simulator-04-text-input-keyboard.v0.15.0.ad', recordedBy: 'v0.15.0', - source: 'test/integration/replays/ios/simulator/04-text-input-keyboard.ad@v0.15.0', + provenance: { + kind: 'mined', + path: 'test/integration/replays/ios/simulator/04-text-input-keyboard.ad', + blob: 'a9f04c26d8ff91ae705bfa2b4f0c61c0b35b8dd3', + }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -262,7 +398,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/ios-simulator-05-app-lifecycle@v0.11.2', file: 'scripts/integration/ios-simulator-05-app-lifecycle.v0.11.2.ad', recordedBy: 'v0.11.2', - source: 'test/integration/replays/ios/simulator/05-app-lifecycle.ad@v0.11.2', + provenance: { + kind: 'mined', + path: 'test/integration/replays/ios/simulator/05-app-lifecycle.ad', + blob: '75cb3ca578c93146731676fb4d0bbf8d71b4e6a9', + }, covers: ['context-header', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -270,7 +410,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/ios-simulator-05-app-lifecycle@v0.15.0', file: 'scripts/integration/ios-simulator-05-app-lifecycle.v0.15.0.ad', recordedBy: 'v0.15.0', - source: 'test/integration/replays/ios/simulator/05-app-lifecycle.ad@v0.15.0', + provenance: { + kind: 'mined', + path: 'test/integration/replays/ios/simulator/05-app-lifecycle.ad', + blob: '32d2a15f5006169e8578cde608703dd5e630f577', + }, covers: ['context-header', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -278,7 +422,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/ios-simulator-06-swipe-gestures@v0.11.2', file: 'scripts/integration/ios-simulator-06-swipe-gestures.v0.11.2.ad', recordedBy: 'v0.11.2', - source: 'test/integration/replays/ios/simulator/06-swipe-gestures.ad@v0.11.2', + provenance: { + kind: 'mined', + path: 'test/integration/replays/ios/simulator/06-swipe-gestures.ad', + blob: 'ca067860a93fbd1ca976a02401c3d86993cb2028', + }, covers: ['context-header', 'retired-gesture', 'wait-landmark'], verdict: { kind: 'fails', @@ -290,7 +438,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/ios-simulator-06-swipe-gestures@v0.15.0', file: 'scripts/integration/ios-simulator-06-swipe-gestures.v0.15.0.ad', recordedBy: 'v0.15.0', - source: 'test/integration/replays/ios/simulator/06-swipe-gestures.ad@v0.15.0', + provenance: { + kind: 'mined', + path: 'test/integration/replays/ios/simulator/06-swipe-gestures.ad', + blob: '8f892f1c7af020680a4f921c60cbc4e30ebb021f', + }, covers: ['context-header', 'retired-gesture', 'wait-landmark'], verdict: { kind: 'fails', @@ -302,7 +454,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/linux-01-desktop-smoke@v0.11.8', file: 'scripts/integration/linux-01-desktop-smoke.v0.11.8.ad', recordedBy: 'v0.11.8', - source: 'test/integration/replays/linux/01-desktop-smoke.ad@v0.11.8', + provenance: { + kind: 'mined', + path: 'test/integration/replays/linux/01-desktop-smoke.ad', + blob: 'c707cfdb2f172437cfb094d7e9eb7bb98213b450', + }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -310,7 +466,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'integration/linux-01-desktop-smoke@v0.15.0', file: 'scripts/integration/linux-01-desktop-smoke.v0.15.0.ad', recordedBy: 'v0.15.0', - source: 'test/integration/replays/linux/01-desktop-smoke.ad@v0.15.0', + provenance: { + kind: 'mined', + path: 'test/integration/replays/linux/01-desktop-smoke.ad', + blob: '1d88b91a84ff6a59ff84c5a5ab79c3c75620543c', + }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -318,7 +478,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'examples/checkout-form-android@v0.15.1', file: 'scripts/examples/checkout-form-android.v0.15.1.ad', recordedBy: 'v0.15.1', - source: 'examples/test-app/replays/checkout-form-android.ad@v0.15.1', + provenance: { + kind: 'mined', + path: 'examples/test-app/replays/checkout-form-android.ad', + blob: '53b62ff99f9ad277d749cd9e1ff76eac2769eb28', + }, covers: ['context-header', 'env-vars', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -326,7 +490,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'examples/checkout-form-android@v0.16.8', file: 'scripts/examples/checkout-form-android.v0.16.8.ad', recordedBy: 'v0.16.8', - source: 'examples/test-app/replays/checkout-form-android.ad@v0.16.8', + provenance: { + kind: 'mined', + path: 'examples/test-app/replays/checkout-form-android.ad', + blob: '8d1d0ada467ca551f66ce69dabf6cc01527a31d5', + }, covers: ['context-header', 'env-vars', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -334,7 +502,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'examples/checkout-form@v0.15.1', file: 'scripts/examples/checkout-form.v0.15.1.ad', recordedBy: 'v0.15.1', - source: 'examples/test-app/replays/checkout-form.ad@v0.15.1', + provenance: { + kind: 'mined', + path: 'examples/test-app/replays/checkout-form.ad', + blob: '8bd23afbd93fe7ec8845882faf346509e9e00d3f', + }, covers: ['context-header', 'env-vars', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -342,7 +514,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'examples/checkout-form@v0.16.8', file: 'scripts/examples/checkout-form.v0.16.8.ad', recordedBy: 'v0.16.8', - source: 'examples/test-app/replays/checkout-form.ad@v0.16.8', + provenance: { + kind: 'mined', + path: 'examples/test-app/replays/checkout-form.ad', + blob: '4c89212dfa191309377a538a16b95bafa8d3d754', + }, covers: ['context-header', 'env-vars', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -350,7 +526,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'examples/gesture-lab@v0.16.0', file: 'scripts/examples/gesture-lab.v0.16.0.ad', recordedBy: 'v0.16.0', - source: 'examples/test-app/replays/gesture-lab.ad@v0.16.0', + provenance: { + kind: 'mined', + path: 'examples/test-app/replays/gesture-lab.ad', + blob: 'a47066ee1db7178ebbac3cdeb149cdffb8b1f303', + }, covers: ['context-header', 'env-vars', 'quoting', 'retired-gesture', 'wait-landmark'], verdict: { kind: 'fails', @@ -363,7 +543,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'examples/gesture-lab@v0.16.8', file: 'scripts/examples/gesture-lab.v0.16.8.ad', recordedBy: 'v0.16.8', - source: 'examples/test-app/replays/gesture-lab.ad@v0.16.8', + provenance: { + kind: 'mined', + path: 'examples/test-app/replays/gesture-lab.ad', + blob: '2d6b3aee582d6a04b02e6d51a9d8e6769ed65816', + }, covers: ['context-header', 'env-vars', 'quoting', 'retired-gesture', 'wait-landmark'], verdict: { kind: 'fails', @@ -375,7 +559,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'examples/gesture-lab@v0.20.0', file: 'scripts/examples/gesture-lab.v0.20.0.ad', recordedBy: 'v0.20.0', - source: 'examples/test-app/replays/gesture-lab.ad@v0.20.0', + provenance: { + kind: 'mined', + path: 'examples/test-app/replays/gesture-lab.ad', + blob: 'ac3946e335a97237e5d53fbb3fec8472b955b1f6', + }, covers: ['context-header', 'env-vars', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -383,7 +571,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'examples/gesture-lab-android@v0.20.0', file: 'scripts/examples/gesture-lab-android.v0.20.0.ad', recordedBy: 'v0.20.0', - source: 'examples/test-app/replays/gesture-lab-android.ad@v0.20.0', + provenance: { + kind: 'mined', + path: 'examples/test-app/replays/gesture-lab-android.ad', + blob: 'cc626f1c00c42fb59fc7f3bdee82775ad25a76f2', + }, covers: ['context-header', 'env-vars', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, }, @@ -391,7 +583,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'docs/vars-parameterized@v0.15.1', file: 'scripts/docs/vars-parameterized.v0.15.1.ad', recordedBy: 'v0.15.1', - source: 'website/docs/docs/replay-e2e.md@v0.15.1 ("Parametrise `.ad` scripts")', + provenance: { + kind: 'derived', + from: 'website/docs/docs/replay-e2e.md@v0.15.1 ("Parametrise `.ad` scripts")', + sha256: 'b2797c29dd85b538505a21699ae9d4a91096ccff54ecf3694d4c194743beeae0', + }, covers: ['context-header', 'env-vars'], verdict: { kind: 'parses' }, note: 'The shipped `${VAR}` surface as documented by the release that introduced it.', @@ -400,8 +596,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'docs/vars-quoting-fallback@v0.15.1', file: 'scripts/docs/vars-quoting-fallback.v0.15.1.ad', recordedBy: 'v0.15.1', - source: - 'website/docs/docs/replay-e2e.md@v0.15.1 ("Substitution happens inside parsed string values" / "Fallback and escape")', + provenance: { + kind: 'derived', + from: 'website/docs/docs/replay-e2e.md@v0.15.1 ("Substitution happens inside parsed string values" / "Fallback and escape")', + sha256: '71859426c7f4798f562482dac2024ae996d3890ff89989ef2dfd9a46e40ca4f9', + }, covers: ['context-header', 'env-vars', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, note: 'Quoted selector values, escaped inner quotes, and `${VAR:-default}` in one script.', @@ -410,7 +609,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'docs/record-as-parameterized-fill@v0.15.1', file: 'scripts/docs/record-as-parameterized-fill.v0.15.1.ad', recordedBy: 'v0.15.1', - source: 'website/docs/docs/replay-e2e.md@v0.15.1 + src/replay/vars.ts@v0.15.1', + provenance: { + kind: 'derived', + from: 'website/docs/docs/replay-e2e.md@v0.15.1 + src/replay/vars.ts@v0.15.1', + sha256: '6d7e24c05eff2f41c9824f1b43a99a8d611dd2eaf3d5e27edf7707dd5b040a87', + }, covers: ['context-header', 'env-vars', 'quoting'], verdict: { kind: 'parses' }, note: '`--record-as` projects a recorded value into exactly this shipped surface: an `env` declaration plus a `${VAR}` fill argument. The flag itself is not in any tag as of the v0.20.0 baseline, so the corpus freezes the released script surface, not the unreleased recorder flag.', @@ -419,8 +622,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'docs/vars-reserved-env-key@v0.15.1', file: 'scripts/docs/vars-reserved-env-key.v0.15.1.ad', recordedBy: 'v0.15.1', - source: - 'website/docs/docs/replay-e2e.md@v0.15.1 ("User-defined keys starting with AD_ are rejected")', + provenance: { + kind: 'derived', + from: 'website/docs/docs/replay-e2e.md@v0.15.1 ("User-defined keys starting with AD_ are rejected")', + sha256: '68121c5f5f2a60cfd813d879030a411b87cae6dab4570e778f716a1c7e3bd5b6', + }, covers: ['context-header', 'env-vars'], verdict: { kind: 'fails', @@ -433,7 +639,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'docs/vars-env-after-action@v0.15.1', file: 'scripts/docs/vars-env-after-action.v0.15.1.ad', recordedBy: 'v0.15.1', - source: 'src/replay/script.ts@v0.15.1 (env ordering guard)', + provenance: { + kind: 'derived', + from: 'src/replay/script.ts@v0.15.1 (env ordering guard)', + sha256: 'e483866a665f952022ceefc35eb97eed87056b52f5cfab7f87849552c47f372d', + }, covers: ['context-header', 'env-vars'], verdict: { kind: 'fails', @@ -446,7 +656,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'docs/context-header-full@v0.15.1', file: 'scripts/docs/context-header-full.v0.15.1.ad', recordedBy: 'v0.15.1', - source: 'website/docs/docs/replay-e2e.md@v0.15.1 ("Run a lightweight `.ad` suite")', + provenance: { + kind: 'derived', + from: 'website/docs/docs/replay-e2e.md@v0.15.1 ("Run a lightweight `.ad` suite")', + sha256: 'af40bb495926c3c75681921f18ce6a673516279ce64e87225d5dfd04ebb805c4', + }, covers: ['context-header', 'wait-landmark', 'quoting'], verdict: { kind: 'parses' }, note: 'Every documented context key on one header: platform, target, timeout, retries.', @@ -455,8 +669,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'docs/context-header-conflicting-platform@v0.15.1', file: 'scripts/docs/context-header-conflicting-platform.v0.15.1.ad', recordedBy: 'v0.15.1', - source: - 'website/docs/docs/replay-e2e.md@v0.15.1 ("duplicate keys in the context header fail fast")', + provenance: { + kind: 'derived', + from: 'website/docs/docs/replay-e2e.md@v0.15.1 ("duplicate keys in the context header fail fast")', + sha256: '8308e62b032db460546801b173dcad7f28af1a080b824e7eed118dc67b41c039', + }, covers: ['context-header'], verdict: { kind: 'fails', @@ -469,8 +686,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'docs/wait-landmark-forms@v0.20.0', file: 'scripts/docs/wait-landmark-forms.v0.20.0.ad', recordedBy: 'v0.20.0', - source: - 'src/commands/capture/wait-command-contract.ts@v0.20.0 + src/replay/script-utils.ts@v0.20.0', + provenance: { + kind: 'derived', + from: 'src/commands/capture/wait-command-contract.ts@v0.20.0 + src/replay/script-utils.ts@v0.20.0', + sha256: '98501e4372c0ebc1583bf0424792d31872943e5917d84da7db20b042a37292b6', + }, covers: ['context-header', 'wait-landmark', 'quoting'], verdict: { kind: 'parses' }, note: 'All released wait kinds a recording can carry: duration, `stable`, selector landmark (#1349), and a generation-pinned `@ref`.', @@ -479,7 +699,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'docs/target-v1-annotated-click@v0.20.0', file: 'scripts/docs/target-v1-annotated-click.v0.20.0.ad', recordedBy: 'v0.20.0', - source: 'docs/adr/0012-interactive-replay.md@v0.20.0 + src/replay/target-identity.ts@v0.20.0', + provenance: { + kind: 'derived', + from: 'docs/adr/0012-interactive-replay.md@v0.20.0 + src/replay/target-identity.ts@v0.20.0', + sha256: '82a86018ead69e03845e1ff19981ac94cc4f4f90b9a72a5355d110c821e8e67a', + }, covers: ['context-header', 'target-annotation', 'quoting'], verdict: { kind: 'parses' }, note: 'ADR 0012 identity annotation bound to its action line.', @@ -488,7 +712,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'docs/target-v1-unbound-annotation@v0.20.0', file: 'scripts/docs/target-v1-unbound-annotation.v0.20.0.ad', recordedBy: 'v0.20.0', - source: 'src/replay/script.ts@v0.20.0 (annotation binding guard)', + provenance: { + kind: 'derived', + from: 'src/replay/script.ts@v0.20.0 (annotation binding guard)', + sha256: 'f270fb0ee7757556e3b0f51893a2054360f291693fee6c425c3cf2cf3771aa33', + }, covers: ['context-header', 'target-annotation'], verdict: { kind: 'fails', @@ -501,8 +729,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'docs/gesture-rotate-velocity@v0.16.8', file: 'scripts/docs/gesture-rotate-velocity.v0.16.8.ad', recordedBy: 'v0.16.8', - source: - 'src/commands/cli-grammar/gesture.ts@v0.16.8 (rotateGesturePositionals emitted a trailing velocity)', + provenance: { + kind: 'derived', + from: 'src/commands/cli-grammar/gesture.ts@v0.16.8 (rotateGesturePositionals emitted a trailing velocity)', + sha256: '5675cc0f4fda8f3b671b2df4c1bafa17b6616fe0add38165a28bb570b3ce03af', + }, covers: ['context-header', 'wait-landmark', 'quoting', 'retired-gesture'], verdict: { kind: 'fails', @@ -515,8 +746,11 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ id: 'docs/gesture-swipe-preset-duration@v0.16.8', file: 'scripts/docs/gesture-swipe-preset-duration.v0.16.8.ad', recordedBy: 'v0.16.8', - source: - 'src/commands/cli-grammar/gesture.ts@v0.16.8 (swipePresetPositionals emitted a trailing durationMs)', + provenance: { + kind: 'derived', + from: 'src/commands/cli-grammar/gesture.ts@v0.16.8 (swipePresetPositionals emitted a trailing durationMs)', + sha256: '33548361f17388b53ef02655b16a1ca78638b2c2e29035631cba61393a068e0c', + }, covers: ['context-header', 'retired-gesture'], verdict: { kind: 'fails', From 2aef0cd547af8f39d293999a3f74c9e90bae27df Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Pierzcha=C5=82a?= Date: Mon, 27 Jul 2026 15:20:32 +0000 Subject: [PATCH 3/8] test: lock replay-compat provenance kind by corpus area and verify it in CI Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- .github/workflows/ci.yml | 20 ++++++++ scripts/check-affected/checks.ts | 9 ++++ scripts/check-affected/model.test.ts | 10 +++- scripts/check-affected/model.ts | 45 ++++++++++++------ scripts/check-affected/run.test.ts | 1 + scripts/check-replay-compat-provenance.ts | 10 +++- test/replay-compat/corpus.test.ts | 28 ++++++++++++ test/replay-compat/provenance-rules.ts | 56 +++++++++++++++++++++++ 8 files changed, 161 insertions(+), 18 deletions(-) create mode 100644 test/replay-compat/provenance-rules.ts diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 464487272..01d1a27d5 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -216,6 +216,26 @@ jobs: - name: Check for production-unused exports run: pnpm check:production-exports + replay-compat-provenance: + # The frozen replay-compat corpus (#1417) claims each entry was published by + # a released tag. Only a full-history checkout can re-derive that claim, so + # this job exists separately from the shallow-clone-safe unit lane. + name: Replay-Compat Provenance + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - name: Checkout + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + fetch-depth: 0 + fetch-tags: true + + - name: Setup toolchain + uses: ./.github/actions/setup-node-pnpm + + - name: Verify corpus entries against their released blobs + run: pnpm check:replay-compat + coverage: # Runs the full unit + provider-integration suites under coverage with # thresholds, so a separate unit-tests job would rerun the same tests. diff --git a/scripts/check-affected/checks.ts b/scripts/check-affected/checks.ts index 5e2dc5da3..3130cfd9c 100644 --- a/scripts/check-affected/checks.ts +++ b/scripts/check-affected/checks.ts @@ -152,6 +152,15 @@ export const CHECK_CATALOG: readonly CheckSpec[] = [ ciJobs: [], localRunnable: true, }, + { + id: 'replay-compat', + label: 'Replay-compat corpus provenance (released blobs)', + kind: { type: 'script', script: 'check:replay-compat' }, + // Needs full history and tags, so it runs in its own fetch-depth: 0 job + // rather than inside the shallow-clone-safe unit lane. + ciJobs: ['Replay-Compat Provenance'], + localRunnable: true, + }, ]; export function getCheckSpec(id: CheckId): CheckSpec { diff --git a/scripts/check-affected/model.test.ts b/scripts/check-affected/model.test.ts index 16f51fb42..584e50990 100644 --- a/scripts/check-affected/model.test.ts +++ b/scripts/check-affected/model.test.ts @@ -114,10 +114,17 @@ test('a non-.ts fixture under an owned root fails open (format alone is not owne assert.equal(result.failOpenReasons[0]?.rule, 'ambiguous-path'); }); -test('a frozen replay-compat corpus script selects the unit lane that asserts it', () => { +test('a frozen replay-compat corpus script selects the unit lane and the provenance verifier', () => { const result = plan(['test/replay-compat/scripts/examples/gesture-lab.v0.16.8.ad']); assert.equal(result.failOpen, false); assert.ok(result.checks.includes('unit')); + assert.ok(result.checks.includes('replay-compat')); +}); + +test('a replay-compat manifest edit selects the provenance verifier', () => { + const result = plan(['test/replay-compat/manifest.ts']); + assert.equal(result.failOpen, false); + assert.ok(result.checks.includes('replay-compat')); }); test('skills guidance change selects format + skillgym, not docs-only', () => { @@ -181,6 +188,7 @@ test('every catalog command resolves against package scripts', () => { 'build:macos-helper': 'x', 'test:smoke:web': 'x', 'test:skillgym': 'x', + 'check:replay-compat': 'x', }; for (const spec of CHECK_CATALOG) { const command = resolveCommand(spec, scripts, 'origin/main'); diff --git a/scripts/check-affected/model.ts b/scripts/check-affected/model.ts index 86a810440..5f519d235 100644 --- a/scripts/check-affected/model.ts +++ b/scripts/check-affected/model.ts @@ -40,7 +40,8 @@ export type CheckId = | 'android-helpers' | 'macos-helper' | 'web-smoke' - | 'skillgym'; + | 'skillgym' + | 'replay-compat'; // The complete local check universe. A fail-open plan selects all of these; // keep it in sync with the catalog in checks.ts (asserted by the self-test). @@ -63,6 +64,7 @@ export const ALL_CHECKS: readonly CheckId[] = [ 'macos-helper', 'web-smoke', 'skillgym', + 'replay-compat', ]; export type SelectionReason = { @@ -228,20 +230,33 @@ const nodeIntegrationOwnership: OwnershipRule = ({ file }) => ? [reason('integration-node', file, 'node-integration', 'node --test integration smoke')] : []; -// The frozen replay-compat corpus (#1417) is `.ad` fixture data whose only -// consumer is `test/replay-compat/corpus.test.ts` in the unit lane, so a corpus -// addition has a derivable owner instead of failing open on its extension. -const replayCompatOwnership: OwnershipRule = ({ file }) => - file.startsWith('test/replay-compat/') && file.endsWith('.ad') - ? [ - reason( - 'unit', - file, - 'own:replay-compat', - 'frozen replay-compat corpus is asserted by the unit-lane corpus test', - ), - ] - : []; +// The frozen replay-compat corpus (#1417). `.ad` fixture data would otherwise +// fail open on its extension: its only consumer is the unit-lane corpus test. +// Any corpus change — script or manifest — also runs the history-backed +// provenance verifier, which is the only gate that can prove an entry's blob +// really came from the release tag it names. +const replayCompatOwnership: OwnershipRule = ({ file }) => { + if (!file.startsWith('test/replay-compat/')) return []; + const selections = [ + reason( + 'replay-compat', + file, + 'own:replay-compat-provenance', + 'corpus provenance is re-derived from released git blobs', + ), + ]; + if (file.endsWith('.ad')) { + selections.push( + reason( + 'unit', + file, + 'own:replay-compat', + 'frozen replay-compat corpus is asserted by the unit-lane corpus test', + ), + ); + } + return selections; +}; // SkillGym validates skill guidance (`skills/`) and owns its harness // (`test/skillgym/`); the Testing Matrix in docs/agents/testing.md routes diff --git a/scripts/check-affected/run.test.ts b/scripts/check-affected/run.test.ts index 13bcf11fb..d69ba6f59 100644 --- a/scripts/check-affected/run.test.ts +++ b/scripts/check-affected/run.test.ts @@ -101,6 +101,7 @@ const ALL_SCRIPTS: Record = { 'test:integration:node': 'x', 'test:integration:progress:check': 'x', 'test:skillgym': 'x', + 'check:replay-compat': 'x', }; const ARGS = { base: 'origin/main', head: 'HEAD', json: false, run: true }; diff --git a/scripts/check-replay-compat-provenance.ts b/scripts/check-replay-compat-provenance.ts index 1a4f7671d..c38c7eb30 100644 --- a/scripts/check-replay-compat-provenance.ts +++ b/scripts/check-replay-compat-provenance.ts @@ -11,7 +11,11 @@ import fs from 'node:fs'; import path from 'node:path'; -import { REPLAY_COMPAT_CORPUS, REPLAY_COMPAT_RELEASED_TAGS } from '../test/replay-compat/manifest.ts'; +import { + REPLAY_COMPAT_CORPUS, + REPLAY_COMPAT_RELEASED_TAGS, +} from '../test/replay-compat/manifest.ts'; +import { findProvenanceKindViolations } from '../test/replay-compat/provenance-rules.ts'; import { runCmdSync } from '../src/utils/exec.ts'; const repoRoot = path.resolve(import.meta.dirname, '..'); @@ -27,7 +31,9 @@ if (git(['rev-parse', '--is-shallow-repository']) === 'true') { ); } -const failures: string[] = []; +// An entry that escaped `mined` would also escape the history check below, so +// the area rule is enforced here as well as in the unit lane. +const failures: string[] = [...findProvenanceKindViolations(REPLAY_COMPAT_CORPUS)]; const existingTags = new Set(git(['tag', '--list']).split('\n').filter(Boolean)); if (existingTags.size === 0) { diff --git a/test/replay-compat/corpus.test.ts b/test/replay-compat/corpus.test.ts index 48d7c34ad..4193287ee 100644 --- a/test/replay-compat/corpus.test.ts +++ b/test/replay-compat/corpus.test.ts @@ -11,6 +11,7 @@ import { type ReplayCompatCoverage, type ReplayCompatEntry, } from './manifest.ts'; +import { findProvenanceKindViolations, requiredProvenanceKind } from './provenance-rules.ts'; const CORPUS_DIR = fileURLToPath(new URL('.', import.meta.url)); @@ -86,6 +87,33 @@ describe('replay-compat corpus', () => { }, ); + // Reclassifying a mined entry as `derived` would let edited bytes be re-pinned + // by digest inside this same manifest and skipped by the historical verifier, + // so the corpus area — not the manifest line — fixes the kind. + test('provenance kind is fixed by corpus area', () => { + expect(findProvenanceKindViolations(REPLAY_COMPAT_CORPUS)).toEqual([]); + for (const entry of REPLAY_COMPAT_CORPUS) { + expect(entry.provenance.kind, entry.id).toBe(requiredProvenanceKind(entry.file)); + } + }); + + test('a mined entry relabelled as derived is rejected', () => { + const mined = REPLAY_COMPAT_CORPUS.find((entry) => entry.provenance.kind === 'mined'); + expect(mined).toBeDefined(); + const reclassified: ReplayCompatEntry = { + ...(mined as ReplayCompatEntry), + provenance: { kind: 'derived', from: 'hand-written', sha256: 'f'.repeat(64) }, + }; + expect(findProvenanceKindViolations([reclassified])).toHaveLength(1); + expect(findProvenanceKindViolations([reclassified])[0]).toContain('must stay "mined"'); + }); + + test('a corpus area with no declared provenance kind is rejected', () => { + expect(() => requiredProvenanceKind('scripts/handwritten/whatever.v0.20.0.ad')).toThrow( + /no declared provenance kind/, + ); + }); + test('every corpus script is claimed by exactly one manifest entry', () => { const claimed = REPLAY_COMPAT_CORPUS.map((entry) => entry.file); expect([...claimed].sort()).toEqual(listCorpusScriptFiles().sort()); diff --git a/test/replay-compat/provenance-rules.ts b/test/replay-compat/provenance-rules.ts new file mode 100644 index 000000000..f73dde136 --- /dev/null +++ b/test/replay-compat/provenance-rules.ts @@ -0,0 +1,56 @@ +/** + * Which provenance kind each corpus area is allowed to use (#1417). + * + * The kind decides how strongly an entry is pinned: `mined` bytes are re-derived + * from git history by `pnpm check:replay-compat`, `derived` bytes are only + * digest-pinned inside the same editable manifest. Without this rule an entry + * could be edited, relabelled `derived`, given a matching digest, and quietly + * escape the historical verifier. So the area — not the manifest line — decides: + * anything mined out of the replay suites must stay `mined` forever. + */ + +import type { ReplayCompatEntry, ReplayCompatProvenance } from './manifest.ts'; + +const CORPUS_AREA_PROVENANCE_KIND: Readonly> = { + // Mined from the git history of `test/integration/replays`. + integration: 'mined', + // Mined from the git history of `examples/test-app/replays`. + examples: 'mined', + // Composed from a release's own grammar/docs: no historical blob exists. + docs: 'derived', +}; + +function corpusArea(file: string): string { + const area = file.split('/')[1]; + if (!area || !file.startsWith('scripts/')) { + throw new Error(`Corpus file "${file}" must live under scripts//.`); + } + return area; +} + +export function requiredProvenanceKind(file: string): ReplayCompatProvenance['kind'] { + const area = corpusArea(file); + const kind = CORPUS_AREA_PROVENANCE_KIND[area]; + if (!kind) { + throw new Error( + `Corpus area "${area}" has no declared provenance kind. Add it to CORPUS_AREA_PROVENANCE_KIND ` + + `in test/replay-compat/provenance-rules.ts before adding entries under it.`, + ); + } + return kind; +} + +/** Human-readable violations, so the same rule can fail a test or a CI script. */ +export function findProvenanceKindViolations( + entries: readonly ReplayCompatEntry[], +): readonly string[] { + return entries.flatMap((entry) => { + const required = requiredProvenanceKind(entry.file); + if (entry.provenance.kind === required) return []; + return [ + `${entry.id}: entries under scripts/${corpusArea(entry.file)}/ must stay "${required}", ` + + `but this one is "${entry.provenance.kind}". Reclassifying a mined script drops it out of ` + + `pnpm check:replay-compat; keep the released blob id instead.`, + ]; + }); +} From aba12705599cd478965ada50117920c711af2084 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Pierzcha=C5=82a?= Date: Mon, 27 Jul 2026 15:20:46 +0000 Subject: [PATCH 4/8] docs: describe corpus provenance-kind lock and CI job Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- test/replay-compat/README.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/test/replay-compat/README.md b/test/replay-compat/README.md index a8e87aa59..b2f707dcc 100644 --- a/test/replay-compat/README.md +++ b/test/replay-compat/README.md @@ -44,10 +44,17 @@ surface, and `provenance` pins the bytes: - `derived` entries (`scripts/docs/`) have no historical blob to point at, so their bytes are pinned by SHA-256. +The kind is not a free choice: `provenance-rules.ts` fixes it by corpus area (`scripts/integration/` +and `scripts/examples/` must stay `mined`, `scripts/docs/` is `derived`), so an edited script cannot +be relabelled `derived`, re-pinned by digest, and thereby skipped by the history check. Both the unit +test and the verifier enforce that rule, and a new area must declare its kind before entries can live +under it. + `pnpm check:replay-compat` closes the other half: it re-derives every mined id from git history (`git rev-parse :`) and checks every cited tag against `git tag --list`, so an -entry cannot claim a version that was never cut. It needs full history and tags, so it is a -maintainer/reviewer check rather than part of the shallow-clone-safe unit lane. +entry cannot claim a version that was never cut. It needs full history and tags, so it runs in its +own `Replay-Compat Provenance` CI job (`fetch-depth: 0`) rather than inside the shallow-clone-safe +unit lane, and any change under `test/replay-compat/` selects it in `pnpm check:affected`. **Released surfaces only.** A grammar state that only ever existed between commits is not compat surface — `git tag --contains ` decides, and unreleased shapes stay out (AGENTS.md, From 09a055719d2515033ab6a7df21afb24580f50cba Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Pierzcha=C5=82a?= Date: Mon, 27 Jul 2026 16:30:32 +0000 Subject: [PATCH 5/8] test: prune replay-compat corpus to minimal witnesses per shipped form Reviewer feedback on #1436: the mechanism earns its place, the dataset did not. Drop the 30 corpus entries whose bytes repeat a syntactic form or a migration refusal another entry already witnesses (platform twins and adjacent-release re-recordings), leaving 22 deliberate entries; make note required and state per entry which form or refusal it is the sole witness of. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- test/replay-compat/README.md | 24 +- test/replay-compat/corpus.test.ts | 11 + test/replay-compat/manifest.ts | 405 +----------------- .../scripts/examples/checkout-form.v0.15.1.ad | 21 - .../scripts/examples/checkout-form.v0.16.8.ad | 22 - .../scripts/examples/gesture-lab.v0.16.8.ad | 40 -- .../scripts/examples/gesture-lab.v0.20.0.ad | 47 -- .../android-01-settings.v0.11.2.ad | 15 - .../android-02-deep-navigation.v0.12.5.ad | 18 - .../android-03-scroll-discovery.v0.11.2.ad | 15 - ...os-device-01-physical-lifecycle.v0.11.2.ad | 8 - ...os-device-01-physical-lifecycle.v0.15.0.ad | 8 - ...os-device-01-physical-lifecycle.v0.17.0.ad | 8 - .../ios-simulator-01-settings.v0.11.0.ad | 12 - .../ios-simulator-01-settings.v0.11.2.ad | 12 - .../ios-simulator-01-settings.v0.15.0.ad | 12 - .../ios-simulator-01-settings.v0.17.0.ad | 12 - ...os-simulator-02-deep-navigation.v0.11.2.ad | 11 - ...os-simulator-02-deep-navigation.v0.15.0.ad | 11 - ...os-simulator-02-deep-navigation.v0.17.0.ad | 11 - ...s-simulator-03-scroll-discovery.v0.11.2.ad | 11 - ...s-simulator-03-scroll-discovery.v0.15.0.ad | 11 - ...imulator-04-text-input-keyboard.v0.11.2.ad | 11 - ...imulator-04-text-input-keyboard.v0.15.0.ad | 11 - .../ios-simulator-05-app-lifecycle.v0.11.2.ad | 13 - .../ios-simulator-05-app-lifecycle.v0.15.0.ad | 13 - ...ios-simulator-06-swipe-gestures.v0.11.2.ad | 16 - ...ios-simulator-06-swipe-gestures.v0.15.0.ad | 16 - .../linux-01-desktop-smoke.v0.11.8.ad | 12 - .../linux-01-desktop-smoke.v0.15.0.ad | 12 - .../macos-01-system-settings.v0.11.0.ad | 16 - .../macos-01-system-settings.v0.11.2.ad | 16 - .../macos-01-system-settings.v0.17.0.ad | 15 - 33 files changed, 51 insertions(+), 845 deletions(-) delete mode 100644 test/replay-compat/scripts/examples/checkout-form.v0.15.1.ad delete mode 100644 test/replay-compat/scripts/examples/checkout-form.v0.16.8.ad delete mode 100644 test/replay-compat/scripts/examples/gesture-lab.v0.16.8.ad delete mode 100644 test/replay-compat/scripts/examples/gesture-lab.v0.20.0.ad delete mode 100644 test/replay-compat/scripts/integration/android-01-settings.v0.11.2.ad delete mode 100644 test/replay-compat/scripts/integration/android-02-deep-navigation.v0.12.5.ad delete mode 100644 test/replay-compat/scripts/integration/android-03-scroll-discovery.v0.11.2.ad delete mode 100644 test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.11.2.ad delete mode 100644 test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.15.0.ad delete mode 100644 test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.17.0.ad delete mode 100644 test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.11.0.ad delete mode 100644 test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.11.2.ad delete mode 100644 test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.15.0.ad delete mode 100644 test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.17.0.ad delete mode 100644 test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.11.2.ad delete mode 100644 test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.15.0.ad delete mode 100644 test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.17.0.ad delete mode 100644 test/replay-compat/scripts/integration/ios-simulator-03-scroll-discovery.v0.11.2.ad delete mode 100644 test/replay-compat/scripts/integration/ios-simulator-03-scroll-discovery.v0.15.0.ad delete mode 100644 test/replay-compat/scripts/integration/ios-simulator-04-text-input-keyboard.v0.11.2.ad delete mode 100644 test/replay-compat/scripts/integration/ios-simulator-04-text-input-keyboard.v0.15.0.ad delete mode 100644 test/replay-compat/scripts/integration/ios-simulator-05-app-lifecycle.v0.11.2.ad delete mode 100644 test/replay-compat/scripts/integration/ios-simulator-05-app-lifecycle.v0.15.0.ad delete mode 100644 test/replay-compat/scripts/integration/ios-simulator-06-swipe-gestures.v0.11.2.ad delete mode 100644 test/replay-compat/scripts/integration/ios-simulator-06-swipe-gestures.v0.15.0.ad delete mode 100644 test/replay-compat/scripts/integration/linux-01-desktop-smoke.v0.11.8.ad delete mode 100644 test/replay-compat/scripts/integration/linux-01-desktop-smoke.v0.15.0.ad delete mode 100644 test/replay-compat/scripts/integration/macos-01-system-settings.v0.11.0.ad delete mode 100644 test/replay-compat/scripts/integration/macos-01-system-settings.v0.11.2.ad delete mode 100644 test/replay-compat/scripts/integration/macos-01-system-settings.v0.17.0.ad diff --git a/test/replay-compat/README.md b/test/replay-compat/README.md index b2f707dcc..8927b3611 100644 --- a/test/replay-compat/README.md +++ b/test/replay-compat/README.md @@ -27,12 +27,25 @@ enough to put in front of whoever recorded the script. ## What is in the corpus - `scripts/integration/` and `scripts/examples/` — mined from the git history of - `test/integration/replays` and `examples/test-app/replays`: one entry per distinct file content at - a release tag. + `test/integration/replays` and `examples/test-app/replays`. - `scripts/docs/` — surfaces the released grammar wrote that those two suites never exercised (`${VAR}` parameterization, reserved/ordering `env` refusals, `wait stable` and landmark waits, `target-v1` annotations, retired `gesture rotate` velocity and `gesture swipe` durationMs). +## What earns an entry + +This is a **parser**-compatibility corpus, not a device matrix. An entry has to be the only witness +of something: + +- a shipped syntactic form no other entry's bytes contain, or +- a distinct migration refusal (error code + hint) no other entry already provokes. + +So the same flow recorded on another platform, or re-recorded at an adjacent release with only +coordinates and labels changed, does **not** get an entry: it costs maintenance and adds no parser +leverage. Each entry's `note` says which form or refusal it is the sole witness of; if a new entry +cannot claim one, it does not belong here. Keep the corpus in the tens of entries, not a mirror of +the replay-fixture tree. + ## How "frozen" and "released" are enforced, not asserted Provenance lives in `manifest.ts`. `recordedBy` is the released tag whose grammar produced the @@ -64,9 +77,10 @@ surface — `git tag --contains ` decides, and unreleased shapes stay ou 1. Find the surface at a release tag (`git show :`, or the released grammar/docs that emitted it) and copy it verbatim into `scripts//..ad`. -2. Add a `manifest.ts` entry with `recordedBy`, `provenance`, `covers`, and the observed verdict. For - a mined entry the blob id is `git rev-parse :`; for a derived one it is - `shasum -a 256 `. Add the tag to `REPLAY_COMPAT_RELEASED_TAGS` if it is new. +2. Add a `manifest.ts` entry with `recordedBy`, `provenance`, `covers`, the observed verdict, and a + `note` naming the form or refusal it is the sole witness of. For a mined entry the blob id is + `git rev-parse :`; for a derived one it is `shasum -a 256 `. Add the tag to + `REPLAY_COMPAT_RELEASED_TAGS` if it is new. 3. Run `pnpm exec vitest run --project unit-core test/replay-compat` and `pnpm check:replay-compat`. Add an entry when a change retires, renames, or narrows a `.ad` form — the corpus is the record of diff --git a/test/replay-compat/corpus.test.ts b/test/replay-compat/corpus.test.ts index 4193287ee..fa5939b17 100644 --- a/test/replay-compat/corpus.test.ts +++ b/test/replay-compat/corpus.test.ts @@ -129,6 +129,17 @@ describe('replay-compat corpus', () => { } }); + // The corpus is a parser-compat suite, not a device matrix: every entry must + // say which form or refusal it is the sole witness of, and the count stays a + // reviewable set of deliberate entries rather than a mirror of the replay + // fixtures. Raising the ceiling is allowed; doing it silently is not. + test('every entry states why it exists, and the corpus stays small', () => { + for (const entry of REPLAY_COMPAT_CORPUS) { + expect(entry.note.trim(), entry.id).not.toBe(''); + } + expect(REPLAY_COMPAT_CORPUS.length).toBeLessThanOrEqual(30); + }); + test('the corpus covers every compat surface it is required to freeze', () => { const covered = new Set(REPLAY_COMPAT_CORPUS.flatMap((entry) => entry.covers)); expect([...covered].sort()).toEqual([...REQUIRED_COVERAGE].sort()); diff --git a/test/replay-compat/manifest.ts b/test/replay-compat/manifest.ts index 34b94343f..d59bc5b8a 100644 --- a/test/replay-compat/manifest.ts +++ b/test/replay-compat/manifest.ts @@ -1,7 +1,7 @@ /** - * The frozen `.ad` replay-compat corpus (#1417): one entry per historical script - * surface a RELEASED version wrote, paired with the verdict today's parser must - * return for it. + * The frozen `.ad` replay-compat corpus (#1417): one minimal witness per + * materially distinct script surface a RELEASED version wrote, paired with the + * verdict today's parser must return for it. * * The scripts under `scripts/` are FROZEN. A grammar change that flips a verdict * edits the verdict here — never the script — and says why in the same PR. See @@ -18,13 +18,9 @@ import type { AppErrorCode } from '../../src/kernel/errors.ts'; export const REPLAY_COMPAT_RELEASED_TAGS: readonly string[] = [ 'v0.11.0', 'v0.11.2', - 'v0.11.8', - 'v0.12.5', - 'v0.15.0', 'v0.15.1', 'v0.16.0', 'v0.16.8', - 'v0.17.0', 'v0.20.0', ]; @@ -66,14 +62,19 @@ export type ReplayCompatEntry = { provenance: ReplayCompatProvenance; covers: ReplayCompatCoverage[]; verdict: ReplayCompatVerdict; - /** Why this entry exists, when the path and verdict do not say it. */ - note?: string; + /** The shipped form or refusal this entry is the sole witness of. */ + note: string; }; /** * Mined from the git history of `test/integration/replays` and - * `examples/test-app/replays` (one entry per distinct content at a release tag), - * plus surfaces the released grammar wrote that those suites never exercised. + * `examples/test-app/replays`, plus surfaces the released grammar wrote that + * those suites never exercised. + * + * This is a parser-compatibility corpus, not a device matrix: an entry earns its + * place by being the only witness of a shipped syntactic form or of a distinct + * migration refusal. Repeating a form across platforms or adjacent releases adds + * maintenance weight and no parser leverage, so those samples stay out. * Git-history-only grammar states that never shipped are deliberately absent. */ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ @@ -88,18 +89,7 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ }, covers: ['context-header', 'quoting'], verdict: { kind: 'parses' }, - }, - { - id: 'integration/android-01-settings@v0.11.2', - file: 'scripts/integration/android-01-settings.v0.11.2.ad', - recordedBy: 'v0.11.2', - provenance: { - kind: 'mined', - path: 'test/integration/replays/android/01-settings.ad', - blob: '47b9ff049772143e1bd8b0cd2e64cb381a48ae2d', - }, - covers: ['context-header', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, + note: 'Sole shipped witness of `appstate`, `snapshot -i`, `is exists ""`, and bare `back`.', }, { id: 'integration/ios-device-01-physical-lifecycle@v0.11.0', @@ -112,126 +102,7 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, - }, - { - id: 'integration/ios-device-01-physical-lifecycle@v0.11.2', - file: 'scripts/integration/ios-device-01-physical-lifecycle.v0.11.2.ad', - recordedBy: 'v0.11.2', - provenance: { - kind: 'mined', - path: 'test/integration/replays/ios/device/01-physical-lifecycle.ad', - blob: '797de36dc0d5c5aa2287176de72e82a3ccb21bff', - }, - covers: ['context-header', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, - { - id: 'integration/ios-device-01-physical-lifecycle@v0.15.0', - file: 'scripts/integration/ios-device-01-physical-lifecycle.v0.15.0.ad', - recordedBy: 'v0.15.0', - provenance: { - kind: 'mined', - path: 'test/integration/replays/ios/device/01-physical-lifecycle.ad', - blob: '7a5ea2f64756b5400f1e987a32d78b058734461c', - }, - covers: ['context-header', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, - { - id: 'integration/ios-device-01-physical-lifecycle@v0.17.0', - file: 'scripts/integration/ios-device-01-physical-lifecycle.v0.17.0.ad', - recordedBy: 'v0.17.0', - provenance: { - kind: 'mined', - path: 'test/integration/replays/ios/device/01-physical-lifecycle.ad', - blob: 'fb6e1c0b90d331a5c7075f37f6690412d0c926f4', - }, - covers: ['context-header', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, - { - id: 'integration/ios-simulator-01-settings@v0.11.0', - file: 'scripts/integration/ios-simulator-01-settings.v0.11.0.ad', - recordedBy: 'v0.11.0', - provenance: { - kind: 'mined', - path: 'test/integration/replays/ios/simulator/01-settings.ad', - blob: '704f28b6d38dadf820a658320dca4c611ea9684d', - }, - covers: ['context-header', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, - { - id: 'integration/ios-simulator-01-settings@v0.11.2', - file: 'scripts/integration/ios-simulator-01-settings.v0.11.2.ad', - recordedBy: 'v0.11.2', - provenance: { - kind: 'mined', - path: 'test/integration/replays/ios/simulator/01-settings.ad', - blob: 'b9482558fcde1f424dfba55a5dd4dfaecddb7bb4', - }, - covers: ['context-header', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, - { - id: 'integration/ios-simulator-01-settings@v0.15.0', - file: 'scripts/integration/ios-simulator-01-settings.v0.15.0.ad', - recordedBy: 'v0.15.0', - provenance: { - kind: 'mined', - path: 'test/integration/replays/ios/simulator/01-settings.ad', - blob: 'f34cb81707b2a8022d466e0ac2b45fcd43f4dcc7', - }, - covers: ['context-header', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, - { - id: 'integration/ios-simulator-01-settings@v0.17.0', - file: 'scripts/integration/ios-simulator-01-settings.v0.17.0.ad', - recordedBy: 'v0.17.0', - provenance: { - kind: 'mined', - path: 'test/integration/replays/ios/simulator/01-settings.ad', - blob: '418401db198bdb8b96ed80a25d1474effa693d35', - }, - covers: ['context-header', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, - { - id: 'integration/macos-01-system-settings@v0.11.0', - file: 'scripts/integration/macos-01-system-settings.v0.11.0.ad', - recordedBy: 'v0.11.0', - provenance: { - kind: 'mined', - path: 'test/integration/replays/macos/01-system-settings.ad', - blob: 'e3f7f39b53d9a56a14a233129b54f9f110640836', - }, - covers: ['context-header', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, - { - id: 'integration/macos-01-system-settings@v0.11.2', - file: 'scripts/integration/macos-01-system-settings.v0.11.2.ad', - recordedBy: 'v0.11.2', - provenance: { - kind: 'mined', - path: 'test/integration/replays/macos/01-system-settings.ad', - blob: '7df54a82235767e42355b204406421d02bae122b', - }, - covers: ['context-header', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, - { - id: 'integration/macos-01-system-settings@v0.17.0', - file: 'scripts/integration/macos-01-system-settings.v0.17.0.ad', - recordedBy: 'v0.17.0', - provenance: { - kind: 'mined', - path: 'test/integration/replays/macos/01-system-settings.ad', - blob: 'd43f790938cbde47f99e135737589002f014193f', - }, - covers: ['context-header', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, + note: 'Sole shipped witness of an unquoted `open ` target and an argument-less `snapshot`.', }, { id: 'integration/android-02-deep-navigation@v0.11.2', @@ -244,30 +115,7 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ }, covers: ['context-header', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, - }, - { - id: 'integration/android-02-deep-navigation@v0.12.5', - file: 'scripts/integration/android-02-deep-navigation.v0.12.5.ad', - recordedBy: 'v0.12.5', - provenance: { - kind: 'mined', - path: 'test/integration/replays/android/02-deep-navigation.ad', - blob: '2072595e8e03f76662af0f7dba34d8d7f6a0333f', - }, - covers: ['context-header', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, - { - id: 'integration/android-03-scroll-discovery@v0.11.2', - file: 'scripts/integration/android-03-scroll-discovery.v0.11.2.ad', - recordedBy: 'v0.11.2', - provenance: { - kind: 'mined', - path: 'test/integration/replays/android/03-scroll-discovery.ad', - blob: 'f37427fae15ba0d905bad924585ea898d7ff5985', - }, - covers: ['context-header', 'wait-landmark'], - verdict: { kind: 'parses' }, + note: 'Sole shipped witness of `screenshot `, a bare `click @ref`, and `find text "…" exists`.', }, { id: 'integration/android-04-text-input-keyboard@v0.11.2', @@ -280,6 +128,7 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ }, covers: ['context-header', 'wait-landmark'], verdict: { kind: 'parses' }, + note: 'Sole shipped witness of `type ""`.', }, { id: 'integration/android-05-app-lifecycle@v0.11.2', @@ -292,6 +141,7 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ }, covers: ['context-header', 'wait-landmark'], verdict: { kind: 'parses' }, + note: 'Sole shipped witness of `home`.', }, { id: 'integration/android-06-swipe-gestures@v0.11.2', @@ -310,170 +160,6 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ }, note: '#1393 retired the trailing swipe durationMs; a v0.11.2 Android recording carries it and must get the migration, not a silent default-duration swipe.', }, - { - id: 'integration/ios-simulator-02-deep-navigation@v0.11.2', - file: 'scripts/integration/ios-simulator-02-deep-navigation.v0.11.2.ad', - recordedBy: 'v0.11.2', - provenance: { - kind: 'mined', - path: 'test/integration/replays/ios/simulator/02-deep-navigation.ad', - blob: '977f082d918c5ec3999d6e71904331465c1ffa85', - }, - covers: ['context-header', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, - { - id: 'integration/ios-simulator-02-deep-navigation@v0.15.0', - file: 'scripts/integration/ios-simulator-02-deep-navigation.v0.15.0.ad', - recordedBy: 'v0.15.0', - provenance: { - kind: 'mined', - path: 'test/integration/replays/ios/simulator/02-deep-navigation.ad', - blob: 'edc97d2854c813ed4230fe40533628d8ead897bc', - }, - covers: ['context-header', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, - { - id: 'integration/ios-simulator-02-deep-navigation@v0.17.0', - file: 'scripts/integration/ios-simulator-02-deep-navigation.v0.17.0.ad', - recordedBy: 'v0.17.0', - provenance: { - kind: 'mined', - path: 'test/integration/replays/ios/simulator/02-deep-navigation.ad', - blob: '86c27387f4d91e6bfcc5c2a424de41159da68cf5', - }, - covers: ['context-header', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, - { - id: 'integration/ios-simulator-03-scroll-discovery@v0.11.2', - file: 'scripts/integration/ios-simulator-03-scroll-discovery.v0.11.2.ad', - recordedBy: 'v0.11.2', - provenance: { - kind: 'mined', - path: 'test/integration/replays/ios/simulator/03-scroll-discovery.ad', - blob: '898bf8423074fc3e8fb05c1dfef8f1c674eb0529', - }, - covers: ['context-header', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, - { - id: 'integration/ios-simulator-03-scroll-discovery@v0.15.0', - file: 'scripts/integration/ios-simulator-03-scroll-discovery.v0.15.0.ad', - recordedBy: 'v0.15.0', - provenance: { - kind: 'mined', - path: 'test/integration/replays/ios/simulator/03-scroll-discovery.ad', - blob: '2646c29e9b2323d4652a72c74ceeb1c88f49ce75', - }, - covers: ['context-header', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, - { - id: 'integration/ios-simulator-04-text-input-keyboard@v0.11.2', - file: 'scripts/integration/ios-simulator-04-text-input-keyboard.v0.11.2.ad', - recordedBy: 'v0.11.2', - provenance: { - kind: 'mined', - path: 'test/integration/replays/ios/simulator/04-text-input-keyboard.ad', - blob: '0a8be87e3e2caacdf6b35820bb16cc3d492c8fea', - }, - covers: ['context-header', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, - { - id: 'integration/ios-simulator-04-text-input-keyboard@v0.15.0', - file: 'scripts/integration/ios-simulator-04-text-input-keyboard.v0.15.0.ad', - recordedBy: 'v0.15.0', - provenance: { - kind: 'mined', - path: 'test/integration/replays/ios/simulator/04-text-input-keyboard.ad', - blob: 'a9f04c26d8ff91ae705bfa2b4f0c61c0b35b8dd3', - }, - covers: ['context-header', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, - { - id: 'integration/ios-simulator-05-app-lifecycle@v0.11.2', - file: 'scripts/integration/ios-simulator-05-app-lifecycle.v0.11.2.ad', - recordedBy: 'v0.11.2', - provenance: { - kind: 'mined', - path: 'test/integration/replays/ios/simulator/05-app-lifecycle.ad', - blob: '75cb3ca578c93146731676fb4d0bbf8d71b4e6a9', - }, - covers: ['context-header', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, - { - id: 'integration/ios-simulator-05-app-lifecycle@v0.15.0', - file: 'scripts/integration/ios-simulator-05-app-lifecycle.v0.15.0.ad', - recordedBy: 'v0.15.0', - provenance: { - kind: 'mined', - path: 'test/integration/replays/ios/simulator/05-app-lifecycle.ad', - blob: '32d2a15f5006169e8578cde608703dd5e630f577', - }, - covers: ['context-header', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, - { - id: 'integration/ios-simulator-06-swipe-gestures@v0.11.2', - file: 'scripts/integration/ios-simulator-06-swipe-gestures.v0.11.2.ad', - recordedBy: 'v0.11.2', - provenance: { - kind: 'mined', - path: 'test/integration/replays/ios/simulator/06-swipe-gestures.ad', - blob: 'ca067860a93fbd1ca976a02401c3d86993cb2028', - }, - covers: ['context-header', 'retired-gesture', 'wait-landmark'], - verdict: { - kind: 'fails', - code: 'INVALID_ARGS', - hint: 'swipe accepts 4 arguments: x1 y1 x2 y2 (line 6). The trailing durationMs positional was removed: use "gesture pan 197 650 0 -350 300" for the same timed drag, or "swipe 197 650 197 300" for a default-duration swipe.', - }, - }, - { - id: 'integration/ios-simulator-06-swipe-gestures@v0.15.0', - file: 'scripts/integration/ios-simulator-06-swipe-gestures.v0.15.0.ad', - recordedBy: 'v0.15.0', - provenance: { - kind: 'mined', - path: 'test/integration/replays/ios/simulator/06-swipe-gestures.ad', - blob: '8f892f1c7af020680a4f921c60cbc4e30ebb021f', - }, - covers: ['context-header', 'retired-gesture', 'wait-landmark'], - verdict: { - kind: 'fails', - code: 'INVALID_ARGS', - hint: 'swipe accepts 4 arguments: x1 y1 x2 y2 (line 6). The trailing durationMs positional was removed: use "gesture pan 197 650 0 -350 300" for the same timed drag, or "swipe 197 650 197 300" for a default-duration swipe.', - }, - }, - { - id: 'integration/linux-01-desktop-smoke@v0.11.8', - file: 'scripts/integration/linux-01-desktop-smoke.v0.11.8.ad', - recordedBy: 'v0.11.8', - provenance: { - kind: 'mined', - path: 'test/integration/replays/linux/01-desktop-smoke.ad', - blob: 'c707cfdb2f172437cfb094d7e9eb7bb98213b450', - }, - covers: ['context-header', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, - { - id: 'integration/linux-01-desktop-smoke@v0.15.0', - file: 'scripts/integration/linux-01-desktop-smoke.v0.15.0.ad', - recordedBy: 'v0.15.0', - provenance: { - kind: 'mined', - path: 'test/integration/replays/linux/01-desktop-smoke.ad', - blob: '1d88b91a84ff6a59ff84c5a5ab79c3c75620543c', - }, - covers: ['context-header', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, { id: 'examples/checkout-form-android@v0.15.1', file: 'scripts/examples/checkout-form-android.v0.15.1.ad', @@ -485,6 +171,7 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ }, covers: ['context-header', 'env-vars', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, + note: 'Sole shipped witness of `open … --launch-url`, `keyboard dismiss`, `scroll `, and `fill id=… "…"` with `${VAR}` header values.', }, { id: 'examples/checkout-form-android@v0.16.8', @@ -497,30 +184,7 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ }, covers: ['context-header', 'env-vars', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, - }, - { - id: 'examples/checkout-form@v0.15.1', - file: 'scripts/examples/checkout-form.v0.15.1.ad', - recordedBy: 'v0.15.1', - provenance: { - kind: 'mined', - path: 'examples/test-app/replays/checkout-form.ad', - blob: '8bd23afbd93fe7ec8845882faf346509e9e00d3f', - }, - covers: ['context-header', 'env-vars', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, - }, - { - id: 'examples/checkout-form@v0.16.8', - file: 'scripts/examples/checkout-form.v0.16.8.ad', - recordedBy: 'v0.16.8', - provenance: { - kind: 'mined', - path: 'examples/test-app/replays/checkout-form.ad', - blob: '4c89212dfa191309377a538a16b95bafa8d3d754', - }, - covers: ['context-header', 'env-vars', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, + note: 'Sole shipped witness of a `react-native ` step inside a recording.', }, { id: 'examples/gesture-lab@v0.16.0', @@ -537,35 +201,7 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ code: 'INVALID_ARGS', hint: 'gesture fling accepts at most 4 arguments: direction x y [distance] (line 15). The trailing durationMs positional was removed: use "gesture fling up 195 443 80", or gesture pan for timed movement.', }, - note: '#1393 retired the trailing fling durationMs; the shipped gesture-lab example wrote it until v0.20.0.', - }, - { - id: 'examples/gesture-lab@v0.16.8', - file: 'scripts/examples/gesture-lab.v0.16.8.ad', - recordedBy: 'v0.16.8', - provenance: { - kind: 'mined', - path: 'examples/test-app/replays/gesture-lab.ad', - blob: '2d6b3aee582d6a04b02e6d51a9d8e6769ed65816', - }, - covers: ['context-header', 'env-vars', 'quoting', 'retired-gesture', 'wait-landmark'], - verdict: { - kind: 'fails', - code: 'INVALID_ARGS', - hint: 'gesture fling accepts at most 4 arguments: direction x y [distance] (line 16). The trailing durationMs positional was removed: use "gesture fling up 195 443 80", or gesture pan for timed movement.', - }, - }, - { - id: 'examples/gesture-lab@v0.20.0', - file: 'scripts/examples/gesture-lab.v0.20.0.ad', - recordedBy: 'v0.20.0', - provenance: { - kind: 'mined', - path: 'examples/test-app/replays/gesture-lab.ad', - blob: 'ac3946e335a97237e5d53fbb3fec8472b955b1f6', - }, - covers: ['context-header', 'env-vars', 'quoting', 'wait-landmark'], - verdict: { kind: 'parses' }, + note: '#1393 retired the trailing fling durationMs; the shipped gesture-lab example wrote it until v0.20.0. This is the only entry for that refusal — the v0.16.8 and iOS copies of the same script repeat it.', }, { id: 'examples/gesture-lab-android@v0.20.0', @@ -578,6 +214,7 @@ export const REPLAY_COMPAT_CORPUS: ReplayCompatEntry[] = [ }, covers: ['context-header', 'env-vars', 'quoting', 'wait-landmark'], verdict: { kind: 'parses' }, + note: 'The post-#1393 gesture positionals as they ship today: pan (incl. `--pointer-count`), fling, pinch, rotate, transform.', }, { id: 'docs/vars-parameterized@v0.15.1', diff --git a/test/replay-compat/scripts/examples/checkout-form.v0.15.1.ad b/test/replay-compat/scripts/examples/checkout-form.v0.15.1.ad deleted file mode 100644 index 8bd23afbd..000000000 --- a/test/replay-compat/scripts/examples/checkout-form.v0.15.1.ad +++ /dev/null @@ -1,21 +0,0 @@ -context platform=ios timeout=60000 -env APP_TARGET="Agent Device Tester" -env APP_URL="" -open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" -wait "label=\"Form\"" 30000 -click "label=\"Form\"" -wait "Checkout form" 5000 -fill id="field-name" "Ada Lovelace" -fill id="field-email" "ada@example.com" -keyboard dismiss -wait "Checkout form" 5000 -scroll down 0.6 -click id="shipping-pickup" -click id="payment-cash" -wait "Delivery choices" 5000 -scroll down 0.7 -click id="checkbox-agree" -click id="submit-order" -wait "Order summary" 5000 -wait "Ada Lovelace chose pickup with cash payment." 5000 -close diff --git a/test/replay-compat/scripts/examples/checkout-form.v0.16.8.ad b/test/replay-compat/scripts/examples/checkout-form.v0.16.8.ad deleted file mode 100644 index 4c89212df..000000000 --- a/test/replay-compat/scripts/examples/checkout-form.v0.16.8.ad +++ /dev/null @@ -1,22 +0,0 @@ -context platform=ios timeout=60000 -env APP_TARGET="Agent Device Tester" -env APP_URL="" -open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" -react-native dismiss-overlay -wait "label=\"Form\"" 30000 -click "label=\"Form\"" -wait "Checkout form" 5000 -fill id="field-name" "Ada Lovelace" -fill id="field-email" "ada@example.com" -keyboard dismiss -wait "Checkout form" 5000 -scroll down 0.6 -click id="shipping-pickup" -click id="payment-cash" -wait "Delivery choices" 5000 -scroll down 0.7 -click id="checkbox-agree" -click id="submit-order" -wait "Order summary" 5000 -wait "Ada Lovelace chose pickup with cash payment." 5000 -close diff --git a/test/replay-compat/scripts/examples/gesture-lab.v0.16.8.ad b/test/replay-compat/scripts/examples/gesture-lab.v0.16.8.ad deleted file mode 100644 index 2d6b3aee5..000000000 --- a/test/replay-compat/scripts/examples/gesture-lab.v0.16.8.ad +++ /dev/null @@ -1,40 +0,0 @@ -context platform=ios timeout=60000 - -env APP_TARGET="Agent Device Tester" -env APP_URL="" - -open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" -react-native dismiss-overlay -wait "Gesture lab" 30000 - -gesture fling left 195 443 180 -wait "fling 1" 5000 - -gesture fling right 195 443 180 -wait "fling 2" 5000 - -gesture fling up 195 443 80 80 -wait "fling 3" 5000 - -gesture fling down 195 443 80 80 -wait "fling 4" 5000 - -gesture pan 195 443 -80 0 -wait "x -" 5000 - -gesture pan 195 443 160 0 -wait "x 72" 5000 - -gesture pan 195 443 0 -80 -wait "y -" 5000 - -gesture pan 195 443 0 160 -wait "y 56" 5000 - -gesture pinch 1.25 195 443 -wait "pinch changed yes" 5000 - -gesture rotate 35 195 443 -wait "rotate changed yes" 5000 - -close diff --git a/test/replay-compat/scripts/examples/gesture-lab.v0.20.0.ad b/test/replay-compat/scripts/examples/gesture-lab.v0.20.0.ad deleted file mode 100644 index ac3946e33..000000000 --- a/test/replay-compat/scripts/examples/gesture-lab.v0.20.0.ad +++ /dev/null @@ -1,47 +0,0 @@ -context platform=ios kind=simulator timeout=180000 - -env APP_TARGET="Agent Device Tester" -env APP_URL="" - -open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" -wait "Gesture lab" 30000 -wait "gesture canary ready" 5000 -wait "two-pointer pan activations 0" 5000 - -gesture pan 110 443 48 0 500 -wait "two-pointer pan activations 0" 5000 - -gesture pan 110 443 48 0 500 --pointer-count 2 -wait "two-pointer pan activations 1" 5000 - -gesture fling left 280 443 120 -wait "fling 1" 5000 - -gesture fling right 280 443 120 -wait "fling 2" 5000 - -open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" -wait "Gesture lab" 30000 -wait "gesture canary ready" 5000 -wait "pan changed no, pinch changed no, rotate changed no" 30000 - -gesture pinch 1.5 280 443 -wait "pan changed no, pinch changed yes, rotate changed no" 5000 - -open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" -wait "Gesture lab" 30000 -wait "gesture canary ready" 5000 -wait "pan changed no, pinch changed no, rotate changed no" 30000 - -gesture rotate 35 280 443 -wait "pan changed no, pinch changed no, rotate changed yes" 5000 - -open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" -wait "Gesture lab" 30000 -wait "gesture canary ready" 5000 -wait "pan changed no, pinch changed no, rotate changed no" 30000 - -gesture transform 280 443 24 -24 1.35 30 300 -wait "pan changed yes, pinch changed yes, rotate changed yes" 5000 - -close diff --git a/test/replay-compat/scripts/integration/android-01-settings.v0.11.2.ad b/test/replay-compat/scripts/integration/android-01-settings.v0.11.2.ad deleted file mode 100644 index 47b9ff049..000000000 --- a/test/replay-compat/scripts/integration/android-01-settings.v0.11.2.ad +++ /dev/null @@ -1,15 +0,0 @@ -# Dogfood Android Settings flow through the replay suite runner. -context platform=android -open settings --relaunch -appstate -snapshot -i -is exists "label=Wait || label=\"Close app\" || label=Apps || label=\"Apps & notifications\" || label=\"Network & internet\" || label=\"Network and internet\" || label=\"Connected devices\" || label=Display || label=Battery || label=Notifications || label=Security || label=Privacy" -click "label=Wait || label=\"Close app\" || label=Apps || label=\"Apps & notifications\" || label=\"Network & internet\" || label=\"Network and internet\" || label=\"Connected devices\" || label=Display || label=Battery || label=Notifications || label=Security || label=Privacy" -back -wait 1000 -snapshot -i -is exists "label=Apps || label=\"Apps & notifications\" || label=\"Network & internet\" || label=\"Network and internet\" || label=\"Connected devices\" || label=Display || label=Battery || label=Notifications || label=Security || label=Privacy" -click "label=Apps || label=\"Apps & notifications\" || label=\"Network & internet\" || label=\"Network and internet\" || label=\"Connected devices\" || label=Display || label=Battery || label=Notifications || label=Security || label=Privacy" -snapshot -i -appstate -back diff --git a/test/replay-compat/scripts/integration/android-02-deep-navigation.v0.12.5.ad b/test/replay-compat/scripts/integration/android-02-deep-navigation.v0.12.5.ad deleted file mode 100644 index 2072595e8..000000000 --- a/test/replay-compat/scripts/integration/android-02-deep-navigation.v0.12.5.ad +++ /dev/null @@ -1,18 +0,0 @@ -# Deep navigation. -context platform=android - -open settings --relaunch -appstate -snapshot -i -screenshot "./test/screenshots/replays/android-deep-nav-root.png" - -click "label=Notifications" -wait 1000 -snapshot -i -find text "Notification history" exists -screenshot "./test/screenshots/replays/android-deep-nav-detail.png" - -back -wait 1000 -snapshot -i -find text "Search" exists diff --git a/test/replay-compat/scripts/integration/android-03-scroll-discovery.v0.11.2.ad b/test/replay-compat/scripts/integration/android-03-scroll-discovery.v0.11.2.ad deleted file mode 100644 index f37427fae..000000000 --- a/test/replay-compat/scripts/integration/android-03-scroll-discovery.v0.11.2.ad +++ /dev/null @@ -1,15 +0,0 @@ -# Scroll discovery. -context platform=android - -open settings --relaunch -snapshot -i -screenshot "./test/screenshots/replays/android-scroll-top.png" - -scroll down 3 -wait 500 -snapshot -i -screenshot "./test/screenshots/replays/android-scroll-found.png" - -scroll up 3 -wait 500 -snapshot -i diff --git a/test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.11.2.ad b/test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.11.2.ad deleted file mode 100644 index 797de36dc..000000000 --- a/test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.11.2.ad +++ /dev/null @@ -1,8 +0,0 @@ -# Dogfood iOS physical device lifecycle through the replay suite runner. -context platform=ios -open com.apple.Preferences --relaunch -snapshot -click "role=cell label=General" -wait "label=About || label=\"Software Update\"" 5000 -back -is exists "role=cell label=General" diff --git a/test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.15.0.ad b/test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.15.0.ad deleted file mode 100644 index 7a5ea2f64..000000000 --- a/test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.15.0.ad +++ /dev/null @@ -1,8 +0,0 @@ -# Dogfood iOS physical device lifecycle through the replay suite runner. -context platform=ios target=mobile -open com.apple.Preferences --relaunch -snapshot -click "role=cell label=General" -wait "label=About || label=\"Software Update\"" 5000 -back -is exists "role=cell label=General" diff --git a/test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.17.0.ad b/test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.17.0.ad deleted file mode 100644 index fb6e1c0b9..000000000 --- a/test/replay-compat/scripts/integration/ios-device-01-physical-lifecycle.v0.17.0.ad +++ /dev/null @@ -1,8 +0,0 @@ -# Dogfood iOS physical device lifecycle through the replay suite runner. -context platform=ios target=mobile -open com.apple.Preferences --relaunch -snapshot -click "role=cell label=General || role=button label=General" -wait "label=About || label=\"Software Update\"" 5000 -back -is exists "role=cell label=General || role=button label=General" diff --git a/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.11.0.ad b/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.11.0.ad deleted file mode 100644 index 704f28b6d..000000000 --- a/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.11.0.ad +++ /dev/null @@ -1,12 +0,0 @@ -# Dogfood iOS Settings flow through the replay suite runner. -context platform=ios -open com.apple.Preferences -screenshot "./test/screenshots/replays/ios-settings.png" -snapshot -i -appstate -click "role=cell label=General" -wait "label=About || label=\"Software Update\" || text=\"Manage your overall setup and preferences\"" 5000 -snapshot -is exists "label=About || label=\"Software Update\" || text=\"Manage your overall setup and preferences\"" -find text "Software Update" exists -back diff --git a/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.11.2.ad b/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.11.2.ad deleted file mode 100644 index b9482558f..000000000 --- a/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.11.2.ad +++ /dev/null @@ -1,12 +0,0 @@ -# Dogfood iOS Settings flow through the replay suite runner. -context platform=ios -open com.apple.Preferences --relaunch -screenshot "./test/screenshots/replays/ios-settings.png" -snapshot -i -appstate -click "role=cell label=General" -wait "label=About || label=\"Software Update\" || text=\"Manage your overall setup and preferences\"" 5000 -snapshot -is exists "label=About || label=\"Software Update\" || text=\"Manage your overall setup and preferences\"" -find text "Software Update" exists -back diff --git a/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.15.0.ad b/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.15.0.ad deleted file mode 100644 index f34cb8170..000000000 --- a/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.15.0.ad +++ /dev/null @@ -1,12 +0,0 @@ -# Dogfood iOS Settings flow through the replay suite runner. -context platform=ios target=mobile -open com.apple.Preferences --relaunch -screenshot "./test/screenshots/replays/ios-settings.png" -snapshot -i -appstate -click "role=cell label=General" -wait "label=About || label=\"Software Update\" || text=\"Manage your overall setup and preferences\"" 5000 -snapshot -is exists "label=About || label=\"Software Update\" || text=\"Manage your overall setup and preferences\"" -find text "Software Update" exists -back diff --git a/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.17.0.ad b/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.17.0.ad deleted file mode 100644 index 418401db1..000000000 --- a/test/replay-compat/scripts/integration/ios-simulator-01-settings.v0.17.0.ad +++ /dev/null @@ -1,12 +0,0 @@ -# Dogfood iOS Settings flow through the replay suite runner. -context platform=ios target=mobile -open com.apple.Preferences --relaunch -screenshot "./test/screenshots/replays/ios-settings.png" -snapshot -i -appstate -click "role=cell label=General || role=button label=General" -wait "label=About || label=\"Software Update\" || text=\"Manage your overall setup and preferences\"" 5000 -snapshot -is exists "label=About || label=\"Software Update\" || text=\"Manage your overall setup and preferences\"" -find text "Software Update" exists -back diff --git a/test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.11.2.ad b/test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.11.2.ad deleted file mode 100644 index 977f082d9..000000000 --- a/test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.11.2.ad +++ /dev/null @@ -1,11 +0,0 @@ -# Deep navigation. -context platform=ios - -open com.apple.Preferences --relaunch -click "role=cell label=General" -wait 1000 -find text "About" exists - -back -wait 1000 -find text "General" exists diff --git a/test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.15.0.ad b/test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.15.0.ad deleted file mode 100644 index edc97d285..000000000 --- a/test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.15.0.ad +++ /dev/null @@ -1,11 +0,0 @@ -# Deep navigation. -context platform=ios target=mobile - -open com.apple.Preferences --relaunch -click "role=cell label=General" -wait 1000 -find text "About" exists - -back -wait 1000 -find text "General" exists diff --git a/test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.17.0.ad b/test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.17.0.ad deleted file mode 100644 index 86c27387f..000000000 --- a/test/replay-compat/scripts/integration/ios-simulator-02-deep-navigation.v0.17.0.ad +++ /dev/null @@ -1,11 +0,0 @@ -# Deep navigation. -context platform=ios target=mobile - -open com.apple.Preferences --relaunch -click "role=cell label=General || role=button label=General" -wait 1000 -find text "About" exists - -back -wait 1000 -find text "General" exists diff --git a/test/replay-compat/scripts/integration/ios-simulator-03-scroll-discovery.v0.11.2.ad b/test/replay-compat/scripts/integration/ios-simulator-03-scroll-discovery.v0.11.2.ad deleted file mode 100644 index 898bf8423..000000000 --- a/test/replay-compat/scripts/integration/ios-simulator-03-scroll-discovery.v0.11.2.ad +++ /dev/null @@ -1,11 +0,0 @@ -# Scroll discovery. -context platform=ios - -open com.apple.Preferences --relaunch -scroll down 3 -wait 500 -find text "Accessibility" exists - -scroll up 3 -wait 500 -find text "General" exists diff --git a/test/replay-compat/scripts/integration/ios-simulator-03-scroll-discovery.v0.15.0.ad b/test/replay-compat/scripts/integration/ios-simulator-03-scroll-discovery.v0.15.0.ad deleted file mode 100644 index 2646c29e9..000000000 --- a/test/replay-compat/scripts/integration/ios-simulator-03-scroll-discovery.v0.15.0.ad +++ /dev/null @@ -1,11 +0,0 @@ -# Scroll discovery. -context platform=ios target=mobile - -open com.apple.Preferences --relaunch -scroll down 3 -wait 500 -find text "Accessibility" exists - -scroll up 3 -wait 500 -find text "General" exists diff --git a/test/replay-compat/scripts/integration/ios-simulator-04-text-input-keyboard.v0.11.2.ad b/test/replay-compat/scripts/integration/ios-simulator-04-text-input-keyboard.v0.11.2.ad deleted file mode 100644 index 0a8be87e3..000000000 --- a/test/replay-compat/scripts/integration/ios-simulator-04-text-input-keyboard.v0.11.2.ad +++ /dev/null @@ -1,11 +0,0 @@ -# Text input. -context platform=ios - -open com.apple.Preferences --relaunch -click "role=searchField || label=Search || label=\"Search\"" -wait 1000 - -type "General" -wait 1000 - -find text "General" exists diff --git a/test/replay-compat/scripts/integration/ios-simulator-04-text-input-keyboard.v0.15.0.ad b/test/replay-compat/scripts/integration/ios-simulator-04-text-input-keyboard.v0.15.0.ad deleted file mode 100644 index a9f04c26d..000000000 --- a/test/replay-compat/scripts/integration/ios-simulator-04-text-input-keyboard.v0.15.0.ad +++ /dev/null @@ -1,11 +0,0 @@ -# Text input. -context platform=ios target=mobile - -open com.apple.Preferences --relaunch -click "role=searchField || label=Search || label=\"Search\"" -wait 1000 - -type "General" -wait 1000 - -find text "General" exists diff --git a/test/replay-compat/scripts/integration/ios-simulator-05-app-lifecycle.v0.11.2.ad b/test/replay-compat/scripts/integration/ios-simulator-05-app-lifecycle.v0.11.2.ad deleted file mode 100644 index 75cb3ca57..000000000 --- a/test/replay-compat/scripts/integration/ios-simulator-05-app-lifecycle.v0.11.2.ad +++ /dev/null @@ -1,13 +0,0 @@ -# App lifecycle. -context platform=ios - -open com.apple.Preferences --relaunch -appstate - -home -wait 2000 - -open com.apple.Preferences --relaunch -wait 1000 -appstate -find text "General" exists diff --git a/test/replay-compat/scripts/integration/ios-simulator-05-app-lifecycle.v0.15.0.ad b/test/replay-compat/scripts/integration/ios-simulator-05-app-lifecycle.v0.15.0.ad deleted file mode 100644 index 32d2a15f5..000000000 --- a/test/replay-compat/scripts/integration/ios-simulator-05-app-lifecycle.v0.15.0.ad +++ /dev/null @@ -1,13 +0,0 @@ -# App lifecycle. -context platform=ios target=mobile - -open com.apple.Preferences --relaunch -appstate - -home -wait 2000 - -open com.apple.Preferences --relaunch -wait 1000 -appstate -find text "General" exists diff --git a/test/replay-compat/scripts/integration/ios-simulator-06-swipe-gestures.v0.11.2.ad b/test/replay-compat/scripts/integration/ios-simulator-06-swipe-gestures.v0.11.2.ad deleted file mode 100644 index ca067860a..000000000 --- a/test/replay-compat/scripts/integration/ios-simulator-06-swipe-gestures.v0.11.2.ad +++ /dev/null @@ -1,16 +0,0 @@ -# Swipe gestures. -context platform=ios - -open com.apple.Preferences --relaunch - -swipe 197 650 197 300 300 -wait 500 - -swipe 197 600 197 350 300 -wait 500 - -snapshot -i - -swipe 197 300 197 650 300 -wait 500 -find text "General" exists diff --git a/test/replay-compat/scripts/integration/ios-simulator-06-swipe-gestures.v0.15.0.ad b/test/replay-compat/scripts/integration/ios-simulator-06-swipe-gestures.v0.15.0.ad deleted file mode 100644 index 8f892f1c7..000000000 --- a/test/replay-compat/scripts/integration/ios-simulator-06-swipe-gestures.v0.15.0.ad +++ /dev/null @@ -1,16 +0,0 @@ -# Swipe gestures. -context platform=ios target=mobile - -open com.apple.Preferences --relaunch - -swipe 197 650 197 300 300 -wait 500 - -swipe 197 600 197 350 300 -wait 500 - -snapshot -i - -swipe 197 300 197 650 300 -wait 500 -find text "General" exists diff --git a/test/replay-compat/scripts/integration/linux-01-desktop-smoke.v0.11.8.ad b/test/replay-compat/scripts/integration/linux-01-desktop-smoke.v0.11.8.ad deleted file mode 100644 index c707cfdb2..000000000 --- a/test/replay-compat/scripts/integration/linux-01-desktop-smoke.v0.11.8.ad +++ /dev/null @@ -1,12 +0,0 @@ -# Smoke test for Linux desktop automation on CI. -# Opens gnome-calculator, takes a snapshot, and validates the -# accessibility tree contains expected calculator UI elements. -context platform=linux -open gnome-calculator -wait 3000 -screenshot "./test/screenshots/replays/linux-calculator.png" -snapshot -# Verify calculator-specific elements: a window titled Calculator, -# or digit buttons that only a calculator app would expose. -is exists "label=Calculator || label=0 || label=1 || label=5" -snapshot -i diff --git a/test/replay-compat/scripts/integration/linux-01-desktop-smoke.v0.15.0.ad b/test/replay-compat/scripts/integration/linux-01-desktop-smoke.v0.15.0.ad deleted file mode 100644 index 1d88b91a8..000000000 --- a/test/replay-compat/scripts/integration/linux-01-desktop-smoke.v0.15.0.ad +++ /dev/null @@ -1,12 +0,0 @@ -# Smoke test for Linux desktop automation on CI. -# Opens gnome-calculator, takes a snapshot, and validates the -# accessibility tree contains expected calculator UI elements. -context platform=linux -open gnome-calculator -wait "appname=gnome-calculator || windowtitle=Calculator || label=Calculator || label=0 || label=1 || label=5" 15000 -screenshot "./test/screenshots/replays/linux-calculator.png" -snapshot -# Verify calculator-specific elements: a window titled Calculator, -# or digit buttons that only a calculator app would expose. -is exists "appname=gnome-calculator || windowtitle=Calculator || label=Calculator || label=0 || label=1 || label=5" -snapshot -i diff --git a/test/replay-compat/scripts/integration/macos-01-system-settings.v0.11.0.ad b/test/replay-compat/scripts/integration/macos-01-system-settings.v0.11.0.ad deleted file mode 100644 index e3f7f39b5..000000000 --- a/test/replay-compat/scripts/integration/macos-01-system-settings.v0.11.0.ad +++ /dev/null @@ -1,16 +0,0 @@ -# Dogfood macOS System Settings flow through the replay suite runner. -context platform=macos -open "System Settings" -screenshot "./test/screenshots/replays/macos-system-settings.png" -appstate -snapshot -i -click "role=cell label=General" -wait "role=button label=About" 5000 -snapshot -i -is exists "role=button label=About" -click "role=button label=About" -wait "label=\"System Report...\" || label=\"Serial number\" || label=Chip || label=Processor || label=macOS" 5000 -snapshot -i -back -wait "role=button label=About" 5000 -is exists "role=button label=About" diff --git a/test/replay-compat/scripts/integration/macos-01-system-settings.v0.11.2.ad b/test/replay-compat/scripts/integration/macos-01-system-settings.v0.11.2.ad deleted file mode 100644 index 7df54a822..000000000 --- a/test/replay-compat/scripts/integration/macos-01-system-settings.v0.11.2.ad +++ /dev/null @@ -1,16 +0,0 @@ -# Dogfood macOS System Settings flow through the replay suite runner. -context platform=macos -open "System Settings" --relaunch -screenshot "./test/screenshots/replays/macos-system-settings.png" -appstate -snapshot -i -click "role=cell label=General" -wait "role=button label=About" 5000 -snapshot -i -is exists "role=button label=About" -click "role=button label=About" -wait "label=\"System Report...\" || label=\"Serial number\" || label=Chip || label=Processor || label=macOS" 5000 -snapshot -i -back -wait "role=button label=About" 5000 -is exists "role=button label=About" diff --git a/test/replay-compat/scripts/integration/macos-01-system-settings.v0.17.0.ad b/test/replay-compat/scripts/integration/macos-01-system-settings.v0.17.0.ad deleted file mode 100644 index d43f79093..000000000 --- a/test/replay-compat/scripts/integration/macos-01-system-settings.v0.17.0.ad +++ /dev/null @@ -1,15 +0,0 @@ -# Dogfood macOS System Settings flow through the replay suite runner. -context platform=macos -open "System Settings" --relaunch -screenshot "./test/screenshots/replays/macos-system-settings.png" -appstate -snapshot -i -wait "role=button label=About" 5000 -snapshot -i -is exists "role=button label=About" -click "role=button label=About" -wait "label=\"System Report...\" || label=\"Serial number\" || label=Chip || label=Processor || label=macOS" 5000 -snapshot -i -back -wait "role=button label=About" 5000 -is exists "role=button label=About" From c21e862f8ce1b13ed2748e7e48dfbe9e735955a6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Pierzcha=C5=82a?= Date: Tue, 28 Jul 2026 06:02:53 +0000 Subject: [PATCH 6/8] test: address corpus review nits (typed coverage list, cap rationale, derived-citation note) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- test/replay-compat/README.md | 5 ++++- test/replay-compat/corpus.test.ts | 28 ++++++++++++++++++---------- 2 files changed, 22 insertions(+), 11 deletions(-) diff --git a/test/replay-compat/README.md b/test/replay-compat/README.md index 8927b3611..5db42e608 100644 --- a/test/replay-compat/README.md +++ b/test/replay-compat/README.md @@ -55,7 +55,10 @@ surface, and `provenance` pins the bytes: checked-in bytes and must reproduce that id, so a rewritten script cannot be made green by editing the manifest to match — the id is only obtainable from the released content. - `derived` entries (`scripts/docs/`) have no historical blob to point at, so their bytes are pinned - by SHA-256. + by SHA-256. Only the bytes are machine-checked: the `from` citation naming the released grammar or + doc that emitted the surface is **reviewer-verified**, so a derived entry has to be reviewed against + its cited source at that tag (`git show :`) before it lands. Never hand-transcribe a + hint or a form — copy it from the release. The kind is not a free choice: `provenance-rules.ts` fixes it by corpus area (`scripts/integration/` and `scripts/examples/` must stay `mined`, `scripts/docs/` is `derived`), so an edited script cannot diff --git a/test/replay-compat/corpus.test.ts b/test/replay-compat/corpus.test.ts index fa5939b17..c7f2bcf74 100644 --- a/test/replay-compat/corpus.test.ts +++ b/test/replay-compat/corpus.test.ts @@ -16,17 +16,21 @@ import { findProvenanceKindViolations, requiredProvenanceKind } from './provenan const CORPUS_DIR = fileURLToPath(new URL('.', import.meta.url)); /** - * Every historical surface the corpus exists to hold still. A coverage tag with - * no entry means the corpus stopped covering a compat surface #1417 named. + * Every historical surface the corpus exists to hold still. Keyed by the coverage + * union so the typechecker — not a hand-kept list — forces a new surface to be + * required here; a tag with no entry means the corpus stopped covering a compat + * surface #1417 named. */ -const REQUIRED_COVERAGE: ReplayCompatCoverage[] = [ - 'context-header', - 'env-vars', - 'quoting', - 'retired-gesture', - 'target-annotation', - 'wait-landmark', -]; +const REQUIRED_COVERAGE_KEYS: Record = { + 'context-header': true, + 'env-vars': true, + quoting: true, + 'retired-gesture': true, + 'target-annotation': true, + 'wait-landmark': true, +}; + +const REQUIRED_COVERAGE = Object.keys(REQUIRED_COVERAGE_KEYS) as ReplayCompatCoverage[]; function readCorpusScript(entry: ReplayCompatEntry): Buffer { return readFileSync(join(CORPUS_DIR, entry.file)); @@ -133,6 +137,10 @@ describe('replay-compat corpus', () => { // say which form or refusal it is the sole witness of, and the count stays a // reviewable set of deliberate entries rather than a mirror of the replay // fixtures. Raising the ceiling is allowed; doing it silently is not. + // + // 30 is headroom over the 22 witnesses the shipped grammar needs today: enough + // for the next few retirements without another pruning pass, small enough that + // a re-mirrored fixture tree (52 entries at first cut) cannot land quietly. test('every entry states why it exists, and the corpus stays small', () => { for (const entry of REPLAY_COMPAT_CORPUS) { expect(entry.note.trim(), entry.id).not.toBe(''); From 0c31102183208df3e691014a8a2fc755d61ddb92 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Pierzcha=C5=82a?= Date: Tue, 28 Jul 2026 06:27:23 +0000 Subject: [PATCH 7/8] =?UTF-8?q?docs:=20split=20corpus=20rule=20=E2=80=94?= =?UTF-8?q?=20form=20from=20the=20release,=20verdict=20from=20today's=20pa?= =?UTF-8?q?rser?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- test/replay-compat/README.md | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/test/replay-compat/README.md b/test/replay-compat/README.md index 5db42e608..c26a7bf31 100644 --- a/test/replay-compat/README.md +++ b/test/replay-compat/README.md @@ -57,8 +57,14 @@ surface, and `provenance` pins the bytes: - `derived` entries (`scripts/docs/`) have no historical blob to point at, so their bytes are pinned by SHA-256. Only the bytes are machine-checked: the `from` citation naming the released grammar or doc that emitted the surface is **reviewer-verified**, so a derived entry has to be reviewed against - its cited source at that tag (`git show :`) before it lands. Never hand-transcribe a - hint or a form — copy it from the release. + its cited source at that tag (`git show :`) before it lands. + +**Two sources, never invention.** The *form* comes from the cited release — the script bytes are what +that tag's grammar or docs emitted. The *verdict* comes from today's parser, because a migration +refusal for a retired form did not exist at the tag that emitted it (v0.16.8 wrote +`gesture rotate `; the hint telling you to drop the velocity is current +output). So copy the form from the release and paste the code/hint from an observed run — never +paraphrase either. The kind is not a free choice: `provenance-rules.ts` fixes it by corpus area (`scripts/integration/` and `scripts/examples/` must stay `mined`, `scripts/docs/` is `derived`), so an edited script cannot @@ -80,11 +86,15 @@ surface — `git tag --contains ` decides, and unreleased shapes stay ou 1. Find the surface at a release tag (`git show :`, or the released grammar/docs that emitted it) and copy it verbatim into `scripts//..ad`. -2. Add a `manifest.ts` entry with `recordedBy`, `provenance`, `covers`, the observed verdict, and a - `note` naming the form or refusal it is the sole witness of. For a mined entry the blob id is - `git rev-parse :`; for a derived one it is `shasum -a 256 `. Add the tag to - `REPLAY_COMPAT_RELEASED_TAGS` if it is new. -3. Run `pnpm exec vitest run --project unit-core test/replay-compat` and `pnpm check:replay-compat`. +2. Add a `manifest.ts` entry with `recordedBy`, `provenance`, `covers`, and a `note` naming the form + or refusal it is the sole witness of. For a mined entry the blob id is `git rev-parse :`; + for a derived one it is `shasum -a 256 `. Add the tag to `REPLAY_COMPAT_RELEASED_TAGS` if it + is new. +3. Take the `verdict` from what the current parser does with those bytes: start from + `verdict: { kind: 'parses' }`, run the suite, and if it refuses, paste the reported code and hint + substring verbatim. Do not transcribe a hint from a release, an issue, or memory — the refusal is + produced by today's parser, not by the tag that emitted the form. +4. Run `pnpm exec vitest run --project unit-core test/replay-compat` and `pnpm check:replay-compat`. Add an entry when a change retires, renames, or narrows a `.ad` form — the corpus is the record of what that costs someone with a saved recording. From 9bc74787f794d1f8b9ddc9564dc052d1f0d1c350 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Pierzcha=C5=82a?= Date: Tue, 28 Jul 2026 06:27:55 +0000 Subject: [PATCH 8/8] docs: format corpus README emphasis markers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- test/replay-compat/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/test/replay-compat/README.md b/test/replay-compat/README.md index c26a7bf31..ba7f5b971 100644 --- a/test/replay-compat/README.md +++ b/test/replay-compat/README.md @@ -59,8 +59,8 @@ surface, and `provenance` pins the bytes: doc that emitted the surface is **reviewer-verified**, so a derived entry has to be reviewed against its cited source at that tag (`git show :`) before it lands. -**Two sources, never invention.** The *form* comes from the cited release — the script bytes are what -that tag's grammar or docs emitted. The *verdict* comes from today's parser, because a migration +**Two sources, never invention.** The _form_ comes from the cited release — the script bytes are what +that tag's grammar or docs emitted. The _verdict_ comes from today's parser, because a migration refusal for a retired form did not exist at the tag that emitted it (v0.16.8 wrote `gesture rotate `; the hint telling you to drop the velocity is current output). So copy the form from the release and paste the code/hint from an observed run — never