test: frozen replay-compat corpus with expected verdicts (#1417)

[devin-ai-integration]

Summary

Closes #1417 (umbrella #1412 Track A). A frozen .ad corpus of scripts released versions wrote, each paired with the verdict today's parser owes it, gated in unit-core.

test/replay-compat/manifest.ts is the whole contract:

{ id: 'integration/android-06-swipe-gestures@v0.11.2',
  file: 'scripts/integration/android-06-swipe-gestures.v0.11.2.ad',
  recordedBy: 'v0.11.2',                                   // must be in REPLAY_COMPAT_RELEASED_TAGS
  provenance: { kind: 'mined', path: 'test/integration/replays/android/06-swipe-gestures.ad',
                blob: '686bace…' },                        // git object id of the historical blob
  covers: ['context-header', 'retired-gesture', 'wait-landmark'],
  verdict: { kind: 'fails', code: 'INVALID_ARGS', hint: 'swipe accepts 4 arguments…' },
  note: '#1393 retired the trailing swipe durationMs…' }

corpus.test.ts asserts each verdict through production parseReplayInput(script, undefined) — the same composition, in the same action-before-metadata order, that replay/test hit — so a multi-fault script reports the code/hint replay really surfaces.

Freezing is mechanical, not asserted in prose:

• mined bytes must hash to the pinned git blob id (sha1("blob <len>\0" + bytes)), an id obtainable only from the released content, so "edit the script until the parser agrees" cannot pass as a corpus update; derived bytes (scripts/docs/, forms the release's grammar wrote that no shipped .ad carries) are SHA-256-pinned.
• provenance-rules.ts fixes the kind by corpus area (integration/examples → mined, docs → derived, unknown area throws), closing the loophole where an edited mined script is relabelled derived, re-pinned by digest, and skipped by the history verifier. Negative test included.
• pnpm check:replay-compat re-derives every mined id via git rev-parse <tag>:<path> and validates every cited tag against git tag --list. It needs full history, so it runs in a dedicated Replay-Compat Provenance job (fetch-depth: 0) and is selected by pnpm check:affected for any test/replay-compat/ change.

Coverage spans the surfaces #1417 named: pre-removal gesture forms (#1393 swipe/fling/rotate/gesture swipe durationMs), ${VAR}/quoting/--record-as projections, wait + landmark forms (#1349), target-v1 annotations, and context-header variants including the fail-fast on a conflicting key.

Pruned per review. This is a parser-compat suite, not a device matrix, so the initial 52-entry dataset is down to 22: an entry must be the sole witness of a shipped syntactic form or of a distinct migration refusal. Platform twins and adjacent-release re-recordings that differ only in coordinates/labels are gone (all iOS-simulator/macOS/linux mirrors of Android flows, the iOS copies of checkout-form/gesture-lab, and duplicate fling durationMs witnesses). note is now a required field naming what each entry witnesses, README.md documents the selection rule, and the test caps the corpus at 30 so regrowth has to be deliberate.

Gate cost: 51 tests, ~30ms in unit-core.

Link to Devin session: https://app.devin.ai/sessions/3340a872ac584fea82e03800f512c215
Requested by: @thymikee

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

Size Report

Metric	Base	Current	Diff
JS raw	1.87 MB	1.87 MB	0 B
JS gzip	598.6 kB	598.6 kB	0 B
npm tarball	714.1 kB	714.1 kB	+19 B
npm unpacked	2.50 MB	2.50 MB	+104 B

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	30.5 ms	30.1 ms	-0.4 ms
CLI --help	61.2 ms	61.6 ms	+0.4 ms

Top changed chunks: no changes in the largest emitted chunks.

[thymikee]

Two blocking guardrail gaps remain. (1) The “frozen, released” claim is not mechanically protected: corpus.test.ts only regex-checks recordedBy and repeats it in free-form source/file strings. It does not prove the tag is released or that the checked-in script still matches its historical source, so a future PR could rewrite both the frozen script and metadata while the gate stays green. Pin a released-tag/source-blob identity or digest per entry and verify the current corpus bytes against it. (2) The test reconstructs the parse boundary in the wrong order: it calls readReplayScriptMetadata before parseReplayScriptDetailed, while production parseReplayInput composes them in the reverse order. Multi-fault scripts can therefore assert a different code/hint than replay actually surfaces, and changes to the real composition boundary are unguarded. Exercise parseReplayInput(script, undefined) instead. Exact-head Coverage is also red on the inherited ios-system-ui benchmark gap, so this head is not ready.

[devin-ai-integration]

(1) The "frozen, released" claim is not mechanically protected … (2) the test reconstructs the parse boundary in the wrong order

Both addressed in 9e5a56f; branch is also rebased onto e8b779cb3.

(1) Identity is pinned, not asserted. The free-form source string is gone. Each entry carries a typed provenance:

• mined (40 entries): the git object id of the historical blob. corpus.test.ts hashes the checked-in bytes (sha1("blob <len>\0" + bytes)) and must reproduce that id — so rewriting a frozen script fails the gate, and the id itself is not obtainable except from the released content.
• derived (12 scripts/docs/ entries — released grammar surfaces with no shipped .ad blob to point at): SHA-256 of the bytes.

The "was this tag actually cut" half needs git, so it lives in pnpm check:replay-compat (scripts/check-replay-compat-provenance.ts): every cited tag is checked against git tag --list, every mined id re-derived via git rev-parse <recordedBy>:<path>, and the released blob byte-compared against the corpus file. Deliberately outside the unit lane — it requires full history and tags, which the fast lane can't assume. Current run: Verified 40 mined replay-compat scripts against their released blobs across 10 tags (12 derived entries are digest-pinned).

Tamper check: appending one line to a corpus script now fails … is byte-identical to its pinned source (Tests 1 failed | 106 passed).

(2) Parse boundary. The test now calls parseReplayInput(script, undefined) rather than composing the two calls itself. No verdict moved under the real ordering (all 52 re-derived through parseReplayInput), but the boundary is now guarded instead of duplicated.

Gates: format:check, lint, typecheck, check:layering, check:fallow, check:affected:test, check:replay-compat green; corpus lane 107 tests / 44 ms.

On Coverage: the ios-system-ui benchmark gap reproduces unchanged on a clean origin/main worktree, so it is inherited rather than from this head. I can add the benchmark case (or a waiver) here if you'd rather not land with it red — it's outside #1417, so tell me which you prefer.

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://callstack.github.io/agent-device/pr-preview/pr-1436/
Built to branch gh-pages at 2026-07-27 16:32 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[thymikee]

Re-review at head 9e5a56f: the parse-order issue is fixed, but one P1 provenance blocker remains. The historical verifier only checks entries while they remain kind mined. A mined corpus script can be edited, reclassified as derived, and given a matching digest in the same editable manifest; the verifier then skips it and the unit hash check accepts the new bytes. Lock provenance kind mechanically by corpus area (at minimum integration/examples must remain mined), add a negative reclassification test, and run check:replay-compat in a full-history CI job. The current green checks do not invoke that verifier. Once that provenance boundary is enforced, the shipped parseReplayInput path and action-before-metadata ordering look correct.

[devin-ai-integration]

A mined corpus script can be edited, reclassified as derived, and given a matching digest…

Fixed in 2e80e27 (+ docs commit). The reclassification escape is closed at the rule level, not by convention:

test/replay-compat/provenance-rules.ts fixes the provenance kind by corpus area, so the manifest line no longer gets a vote:

const CORPUS_AREA_PROVENANCE_KIND = { integration: 'mined', examples: 'mined', docs: 'derived' };
requiredProvenanceKind('scripts/examples/gesture-lab.v0.16.8.ad'); // 'mined'
findProvenanceKindViolations(entries); // '…must stay "mined" … keep the released blob id instead'

• Enforced in both places: corpus.test.ts (unit lane) and scripts/check-replay-compat-provenance.ts, so a relabelled entry fails even in the lane that can't reach git history.
• Negative tests added: a mined entry cloned with { kind: 'derived', sha256 } yields exactly one violation, and an undeclared area (scripts/handwritten/…) throws rather than defaulting to the weaker pin — a new area must declare its kind before entries can live under it.

CI now invokes the verifier. New Replay-Compat Provenance job in ci.yml (fetch-depth: 0, fetch-tags: true) runs pnpm check:replay-compat. It's also a first-class replay-compat check in the affected selector: any change under test/replay-compat/ (script or manifest) selects it, covered by two new model.test.ts cases, and the existing "every catalog CI job maps to a real workflow job" parity gate ties the catalog entry to the workflow.

Green locally: format:check, lint, typecheck, check:layering, check:fallow, check:affected:test (33/33), corpus lane 110 tests, and check:replay-compat → Verified 40 mined replay-compat scripts against their released blobs across 10 tags (12 derived entries are digest-pinned).

Coverage's ios-system-ui benchmark gap is still inherited from main; happy to fix it here if you want this head fully green, otherwise it stays outside #1417.

[thymikee]

Re-review at head c1b4af5: clean readiness verdict. The remaining provenance blocker is fixed: integration/examples corpus areas are mechanically locked to mined provenance, unknown areas fail closed, mined-to-derived reclassification has a negative test, and the full-history Replay-Compat Provenance CI job re-derives released blobs with tags. Corpus assertions still use production parseReplayInput in the correct action-before-metadata order. The PR is mergeable/CLEAN and all checks, including provenance, Coverage, and Integration Tests, pass. Marked ready-for-human.

[thymikee]

Re-review at 09a0557: clean and ready for human review.\n\nThe 22-entry corpus retains every #1417 required parser surface and all retired-gesture migration refusals; removed fixtures are platform/adjacent-release duplicates. Required per-entry notes and the 30-entry cap make future additions deliberate. Provenance protections remain intact (area-kind lock, byte/digest pins, and full-history released-blob verification).\n\nValidated: focused corpus unit suite (51/51), local provenance verification, and the reviewed-head CI suite including Replay-Compat Provenance, Coverage, Integration Tests, Typecheck, Lint/Format, Fallow, and affected-check selector. Merge state: CLEAN.