Skip to content

Design: press hit-test and occlusion evidence field (the landing-fact layer of #3335) #4831

Closed as not planned

Description

@strzhao

Design issue. It narrows the landing-evidence layer of #3335 and hangs the hit-test check off the #4456 admission owner instead of a new surface.

Problem

A follow-up audit of 146 recent transcripts (26 session digests, ~2,656 automation events; 5 shard analysts + 1 quantitative pass + 2 adversarial verifiers; 56 verdicts — 34 confirmed, 15 downgraded, 7 merged) found one costliest incident class: a press that reports success while the input provably landed nowhere. On an iOS React Native ActionSheet, three distinct target-resolution forms resolved to the same coordinate. Three consecutive presses each reported success; the tool's own --settle post-action accessibility-tree diff reported zero changes (+0 −0, ~5 unchanged lines) after each press. To be precise: this no-change evidence was printed loudly on every retry and still did not stop the retries — which is exactly the informational-versus-enforcement gap that Option B and question 3 target (+0/-0). A separate incident in another session carries the cleaner root cause: an RN ring control (not a row) whose accessibility frame sat ~147 pt away from the visually rendered control, so every resolved touch point fell outside the rendered hit area; the session degraded into ~95 blind presses. The saga's ~9-hour wall-clock span is duration, not attributed cost — the audit's adversarial pass downgraded the causal attribution (session route pollution inflated the retry count, and the visual-coordinate fallback also failed and required an app bounce), so the honest unit is ~95 wasted presses. The ActionSheet root cause itself was never adjudicated (suspected app-side surface) — which is the sharpest form of this gap: without a landing fact, a wrong-coordinate dispatch and a dead surface are indistinguishable.

The detection for this already existed. --verify reports a flat surface as changedFromBefore: false. The session did not pass --verify — and even when a caller does, a zero-change observation is informational evidence. Nothing escalates it, and nothing names what the tap hit.

Evidence chain

  1. Snapshot resolves a target; the touch point derives from the accessibility frame.
  2. Admission (ADR 0039, admitResolvedTarget) runs motion and readiness. Readiness asks the accessibility layer whether the element is hittable — the same layer whose frame is wrong.
  3. The backend dispatches a synthetic touch at the point (pressPoint in packages/platform-apple/src/interactions.ts). Success is a transport ack.
  4. The result reports dispatch success. No field records which element, if any, the point reached.
  5. The screen does not change. The only signal is changedFromBefore: false (honest since fix(interaction): omit changedFromBefore when no verify baseline was captured #4338: omitted when no baseline). It is informational; nothing consumes it as a refusal or a warning.
  6. The caller retries steps 1–5. Nothing distinguishes "hit the wrong element" from "hit the right element and the app ignored it".

The class: accessibility-frame facts vouch for a point that the rendered layer does not honor. Readiness checks inherit the blind spot, because they read the same tree.

Existing family

Capture-time occlusion already has a typed carrier: SnapshotOcclusionContextEvidence in packages/contracts/src/snapshot-private-evidence.ts. It is deliberately capture-local. Nothing in the family owns the dispatch-time fact.

Increment (narrowed)

Only the hit fact and its escalation:

  1. A field that names the element the input actually reached, plus an occlusion verdict, when such a read ran.
  2. A contract that escalates a reported-success + zero-change press into a strong warning (or a one-shot hit read), instead of leaving changedFromBefore: false informational.

No new admission stage, no new wait loop, no per-default-click probe, no constant-value response field. Those are #3335's and #4456's rejected shapes; this issue keeps them rejected.

Proposed shapes

A. Dispatch-ack hit fact (conditional field). When a hit read runs (shapes B and C are the triggers), the press result carries hitElement (the accessibility element that owns the point) and hitOcclusion (verdict: clear / occluded by <identity> / none). Unpopulated when no read ran — never a constant unobserved, so it passes #3335's field test: some paths report a different value. Trade-off: a native read cost, and one platform capability per platform (iOS: accessibility hit-test at a point; Android: an equivalent probe).

B. Zero-change escalation. When --verify reports zero surface change after a press that reported success, escalate: a typed strong warning on the result at minimum, optionally a one-shot hit read that names what the point reached before the next attempt. The read is paid only on the suspicious path, so this respects #3335's probe rejection and #4456's idle-path cost rule. Trade-offs: post-dispatch only — the wasted input is already sent; and it inherits --verify's opt-in, which the saga shows a caller can skip. Pairing with #4342 decides whether the pixels channel and this field merge into one escalation plan.

C. Pre-dispatch hitTestAtPoint consistency check. An admission check under #4456's ordered list (after motion and readiness): hit-test the resolved touch point; if the receiving element is not the resolved element, refuse with dispatched: no and a typed reason (extends the existing covered/ambiguous reasons). Trade-offs: prevention before any send — the only shape that stops the wasted input. Two honest limits: it validates accessibility-layer self-consistency; when the a11y hit-test uses the same wrong frames, the 147 pt case passes it. And under #4456's idle-target cost rule it would not run on an idle target unless its ADR 0011 row declares the read — exactly where the saga sat.

A and B compose: B decides when the read runs, A decides what the result carries. C prevents a different, cheaper failure class (the point owned by a different element) at admission.

Affected surfaces

  • docs/adr/0039-pre-dispatch-admission.md — check order, budget rows, evidence path (owns shape C's placement)
  • packages/command-runtime/src/commands/interaction/runtime/pre-dispatch-admission.ts (admitResolvedTarget) — shape C's check
  • packages/command-runtime/src/commands/interaction/runtime/post-action-surface.ts (summarizePostActionEvidence, changedFromBefore) — shape B's escalation hook
  • packages/contracts/src/snapshot-private-evidence.ts — where a dispatch-time hit fact lives as typed private evidence (or beside SnapshotOcclusionContextEvidence)
  • packages/platform-apple/src/interactions.ts (pressPoint) — the dispatch ack path that carries shape A's field
  • packages/contracts/src/interaction-guarantees.ts — the outcomeObservation cells that shapes A/B would tighten

Questions for maintainers

  1. Owner: is the hit fact an admission check under Pre-dispatch admission: one owner, budget and evidence path for resolved-target input checks #4456 (refuse before send, shape C), a post-dispatch evidence field (shapes A+B), or both with C as a cheap subset? ADR 0039's rejected-alternatives note already says "element hittability still does not prove that the point receives the input" — that argues for a post-dispatch landing fact that admission cannot provide.
  2. When the hit read names a different element than resolved: refuse, dispatch-with-warning, or dispatch-and-record only?
  3. Escalation tier for reported-success + zero-change: strong warning, refusal of the next same-target press, or a forced one-shot hit read? How does the protection stay real when a caller does not pass --verify?
  4. Platform availability: is an accessibility hit-test at a point cheap enough (per-press cost and line cost) to fit Pre-dispatch admission: one owner, budget and evidence path for resolved-target input checks #4456's budget for non-idle targets — and is any render-layer landing fact available at all, given the 147 pt class defeats a11y-layer reads?
  5. Field placement: extend SnapshotOcclusionContextEvidence, or a new dispatch-time evidence type? The capture-time carrier is deliberately capture-local (object-identity retention), so sharing may violate that rule.

An implementation PR follows only after the design settles here.

🤖 Generated with Claude Code

Activity

  1. thymikee commented on Nov 21, 2026

    @thymikee
    Member

    Triage: duplicate of #3335. By the issue's own analysis, an accessibility hit-test passes the 147 pt case, and the zero-change escalation needs --verify, which that session did not pass, so none of the proposed shapes would have stopped the cited retries. Please add the audit evidence to #3335. Pre-dispatch checks are tracked in #4456. Leaving this open for a maintainer to close.

  2. strzhao commented on Nov 21, 2026

    @strzhao
    ContributorAuthor

    Accepted — duplicate of #3335. Under this issue's own admission rule none of the proposed shapes would have reported a different value for the cited sagas, as the triage notes. The audit evidence (three-form same-point presses with +0 −0 settle diffs; the 147 pt ring case) has been posted on #3335. Leaving this open for a maintainer to close as duplicate.

  3. roy-tong commented on Nov 21, 2026

    @roy-tong

    Hit testing and occlusion belong in the evidence payload, not only in the screenshot. If a tap is recorded as successful but the target was covered, the automation looks fine and the app is broken, so I would store the hit-test verdict alongside the coordinates.

  4. thymikee commented on Nov 21, 2026

    @thymikee
    Member

    Closing as a duplicate of #3335, per the triage above. Please add the audit evidence there.

  5. roy-tong commented on Nov 21, 2026

    @roy-tong

    The detail worth keeping is that the no-change diff was printed on every retry and still did not stop the loop. That suggests the retry policy reads the action result, not the settle evidence. If the settle diff is the expensive check, make it a gate: zero change after a press should end the retry path and surface the coordinate collision instead of being logged as one more observation. A press that lands nowhere is also the one class where retrying cannot help, since the resolver will pick the same coordinate again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    duplicateThis issue or pull request already exists

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions