Repository navigation
Description
Design (XL). Problem, evidence, candidate shapes, affected surfaces, phasing, and focused questions below. An implementation PR follows only after the design settles here.
Problem
On real HarmonyOS devices the ArkUI accessibility dump often comes back nearly empty, and the only supported fallback today ends in unverified bare-coordinate interaction.
Numbers from a usage audit of one 25-minute real-device session on a production HarmonyOS build:
snapshotp95 latency was 6.4 s, and the dump returned 2 nodes for the screen under test.- The agent fell back to screenshot, then an external vision model, then manual coordinate math, then bare
press x y: 14 vision-model calls, 23 screenshots, and 8 bare-coordinate presses in that one session.
No session content is quoted here; only counts.
Every bare-coordinate press bypasses what the tree normally provides: no ref, no post-action verification target, no tree for the tap guard's keyboard-band derivation to read, and no selector value for record and replay.
Evidence
The chain, mapped to code on main:
snapshoton HarmonyOS readsuitest dumpLayout(packages/platform-harmonyos/src/snapshot.ts,snapshotHarmony). When the foreground app publishes little accessibility content, the dump holds a handful of nodes and the quality verdict reportsstate: "sparse"(packages/capture-kit/src/snapshot-quality-verdict.ts).- Both sparse outputs then instruct the agent to leave the tree. The quality warning says: "Its refs and selectors are invalid. Use screenshot as visual truth and coordinate taps; retry snapshot after navigating" (
packages/capture-kit/src/snapshot/snapshot-presentation/quality-warnings.ts). The output hint says the same in more detail (packages/command-output/src/output/snapshot.ts). The skill agrees: screenshot, inspect, coordinates, then retrysnapshot -i(skills/agent-device/SKILL.md). - Once the agent is on pixels, nothing in the product turns a pixel back into an action. HarmonyOS interaction primitives are coordinate-only:
pressHarmony(x, y)dispatchesuitest uiInput(packages/platform-harmonyos/src/input-actions.ts), andcreateHarmonyInteractorbindstap/doubleTap/longPress/focusall to coordinates (packages/platform-harmonyos/src/interactor.ts). Selector resolution needs tree nodes, which do not exist here. - The post-processing that could bridge pixels and targets is not documented for this platform. The
screenshothelp enumerates--crop-onplatforms as "currently iOS simulators, Android emulators, CoreDevice physical iPhones, and Samsung Tizen and LG webOS TV apps", with no HarmonyOS, and--overlay-refssits in the same sentence with no HarmonyOS mention (packages/command-surface/src/capture/screenshot.ts). On a 2-node tree--overlay-refswould have nothing to annotate anyway. - The snapshot pipeline carries one platform fact today: the keyboard band, measured from
hidumper -s WindowManagerService -a -aand attached to every snapshot response (packages/platform-harmonyos/src/keyboard-band.ts, feat(harmonyos): report the keyboard band in snapshot responses #4285, evidence indocs/evidence/harmonyos-keyboard-band-2026-11-08.md). That is the pattern this issue asks to extend, including itsunmeasurablehonesty state.
The same problem has a web precedent: #3509 ("on-screen text reaches no reader"). It closed by putting the on-screen text into the tree (#3515), not by adding a side channel. On HarmonyOS the app-side tree cannot be fixed from here, so the open question is what owned channel, if any, carries visual truth into a verified action. #1296 is the sister question for surfaces no app tree can ever describe (iOS system UI targets).
Proposed shapes
Three shapes. They are not mutually exclusive, and "no" is an acceptable answer to some of them.
A. Built-in OCR or control detection as a fallback target source
Add a bounded on-host read of the screenshot (text regions or widget-shaped regions) that attaches candidate targets with normalized rects to a sparse-tree response.
- For: closes the loop inside the product. One command returns pixels plus candidate targets. No external model needed.
- Against: the project removed diff-side OCR once already.
ScreenshotDiffResult.ocrandnonTextDeltasare deprecated "no longer emitted" fields (packages/capture-kit/src/screenshot-diff.ts, emission removed in Simplify screenshot diff output #1495). An accuracy floor, an engine dependency, and a maintenance owner all come back with them. Host-side image analysis also raises the command-surface budget question. - Honest framing: this only has to be good enough to propose a rect that verification can falsify. It does not have to read arbitrary text.
B. press <text> as a visual-semantic interaction channel
Accept text (or a label-like selector) on interaction commands. On a sparse tree, the platform resolves it against on-screen text, dispatches one bounded tap, and reports the rect it acted on.
- For: matches how agents already phrase intent. Keeps tap authority in the daemon. The reported rect gives verification something to check, and with feat(interaction): verify coordinate taps with screenshot fallback #4297's readback the same rect can be diffed after the tap.
- Against: needs an on-device or on-host text locator per platform, which is the same engine question as A. The ambiguity policy (two matching labels on screen) must be defined before shipping. It risks becoming a second selector grammar to maintain.
- Note: on HarmonyOS the only public shell channel is
uitest uiInputcoordinates, so a text channel still lands as one resolved coordinate tap.
C. Declare the bypass, verify it, and stop there (the screenshot-diff line)
Accept that grounding stays external (the agent's own vision model), and make the product own the safety rails: declarative sparse guidance (below) plus the #4297/#4342 post-action readback, where --expect-screenshot-diff verifies that pixels changed. computePngChangedPixelRatio (packages/capture-kit/src/png-changed-pixel-ratio.ts) is the existing primitive.
- For: no new engine. Builds on two open items that already have design momentum (feat(interaction): verify coordinate taps with screenshot fallback #4297, Design: --expect-screenshot-diff as an owned post-action readback (design for #4297 part 2) #4342). Keeps the "pixels changed is not action succeeded" honesty that Design: --expect-screenshot-diff as an owned post-action readback (design for #4297 part 2) #4342 already words well.
- Against: the agent still does external coordinate math with no product-owned rect. The audit failure mode is not "the tap did nothing" but "the tap landed on the wrong pixel". A same-screen diff inside a wrong rect still reads as success unless the rect itself is owned and its provenance is known.
Declarative sparse guidance (applies to every shape)
Turn the sparse warning from prose into a declared structure, so callers and skills stop re-deriving policy from sentences. Example shape: machine-readable fields on the sparse verdict, such as treeRefsValid: false, visualTruthAvailable: true, and a declared retry path, alongside the existing sentences. This must stay compatible with the current SKILL.md sentence (sparse, then screenshot and coordinates, then retry snapshot -i); the change is where the policy lives (a typed fact, not only prose), not what it says. The keyboard-band fact from #4285 is the precedent for snapshot-carried platform facts.
Affected surfaces
packages/platform-harmonyos/src/snapshot.ts,screenshot.ts,input-actions.ts,interactor.ts— HarmonyOS read and act primitives, coordinate-only today.packages/capture-kit/src/screenshot-diff.ts,png-changed-pixel-ratio.ts— diff primitives; the deprecated OCR fields mark where built-in analysis once lived.packages/capture-kit/src/snapshot-quality-verdict.ts,packages/capture-kit/src/snapshot/snapshot-presentation/quality-warnings.ts,packages/command-output/src/output/snapshot.ts— the sparse verdict and its prose.packages/capture-kit/src/screenshot-overlay.ts,screenshot-overlay-android.ts— the refactor(snapshot): assign and scope the Wave 4 shared presentation debt (#1739) #1983 pattern: shared ranking, projection and artifact assembly in shared code, platform-specific classification in a platform module.packages/command-surface/src/capture/screenshot.ts,packages/command-registry/src/flag-definitions-workflow.ts— flag and help copy that name, or omit, platforms.skills/agent-device/SKILL.md— the sparse-tree sentence.- New ADR candidate per
docs/adr/FORMAT.md: what owns visual grounding, if anything.
Phasing (tied to #4317)
#4317 moves HarmonyOS behind a platform plugin interface for 0.22. That decides the landing order:
- Before the plugin extraction: declarative sparse guidance only. It touches shared verdict and presentation types plus one skill sentence, adds no platform capability, and the plugin then inherits it for free.
- Inside the plugin, after Extract HarmonyOS behind a platform plugin interface, then ship it as @agent-device/harmonyos (0.22) #4317: any HarmonyOS grounding capability (shape A or B) lands as a plugin-provided fact or operation, behind the same interface the keyboard band already uses, so core never learns about OCR or text channels.
- Shared and design-gated: the shape C rails (feat(interaction): verify coordinate taps with screenshot fallback #4297/Design: --expect-screenshot-diff as an owned post-action readback (design for #4297 part 2) #4342) are platform-neutral and can proceed independently. If they land first, a later HarmonyOS grounding channel should feed its resolved rect into the same readback contract instead of adding a second verification path.
Questions for maintainers
- Is a built-in grounding engine (A or B) wanted at all, given the project removed diff-side OCR once (Simplify screenshot diff output #1495) and audited agents reach for their own vision model anyway? Or is the honest answer C: external grounding, owned verification?
- If grounding lands, where does it live: inside the HarmonyOS plugin boundary (Extract HarmonyOS behind a platform plugin interface, then ship it as @agent-device/harmonyos (0.22) #4317), or as a core capability with platform adapters? Does the keyboard-band fact pattern generalize to "target candidate" facts?
- For B (
press <text>): is a second interaction surface acceptable, and what is the ambiguity policy — refuse on multiple matches, rank like overlay refs do (MAX_OVERLAY_REFSinpackages/capture-kit/src/screenshot-overlay.ts), or require a rect hint? - Should the sparse verdict grow declarative fields now (cheap, shared) even if no grounding shape is chosen, so callers stop parsing prose?
- Does the shape C readback (Design: --expect-screenshot-diff as an owned post-action readback (design for #4297 part 2) #4342) need a provenance field for the compared rect, so externally derived and product derived rects stay distinguishable in verification results?
🤖 Generated with Claude Code
Activity