Skip to content

Design (XL): visual grounding channel for sparse accessibility trees on HarmonyOS #4832

Open

Description

@strzhao

Design (XL). Problem, evidence, candidate shapes, affected surfaces, phasing, and focused questions below. An implementation PR follows only after the design settles here.

Problem

On real HarmonyOS devices the ArkUI accessibility dump often comes back nearly empty, and the only supported fallback today ends in unverified bare-coordinate interaction.

Numbers from a usage audit of one 25-minute real-device session on a production HarmonyOS build:

  • snapshot p95 latency was 6.4 s, and the dump returned 2 nodes for the screen under test.
  • The agent fell back to screenshot, then an external vision model, then manual coordinate math, then bare press x y: 14 vision-model calls, 23 screenshots, and 8 bare-coordinate presses in that one session.

No session content is quoted here; only counts.

Every bare-coordinate press bypasses what the tree normally provides: no ref, no post-action verification target, no tree for the tap guard's keyboard-band derivation to read, and no selector value for record and replay.

Evidence

The chain, mapped to code on main:

  1. snapshot on HarmonyOS reads uitest dumpLayout (packages/platform-harmonyos/src/snapshot.ts, snapshotHarmony). When the foreground app publishes little accessibility content, the dump holds a handful of nodes and the quality verdict reports state: "sparse" (packages/capture-kit/src/snapshot-quality-verdict.ts).
  2. Both sparse outputs then instruct the agent to leave the tree. The quality warning says: "Its refs and selectors are invalid. Use screenshot as visual truth and coordinate taps; retry snapshot after navigating" (packages/capture-kit/src/snapshot/snapshot-presentation/quality-warnings.ts). The output hint says the same in more detail (packages/command-output/src/output/snapshot.ts). The skill agrees: screenshot, inspect, coordinates, then retry snapshot -i (skills/agent-device/SKILL.md).
  3. Once the agent is on pixels, nothing in the product turns a pixel back into an action. HarmonyOS interaction primitives are coordinate-only: pressHarmony(x, y) dispatches uitest uiInput (packages/platform-harmonyos/src/input-actions.ts), and createHarmonyInteractor binds tap/doubleTap/longPress/focus all to coordinates (packages/platform-harmonyos/src/interactor.ts). Selector resolution needs tree nodes, which do not exist here.
  4. The post-processing that could bridge pixels and targets is not documented for this platform. The screenshot help enumerates --crop-on platforms as "currently iOS simulators, Android emulators, CoreDevice physical iPhones, and Samsung Tizen and LG webOS TV apps", with no HarmonyOS, and --overlay-refs sits in the same sentence with no HarmonyOS mention (packages/command-surface/src/capture/screenshot.ts). On a 2-node tree --overlay-refs would have nothing to annotate anyway.
  5. The snapshot pipeline carries one platform fact today: the keyboard band, measured from hidumper -s WindowManagerService -a -a and attached to every snapshot response (packages/platform-harmonyos/src/keyboard-band.ts, feat(harmonyos): report the keyboard band in snapshot responses #4285, evidence in docs/evidence/harmonyos-keyboard-band-2026-11-08.md). That is the pattern this issue asks to extend, including its unmeasurable honesty state.

The same problem has a web precedent: #3509 ("on-screen text reaches no reader"). It closed by putting the on-screen text into the tree (#3515), not by adding a side channel. On HarmonyOS the app-side tree cannot be fixed from here, so the open question is what owned channel, if any, carries visual truth into a verified action. #1296 is the sister question for surfaces no app tree can ever describe (iOS system UI targets).

Proposed shapes

Three shapes. They are not mutually exclusive, and "no" is an acceptable answer to some of them.

A. Built-in OCR or control detection as a fallback target source

Add a bounded on-host read of the screenshot (text regions or widget-shaped regions) that attaches candidate targets with normalized rects to a sparse-tree response.

  • For: closes the loop inside the product. One command returns pixels plus candidate targets. No external model needed.
  • Against: the project removed diff-side OCR once already. ScreenshotDiffResult.ocr and nonTextDeltas are deprecated "no longer emitted" fields (packages/capture-kit/src/screenshot-diff.ts, emission removed in Simplify screenshot diff output #1495). An accuracy floor, an engine dependency, and a maintenance owner all come back with them. Host-side image analysis also raises the command-surface budget question.
  • Honest framing: this only has to be good enough to propose a rect that verification can falsify. It does not have to read arbitrary text.

B. press <text> as a visual-semantic interaction channel

Accept text (or a label-like selector) on interaction commands. On a sparse tree, the platform resolves it against on-screen text, dispatches one bounded tap, and reports the rect it acted on.

  • For: matches how agents already phrase intent. Keeps tap authority in the daemon. The reported rect gives verification something to check, and with feat(interaction): verify coordinate taps with screenshot fallback #4297's readback the same rect can be diffed after the tap.
  • Against: needs an on-device or on-host text locator per platform, which is the same engine question as A. The ambiguity policy (two matching labels on screen) must be defined before shipping. It risks becoming a second selector grammar to maintain.
  • Note: on HarmonyOS the only public shell channel is uitest uiInput coordinates, so a text channel still lands as one resolved coordinate tap.

C. Declare the bypass, verify it, and stop there (the screenshot-diff line)

Accept that grounding stays external (the agent's own vision model), and make the product own the safety rails: declarative sparse guidance (below) plus the #4297/#4342 post-action readback, where --expect-screenshot-diff verifies that pixels changed. computePngChangedPixelRatio (packages/capture-kit/src/png-changed-pixel-ratio.ts) is the existing primitive.

Declarative sparse guidance (applies to every shape)

Turn the sparse warning from prose into a declared structure, so callers and skills stop re-deriving policy from sentences. Example shape: machine-readable fields on the sparse verdict, such as treeRefsValid: false, visualTruthAvailable: true, and a declared retry path, alongside the existing sentences. This must stay compatible with the current SKILL.md sentence (sparse, then screenshot and coordinates, then retry snapshot -i); the change is where the policy lives (a typed fact, not only prose), not what it says. The keyboard-band fact from #4285 is the precedent for snapshot-carried platform facts.

Affected surfaces

  • packages/platform-harmonyos/src/snapshot.ts, screenshot.ts, input-actions.ts, interactor.ts — HarmonyOS read and act primitives, coordinate-only today.
  • packages/capture-kit/src/screenshot-diff.ts, png-changed-pixel-ratio.ts — diff primitives; the deprecated OCR fields mark where built-in analysis once lived.
  • packages/capture-kit/src/snapshot-quality-verdict.ts, packages/capture-kit/src/snapshot/snapshot-presentation/quality-warnings.ts, packages/command-output/src/output/snapshot.ts — the sparse verdict and its prose.
  • packages/capture-kit/src/screenshot-overlay.ts, screenshot-overlay-android.ts — the refactor(snapshot): assign and scope the Wave 4 shared presentation debt (#1739) #1983 pattern: shared ranking, projection and artifact assembly in shared code, platform-specific classification in a platform module.
  • packages/command-surface/src/capture/screenshot.ts, packages/command-registry/src/flag-definitions-workflow.ts — flag and help copy that name, or omit, platforms.
  • skills/agent-device/SKILL.md — the sparse-tree sentence.
  • New ADR candidate per docs/adr/FORMAT.md: what owns visual grounding, if anything.

Phasing (tied to #4317)

#4317 moves HarmonyOS behind a platform plugin interface for 0.22. That decides the landing order:

  1. Before the plugin extraction: declarative sparse guidance only. It touches shared verdict and presentation types plus one skill sentence, adds no platform capability, and the plugin then inherits it for free.
  2. Inside the plugin, after Extract HarmonyOS behind a platform plugin interface, then ship it as @agent-device/harmonyos (0.22) #4317: any HarmonyOS grounding capability (shape A or B) lands as a plugin-provided fact or operation, behind the same interface the keyboard band already uses, so core never learns about OCR or text channels.
  3. Shared and design-gated: the shape C rails (feat(interaction): verify coordinate taps with screenshot fallback #4297/Design: --expect-screenshot-diff as an owned post-action readback (design for #4297 part 2) #4342) are platform-neutral and can proceed independently. If they land first, a later HarmonyOS grounding channel should feed its resolved rect into the same readback contract instead of adding a second verification path.

Questions for maintainers

  1. Is a built-in grounding engine (A or B) wanted at all, given the project removed diff-side OCR once (Simplify screenshot diff output #1495) and audited agents reach for their own vision model anyway? Or is the honest answer C: external grounding, owned verification?
  2. If grounding lands, where does it live: inside the HarmonyOS plugin boundary (Extract HarmonyOS behind a platform plugin interface, then ship it as @agent-device/harmonyos (0.22) #4317), or as a core capability with platform adapters? Does the keyboard-band fact pattern generalize to "target candidate" facts?
  3. For B (press <text>): is a second interaction surface acceptable, and what is the ambiguity policy — refuse on multiple matches, rank like overlay refs do (MAX_OVERLAY_REFS in packages/capture-kit/src/screenshot-overlay.ts), or require a rect hint?
  4. Should the sparse verdict grow declarative fields now (cheap, shared) even if no grounding shape is chosen, so callers stop parsing prose?
  5. Does the shape C readback (Design: --expect-screenshot-diff as an owned post-action readback (design for #4297 part 2) #4342) need a provenance field for the compared rect, so externally derived and product derived rects stay distinguishable in verification results?

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions