Repository navigation
List and act on controls inside shadow roots as d refs - #107
Merged
senamakel merged 4 commits intoOct 17, 2026
Merged
Conversation
BrowserSurface::press_point built the raw mouse events of a press inline. They move to convert::press_at, with convert::move_to for the move alone, so every press the crate sends by position is the same trusted move, button down, and button up.
Pages built from web components keep their controls inside open shadow roots, often as plain spans and divs drawn with a pointer and wired to a click by the component's script. agent-browser looks for such elements in the light DOM only, so they came out as bare text, or, in a compact snapshot, not at all: there was nothing to press. Browser::snapshot now follows agent-browser's snapshot with one look through the open shadow roots (deep/deep.js) for what a person can press or type into that the tree has no ref for: an own pointer, an onclick, a tab stop, or an editable region, drawn and outermost, at most 40. Each is marked data-tc-deep and listed as dN, on the line that shows its words or in a group after the tree that a max_chars cut keeps whole. A number is never reused in a session, and an element keeps it across snapshots. A page with none comes out unchanged. Browser::perform acts on a d ref itself: the element is found by its mark (gone: StaleRef), brought into view, hit-tested through open shadow roots (covered: NotActionable naming the cover), and pressed with trusted mouse events. Type and Fill press into a field and keep the focus check; Focus, Press, Hover, GetText, and IsVisible work too, and any other action is InvalidInput.
Tiny Sweeper reviewTiny Sweeper reviewed this change across 5 lane(s) and found 0 active actionable finding(s). Detailed lane evidence and any incomplete work are listed below. State: Incomplete Review snapshot
Completeness: Incomplete FeaturesNone identified with supported citations. TestsNo supported feature-to-test mapping was produced. Test execution is not inferred. FindingsNo active actionable findings. Could not review: crates/tinycomputer-browser/src/deep/deep.js, crates/tinycomputer-browser/src/deep/deep_tests/live_results_tests.rs, crates/tinycomputer-browser/src/deep/deep_tests/live_tests.rs, crates/tinycomputer-browser/src/deep/merge.rs, crates/tinycomputer-browser/src/lib.rs, crates/tinycomputer-browser/src/sessions/deep.rs, crates/tinycomputer-browser/src/sessions/focus.rs, crates/tinycomputer-browser/src/sessions/mod.rs, crates/tinycomputer-browser/src/sessions/page.rs, crates/tinycomputer-browser/src/sessions/sessions_tests.rs, crates/tinycomputer-browser/src/sessions/sessions_tests/deep_keys_tests.rs, crates/tinycomputer-browser/src/sessions/sessions_tests/deep_page.rs, crates/tinycomputer-browser/src/sessions/sessions_tests/deep_tests.rs, crates/tinycomputer-browser/src/surface/cover.rs, crates/tinycomputer-browser/src/surface/surface_tests/perception_tests.rs, docs/crates/tinycomputer-browser/README.md, docs/crates/tinycomputer-browser/interacting.md, docs/crates/tinycomputer-bus/browser.md Before merge
How this fits togetherflowchart LR
n0["Browser<br/>changed"]:::changed
n1["Session<br/>changed"]:::changed
n2["sent"]:::impacted
n3["open"]:::impacted
n4["...ake_is_refused_before_the_page_is_touched"]:::impacted
n5["page"]:::impacted
n6["..._the_deep_controls_with_refs_of_their_own"]:::impacted
n0 -->|uses| n1
n3 -->|uses| n0
n4 -->|calls| n2
n4 -->|tests| n2
n4 -->|calls| n3
n4 -->|tests| n3
n4 -->|calls| n5
n4 -->|tests| n5
n6 -->|calls| n2
n6 -->|tests| n2
n6 -->|calls| n3
n6 -->|tests| n3
n6 -->|calls| n5
n6 -->|tests| n5
classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff
Agent review detailscritique
security
tests
commits
description
Evidence and run details
|
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
Comment |
clippy's assert_is_empty (Rust 1.99, the CI toolchain) refuses assert!(x.is_empty()); the rest of the workspace states these as assert_eq!(x.len(), 0).
Review fixes for the d refs of controls inside shadow roots: - Placement: deep.js now counts where each control's words show among the page's text, the way the tree shows its lines, and merge.rs puts a control on that occurrence only when the tree shows the words as many times; otherwise it goes to the trailing group. An unlisted row with the same words (a sold-out "Select") no longer moves every later control up a row. - Identity: Minted keeps the words each number was handed out with. A number survives a look only while its element shows the same words, and aim, focus, and text refuse a node re-drawn in place as StaleRef. An editable region is named by its label, never its typed content. - Pressing: the aim waits for the element to hold still for two frames, picks a point that reaches the element and not a control nested in it (a field, when aiming for text), and arms a capture guard that stops any part of the press whose path misses the element, reported as NotActionable. - A look whose reply is lost advances the numbering by 40, so the numbers it may have marked are never handed out. - Text: end/select wait two frames before the focus check, and Type presses End where a field's caret cannot be set (email, number). - No look while a raw frame command has agent-browser read a frame. - Node budgets on per-action scans and text reads; a pointer label whose field the tree already refs is not listed again.
![tinysweeper[bot]](https://avatars.githubusercontent.com/in/4526585?s=60&v=4){kind=small}
There was a problem hiding this comment.
tinysweeper found nothing blocking, but could not review everything, so this is not an approval: crates/tinycomputer-browser/src/deep/deep.js, crates/tinycomputer-browser/src/deep/deep_tests/live_results_tests.rs, crates/tinycomputer-browser/src/deep/deep_tests/live_tests.rs, crates/tinycomputer-browser/src/deep/merge.rs, crates/tinycomputer-browser/src/lib.rs, crates/tinycomputer-browser/src/sessions/deep.rs, crates/tinycomputer-browser/src/sessions/focus.rs, crates/tinycomputer-browser/src/sessions/mod.rs and 10 more.
$0.0175 · 1,232,007 in / 38,741 out · 125,440 cached (10%) · ladder/vectors-oai3, glm-5.3-flash, · 738 embedded
critique: $0.0023 · 235,620 in / 4,443 out · 0 cached (0%) · glm-5.3-flash,
security: $0.0069 · 735,151 in / 20,766 out · 71,680 cached (10%) · glm-5.3-flash
tests: $0.0006 · 68,983 in / 7 out · 0 cached (0%) · glm-5.3-flash
description: $0.0017 · 127,703 in / 9,162 out · 53,760 cached (42%) · glm-5.3-flash
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Controls inside open shadow roots now get refs (
d1,d2, …) in a snapshot, and actions on those refs work.Pages built from web components keep their controls inside shadow roots. On one live booking page, almost everything sits in an
<ola-app>element's 60 open shadow roots. The log-in link and every ride row there are plainspan/divs with a pointer cursor and a click listener: no role, no tab stop, noonclickattribute. Chrome's accessibility tree has them, andDOM.getDocument({pierce: true})reaches them. But agent-browser's snapshot gives refs only to native/ARIA controls and to pointer elements its cursor pass finds, and that pass walks the light DOM only. So none of these controls got a ref. In compact mode, rows after the last ref were dropped entirely: an agent saw "LOG IN" as bare text and no ride list at all.The agent-browser fork cannot take this change, so it lives in tinycomputer-browser.
How
deep/(deep.js,mod.rs,merge.rs): after agent-browser's snapshot, a look walks open shadow roots and picks the controls the tree gave no ref.onclick, a tab stop, or an editable region.data-tc-deep.placeturns the line that shows a control's words into- clickable "LOG IN" [ref=d1]. Controls with no such line go into a trailing group that amax_charscut never drops.sessions/deep.rs: actions ondrefs.convert::press_at, shared with the surface's point press), neverelement.click().focus.rs: keys only where the action points.InvalidInput.sessions/page.rs:snapshotruns the look (capped at 3 s; a look that fails or runs late hands out nodrefs), andperformroutesdrefs to the deep path. Page/title reporting and the origin checks are shared by both paths.convert: the raw mouse press moves intoconvert::press_at/move_to.BrowserSurface::press_pointuses it and sends the same events as before.interacting.md, and a sentence in the busbrowser.md.Testing
agent-browserfeature, 239/239 with it (34 new).TINYCOMPUTER_BROWSER_EXECUTABLEset to a regular Chrome 155): the 4 new ones pass. The fixture is a web component with a "LOG IN" pointer span, a nested list with "Mini" / "3 min" rows and a nameless row, trailing text, a contenteditable, and a closed root. They check:drefs and their names appear;isTrusted === true;rustfmt,clippy(pedantic,-D warnings, all targets, with the feature) andrustdoc -D warningsare clean.Notes
onclick);d-ref press whose click handler opens a JavaScript dialog can block until the engine's 30 s timeout. The surface's existing point press has the same limit.d\d+. Today only@d3passesTarget::parseas a ref.Since the review (
28121790)An independent review found one serious placement bug and several smaller risks. All are fixed:
deep.jsnow counts each control's place among the page's text with the same words, in the tree's order.merge.rsuses that place only when the tree shows the words the same number of times; otherwise the control goes to the trailing group.StaleRefon a mismatch. Editable regions are named by their label, not by typed content.NotActionable.framecommand is active.<label>whose field already has a ref is not listed again.Tests now: