feat(selenium-devtools-py): A11y tree and element overlay in Python traces - #343
feat(selenium-devtools-py): A11y tree and element overlay in Python traces#343vishnuv688 wants to merge 20 commits into
Conversation
Greptile SummaryThe PR adds accessibility trees, element rectangles, and overlay points to Python Selenium trace archives by serving shared browser scripts and streaming per-action snapshots through the backend.
Confidence Score: 2/5The PR is not yet safe to merge because portable traces can expose non-password secrets and multi-session runs can still attach snapshots to the wrong commands. Element capture preserves non-password values and hrefs through verbatim archive serialization, while the Python instrumentation continues to append snapshots from every driver to one session-unattributed global list. Files Needing Attention: packages/shared/src/element-scripts.ts and packages/selenium-devtools-py/src/selenium_devtools/instrumentation.py
|
| Filename | Overview |
|---|---|
| packages/shared/src/element-scripts.ts | Centralizes browser extraction scripts and adds password-only value redaction, but other secret-bearing values and hrefs remain persisted. |
| packages/selenium-devtools-py/src/selenium_devtools/instrumentation.py | Captures accessibility and element data after commands, while snapshots remain globally accumulated without session identity. |
| packages/selenium-devtools-py/src/selenium_devtools/trace_export.py | Streams captured action snapshots to the backend in bounded batches before trace export. |
| packages/backend/src/trace-export.ts | Passes accumulated action snapshots into trace generation and serializes raw accessibility trees for the viewer. |
| packages/trace/src/a11y-snapshot.ts | Provides the extracted web accessibility-tree serializer used by backend trace generation. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
DOM[Browser DOM] --> Scripts[Shared element and accessibility scripts]
Scripts --> Python[Python per-action capture]
Python --> Socket[Action snapshot WebSocket batches]
Socket --> Backend[Backend active run]
Backend --> Trace[Trace serializer]
Trace --> Archive[Portable trace archive]
Reviews (3): Last reviewed commit: "fix(examples): a login example that logs..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| snapshot: ActionSnapshot = {"timestamp": timestamp, "command": command} | ||
| if shot: | ||
| snapshot["screenshot"] = shot | ||
| # Two reads, two panes: `elements` carries the interactable boxes, the | ||
| # accessibility tree becomes the A11y tab's text. Capturing only the first | ||
| # left that tab reporting "no accessibility snapshot for this command" with | ||
| # 39 element files sitting in the same archive. | ||
| for key, script in ( | ||
| ("elements", scripts["elements"]), | ||
| ("accessibilityTree", scripts["accessibilityTree"]), | ||
| ): | ||
| try: | ||
| value = run(f"return {script}") | ||
| except Exception as exc: # noqa: BLE001 — a missing pane, not a failed run | ||
| _log.debug("%s read failed: %s", key, exc) | ||
| continue | ||
| if isinstance(value, list) and value: | ||
| snapshot[key] = value | ||
| if "elements" not in snapshot and "accessibilityTree" not in snapshot: | ||
| return | ||
| _state["action_snapshots"].append(snapshot) |
There was a problem hiding this comment.
If two WebDriver sessions issue interleaved commands, their snapshots enter one global list without session identity and are later matched to commands only by timestamp, causing actions to display another session's screenshot, accessibility tree, element rectangles, or overlay point.
Knowledge Base Used:
| value: elType.toLowerCase() === 'password' ? '' : inputEl.value || '', | ||
| href: htmlEl.getAttribute('href') || '', |
There was a problem hiding this comment.
When a traced page contains an OTP, API token, or other secret in a non-password input—or a signed or bearer token in an href—the element script captures it unmodified and the trace writer serializes it into a portable *-elements.json resource, exposing the secret to anyone with the archive.
How this was verified: The captured values flow through Python action-snapshot streaming to verbatim JSON serialization in the trace archive.
Knowledge Base Used:
1 participant
What & why
Type of change
Packages touched
shared(types and contracts)core(framework-agnostic capture/reporting)elements(published element/snapshot API —@wdio/elements)service(WebdriverIO adapter)nightwatch-devtools(Nightwatch adapter)selenium-devtools(Selenium adapter)backend(server)app(UI)script(page-injected runtime)selenium-devtools-pytraceNotes for reviewers
Screenshots / recordings