Repository navigation
Make live view usable on touch devices - #448
masnwilliams wants to merge 10 commits into
Conversation
Add a gesture layer for touch input: tap clicks, one-finger swipes scroll with momentum, long-press right-clicks or drags, and two-finger pinch zooms and pans the video locally. Raise the soft keyboard from the tap when the remote cursor is a text cursor, type Android input and composition events, and send Shift for upper case letters from soft keyboards. Replace the small keyboard icon with a labeled button inside the safe area and fix the layout below 360px and under mobile browser toolbars.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
…sor shapes Lift the keyboard button by the height the visual viewport loses to an overlaid soft keyboard, and only apply the classification of the newest cursor image.
The keyboard button is positioned inside the player, so lift it by the part of the player the visual viewport no longer shows.
Replace the fixed keyboard button with a draggable control that sits in a band outside the video: below it when there is vertical room, in a strip beside it when there is horizontal room, shrinking the video by up to 10% to make one. Only when there is no room does it overlay the stream as an edge-snapping handle that lets taps through while faded. The control fades when idle in every mode and its position is remembered.
Dragging kept a 44px assumption, so the wider bottom-band control jumped under the finger and landed away from the drop point. Measure the control, keep the finger's offset while dragging, and keep it out of the top and bottom safe areas.
Over a real network the cursor update for a tap usually arrives after the finger lifts, so the keyboard could only be offered through the chip. Track where the current cursor shape was seen and decide inside the tap when it lands on that spot again, such as a second tap on the same field.
A cursor image could be attributed to a newer pointer position while an update for an earlier one was still in flight, and the wait timeout used the latest pointer instead of the tap. Attribute images only when no earlier update can be pending, and pin a timed-out tap to its own point while the pointer is still there.
Pages often show another cursor before the I-beam after a click, and the first update after a tap was taken as the answer, so taps on such fields never offered the keyboard. Any text cursor within the window now offers it, only the end of the window means the tap was not on text, and the same-spot shortcut remembers where a text cursor was seen.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit b978a25. Configure here.
Neko sends no cursor image when the shape does not change, so a tap on another text field while the I-beam was showing ended the wait with no update and dismissed the keyboard. Treat an unchanged I-beam at the end of the wait as text.
|
We ran into this with a real user on an iPhone (iOS Safari), on the live view that production serves today (bundle app.416afd11.js, which still has Tapping a field in the stream focuses the remote field, but the iOS keyboard never opens, and the user could not find any way to type. The small keyboard icon at the bottom-right does open it. In an emulated iPhone (Playwright WebKit, iPhone 15 Pro) the text then arrived wrong: "Hello" came through as "hello", "!" was lost, and text from autocorrect or predictive suggestions (input events only) was dropped. On the default 1920x1080 window the stream is about 393x221 px on a phone, so the fields are hard to hit too. CDP Our case: an agent drives the browser, and a person on a phone types a login and a 2FA code in the live view. This PR (tap-to-type, input/composition handling, Shift for capitals, the 16 px font) covers what we hit. Happy to test a build on a physical iPhone if that helps. |
iOS only shows the keyboard for focus() inside a gesture, and over a real network the I-beam usually arrives after the finger lifts, so auto mode fell back to the Tap to type button. Focus the overlay inside the tap with inputmode=none and remove inputmode when a text cursor arrives, which WebKit treats as a request to show the keyboard. Other browsers focus late instead. A tap under an I-beam that is already showing now raises the keyboard at the end of the window, the remembered text spot is measured in screen pixels, and the button is kept for a text cursor that arrives after the window.
2 participants
Summary
Live view is hard to use from a phone or tablet: a one-finger swipe is sent as a mouse drag (it selects text instead of scrolling), a desktop-sized stream is too small to read or hit, and the soft keyboard only opens from a small icon. This PR adds a touch layer to the live view client. It is client-only: the remote browser still receives the same mouse, wheel and key messages it does today.
Changes
Gestures (
src/utils/touch-gestures.ts)src/utils/zoom-pan.ts), up to 3x the fitted size, or 1.5x the stream's native pixel size for large streams (at most 8x). The zoomed video can fill the letterbox bars. Double-tap resets the zoom, and a "2.5× · Reset" chip shows while zoomed. Pointer coordinates are mapped through the zoom, so taps land on the right remote pixel.Keyboard
touchendusing the remote cursor shape: neko already sends cursor images over the data channel, and Chromium shows a text (I-beam) cursor over editable fields. The pointer moves to the touch point ontouchstart, so the cursor update usually arrives before the finger lifts. If it arrives later, a "Tap to type" button appears at the tap point instead. Pages often show another cursor briefly before the I-beam after a click, so any text cursor within 800ms of the tap counts; only the end of that window means the tap was not on text. The client also remembers where a text cursor was seen, so a tap on that spot, such as a second tap on the same field, raises the keyboard inside the tap. Tapping a non-text area closes the keyboard. Seesrc/utils/cursor-shape.ts.inputand composition events, which the Guacamole keyboard ignores. These are now typed, including composition updates and backspace.src/components/touch-controls.vue) that stays off the stream whenever it can:localStorage, the control moves up above an overlaid soft keyboard (usingvisualViewport), and it shows on any touch device rather than only below 768px or with?embed.autocapitalize,autocorrectandautocompleteoff, and a 16px font so iOS does not zoom the page when it focuses.Layout
.neko-mainno longer hasmin-width: 360px, which pushed the controls off-screen in narrow embeds.100dvh, so the bottom controls are not hidden under mobile browser toolbars.viewport-fit=covermakes the safe-area insets apply.URL option
?tapKeyboard=auto(default): raise the keyboard when the tap lands on a text cursor.chip: show the "Tap to type" button there instead.always: raise it on every tap.off: keyboard button only.Desktop mouse and keyboard handling is unchanged. The new input paths only run for touch events, or on touch devices (Shift wrapping and
inputhandling). The touch-only branch infocusOverlayis removed because touch no longer synthesizes mouse events.Testing
bun test tests: 39 pass, including the newtests/touch-input.test.ts(gesture classification, zoom math and clamping, keysym and Shift mapping, composition diffing) andtests/touch-controls.test.ts(band, strip, shrink and overlay layout; drag and snap; stored position). Lint shows no new warnings compared withmain, andnpm run buildsucceeds.contextmenu.inputevents, IME composition with a keyCode 229 backspace, and key events with capitals all arrived correctly.input-event text was dropped, and capitals arrived in lower case.Not verified yet:
touchend, real Gboard behaviour, and timing over cellular networks.Known gaps
automode tapping plain paragraph text can open the keyboard.?tapKeyboard=chipis the less intrusive option.Future work
maxTouchPoints0 and a fine pointer, so pages would see touch events from a browser that says it has no touch support. Native touch should come with a consistent mobile device configuration, not on its own.Note
Medium Risk
Large changes to touch input, keyboard focus, and remote keystroke generation in
video.vue, but desktop paths are gated; misclassification or timing bugs could affect remote control on touch devices only.Overview
Adds a client-only touch layer to the chromium-headful live view so phones and tablets can scroll, zoom, and type without changing what the remote browser receives (still mouse, wheel, and key messages).
Gestures replace touch-to-mouse synthesis with dedicated handlers: tap/click, one-finger scroll (with fling) via wheel events, long-press right-click and drag, and pinch zoom on the local video with coordinate mapping and a reset chip.
Keyboard uses remote cursor images from the WebRTC data channel (new
cursorevent + I-beam classification) to open the soft keyboard on text taps, including iOS gesture/focus workarounds and a draggable keyboard control (touch-controls.vue) with band/side/overlay layout. Android/IME text is forwarded frominput/composition events with Shift wrapping for capitals.?tapKeyboard=tunes auto/chip/always/off behavior.Layout tweaks:
100dvh,viewport-fit=cover, and relaxedmin-widthon narrow embeds. Tests cover gestures, zoom, control layout, and tap-keyboard logic.Reviewed by Cursor Bugbot for commit 77dd1eb. Bugbot is set up for automated code reviews on this repo. Configure here.