Repository navigation
Feature request: audio controls, and a Quest 3 immersive mode that already works on a fork #128
Description
I fly with PixelPilot as the ground station on a Meta Quest 3 and have been building on a fork
for a while. Two things I would like to put forward: audio, which already works but nobody would
know it, and an immersive mode for VR headsets, which is what #92 and #95 are asking for.
Audio works — it just is not switched on, and the app gives no control over it
PixelPilot has had an Opus decoder since September 2024 and plays audio that arrives as RTP
payload type 98 in the video stream. majestic ships with audio off, though, so in practice it is
silent. On the camera:
cli -s .audio.enabled true
/etc/init.d/S95majestic restart
With that, the camera's microphone comes through in the headset. It costs about 13 kbit/s at the
default 8 kHz Opus, against tens of Mbit/s of video.
What I think is missing in the app:
- A switch. Audio playback starts unconditionally today; there is no way to turn it off, and
no mute. - Volume.
- Some sign that audio is arriving. At the moment "the camera sends nothing", "the decoder
failed to start" and "it is playing but quiet" all look the same: silence. - A line in the docs that majestic needs
audio.enabled.
Getting it working also turned up six defects in the decoder, several reachable from radio
input, fixed in #127 — and a crash when the app is launched in the background, which is what
happens when the adapter is plugged in with the screen off, fixed in #126.
A Quest 3 immersive mode
On the fork, the video runs in a native OpenXR session instead of a flat Android window: the
decoder renders straight into a compositor layer, so there is no extra copy and no "grab the
window and bring it along". The flat mode is unchanged, and the immersive one is opt-in.
- Head-locked by default, with world-locked as a setting — the screen moves with your head,
which is exactly VR Screen mooving with the head #92. - A HUD drawn from the MAVLink telemetry, on separate layers at different depths: an
artificial horizon from pitch and roll (Add horizon line ? #95), a dashboard (battery per cell, current, altitude,
speed, heading, link quality, and — with a small script on the air unit — its temperature
and CPU load), a minimap, and a height profile. - Offline maps, since the headset rarely has internet in the field: the minimap can use
orthophotos and the height profile a terrain model, both from open data and built ahead of
time by a script in the repository. - Instruments can be moved by pointing at them with a controller and holding the trigger.
- A flight log — telemetry and link statistics, five samples a second, as CSV.
It needs XR_KHR_android_surface_swapchain; the Meta extensions it uses (passthrough, hand
tracking) are optional, so other OpenXR headsets should work, but I have only tested on a
Quest 3.
Branch: https://github.com/iflyhere/PixelPilot/tree/feature/xr-goggle-mode
Would you want this upstream, and in what shape?
It is a fair amount of code, so I would rather ask than drop one large PR on you. One way to
split it: first the link moving into a foreground service (which the immersive mode needs, so
the link survives switching modes), then the OpenXR module, then the HUD on top. Happy to do it
differently if you prefer.
Activity