refactor(camera): use generated CDR values in Python producers by TomCC7 · Pull Request #4655 · dimensionalOS/dimos

TomCC7 · 2026-10-20T02:41:57Z

Linear project: Message Encoding Refactor

Webcam, CameraModule and GstreamerCameraModule now produce the generated dimos.messages values. Webcam retains RGB conversion, stereo slicing and configured/nominal calibration. Camera metadata publishes generated CameraInfo and TFMessage values without mutating the configured mount. GStreamer copies padded BGR rows before unmapping and preserves source PTS plus offsets in integer nanoseconds.

Viewer blueprint construction now defers pubsub allocation until startup. Camera imports previously constructed an LCM socket even when the selected backend was Zenoh. Backend selection and explicit pubsub behavior remain the same. GStreamer stop also closes its Module resources if the pipeline was never started.

This is a dependent draft after #4654. Native V4L2/RealSense/ZED, downstream perception, robot blueprints and default transport cutover remain later slices. No camera, robot or TCP source was operated. No registry publication or new generator release.

Validation

At 7e2452385ddbf60b19cf9c979e17eefb0109dcf7:

45 focused camera/viewer checks passed from source and against the installed main wheel outside checkout imports. Tests mock OpenCV/RPC/publish boundaries; actual generated values and real GStreamer buffers/pipelines are used.
GStreamer acceptance covers padded rows, SDK-buffer lifetime, exact source nanoseconds and ±1ns offsets, invalid source timestamps and shutdown before pipeline startup. These cases ran locally with PyGObject 3.58.1 and existing GStreamer libraries; CI skips them if that optional SDK is absent.
Strict typing passed seven production files. All applicable hooks passed.
Ordinary main wheel build passed, including automatic built-in message generation. The wheel SHA256 is 8dc489722116223003357f0dc6938ae07221c94493470eccd31919b70cd76c7c; installed pip check passed.
The exact documentation Python block ran with generated Image values. CI adds installed-wheel camera checks and scoped typing. CI was not polled; no terminal CI claim.

Try it

The camera-value walkthrough includes port contracts, a short pixel-reader example and offline test commands.

For a fresh source installation, use Python 3.12 and a C++ compiler for the existing main-package A* extension:

demo_dir="$(mktemp -d)"
uv venv --python 3.12 "$demo_dir/venv"
uv pip install --python "$demo_dir/venv/bin/python" \
  'dimos @ git+https://github.com/dimensionalOS/dimos.git@7e2452385ddbf60b19cf9c979e17eefb0109dcf7'
cd "$demo_dir"
"$demo_dir/venv/bin/python" -I - <<'PYTHON'
from dimos.hardware.sensors.camera.webcam import Webcam
from dimos.messages.sensor_msgs.msg import CameraInfo
info = Webcam(width=1280, height=720, stereo_slice="left").camera_info
assert type(info) is CameraInfo
assert (info.width, info.height) == (640, 720)
assert info.header.frame_id == "camera_optical"
print("generated calibration:", info.width, info.height, info.header.frame_id)
PYTHON

This constructs calibration only; it does not start camera capture. Automatic message-source preparation is part of the normal package build. The wheel was tested locally; the equivalent fresh Git install above was not repeated for this camera-only change.

File change summary

Parent-relative to #4654 at ad1a6d50c34431c7de182a0782a2077b21529460: 13 files, 410 additions, 78 deletions.

Category	Files	Added	Deleted
Camera producers and typed specs	5	97	68
Deferred viewer initialization	2	3	3
Camera and viewer regression tests	4	246	7
Documentation and CI	2	64	0

No generated classes or upstream message snapshots are checked in. Native camera WIP is isolated from this PR.

github-actions

go2 replay realtime (arm64)

Details

Benchmark suite	Current: `7e24523`	Previous: `17fdb6b`	Ratio
`peak memory`	`1659.762` MB	`1589.477` MB	`1.04`
`peak threads`	`357` threads	`350` threads	`1.02`
`network (transport)`	`2721.811` MB	`2730.079` MB	`1.00`
`disk write`	`2.125` MB	`1.707` MB	`1.24`
`instructions`	`122.445` G	`115.497` G	`1.06`

This comment was automatically generated by workflow using github-action-benchmark.

codecov · 2026-10-20T02:54:41Z

❌ 27 Tests Failed:

Tests completed	Failed	Passed	Skipped
6515	27	6488	56

View the top 3 failed test(s) by shortest run time

dimos.web.relay_bridge.test_relay_bridge_module::test_relay_key_falls_back_to_global_config

Stack Traces | 0.004s run time

monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0xffa195efde50>

    def test_relay_key_falls_back_to_global_config(monkeypatch) -> None:
        # RELAY_KEY (env or .env) lands in GlobalConfig; the module reads it when
        # its own field is unset. The autouse fixture restores global_config.
        client = FakeClient()
    
        async def fake_connect(url: str, role: str, **kwargs: Any) -> FakeClient:
            return client
    
        patch_relay(monkeypatch, fake_connect)
        global_config.update(relay_key="global-key-0123456789abcdef")
        module = RelayBridgeModule(
            relay_url="http://127.0.0.1:7780", open_browser=False, robot_id="unit-bot"
        )
        try:
>           module.start()

client     = <dimos.web.relay_bridge.module_test_support.FakeClient object at 0xffa195efc6e0>
fake_connect = <function test_relay_key_falls_back_to_global_config.<locals>.fake_connect at 0xffa194e12480>
module     = <dimos.web.relay_bridge.relay_bridge_module.RelayBridgeModule object at 0xffa195efde80>
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0xffa195efde50>

.../web/relay_bridge/test_relay_bridge_module.py:1047: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
dimos/core/module.py:211: in start
    self._start_main()
        self       = <dimos.web.relay_bridge.relay_bridge_module.RelayBridgeModule object at 0xffa195efde80>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <dimos.web.relay_bridge.relay_bridge_module.RelayBridgeModule object at 0xffa195efde80>

    def _start_main(self) -> None:
        """
        If the subclass defines `async def main(self)` as an async generator
        with exactly one `yield`, run everything before the `yield` as part of
        start().
        """
        main_fn = getattr(type(self), "main", None)
        if main_fn is None:
            return
        if not inspect.isasyncgenfunction(main_fn):
            raise TypeError(
                f"{type(self).__name__}.main must be an `async def` with exactly "
                "one `yield` (an async generator function)"
            )
        loop = self._loop
        if loop is None or not loop.is_running():
>           raise RuntimeError(f"{type(self).__name__}._loop is not running")
E           RuntimeError: RelayBridgeModule._loop is not running

loop       = <_UnixSelectorEventLoop running=False closed=False debug=False>
main_fn    = <function RelayBridgeModule.main at 0xffa25c4bdbc0>
self       = <dimos.web.relay_bridge.relay_bridge_module.RelayBridgeModule object at 0xffa195efde80>

dimos/core/module.py:655: RuntimeError

dimos.hardware.sensors.camera.test_webcam_backend::test_capture_backend[darwin-/dev/video0-/dev/video0-0]