feat(sensors): adapt generated image and pointcloud values

[TomCC7]

Generated Image/CompressedImage, CameraInfo and PointCloud2 values now have explicit array, compression, calibration and geometry adapters. They retain source Headers, padding, byte order and extra cloud fields. Rerun renders these values without adding methods to generated classes, and pairs CameraInfo with images in either arrival order.

Project: Message Encoding Refactor. Based on #4652. This establishes the sensor value/renderer boundary; physical camera/perception modules, robot blueprints and default factories are not cut over in this draft. Those remain implementation scopes, not external blockers.

Use the values

import numpy as np
from dimos.msgs.image import image_from_array, image_view
from dimos.msgs.time import header_now

image = image_from_array(np.zeros((480, 640, 3), dtype=np.uint8),
                         encoding="rgb8", header=header_now("camera_optical"))
pixels = image_view(image)

The factory copies input data; the view is borrowed/read-only. Image and cloud operations stay outside the generated plain-value API. The embedded walkthrough also shows RGBD projection and compression; CI executes those three Python blocks verbatim.

Manual visual test

With this checkout installed normally into the active environment:

python examples/message-viewer/demo_cdr_rerun.py sensors.rrd
rerun sensors.rrd

Use a new output. This exact example sends generated PoseStamped, RGB Image and calibrated RGBD cloud over discovery-disabled loopback Zenoh/CDR. The real RRD shows pose axes, colored points, full Pose/Header values and the RGB pixels. It refuses overwrite. The demo explicitly anchors its map frame; runtime frame names remain unchanged.

Parent-relative file changes

Category	Files	Added	Deleted	Review focus
Value adapters	5	1322	0	Pixel/point layout, explicit units, calibration/rigid transforms, timestamp normalization.
Viewer adapters/bridge	2	122	15	Correct RGB dimensions/bytes, endian depth, cloud colors and frame-aware calibration pairing.
Tests	9	1569	1	Codec/native array behavior, lifetime, malformed layout, exact doc snippets and actual RRD exchange.
Existing example	1	56	17	Extend the already tested loopback example with image/RGBD cloud; no additional example launcher.
Existing guide/CI	2	67	7	Short copyable blocks and installed-wheel/type coverage.

Total: 19 files, 3136 additions / 40 deletions, parent 70c08e85f4954e021111e55a11c0a5d1d35651e1, head ad1a6d50c34431c7de182a0782a2077b21529460. No generated classes, vendored upstream sources, native locks or hardware SDK changes are checked in. The largest file is the 462-line point-cloud test; the largest production sources are pointcloud.py (380 lines) and geometry.py (376).

Validation and limits

216 focused/compatibility checks passed, including actual JPEG/PNG/JXL, Open3D tensor downsampling, both camera/image arrival orders, the exact documentation blocks, the isolated loopback example and no-overwrite. Strict typing passes on eight production files; applicable configured hooks pass. A real SDK issue with single-row/column RGB inference is covered: the adapter now supplies explicit width/height and bytes. Depth rendering retains uint16 units and normalizes endian without changing source values.

This does not promise ROS-node integration, legacy recording conversion, human visual acceptance, hardware acceptance or terminal CI success. JXL remains an explicit consumer extension. M20 SDK/Foxglove sign-in and later domain/default/cleanup scopes are separate.

Final ad1a6d50 also sends CameraInfo over CDR and asserts the actual RRD pinhole. This fixes the 3D missing-calibration indicator found during real-browser acceptance. The same216 checks pass against the installed production wheel with the current exact example; pip check passes in the private proof environment.

[github-actions]

go2 replay realtime (arm64)

Details

Benchmark suite	Current: ad1a6d5	Previous: 17fdb6b	Ratio
peak memory	1639.457 MB	1589.477 MB	1.03
peak threads	364 threads	350 threads	1.04
network (transport)	2730.031 MB	2730.079 MB	1.00
disk write	1.934 MB	1.707 MB	1.13
instructions	121.417 G	115.497 G	1.05

This comment was automatically generated by workflow using github-action-benchmark.

[codecov]

❌ 4 Tests Failed:

Tests completed	Failed	Passed	Skipped
6509	4	6505	55

View the top 3 failed test(s) by shortest run time

dimos.codebase_checks.test_docs_branding::test_docs_use_current_branding

Stack Traces | 0.013s run time

def test_docs_use_current_branding() -> None:
        """Fail if any file under docs/ spells the brand "DimOS" instead of "dimOS"."""
        hits = find_old_branding()
        if hits:
            listing = "\n".join(
                f"  - {p.relative_to(DIMOS_PROJECT_ROOT)}:{lineno}: {line.strip()}"
                for p, lineno, line in hits
            )
>           raise AssertionError(f'Found "DimOS" in docs/:\n{listing}\n\nThe brand is spelled "dimOS".')
E           AssertionError: Found "DimOS" in docs/:
E             - docs/usage/visualization.md:179: DimOS environment:
E           
E           The brand is spelled "dimOS".

hits       = [(PosixPath('.../docs/usage/visualization.md'), 179, 'DimOS environment:')]
listing    = '  - docs/usage/visualization.md:179: DimOS environment:'

dimos/codebase_checks/test_docs_branding.py:43: AssertionError

dimos.codebase_checks.test_inline_heavy_imports::test_heavy_imports_are_inline

Stack Traces | 1.99s run time

def test_heavy_imports_are_inline() -> None:
        """Fail if any file imports cv2/open3d/rerun at module level."""
        hits = find_eager_heavy_imports()
        if hits:
            listing = "\n".join(
                f"  - dimos/{f}:{line}: `{module}`"
                for f, lines in sorted(hits.items())
                for line, module in lines
            )
>           raise AssertionError(
                f"Found module-level import(s) of {'/'.join(HEAVY_MODULES)}:\n{listing}\n\n"
                "These libraries load large native extensions into every process that "
                "transitively imports the module. Import them inside the function or "
                "method that uses them; imports needed only for type annotations go "
                "under `if TYPE_CHECKING:`."
            )
E           AssertionError: Found module-level import(s) of cv2/open3d/rerun:
E             - dimos/msgs/image.py:22: `cv2`
E             - dimos/msgs/pointcloud.py:24: `open3d`
E           
E           These libraries load large native extensions into every process that transitively imports the module. Import them inside the function or method that uses them; imports needed only for type annotations go under `if TYPE_CHECKING:`.

hits       = {'msgs/image.py': [(22, 'cv2')], 'msgs/pointcloud.py': [(24, 'open3d')]}
listing    = '  - dimos/msgs/image.py:22: `cv2`\n  - dimos/msgs/pointcloud.py:24: `open3d`'

dimos/codebase_checks/test_inline_heavy_imports.py:92: AssertionError

dimos.core.test_build_native_modules::test_module_locks_pin_the_shared_code_as_it_is_now

Stack Traces | 3.61s run time

@pytest.mark.skipif(not _IN_GIT_CHECKOUT, reason="needs a git checkout to read refs")
    def test_module_locks_pin_the_shared_code_as_it_is_now() -> None:
        """Every module's pin on native/cpp or native/rust must hold this commit's tree.
    
        `nix flake lock` and `cargo build --locked` do not notice that the shared code
        moved -- they only check that the lock is complete -- so a change there can land
        while every module still builds against the revision before it. The revision is
        free to be older than HEAD; what must match is the content it pins.
        """
        stale = {
            _rel(lock): names
            for lock in _RELOCK.flake_locks()
            if (names := _RELOCK.stale_flake_inputs(lock))
        }
        stale |= {
            _rel(manifest): why
            for manifest in _RELOCK.cargo_manifests()
            if (why := _RELOCK.stale_cargo_pins(manifest))
        }
>       assert not stale, (
            "these modules pin shared code that is not the code in this commit, so they build "
            f"against the older version: {stale} -- push, run bin/relock-shared-flakes, commit"
        )
E       AssertionError: these modules pin shared code that is not the code in this commit, so they build against the older version: {'.../memory/rust/Cargo.toml': ['dimos-module @ 6663bcb6e5'], '.../realsense/rust/Cargo.toml': ['dimos-module @ 6663bcb6e5'], '.../v4l2/rust/Cargo.toml': ['dimos-module @ 6663bcb6e5'], '.../livox/rust/Cargo.toml': ['dimos-module @ 6663bcb6e5'], '.../pointlio/rust/Cargo.toml': ['dimos-module @ 6663bcb6e5'], '.../lidar/virtual_mid360/Cargo.toml': ['dimos-module @ 6663bcb6e5'], '.../dim_slam/rust/Cargo.toml': ['dimos-module @ 6663bcb6e5'], '.../ray_tracing/rust/Cargo.toml': ['dimos-module @ 6663bcb6e5'], '.../mls_planner/rust/Cargo.toml': ['dimos-module @ 6663bcb6e5'], '.../local_planner/rust/Cargo.toml': ['dimos-module @ 6663bcb6e5'], '.../fancy/rust/Cargo.toml': ['dimos-module @ 6663bcb6e5'], '.../depth2depth_cloud/rust/Cargo.toml': ['dimos-module @ 6663bcb6e5'], '.../dds/rust/Cargo.toml': ['dimos-module @ 6663bcb6e5'], '.../native-modules/rust/Cargo.toml': ['dimos-module @ 6663bcb6e5']} -- push, run bin/relock-shared-flakes, commit
E       assert not {'.../memory/rust/Cargo.toml': ['dimos-module @ 6663bcb6e5'], '.../realsense/rust/Cargo.toml': ['dimos-module @ 6663bcb6e5'], '.../v4l2/rust/Cargo.toml': ['dimos-module @ 6663bcb6e5'], '.../livox/rust/Cargo.toml': ['dimos-module @ 6663bcb6e5'], ...}

names      = []
stale      = {'.../memory/rust/Cargo.toml': ['dimos-module @ 6663bcb6e5'], '.../camera/realsense/...'dimos-module @ 6663bcb6e5'], '.../livox/rust/Cargo.toml': ['dimos-module @ 6663bcb6e5'], ...}
why        = ['dimos-module @ 6663bcb6e5']

dimos/core/test_build_native_modules.py:476: AssertionError

dimos.cli.test_cli_startup::test_go2_blueprint_import_does_not_pull_heavy_deps

Stack Traces | 5.08s run time

def test_go2_blueprint_import_does_not_pull_heavy_deps() -> None:
        # Importing the blueprint builds LCM handles. memq keeps them off the network.
        env = {**os.environ, "LCM_DEFAULT_URL": "memq://"}
        leaked = _leaked_modules(
            "import dimos.robot.unitree.go2.blueprints.smart.unitree_go2",
            BLUEPRINT_MUST_NOT_IMPORT,
            env=env,
        )
>       assert leaked == []
E       AssertionError: assert ['open3d', 'cv2'] == []
E         
E         Left contains 2 more items, first extra item: #x1B[0m#x1B[33m'#x1B[39;49;00m#x1B[33mopen3d#x1B[39;49;00m#x1B[33m'#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
E         
E         Full diff:
E         #x1B[0m#x1B[91m- []#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
E         #x1B[92m+ [#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
E         #x1B[92m+     'open3d',#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
E         #x1B[92m+     'cv2',#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
E         #x1B[92m+ ]#x1B[39;49;00m#x1B[90m#x1B[39;49;00m

env        = {'ACCEPT_EULA': 'Y', 'ACTIONS_ID_TOKEN_REQUEST_TOKEN': 'eyJhbGciOiJSUzI1NiIsImtpZCI6IjM4ODI2YjE3LTZhMzAtNWY5Yi1iMTY5LT...-version=2.0', 'ACTIONS_ORCHESTRATION_ID': '0dc1b01a-9ee8-89a5-9d91-90601f5ce33b.tests.ubuntu-24_04-arm_3_12_fal', ...}
leaked     = ['open3d', 'cv2']

dimos/cli/test_cli_startup.py:135: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}