[TomCC7]

Linear project: Message Encoding Refactor

Problem

Reviewers need the generator, bulk message definitions/packages and transport integration in their respective stack layers.

Solution

External custom-message authoring, dependency type reuse, dimos build and Python package backend. Packages upstream generator resources and dependencies in the standalone backend, preserving compiler-free Python source/wheel/editable installation and shared dependency identities.

Layer 4/8 of native stack #4446. Published HEAD: 85fd0a408cbc8ae6b415f8b284176a8d9c801c90. Base: cc/feat/cdr-mcap-viewers. Normal forward commits/merges; existing ancestry and reviews are preserved, with no force push.

Parent-relative git diff (binary lockfile excluded from textual line counts): 133 files changed, 63702 insertions(+), 11574 deletions(-).

Layout-repair validation: unchanged namespace-policy test plus generator tests: 41 passed; ownership/project checks: 19 passed; clean source/wheel/sdist/editable installation with Cargo/rustc/CMake/C++ disabled: 1 passed. A fresh backend wheel has api modules/stubs and no stale package initializers. C++/Rust/schema outputs are byte-identical to the pre-layout validated outputs; final built-in generation --check passed. Scoped strict typing and hooks passed. These focused checks are not a complete new-HEAD CI pass.

Production Rust declarations now come from pinned ros2msg 0.5.3 in a normal Cargo build.rs; Python installation only packages its sources and does not invoke Cargo. C++ declaration and CDR field traversal use pinned upstream ROSIDL/FastRTPS templates, with documented strict-bool patches. Python uses rosbags-generated declarations and CDR traversal with scoped malformed-input checks; the separate handwritten wire-layout walker was removed. Default, bounds, ownership and Sequence compatibility adapters remain. PR1 core: ten files including both helpers total 1,380 physical lines versus the original eight-file 1,463. Including maintained typing stubs gives 1,493 -> 1,444, a net 49-line reduction (83 is core-only). Full-stack message_codegen maintained non-test sources total 2,573 lines including stubs; CLI/SDK sources elsewhere are outside that count. Vendored sources and generated/copied artifacts are disclosed separately. Runtime native consumers use the distributed message crate; the redundant four-file Python/Cargo bootstrap (126 lines) was removed. No generator code was hidden in uncounted helpers.

Exact-new-HEAD remote CI is pending; no reruns or manual dispatches were requested for this batch. Historical runs do not verify these commits. Benchmark investigation remains paused, without a waiver or weakened gate.

Complete proposal walkthrough. M20 SDK/access, host multicast and human viewer acceptance remain external; no hardware operation, registry release or merge is claimed.

Replaces closed, unmerged #4362. Original refs/history remain preserved.

Upstream generator contracts, compatibility adapters and maintenance counts.

Namespace repair: five nonempty upstream initializers and two matching typing stubs are relocated to ordinary api modules; the empty parser initializer is omitted. Imports/EmPy templates reference modules directly. Licenses and original upstream source paths/hashes remain recorded, with the relocation and omitted empty file documented in manifests. test_no_init_files is unchanged; no blanket vendor exemption was added. The earlier mypy vendor-discovery boundary remains separate, and first-party strict settings remain enabled. The coauthor policy check remains unresolved; no attribution removal/history rewrite, force push, benchmark gate change, rerun or manual CI dispatch was requested.

[codecov]

❌ 3 Tests Failed:

Tests completed	Failed	Passed	Skipped
6439	3	6436	56

View the top 3 failed test(s) by shortest run time

::dimos.message_codegen.test_builtin_codec

Stack Traces | 0s run time

ImportError while importing test module '.../dimos/message_codegen/test_builtin_codec.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
dimos/message_codegen/test_builtin_codec.py:21: in <module>
    from dimos_generated.builtin_interfaces.msg import Time
E   ModuleNotFoundError: No module named 'dimos_generated'

dimos.codebase_checks.test_docs_branding::test_docs_use_current_branding

Stack Traces | 0.06s run time

def test_docs_use_current_branding() -> None:
        """Fail if any file under docs/ spells the brand "DimOS" instead of "dimOS"."""
        hits = find_old_branding()
        if hits:
            listing = "\n".join(
                f"  - {p.relative_to(DIMOS_PROJECT_ROOT)}:{lineno}: {line.strip()}"
                for p, lineno, line in hits
            )
>           raise AssertionError(f'Found "DimOS" in docs/:\n{listing}\n\nThe brand is spelled "dimOS".')
E           AssertionError: Found "DimOS" in docs/:
E             - docs/development/message-source-packages.md:1: # CDR generation in DimOS
E             - docs/development/message-source-packages.md:3: DimOS uses the published `dimos-cdrgen==0.1.0` package directly. Its parser,
E             - docs/development/message-source-packages.md:17: There is no DimOS compatibility distribution. Existing consumers of
E             - docs/development/message-source-packages.md:21: DimOS retains two adapters:
E             - docs/development/message-source-packages.md:27: for DimOS C++ consumers.
E             - docs/development/message-source-packages.md:42: distribution, Python type identity and the DimOS C++ header against literal CDR
E           
E           The brand is spelled "dimOS".

hits       = [(PosixPath('.../docs/development/message-source-packages.md'), 1, '# CDR generation in DimO...t/message-source-packages.md'), 42, 'distribution, Python type identity and the DimOS C++ header against literal CDR')]
listing    = '  - docs/development/message-source-packages.md:1: # CDR generation in DimOS\n  - docs/development/message-source-pac...lopment/message-source-packages.md:42: distribution, Python type identity and the DimOS C++ header against literal CDR'

dimos/codebase_checks/test_docs_branding.py:43: AssertionError

dimos.message_codegen.test_integration::test_installed_message_consumer

Stack Traces | 4.96s run time

tmp_path = PosixPath('.../pytest-0/popen-gw3/test_installed_message_consume0')

    def test_installed_message_consumer(tmp_path: Path) -> None:
        project = tmp_path / "consumer"
        source = project / ".../consumer_msgs/msg/Tick.msg"
        source.parent.mkdir(parents=True)
        source.write_text("int32 value\n")
        (project / "pyproject.toml").write_text("""[build-system]
    requires = ["setuptools>=77", "wheel", "dimos-cdrgen==0.1.0"]
    build-backend = "setuptools.build_meta"
    [project]
    name = "dimos-cdr-consumer"
    version = "0.1.0"
    dependencies = ["dimos-cdrgen==0.1.0"]
    [tool.dimos.messages]
    source = "interfaces"
    """)
        (project / "setup.py").write_text(
            "from pathlib import Path\nfrom setuptools import setup\n"
            "from dimos_cdrgen.source_package import setup_arguments\n"
            "setup(**setup_arguments(Path(__file__).resolve().parent))\n"
        )
        run(sys.executable, "-m", "build", "--no-isolation", cwd=project)
        wheel = next((project / "dist").glob("*.whl"))
        installed = tmp_path / "installed"
>       run(
            sys.executable,
            "-m",
            "pip",
            "install",
            "--no-deps",
            "--target",
            str(installed),
            str(wheel),
            cwd=tmp_path,
        )

installed  = PosixPath('.../pytest-0/popen-gw3/test_installed_message_consume0/installed')
project    = PosixPath('.../pytest-0/popen-gw3/test_installed_message_consume0/consumer')
source     = PosixPath('.../pytest-0/popen-gw3/test_installed_message_consume0/consumer/.../consumer_msgs/msg/Tick.msg')
tmp_path   = PosixPath('.../pytest-0/popen-gw3/test_installed_message_consume0')
wheel      = PosixPath('.../pytest-0/popen-gw3/test_installed_message_consume0/consumer/dist/dimos_cdr_consumer-0.1.0-py3-none-any.whl')

dimos/message_codegen/test_integration.py:83: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
dimos/message_codegen/test_integration.py:32: in run
    subprocess.run(command, cwd=cwd, check=True, capture_output=True, text=True)
        command    = ('.../dimos/dimos/.venv/bin/python', '-m', 'pip', 'install', '--no-deps', '--target', ...)
        cwd        = PosixPath('.../pytest-0/popen-gw3/test_installed_message_consume0')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

input = None, capture_output = True, timeout = None, check = True
popenargs = (('.../dimos/dimos/.venv/bin/python', '-m', 'pip', 'install', '--no-deps', '--target', ...),)
kwargs = {'cwd': PosixPath('.../pytest-0/popen-gw3/test_installed_message_consume0'), 'stderr': -1, 'stdout': -1, 'text': True}
process = <Popen: returncode: 1 args: ('.../dimos/dimos/.venv/bin/python...>
stdout = ''
stderr = '.../dimos/dimos/.venv/bin/python: No module named pip\n'
retcode = 1

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.
    
        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them,
        or pass capture_output=True to capture both.
    
        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.
    
        If timeout (seconds) is given and the process takes too long,
         a TimeoutExpired exception will be raised.
    
        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.
    
        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.
    
        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE
    
        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE
    
        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '('.../dimos/dimos/.venv/bin/python', '-m', 'pip', 'install', '--no-deps', '--target', '.../pytest-0/popen-gw3/test_installed_message_consume0/installed', '.../pytest-0/popen-gw3/test_installed_message_consume0/consumer/dist/dimos_cdr_consumer-0.1.0-py3-none-any.whl')' returned non-zero exit status 1.

capture_output = True
check      = True
input      = None
kwargs     = {'cwd': PosixPath('.../pytest-0/popen-gw3/test_installed_message_consume0'), 'stderr': -1, 'stdout': -1, 'text': True}
popenargs  = (('.../dimos/dimos/.venv/bin/python', '-m', 'pip', 'install', '--no-deps', '--target', ...),)
process    = <Popen: returncode: 1 args: ('.../dimos/dimos/.venv/bin/python...>
retcode    = 1
stderr     = '.../dimos/dimos/.venv/bin/python: No module named pip\n'
stdout     = ''
timeout    = None

../...../_temp/uv-python-dir/cpython-3.12.15-linux-aarch64-gnu/lib/python3.12/subprocess.py:571: CalledProcessError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[github-actions]

go2 replay realtime (arm64)

Details

Benchmark suite	Current: ce665bf	Previous: 4bee180	Ratio
peak memory	1598.879 MB	1566.871 MB	1.02
peak threads	369 threads	364 threads	1.01
network (transport)	2729.56 MB	2730.605 MB	1.00
disk write	1.66 MB	1.449 MB	1.15
instructions	115.511 G	115.402 G	1.00

This comment was automatically generated by workflow using github-action-benchmark.