[TomCC7]

Historical combined CDR PR — closed and unmerged. Current work is GitHub native stack #4372, eight layers. Open #4247 and use its native 1/8 stack badge to navigate all layers:

#4247 → #4366 → #4367 → #4368 → #4369 → #4370 → #4371 → #4387.

The runtime portion is now #4369; authoring, recording/replay and retirement have their separate layers. This historical PR and its branch/reviews remain preserved.

---

Problem and behavior

Fourth draft in the six-stage CDR stack, based on #4254. Typed runtime streams, native modules, browser command channels and recording paths use generated ROS2-layout messages and encapsulated CDR. Raw LCM and Zenoh remain transports; Python-object serialization stays separate.

The change extracts geometry, timestamp, NumPy/Open3D and viewer helpers, migrates runtime consumers to nested generated fields, integrates C++/Rust SDK codecs, advertises schemas to browser clients, and passes complete metadata to storage/replay. WebXR and phone command frames identify their channel and qualified message type explicitly. Legacy unframed bytes and old typed SQLite codecs are rejected rather than decoded through fallback paths. This is an intentional API/wire break.

External message authors now configure a project once and use dimos build or pip install .. The CLI and PEP 517/660 backend share one lightweight implementation; custom builds do not build the DimOS runtime. Built-in messages are a separate prebuilt package. Python, C++ and Rust reuse dependency-owned types, with exact versions, ABI/schema checks and full MCAP schema closure. The guide embeds the real compiled module examples rather than relying on shell-script links. See the guide and cumulative demo instructions.

Verification

Published HEAD: cb982817bde248baa70a6fa69cc9ecc275c43707.

This latest commit changes diagnostics only: xdist prints a dead worker's crash dump immediately, and long-running CI tests emit non-fatal thread stacks. Assertions, runtime code and execution limits are unchanged. A deliberately aborted isolated worker verified that the actual traceback reaches the controller while pytest still fails. On Python 3.11, the exact prior message wheel passed four focused ObserveSkill/collision tests and ten repeated ObserveSkill tests using in-memory LCM; this is not host UDP acceptance.

Latest exact-HEAD main CI, codegen/Jazzy, and nonpublishing full wheel matrix are terminal success, without a retry on this HEAD. Main includes Python 3.10/3.11/3.12 x86_64 and Python 3.12 ARM64; self-hosted jobs remain skipped. The previously affected ObserveSkill and collision-policy tests passed on Python 3.11. The earlier intermittent crash was not reproduced or assigned a proven root cause; its evidence remains below, and the diagnostic improvement does not claim a runtime fix.

Latest UX/build validation:

• 41 local acceptance tests pass: cross-package ownership, schema/version conflicts, incremental generation, deletion/rename, corruption repair, and clean isolated wheel/source/sdist/editable installs. Wheel install runs with no compiler on PATH; custom Python installs do not install DimOS or build Rust.
• Additional native ownership/MCAP acceptance passes: Python construction and assignment use the dependency Header class, C++ checks the same declaration, Rust uses the dependency type and codec trait, and an empty independent registry decodes the recording from embedded schemas only.
• The documented installed-backend build produces Python, CMake and Cargo artifacts. The real coordinator passes 20.5 → Python 21.5 → C++ 22.5 → Rust 23.5, preserving Header and sequence. CMake presets, exported SDK consumption, locked Rust build, documentation-source drift checks, strict typing across 10 files and repository hooks pass locally.
• At prior HEAD 7a862e470, message-codegen and independent Jazzy reference and nonpublishing full wheel matrix are success. Main CI attempt 1 ended red: Python 3.10 had one existing Zenoh async-RPC test exceed its unchanged 100 ms queue wait (6,335 passed); fail-fast cancelled Python 3.11. Both Python 3.12 lanes, native, Rust, Web, lint and executable documentation passed. Attempt 2 passed Python 3.10, but Python 3.11 lost a worker during test_observe_returns_published_frame and was cancelled at the existing 30-minute job limit with a collision-policy test outstanding. The main aggregate remains failure. The available log has no crash traceback; local collision-policy tests pass, while local ObserveSkill reproduction is blocked by the existing host LCM startup limitation. No further rerun, timeout relaxation or gate change was made. Earlier green runs below do not substitute for this final main result.
• Built-in native CMake/Rust artifacts use the message component version, not the runtime release version, and exclude installed custom providers. The relocated CMake archive round-trips CDR outside the checkout; codegen CI now verifies and retains these artifacts. Local offline codebase checks pass (136 tests).
• Artifacts remain proposal review packages, not a registry release. Message module names deliberately match normalized distribution names; dependency resolution rejects aliases that would obscure package ownership.

Prior migration evidence (before the new UX/build batch):

• Real detection fixtures now use canonical CDR from the hash-verified project recording. All 26 detection-type tests pass, including original real-scene bounding-box/point-cloud numerical tolerances and five-frame object history; only the manual live-publish demo is deselected. All 15 payloads reproduce byte-for-byte and independently decode with exact stamps. The new LFS fixture is 5.8 MB instead of the original 1.21 GB archive.
• Three offline G1 Rerun callback tests pass after replacing the remaining rich-cloud and stamped-point field accesses. Strict scoped typing and all changed-file hooks pass.
• At runtime HEAD b6884a186, the cumulative Python → C++ → Rust → Python Zenoh blueprint recorded nine CDR rows through the native Rust recorder with zero encoding errors, stopped every producer, independently decoded all rows, and replayed byte-identical per-stream values.
• At runtime HEAD b6884a186, a fresh-sdist root wheel build succeeded offline; a fresh isolated installation with no source PYTHONPATH verifies 153 provider types, CDR fields/stamps, complete schemas, helper modules, web assets, raw LCM retained and old dimos-lcm dependency absent.
• Earlier unchanged-source gates include 624 generated-message/voxel tests, all 18 memory pipeline tests using cached CPU CLIP, executable transform/eval docs, installed external wheel/sdist/CMake/Cargo consumers, external native Zenoh exchange/recording, and independent Foxglove-library decoding. DimSim Vite build, frozen Deno CLI check, three offline codec tests and schema drift check passed.

Exact-HEAD terminal results at 9cd5cedd3ea90864565048d747846f7ec3755389:

• Main CI: success, including Python 3.10/3.11/3.12 x86_64, Python 3.12 ARM64, Rust, native C++/Rust, Web, lint and executable documentation.
• Message-codegen and independent Jazzy reference: success.
• Nonpublishing full release wheel matrix: success, Python 3.10/3.11/3.12 on Linux x86_64, Linux ARM64 and macOS ARM64, plus sdist validation. Three platform review-artifact bundles are available in that run. No PyPI/TestPyPI publication occurred.

Self-hosted and large-test jobs were skipped; software CI does not establish hardware acceptance. The runtime-head main/codegen runs also passed before the final workflow-only change.

See real-data provenance and regression evidence, cumulative recording, and message authoring user stories. Actual Rerun browser screenshots and MCAP files are retained as review artifacts; human visual acceptance has not occurred. The final colored image/telemetry demo ran at 9cd5cedd3; its actual screenshot and MCAP have now been saved through the supported Library upload workflow, with delivery to the user still pending. Earlier b6884a186 screenshot/MCAP are saved in Library but have not been delivered to the user; the older 75b31da0b screenshot was delivered separately.

Remaining acceptance gates

This remains an unmerged draft proposal. The source/build dependency audit finds remaining old message-repository references only in M20 deployment and native DrDDS bridge files. Its proprietary SDK and ABI-compatible build environment are unavailable, so complete stage-6 dependency retirement is blocked there. No claim of whole-repository dependency removal is made.

Default-host LCM multicast remains unavailable without host-network changes; final local cumulative LCM acceptance is open. The same transport has earlier and CI conformance evidence, which is distinct from this final-host demo. Foxglove browser access redirects to sign-in; no credentials or account were created. Rerun rendered real captured data and custom fields, but human image/pose/custom-field approval and full stage-3/5 visual acceptance remain open. No physical robot operation, RPC prewarming, timeout relaxation, or runtime legacy typed-recording fallback was introduced.

The OpenSpec checklist remains authoritative. Stages 3.5–3.6, cumulative human/transport gates in 4–5, and complete dependency retirement/final acceptance in 6 cannot be marked complete from focused tests or screenshots alone. The latest exact-HEAD main CI, codegen/Jazzy and full wheel matrix passed. The prior Python 3.11 crash/job-timeout remains a recorded intermittent reliability concern, not a proven fixed defect. The external/manual gates above remain open.

The independent #3854–#3942 stack is not blocked by this CDR proposal. EpisodeStatus remains a future compatibility note; this draft imports none of that stack.

AI assistance

Codex implemented and tested these draft changes. Human review is pending.

Checklist

• I have read and approved the CLA.

[TomCC7]

Superseded by the user-approved seven-layer review chain: #4247 → #4252 → #4254 → #4362 → #4363 → #4364 → #4365. Four new stacked drafts split authoring, runtime/minimum storage compatibility, integrated recording/replay, and retirement. Their final tracked source tree equals this preserved complete reference. This PR is closed only to remove the combined review surface; feat/cdr-runtime-cutover and its history remain intact. New-layer CI is pending; M20/multicast/manual gates remain open. No merge or force push.