Skip to content

feat: integrate explicit CDR recording conversion and fidelity audit - #4704

Closed
TomCC7 wants to merge 2 commits into
cc/feat/cdr-keyboard-teleopfrom
cc/feat/cdr-recording-converter-integration
Closed

TomCC7 wants to merge 2 commits into
cc/feat/cdr-keyboard-teleopfrom
cc/feat/cdr-recording-converter-integration

Conversation

@TomCC7

@TomCC7 TomCC7 commented Oct 20, 2026

Copy link
Copy Markdown
Member

Add explicit dimos mem convert for historical MCAP/memory SQLite recordings. Conversion uses an allowlisted retained LCM decoder and the installed generated CDR registry, embeds full dependency schemas, and preserves an adjacent per-row audit. Runtime transports never fall back to legacy decoding.

Parent: #4703. Message Encoding Refactor

This integrates the previously reviewed converter design into the restructured stack with the current registry/value ownership and storage contracts; the prior converter branch and its evidence are preserved.

Changes Files Added Removed
Conversion, batch preflight and existing mem CLI 3 793 0
Fidelity/failure/CLI tests 2 578 0
Self-contained workflow documentation 1 79 0
Installed-package CI 1 7 0

Manual test with an existing trusted local recording (replace old.db with its path, choose new output names):

dimos mem convert old.db review.mcap --dry-run
dimos mem convert old.db review.mcap
dimos mem summary review.mcap

Use .db output to preserve existing vectors; MCAP is recording-only and rejects vectors. For a directory, dimos mem convert ./old-recordings ./cdr-recordings --format mcap --dry-run inventories every candidate without outputs or downloads. Removing --dry-run performs the batch. See the embedded-code walkthrough.

Existing files/reports and competing writers are never overwritten. Originals are read-only. A late single-file payload error publishes no output; partial batch success is explicitly reported with nonzero exit. Pickle, unsupported/custom types, external component stores and attachments are rejected. Header.seq and SQLite IDs/value/tags are audited rather than silently dropped. JPEG bytes are not recompressed. Dry-run validates metadata/schema/counts; full payload integrity is checked during conversion.

Validation:

  • 34 source + 34 isolated installed-wheel tests; strict 3 production files; configured scoped hooks passed except unavailable Nix lint.
  • Actual installed root CLI dry-run/conversion of already hash-verified cached project go2_mid360_stairs.db: all 4,451 messages, five streams. mem summary reads every stream.
  • Actual go2_short.db -> CDR SQLite: all 2,546 messages, four streams, 108 exact 512-dimensional vectors and their observation associations preserved.
  • Independent rosbags decoder from embedded schemas compares every mapped field/JPEG byte, source/envelope timestamps, sequences and vector IDs; both source SHA256 values unchanged. Source SHA256: stairs dbb725b763f9d268c47c56e285f5c0150923f488e8fa03b6cdffd26d0dbd125e; go2_short 4f056a042d58ad1dbc8248e408bd61d98bb90e5989cf0520c5400a1088dfe1b9.
  • Wheel SHA256 3a152b4b052af8cc66b82f1732ece372d6acfdd15c6906566accdacbc75525fa. Vector tests use official CPU PyTorch only, no model weights or GPU work.

Remaining: dimos mem rerun still assumes legacy to_rerun() methods and skips plain generated values; its external-adapter migration is the next layer. Foxglove local human acceptance/sign-in remains external. Full consumer graphs and legacy dependency retirement remain open. No CI polling, package publication, hardware, history rewrite or merge.

@TomCC7 TomCC7 closed this Oct 20, 2026
@TomCC7
TomCC7 deleted the cc/feat/cdr-recording-converter-integration branch October 20, 2026 18:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant