Antim/sim integration merged by leshy · Pull Request #1390 · dimensionalOS/dimos

leshy · 2026-03-01T16:38:05Z

No description provided.

Add native DimSim bridge and nav blueprint for browser-based 3D simulation with LCM transport. Uses globally installed dimsim CLI (https://jsr.io/@antim/dimsim). - DimSimBridge (NativeModule) manages the dimsim subprocess - DimSimTF publishes transform tree from odom - sim-nav blueprint wires bridge + TF + voxel mapping + A* + frontier exploration Usage: dimos run sim-nav TODO: - General eval workflow integration - Test headless integration

leshy · 2026-03-01T16:43:47Z

@greptileai go for it

greptile-apps

_{9 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps · 2026-03-01T16:51:43Z

+    def move(self, twist: Twist, duration: float = 0.0) -> bool:
+        """Send movement command to the simulator via cmd_vel."""
+        self.cmd_vel.publish(twist)


duration parameter is unused. Other move() implementations in the codebase (unitree, drone) use this to execute timed movements.

Dreamsorcerer · 2026-03-04T19:02:08Z

+    },
+}
+
+match global_config.viewer_backend:


I'd suggest we should be moving to a config that is not created at import time. To facilitate that in future, maybe this should all be wrapped into a function? Then if we remove the global_config variable in future this API wouldn't need to change.

- Remove out-of-order odom drop in tf_module — server-side physics publishes at 50Hz from a single source, timestamp filtering caused jitter by dropping near-identical timestamps over UDP multicast Sensor rates: - Odom: 50 Hz - Lidar: 10 Hz, 15,000 points, 4m range - RGB: 2 Hz, 960×432 JPEG - Depth: 2 Hz, 960×432 16UC1 Usage: dimos run sim-nav # normal DIMSIM_HEADLESS=1 DIMSIM_RENDER=gpu dimos run sim-nav # headless Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Viswa4599 · 2026-03-10T02:17:14Z

TODO

CI tests for dimsim integration
Integrate eval stack

…prints - bridge.py: download compiled binary from GitHub Releases (no Deno dependency), auto-symlink to ~/.local/bin, support DIMSIM_CONNECT_ONLY and DIMSIM_CHANNELS env vars for parallel eval instances - New e2e test: test_dimsim_nav.py — 6 smoke tests for sim-nav sensor pipeline (color, depth, odom, lidar, cmd_vel control, odom rate). Runs in CI with CPU rendering, force-kills port 8090 on setup/teardown to prevent leaks - Eval tests: EVALS_DIR now reads from ~/.dimsim/evals/ (installed by dimsim setup) instead of reaching into DimSim source tree. Added force-kill cleanup to both sequential and parallel eval fixtures - New sim blueprints: sim-agentic, sim-eval, sim-parallel-eval, sim-spatial, sim-temporal-memory - Simulation developer docs at docs/development/simulation.md

…merged # Conflicts: # dimos/robot/all_blueprints.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

@rpc

…wn, untrack generated artifacts - Fix global_config.viewer_backend → global_config.viewer (AttributeError) - Add @rpc decorator to DimSimBridge.start() override - Update self.config.executable after binary download in _maybe_build() - Add start_new_session=True to all test subprocess.Popen calls - Implement duration-based motion in DimSimTF.move() - Fix docstring filenames to match actual test file names - Untrack assets/temporal_memory/ and add to .gitignore

…it__.py files - Replace all removed shorthand aliases with ClassName.blueprint() pattern (CostMapper, VoxelGridMapper, ReplanningAStarPlanner, WavefrontFrontierExplorer, SpatialMemory, WebsocketVisModule, RerunBridgeModule, FoxgloveBridge) - Replace deleted dimos.agents.agent with MCP agent (McpServer + McpClient) - Replace dimos.perception.experimental.temporal_memory shorthand with direct import - Fix spec.Camera/Pointcloud import path (dimos.spec.perception) - Fix CameraInfo import (module vs class) - Fix LCM() constructor (remove autoconf param) - Fix rerun bridge import (RerunBridgeModule + _resolve_viewer_mode) - Remove all __init__.py files (not allowed in dimos) - Fix all_blueprints.py sorting and add dim-sim-bridge/dim-sim-tf modules - Add skipif_in_ci back to test_dimsim_nav (requires dimsim binary)

- Remove `from __future__ import annotations` from bridge.py and tf_module.py (broke Module port creation from type annotations) - Convert DimSimBridgeConfig from @DataClass to pydantic model (parent NativeModuleConfig is pydantic) - Fix all msg imports: use `from dimos.msgs.*.X import X` (module-level imports returned modules, not classes) - Replace `None` with named `_suppress` function in rerun visual_override (pydantic requires callable, lambdas not picklable) - Add missing PatrollingModule to sim_agentic blueprint - Fix os.killpg PermissionError on macOS in test teardown - Add venv bin to PATH in e2e test subprocess env

Fix update flow: prefer compiled binary over PATH in _resolve_paths(), always sync self.config.executable after version check, create symlink before running setup. Add eval creation section to simulation.md. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Fix update flow: prefer compiled binary over PATH in _resolve_paths(), always sync self.config.executable after version check, create symlink before running setup. Add eval creation section to simulation.md. Remove skipif_in_ci from test_dimsim_nav only. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

greptile-apps · 2026-04-30T05:14:38Z

+    def test_set_background(self, scene):
+        """set_background doesn't error."""
+        scene.set_background(0xFF0000)
+
+    def test_set_fog(self, scene):
+        """set_fog creates fog on the scene."""
+        scene.set_fog(color=0xCCCCCC, near=5, far=30)
+        result = scene.exec(
+            "return scene.fog ? { near: scene.fog.near, far: scene.fog.far } : null;"
+        )
+        assert result is not None
+        assert result["near"] == 5
+        assert result["far"] == 30


set_background and set_fog missing from SceneClient

set_background (line 158) and set_fog (lines 162–163) are called here but neither method exists in dimos/robot/sim/scene_client.py. Running these two tests will immediately raise AttributeError, causing test_set_background and test_set_fog to fail at the call site rather than exercising the intended assertion.

greptile-apps · 2026-04-30T05:14:39Z

+                    try:
+                        logger.info(f"Downloading dimsim {latest_ver} for {system}/{machine}...")
+                        urllib.request.urlretrieve(url, str(dimsim))
+                        dimsim.chmod(
+                            dimsim.stat().st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH
+                        )
+                        # macOS quarantines downloaded binaries — clear all xattrs
+                        if system == "darwin":
+                            subprocess.run(
+                                ["xattr", "-c", str(dimsim)],
+                                capture_output=True,
+                            )
+                        dimsim_path = str(dimsim)
+                        downloaded = True
+                        logger.info("dimsim binary installed.")
+                    except Exception as exc:
+                        logger.warning(
+                            f"Compiled binary not available ({exc}), trying deno fallback..."
+                        )


Downloaded binary executed without integrity verification

The binary is fetched from GitHub Releases and made executable without any checksum or signature check. If the release artifacts are replaced (compromised account, CI/CD supply-chain attack, or DNS hijack), the malicious binary will be silently installed and executed on the next run. HTTPS provides transport security but not artifact integrity. Consider verifying a SHA-256 or GPG signature published alongside the release before chmod +x and execution.

Bumping kern.ipc.maxsockbuf, net.inet.udp.recvspace, and net.inet.udp.maxdgram to 16M (commits f7d7d0c + 2173028) was applied successfully on the runner this time, but sim-eval and sim-nav fixtures still timed out waiting for /color_image. So maxdgram was not the root cause — restoring the workflow to its prior state to avoid touching shared CI infra without a clear win.

greptile-apps · 2026-04-30T06:41:17Z

+    def _wait_for(self, msg_type: str, timeout: float = 120) -> dict:
+        self.ws.settimeout(timeout)
+        while True:
+            raw = self.ws.recv()
+            if isinstance(raw, bytes):
+                continue
+            if not raw:
+                continue
+            try:
+                msg = json.loads(raw)
+            except json.JSONDecodeError:
+                continue
+            msg_ch = msg.get("channel", "")
+            if msg.get("type") == msg_type and (not msg_ch or msg_ch == self.channel):
+                return msg


_wait_for timeout not enforced under 50 Hz message stream

ws.settimeout(timeout) is set once before the loop, but since the bridge sends pose updates at 50 Hz, each individual recv() call returns within ~20 ms — never blocking long enough to raise a WebSocketTimeoutException. This means _wait_for("workflowComplete", ...) will loop indefinitely if the workflow hangs, stalling CI with no termination.

The same deadline-tracking fix described for test_dimsim_eval.py is needed here:

def _wait_for(self, msg_type: str, timeout: float = 120) -> dict: deadline = time.time() + timeout while True: remaining = deadline - time.time() if remaining <= 0: raise TimeoutError(f"Timed out waiting for '{msg_type}' on {self.channel}") self.ws.settimeout(min(remaining, 5.0)) raw = self.ws.recv() if isinstance(raw, bytes) or not raw: continue try: msg = json.loads(raw) except json.JSONDecodeError: continue msg_ch = msg.get("channel", "") if msg.get("type") == msg_type and (not msg_ch or msg_ch == self.channel): return msg

- scene_client.py: add dict[str, Any] type params, cast Any returns, type: ignore for untyped websocket.connect - dimos_cli_call.py: assert process is not None before pid access - sim_parallel_eval.py: import Image class (not module), import DIMOS_PROJECT_ROOT from dimos.constants - test_dimsim_eval.py: skipif_in_ci on TestSimEvalSequential — headless Chrome WebGL doesn't produce frames on the macOS runner so workflows time out before completion. Parallel suite already skipped.

Greptile flagged that EvalClient._wait_for set ws.settimeout(timeout) once before the recv loop, but the bridge streams pose at 50 Hz so each recv() returns in ~20 ms. The single up-front timeout never trips and the loop hangs indefinitely if a workflow gets stuck — pytest-timeout masked it but the safety net would be the only thing terminating us. Switch all three _wait_for methods to deadline-tracked: recompute the remaining budget on every iteration, cap each recv() at min(remaining, 5s) so we exit promptly when the deadline lapses. Also @pytest.mark.skipif_in_ci on TestSceneClient — same headless Chrome WebGL frame-pipeline issue as the eval suites; nav stays enabled because that's the one we're actively diagnosing.

The fixture-launched dimsim subprocess defaulted to "gpu" when the env var wasn't set, which doesn't match dimos.robot.sim.bridge runtime default (cpu). On the headless macOS CI runner GPU rendering doesn't produce frames, so /color_image never publishes and the suite times out.

test_dimsim_nav has been failing in CI on /color_image timeout but captures dimsim subprocess output to DEVNULL, so we have no signal on which stage of the pipeline (Chrome launch, WebGL init, JPEG encoding, multicast publish) is breaking. Two new layered classes: TestDimSimBinaryOnly bare ~/.dimsim/bin/dimsim — proves the simulator, headless Chrome, WebGL, and LCM multicast all work on the runner without dimos. TestDimosSimBasic dimos sim-basic — minimum dimos blueprint above dimsim, no nav/agent stack. Each fixture captures subprocess output to a tempfile and dumps on teardown so pytest surfaces it on failure. If TestDimSimBinaryOnly fails: dimsim itself is broken on the runner. If 1 passes and TestDimosSimBasic fails: dimos blueprint wiring issue. If 1+2 pass and test_dimsim_nav fails: nav-stack issue.

…I logs Previous run had empty captured-output blocks even though dimos sim-basic clearly started and opened port 8090. Cause: Popen(stdout=open(file)) relies on the child's stdio to flush on exit, which Deno and Python with block-buffered stdout do not do reliably under SIGTERM. Add dimos/e2e_tests/_subprocess_log.py with launch_with_streaming_log() that reads PIPE in a daemon thread and writes each line with explicit flush(). Also set PYTHONUNBUFFERED=1 in the dimos subprocess env. Wire it into TestDimSimBinaryOnly, TestDimosSimBasic, and TestSimNav. Replaces the prior DEVNULL fallback in test_dimsim_nav so the dimsim subprocess output now appears inline in CI logs on every run.

Sim-nav diagnostic loop has been blocked by the ~20min wait through unrelated tests. List the dimsim files explicitly before the dimos/ walk so pytest collects them first; deduplication handles the overlap.

greptile-apps · 2026-05-02T03:18:02Z

+# Overridable via DIMSIM_CAMERA_FOV env var.
+_DIMSIM_WIDTH = 640
+_DIMSIM_HEIGHT = 288
+_DIMSIM_FOV_DEG = int(os.environ.get("DIMSIM_CAMERA_FOV", "46"))


Default FOV mismatch causes wrong CameraInfo

_DIMSIM_FOV_DEG defaults to 46 when DIMSIM_CAMERA_FOV is unset, but bridge.py documents DimSim's own default as 80° (# Camera FOV in degrees. None = use DimSim default (80).) and passes no --camera-fov flag when camera_fov is None. The same 46° default is used in sim_basic.py's _static_camera_pinhole. When no env var is set, the simulator runs at 80° but Python publishes a CameraInfo computed for 46°, producing incorrect focal lengths (fx) across the full sensor pipeline (projection, depth scale, SLAM, nav costmaps).

Streamed subprocess output finally revealed the root cause that DEVNULL was hiding for hours: error: Uncaught (in promise) Error: browserType.launch: Chromium distribution 'chrome' is not found at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome Run "npx playwright install chrome" DimSim's bridge launches headless via Playwright with channel: "chrome" because bundled Chromium has broken WebGL on macOS. The path is hard-coded by Playwright, so we install real Chrome via brew cask if the runner doesn't already have it. This crash explains every /color_image timeout we've been chasing — dimsim opens port 8090, then immediately exits 1 when Playwright fails to find Chrome, and the LCM frame pipeline never runs.

v0.2.3 fixed the Chrome dependency but introduced a new failure: error: Uncaught (in promise) TimeoutError: page.waitForFunction: Timeout 30000ms exceeded. That's launcher.ts waiting for window.__dimosBridge after the page loads. Chromium launches and navigates, but the engine never finishes init. No browser console output captured because hookPageConsole filters most messages without DIMSIM_VERBOSE=1. Enable verbose so the next CI run shows every console.log/warn/error from the page, including any WebGL/SwiftShader init errors.

Previous attempt — listing dimsim files explicitly before dimos/ in the same pytest invocation — caused pytest to collect ONLY the explicit files (11 items) and skip the entire rest of the suite (~1900 tests). Bad regression in CI coverage. Split into two steps instead: 1. dimsim e2e tests run first as their own pytest invocation, with continue-on-error so the rest of CI still proceeds when they fail 2. Full dimos/ test suite runs after, exactly as before Sim-nav signal lands in step 1 (~5min after Run tests starts) and the full suite still runs in step 2.

Bridge takes ~78s in CI before first sensor frame: 43s engine init plus 35s for 26MB Rapier snapshot upload and physics init. The 60s waits and 120s pytest timeout were both too tight, causing color_image to fail ~18s before the bridge actually started publishing. - spy fixture and color_image waits: 60s -> 120s - odom publish wait: 30s -> 60s - pytest --timeout: 120s -> 300s

The full dimos/ pytest step has --timeout=120, which is too tight for the sim_nav fixture cold start (~98s engine init + Rapier snapshot) and gets SIGTERMed before sensors finish coming up. Skip the dimsim files in step 2 — they already run in the dedicated e2e step with --timeout=300. Also bump per-test odom/lidar waits 30s -> 200s to absorb the slower publish cadence under headless SwiftShader CPU rendering.

The "Sensor publishing active" log line is misleading — it only confirms the WS is connected. Physics-driven topics (/odom, /lidar) cannot publish until the browser's 26MB Rapier snapshot reaches the bridge AND is restored server-side. Locally on M4 this WS transfer takes ~60s; on the self-hosted CI runner under load it can take much longer, so any wall-clock timeout is fragile. Add wait_for_log_pattern() in _subprocess_log.py that tails the streamed subprocess log for a regex. Use it in sim_nav and dimos_sim_basic fixtures to block until "Rapier snapshot restored" appears (timeout 600s). Bump pytest --timeout to 900s so the first test in each class can carry the full fixture setup budget (port + snapshot + color_image).

DimSim v0.2.6 splits the 26MB Rapier snapshot into ~256KB chunks so the WebSocket pump can drain even when the browser main thread is CPU- saturated by SwiftShader (the failure mode on the EC2 mac runner). Verified locally: TestSimNav 6/6 pass in cpu mode, robot moves 1.84m on cmd_vel, odom at 35Hz.

Software rendering (SwiftShader) is fundamentally too slow for a complex scene on the Mac Mini-class self-hosted runner. The renderer saturates the JS main thread, which starves the WebSocket pump and pushes the 26MB Rapier snapshot transfer past 10 minutes. Even the v0.2.6 chunked send loses to setTimeout(0) yields stretching to ~7s each under render load. The runner already has system Chrome (brew --cask google-chrome) which fixes the broken-WebGL-on-macOS issue with bundled Chromium. Setting DIMSIM_RENDER=gpu + DIMSIM_CHROME_CHANNEL=chrome gives us hardware Metal via ANGLE — main thread idle, snapshot drains in seconds. Also fix TestDimSimBinaryOnly's hardcoded "--render cpu" to honor the DIMSIM_RENDER env override.

greptile-apps · 2026-05-05T01:08:31Z

+        str(PORT),
+        *extra_args,
+    ]
+    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)


stdout=subprocess.PIPE with no reader will deadlock the server process. Once the OS pipe buffer fills (~64 KB of log output), the server blocks on its next write() syscall, stops accepting WebSocket connections, and every SceneClient.exec() call times out — hanging the entire test module. Use subprocess.DEVNULL to discard output, or the launch_with_streaming_log helper from _subprocess_log.py (which drains the pipe in a background thread), as done in test_dimsim_nav.py and test_dimsim_pipeline.py.

Suggested change

proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

proc = subprocess.Popen(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

Greptile P1: _start_server used subprocess.PIPE with no reader. Once the OS pipe buffer (~64 KB) fills with server log output, the server blocks on its next write() syscall, stops accepting WebSocket connections, and every SceneClient.exec() call times out — hanging the entire test module. Switch to launch_with_streaming_log (the helper test_dimsim_nav.py and test_dimsim_pipeline.py already use): subprocess stdout is read line-by- line in a daemon thread, so the pipe never fills. Stash the log path on the Popen object so _stop_server can clean it up without changing every caller's signature, and dump it on startup failure for diagnostics.

Cleanup of the hacks accumulated while debugging cpu-mode timing: - Drop the dedicated 'Run dimsim e2e tests (fast feedback)' step. It existed only to give the dimsim suite a longer budget than the main pytest step. With GPU rendering the full TestSimNav class runs in ~30s, so it fits the regular suite. - Drop continue-on-error: true. It was hiding dimsim failures from the build; now they block CI like every other test. - Drop the --ignore flags (test_dimsim_pipeline.py, test_dimsim_nav.py) from the unified Run tests step. - Move DIMSIM_RENDER=gpu and DIMSIM_CHROME_CHANNEL=chrome to the unified step env so the dimsim tests pick them up. - Bump --timeout 120 -> 240 to absorb the Rapier snapshot warmup that rolls into the first TestSimNav test. - Drop the trailing "macOS CI retrigger" debug comment. Test-side timeouts pulled back from cpu-mode-pessimistic values: - wait_for_log_pattern("Rapier snapshot restored"): 600s -> 60s - spy fixture color_image wait: 120s -> 30s - per-test odom/lidar waits: 200s -> 30s - TestDimSimBinaryOnly + TestDimosSimBasic color_image waits: 120 -> 30 - TestDimosSimBasic odom wait: 200 -> 30

test_odom_rate sampled over 2s = ~9 messages, too noisy against a hard >5Hz threshold (one CI run got 4.5Hz, prior got 6.5Hz on the same code). Widen to 5s (~22 messages) for a stable mean. Also drop TestDimSimBinaryOnly (and its dimsim_only / dimsim_only_spy fixtures plus _ensure_dimsim_binary helper). Bare-binary stage was a diagnostic layer carved out when we were debugging cpu-mode WebGL on the runner; now redundant since TestDimosSimBasic already exercises WebGL, multicast, and JPEG publishing through the dimos blueprint.

DimSim v0.2.7 drops lidar 10Hz -> 5Hz. On the M4 lidar scans take ~30ms each, so 10Hz consumed ~300ms/sec of bridge-thread time. On the slower EC2 mac runner each scan takes ~60-80ms, so 10Hz was using 600-800ms/sec, starving the physics setInterval and capping odom at 4.6Hz. 5Hz halves the per-second blocking budget. Expecting odom rate on the runner to climb from 4.6 -> ~9-10 Hz. Local M4 was unable to confirm the win directly — it has so much headroom that physics rate doesn't move regardless — but per-handler profiling confirmed lidar IS the dominant blocker (300ms/sec) vs pose-cb (4ms/sec) and sensor relay (1ms/sec).

Resolves CONFLICTING state that was blocking GitHub Actions from firing pull_request: synchronize on this branch. One conflict in dimos/robot/all_blueprints.py — both branches added entries to the alphabetically-sorted dict. Interleaved openarm-* (from dev) before sim-* (ours). Verified with the auto-generator test test_all_blueprints_is_current.

codecov · 2026-05-07T02:00:28Z