Improv = REST over serial: push device-model config (incl. eth-only)

.github/workflows/test.yml

@@ -0,0 +1,61 @@
+name: Test
+
+# Fast host-side unit tests for the Python (MoonDeck / build scripts) and JS
+# (web installer) code that the C++ ctest/scenario suites can't reach. Runs on
+# every PR and on pushes to main/next-iteration. Scoped to the paths these tests
+# cover so a docs-only or pure-firmware change doesn't spend a runner here.
+#
+# Today this pins the Improv frame wire format (test/python + test/js assert a
+# shared golden vector so the device C++, Python, and JS builders can't drift).
+# New Python/JS unit suites land under test/python and test/js and run here.
+
+# Paths cover every input to the host-side tests: the Python/JS sources under test
+# (scripts, docs/install), the test files themselves, AND the device-side C++ frame
+# contract (src/core/Improv*.h + the platform handler) — a wire-format change in the
+# firmware must run the cross-language golden-vector tests so it can't drift from the
+# Python/JS builders silently. pull_request gates every PR; push runs main only (a
+# direct-to-main hotfix). A PR branch is covered by pull_request alone — listing
+# feature branches under push too would double-run every PR (push + pull_request).
+on:
+  pull_request:
+    paths: &test-paths
+      - 'scripts/**'
+      - 'docs/install/**'
+      - 'src/core/ImprovFrame.h'
+      - 'src/core/ImprovOpReassembler.h'
+      - 'src/platform/esp32/platform_esp32_improv.cpp'
+      - 'test/python/**'
+      - 'test/js/**'
+      - '.github/workflows/test.yml'
+  push:
+    branches:
+      - main
+    paths: *test-paths
+
+permissions:
+  contents: read
+
+jobs:
+  python:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@v3
+      # pytest + pyserial come from the test file's inline PEP-723 block; passing
+      # them via --with is the explicit, discovery-friendly form (a bare `pytest
+      # <dir>` doesn't honour a test file's own inline deps).
+      - name: pytest
+        run: uv run --with pytest --with pyserial pytest test/python -q
+
+  js:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+      # Node's built-in test runner — no npm install, no package.json. The glob
+      # form is required: a bare directory arg is treated as a module to execute.
+      - name: node --test
+        run: node --test "test/js/**/*.test.mjs"

CLAUDE.md

@@ -55,7 +55,7 @@ The design rationale for each rule below lives in [docs/architecture.md](docs/ar
 
 Then check the recommendation against [§ Principles](#principles) (minimalism, data over objects, concrete first) and propose it as a question, not a fait accompli. The product owner picks; the agent implements only what was picked. If the picked option turns out to need a follow-up change (e.g. an updated naming convention to make the new layout consistent), surface that *before* starting the move so it's a single coherent refactor, not three round-trips.
 
-**Plan before implementing.** Use `/plan` mode before every feature. Review plans for: unnecessary files, inheritance where structs suffice, modifications outside the relevant directory. Reject and regenerate bad plans.
+**Plan before implementing.** Use `/plan` mode before every feature. Review plans for: unnecessary files, inheritance where structs suffice, modifications outside the relevant directory. Reject and regenerate bad plans. **Save every approved plan** to `docs/history/plans/` named `Plan-YYYYMMDD - <title>.md` (ISO-8601 date order so the directory sorts chronologically, e.g. `Plan-20260620 - Improv-as-REST.md` for 2026-06-20), as the first implementation step. The plan is the design record that complements `decisions.md` (the lesson record): the plan says what we set out to build and why; decisions.md captures what we learned doing it. **These saved plans are a reference archive for the product owner — agents WRITE a plan when creating one, but do NOT read the existing plan files for context unless the product owner explicitly points to one** (they're under the "Never automatically" rule below alongside the rest of `docs/history/`). Like the rest of `history/`, plans are pruned under *Mandatory subtraction* once their design is fully absorbed into the code + module specs.
 
 **Use `uv` for every Python invocation.** Never type `python` or `python3` directly; always go through `uv run` (e.g. `uv run scripts/build/build_desktop.py`, `uv run python -c "…"`). This applies to shell commands, CMake `add_custom_command` / `execute_process`, documentation examples, and anything that shells out. In CMake, resolve `find_program(UV_EXECUTABLE NAMES uv REQUIRED HINTS "$ENV{USERPROFILE}/.local/bin" "$ENV{HOME}/.local/bin")` once and use `${UV_EXECUTABLE} run python …` thereafter. Reason: uv manages the project venv and is the project standard ([scripts/MoonDeck.md](scripts/MoonDeck.md)); bare `python3` isn't on PATH on Windows (and macOS Python Launcher pops a Store prompt). If you catch yourself about to type `python`, stop and prefix with `uv run`.
 
@@ -85,7 +85,7 @@ Each commit produces visible output. The product owner picks what to build next.
 
 1. **Pick what to build.** One layout, one effect, one driver, one modifier, one system module: whatever adds the next useful capability.
 2. **Spec it.** Write (or review) the module's spec. A spec-in-progress is a plain `.md` in `docs/backlog/` (like any other forward-looking note); when the module ships, its final spec is written in `docs/moonmodules/<Name>.md` and the temporary backlog draft (if any) is deleted. Most small modules go straight to `docs/moonmodules/` in the same change that implements them — the backlog draft is only for specs worth circulating before the code exists.
-3. **`/plan` it.** Plan references only the relevant `docs/moonmodules/` specs + architecture docs. Plans are not committed to the repo; the implemented code, docs, and commit message together describe what landed.
+3. **`/plan` it.** Plan references only the relevant `docs/moonmodules/` specs + architecture docs. The approved plan is saved to `docs/history/plans/` (see *Plan before implementing*); the implemented code, docs, and commit message together describe what actually landed (which may diverge from the plan — that's expected, the plan is the intent record, not a contract).
 4. **Implement in a branch** (`next-iteration` or feature branch). Test on hardware. Run the commit gates (see Lifecycle Events below). Commit.
 5. **Push.** Product owner pushes. CodeRabbit reviews the PR. Process findings.
 6. **Repeat.**
@@ -114,8 +114,9 @@ The narrow safety net: "this snapshot is internally consistent."
 7. KPI collection, `collect_kpi.py --commit`, if any file under `src/` changed. **The one-liner MUST include `tick:Xus(FPS:Y)` for every supported target** (PC + ESP32 today; Teensy/RPi when added). If a target's tick/FPS is missing (e.g. ESP32 wasn't monitored recently and `esp32/monitor.log` is stale), re-run a short live capture before committing, or note explicitly in the commit body why the value is absent.
 8. Device-model catalog, `check_devices.py`, fast (<1s), if `docs/install/deviceModels.json` or `scripts/check/check_devices.py` changed. Validates the installer catalog: required fields, `firmwares` a non-empty list of non-empty strings (`firmwares[0]` is the default), every `image` resolves on disk, each entry's `System.deviceModel` control equals its entry `name`, module `type`s are factory-registered (or boot-wired singletons), `pins` controls live only on `*LedDriver` modules, and `supported` capabilities stay within the known vocabulary.
 9. Firmware list, `check_firmwares.py`, fast (<1s), if `scripts/build/build_esp32.py`, `docs/install/firmwares.json`, or `scripts/check/check_firmwares.py` changed. Regenerates the firmware projection from the `FIRMWARES` dict and fails on drift from the committed `docs/install/firmwares.json` (so a `FIRMWARES` edit without regenerating is caught). Trigger includes `build_esp32.py` because that dict is the upstream source.
+10. Host-side unit tests (Python + JS), fast (<2s), the suites the C++ ctest can't reach (MoonDeck / build-script logic, web-installer logic). Run `uv run --with pytest --with pyserial pytest test/python -q` if any file under `scripts/` or `test/python/` changed, and `node --test "test/js/**/*.test.mjs"` if any file under `docs/install/` or `test/js/` changed. (Same suites the PR-triggered `.github/workflows/test.yml` runs.) Today these pin the Improv frame wire format — `test/python` + `test/js` assert a shared golden vector so the device C++, Python, and JS frame builders can't drift. New Python/JS unit suites land under `test/python` / `test/js` and run here.
 
-A commit that touches *only* `.github/`, `docs/`, `scripts/` (non-test), `README.md`, `CLAUDE.md`, or `.claude/` therefore runs only the spec check (plus the board-catalog and firmware checks when their specific files changed); the build/test/ESP32/KPI gates are no-ops because their triggers don't fire. This is the intended pre-commit cost for CI-only or doc-only changes.
+A commit that touches *only* `.github/`, `docs/` (excluding `docs/install/`), `README.md`, `CLAUDE.md`, or `.claude/` therefore runs only the spec check (plus the board-catalog and firmware checks when their specific files changed); the build/test/ESP32/KPI gates are no-ops because their triggers don't fire. A `scripts/` or `docs/install/` change adds the relevant host-side unit-test gate but still skips the C++ build/ESP32/KPI gates. This is the intended pre-commit cost for CI-only or doc-only changes.
 
 **Recommended (manual, not blocking):**
 
@@ -200,6 +201,7 @@ docs/
   history/                 ← backward-looking: accumulated wisdom
     README.md              ← index: what's here + cross-repo trends + digest prompt
     decisions.md           ← actions, lessons, proven patterns
+    plans/                 ← approved feature plans (Plan-YYYYMMDD - <title>.md; PO reference, agents don't auto-read)
     *-inventory.md         ← prior-project surveys (v1, v2, moonlight)
     <repo>.md              ← friend-repo monthly activity digests (FastLED, WLED, …)
   moonmodules/             ← one page per MoonModule (specs before code)

docs/architecture.md

@@ -228,7 +228,7 @@ Three distinct things, kept distinct in the vocabulary:
 
 **Firmware** is the compiled binary: chip target plus which radios/peripherals/sdkconfig fragments are included. Today's variants: `esp32` (classic, WiFi **and** RMII Ethernet in one binary — Ethernet comes up only when a PHY is present, pins/PHY per deviceModel), `esp32-eth` (classic, Ethernet only, WiFi excluded), `esp32-16mb` (classic with 16 MB flash, WiFi + Ethernet), `esp32s3-n16r8` / `esp32s3-n8r8` (S3 with WiFi + W5500 SPI Ethernet), `esp32p4-eth` (Waveshare ESP32-P4-NANO, Ethernet only), `esp32p4-eth-wifi` (the same P4 hardware with WiFi via its on-board ESP32-C6 over esp_hosted). Each chip's firmware carries the Ethernet *driver(s)* it can host (RMII EMAC for classic/P4, W5500 SPI for S3); which PHY/pins a deviceModel uses is runtime config. Selected by `build_esp32.py --firmware <key>`, reported by `SystemModule.firmware`, used as the contract target key in scenarios.
 
-**deviceModel** is the physical hardware: chip + PCB + on-board peripherals (PHY, USB-serial, PSRAM, antenna), identified by its product name. Examples: `Olimex ESP32-Gateway Rev G`, `LOLIN D32`, `Generic ESP32 Dev`. A unit cannot identify its own deviceModel (no readable PCB ID on classic ESP32), so MoonDeck deduces it from the firmware where unambiguous (`esp32-eth*` ⇒ Olimex) and otherwise lets the user pick. It is stored on the unit as SystemModule's `deviceModel` Text control (display-only in the UI; HTTP `/api/control` writes still apply). MoonDeck mirrors the picked / deduced value to the unit via `POST /api/control` after each discover and after every dropdown change. The catalog of valid deviceModels lives at [docs/install/deviceModels.json](install/deviceModels.json), shared between MoonDeck and the web installer: MoonDeck reads it for its dropdown and HTTP push; the web installer reads it for its picker, pushes the pick over Improv on first flash, and provides an HTTP fallback (Inject button on *Your devices*) when Improv isn't available on the firmware variant.
+**deviceModel** is the physical hardware: chip + PCB + on-board peripherals (PHY, USB-serial, PSRAM, antenna), identified by its product name. Examples: `Olimex ESP32-Gateway Rev G`, `LOLIN D32`, `Generic ESP32 Dev`. A unit cannot identify its own deviceModel (no readable PCB ID on classic ESP32), so MoonDeck deduces it from the firmware where unambiguous (`esp32-eth*` ⇒ Olimex) and otherwise lets the user pick. It is stored on the unit as SystemModule's `deviceModel` Text control (display-only in the UI; HTTP `/api/control` writes still apply). MoonDeck mirrors the picked / deduced value to the unit via `POST /api/control` after each discover and after every dropdown change. The catalog of valid deviceModels lives at [docs/install/deviceModels.json](install/deviceModels.json), shared between MoonDeck and the web installer: MoonDeck reads it for its dropdown and HTTP push (plain REST on the LAN); the web installer reads it for its picker and pushes the whole entry — deviceModel plus every module/control — over serial during provisioning as REST ops (**"Improv = REST over serial"**, the `APPLY_OP` vendor RPC; see [ImprovProvisioningModule.md](moonmodules/core/ImprovProvisioningModule.md)). Pushing over serial sidesteps the mixed-content block that stops an HTTPS installer page from POSTing to an `http://` device; an already-running device is re-configured via MoonDeck on the LAN.
 
 A deviceModel can run multiple firmwares (the Olimex Gateway runs both `esp32-eth` and the default `esp32`); a firmware can run on multiple deviceModels (`esp32` runs on any classic ESP32 dev kit). The `esp32s3-n16r8` firmware is S3-only and does not run on the Olimex Gateway or other classic-ESP32 hardware. The codebase reserves "deviceModel" exclusively for the physical product and "firmware" exclusively for the compiled binary.