diff --git a/CLAUDE.md b/CLAUDE.md index 3fbebe1..62415b8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -21,7 +21,7 @@ See `docs/architecture.md` for system design. This file contains only rules and - **Robust to any input.** A running device tolerates any sequence of UI actions or API calls: add, delete, replace, or reconfigure any module in any order, at any grid size, and it keeps running. Degraded or idle is acceptable; crashed is not. This robustness is a defining strongpoint of projectMM, and it's guarded by the test framework, not by hope: a discovered crash drives a new test that pins the fix (see the Hard Rule). Out of scope: power loss, malformed OTA, brown-out, and other physical/electrical faults the firmware can't intercept; this principle is about what the software accepts as input. - **No reboot to apply a configuration change.** Every setting takes effect live, on the next render tick — change a pin map, a strand length, an output protocol, a mic pin or rate, anything, on a running device and it just works. There is no init-once-at-boot step, and no *config* change requires a restart, which sets projectMM apart from most LED-controller firmware (where a pin or protocol change means a reboot). Like robustness, this is a defining strongpoint, and it falls out of the architecture for free rather than being hand-built per module: any control whose change reshapes derived state routes through the generic `onBuildState()` rebuild sweep, so drivers, the audio peripheral, effects, layouts, modifiers and network I/O all inherit it. When adding a feature, don't reach for a reboot/restart to apply config; make the change live. Full mechanism + rationale: [architecture.md § Live reconfiguration](docs/architecture.md#live-reconfiguration-every-change-applies-without-a-reboot). The one exception is what you'd expect: a *firmware* OTA flash swaps the binary and needs the usual power cycle — that's not a configuration change, and (like power loss and brown-out) it's the same physical-fault boundary the robustness principle draws. - **Domain-neutral core.** Separate core infrastructure from the light domain as much as practical. When mixing is necessary, use domain-neutral naming so the code stays open to future separation. -- **Present tense only.** Code, comments, and documentation describe the system as it is now. No changelogs, no roadmaps. History lives in git commits. Exceptions: `docs/backlog/` (forward-looking) and `docs/history/` (backward-looking). +- **Present tense only.** Code, comments, and documentation describe the system as it is now. No changelogs, no roadmaps. History lives in git commits. This bans not just future-tense ("will be", "planned") but **absence-narration**: phrases like "no longer", "anymore", "formerly", "used to", "X was removed", or "there's no longer a Y" describe a *change from a past state* a present-tense reader never saw — state what *is*, not what stopped being. (The test: "there is no MCLK pin" is a present-tense *property* — keep it; "there's no SET_DEVICE_MODEL RPC anymore" narrates a removal — cut it, just describe the path that exists.) Exceptions: `docs/backlog/` (forward-looking) and `docs/history/` (backward-looking) — and `decisions.md` lessons, which legitimately contrast before/after because the contrast *is* the lesson. ## Hard Rules diff --git a/docs/architecture.md b/docs/architecture.md index 999134f..b1f8253 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -210,7 +210,7 @@ Only abstract what you actually need. Currently: - **Time**: `millis()`, `micros()`. Monotonic, microsecond resolution. (`esp_timer` / `std::chrono`) - **Memory**: `alloc(size)`, `free(ptr)`. Prefers PSRAM on ESP32, falls back to regular heap. `freeHeap()`, `maxAllocBlock()` for diagnostics. (`heap_caps_malloc` / `std::malloc`) -- **Networking**: `UdpSocket` for ArtNet send. `TcpConnection` / `TcpServer` for HTTP + WebSocket; `TcpConnection::writeChunks` is a non-blocking scatter-gather write so a backpressured browser can't stall the render loop. (lwIP sockets / BSD sockets) +- **Networking**: `UdpSocket` for ArtNet send. `TcpConnection` / `TcpServer` for HTTP + WebSocket; `TcpConnection::writeSome` is a non-blocking partial write (returns bytes written, 0 = would-block) so a backpressured browser can't stall the render loop. (lwIP sockets / BSD sockets) - **Scheduling**: `yield()` (cooperative yield to OS/RTOS), `delayMs(ms)` (blocking sleep, off-path only), `delayUs(us)` (microsecond busy-wait, only for sub-millisecond hardware timing a driver owns — e.g. the WS2812 ≥300 µs inter-frame latch in `RmtLedDriver`; never for general pacing, which uses the non-blocking `millis()` gate), `reboot()`. (`vTaskDelay` / `esp_rom_delay_us` / `esp_restart` on ESP32; `std::this_thread::sleep_for` / `std::exit` on desktop) - **Platform config**: `platform_config.h` per platform: compile-time constants like `hasPsram` and `hasWiFi`. Each platform provides its own version; `types.h` includes it without `#ifdef`. Core code branches on these via `if constexpr` (e.g. NetworkModule drops its WiFi cascade when `hasWiFi` is false), so the dead branch is removed from the binary with no `#ifdef` outside `src/platform/`. @@ -311,6 +311,8 @@ Modules in the light pipeline can be added, replaced, or removed dynamically at - *Shared-struct (pull):* `Drivers` hands every child driver a `Buffer*` (source) plus a `Correction*` (shared brightness/reorder/white), and `Layer` exposes its pixel buffer to `Drivers` directly on the identity-mapping fast path: each consumer holds a `const`-pointer and reads it per frame. The pointers are **(re)bound on every rebuild**, not just at boot: `Drivers::onBuildState()` re-resolves the active `Layer` (`Layers::activeLayer()`) and calls `passBufferToDrivers()`, which re-runs `setSourceBuffer()`/`setLayer()` on each child (clearing them to `nullptr` when there is no active Layer). So a held pointer is valid only until the next rebuild — which is exactly why the consumers re-read it each frame and tolerate a null (the [robustness rule](#robustness)): a Layer add/delete/replace re-binds or clears it live, no dangling reference. - *Push to a core sink:* `PreviewDriver` owns the preview wire format (a one-time coordinate table + per-frame RGB point list) and pushes the bytes to a `BinaryBroadcaster` (the core HTTP server). The server broadcasts them over WebSocket without knowing they're a preview: the format and the light types stay entirely in the driver. See [PreviewDriver](moonmodules/light/drivers/PreviewDriver.md). +**Graceful degradation under transport backpressure.** The preview is the project's worked example of a property worth naming generically, because it is the transport-side sibling of the memory-side [§ Degradation cascade](#degradation-cascade): when a consumer can't keep up, **shed quality, never the connection or the render loop.** The link to a browser is the slow consumer; a full-resolution frame (128² = 16384 lights = ~49 KB) may not drain in the budget one tick allows. Rather than block the loop until it drains (a stall) or drop the client (a black preview), the producer **degrades**: it streams from the producer buffer with no intermediate copy, sheds resolution via a spatial-lattice downsample when the link can't sustain full density, and adapts that factor from whether the previous frame reached every client — the same congestion-responsive, adaptive-bitrate idea behind HLS/DASH/WebRTC, applied to a binary WebSocket. The render loop is never charged more than a bounded slice per tick; the worst case is a coarser or slower preview, which is acceptable for a *view* of the output (the LEDs themselves are unaffected). This is *graceful degradation*: a fast link sees every light at full rate, a slow link sees a faithful coarser sample, and neither stalls the device. The mechanism (resumable cross-tick send + newest-wins backpressure + adaptive lattice) lives in `PreviewDriver` + `HttpServerModule` today; it is payload-agnostic, so a future bulky stream could ride the same transport. + **Naming convention.** Capital `Layouts`, `Layers`, `Drivers` are class names (always capitalised when referring to the class). Lowercase "layouts", "layers", "drivers" is the English plural, used freely when context makes it clear. Singular "layout", "layer", "driver" is an individual instance. ## 3D from the start diff --git a/docs/backlog/backlog.md b/docs/backlog/backlog.md index 30c9cfa..5939884 100644 --- a/docs/backlog/backlog.md +++ b/docs/backlog/backlog.md @@ -292,8 +292,6 @@ Several `platform.h` APIs still use `(buf, len)` pairs where `std::span` would c Device-model injection over Improv shipped as **"Improv = REST over serial"** (the `APPLY_OP` vendor RPC pushes the whole `deviceModels.json` entry over serial during install; the device runs the same apply-core the HTTP REST API does, on WiFi *and* eth-only firmware). That subsumed the earlier multi-step "board injection + Improv as a general data injector" plan — the general injector *is* APPLY_OP. What remains: -**Open follow-up: per-control validator hook on `ControlDescriptor`.** `SystemModule::setDeviceModel()` validates ASCII-printable (rejecting control bytes, embedded NUL); the HTTP `POST /api/control` write path uses the generic `applyControlValue()` in `Control.cpp` which has no per-control validator and writes the raw bytes through. Acceptable today (HTTP-write callers source values from `deviceModels.json` which the project controls), but the right fix is a per-control validator hook on `ControlDescriptor` so any control can declare an inline validation function pointer. Worth doing when the next control with non-trivial input constraints lands, or when the threat model grows (an integration accepts arbitrary external input and POSTs it through). Sketch: `ControlDescriptor` grows a `bool (*validate)(const void*, size_t)` slot defaulting to nullptr; `applyControlValue` calls it before writing and returns `ApplyResult::Malformed` on false; `addText` / `addPassword` get an optional validator argument. Touches ~5 sites; no protocol change. - **Open follow-up: closed-loop APPLY_OP pacing (read-back ack + retry).** The installer paces APPLY_OP frames open-loop (`sendApplyOpFrame` waits a fixed ~120 ms between ops) rather than reading the device's ack back, because a Web Serial duplex read while the writer lock is held is awkward. The delay covers the worst-case single-buffer consume window with headroom, and each op is idempotent (a lost op re-applies cleanly on a re-flash), so this is robust today. The closed-loop upgrade — read the RPC response, retry once on error `0x82` (buffer busy) — removes the fixed delay (faster install) and makes op-loss impossible rather than improbable. Worth doing if a real install is ever observed dropping an op, or when the config push grows large enough that the cumulative fixed delay is noticeable. Touches only `install-orchestrator.js`. **Open follow-up: shared JS helpers across device-UI and web-installer.** `safeLocalGet` / `safeLocalSet` (3-line hostile-storage guards) are duplicated in `src/ui/install-picker.js` (device firmware, embedded as a C string via `embed_ui.cmake`) and `docs/install/devices.js` (web installer page, served from Pages). The two live in different build contexts so the shared extract isn't trivial — it'd need a new `src/ui/safe-storage.js` plus updates to: `embed_ui.cmake` (embed the new file), `ui_embedded.h` generator (new C array), HTTP server file routing (new path served), `release.yml` workflow staging, `preview_installer.py` staging. Five files for one 3-line helper is too much pre-merge. Worth doing when the next shared helper arrives — `relativeTime` and `formatBytes` are candidates. Two helpers earn the build-glue cost; one doesn't. @@ -395,6 +393,22 @@ The preview index-downsamples a large layout to fit the WS send budget (e.g. 128 Not simple — own planning pass. Until then the preview is a faithful strided *sample* (correct shape/colour/motion, not per-pixel). A cheap interim (point-size scaled by stride to fatten samples into their cells) was tried and reverted as not what's wanted — it filled the volume but didn't add real points. +### Self-describing preview frame header (mid term) + +The preview wire format is a private opcode protocol: `0x02` per-frame channels, `0x03` coordinate table, each a hand-rolled byte layout, and the colour payload is **always RGB** regardless of the buffer's `channelsPerLight`. Every new data kind (RGBW display, beam direction, …) means inventing another opcode and another fixed layout by hand. The minimal fix that stops that sprawl: a small **typed header** — `[type][format][count][stride]` where `format` enumerates `{RGB, RGBW, …}` — so one message kind carries any per-light channel layout and the browser shader reads `format` to interpret the payload. Do it concrete-first, when RGBW *display* (below) is actually wanted, not speculatively. Prereq for both items below. + +### RGBW preview end-to-end (mid term) + +The light `Buffer` already holds `channelsPerLight = 4` (RGBW), and the device output drivers handle it, but the **preview only ever sends/draws RGB** — the W channel is invisible in the UI. (The full-res fast path no longer penalises a cpl≥3 buffer — see the short-term fix — but it still drops W on the wire.) Once the self-describing header lands, carry the W channel on the wire and render it in the shader (W as a warm-white tint / brightness lift on the disc). Small, but gated on the header so it isn't another bespoke opcode. + +### Fixture model — moving heads, beams (long term) + +Today a "light" is a point at a static coordinate with a colour. A **moving head** is a fixture that emits a *beam* in a direction it controls live (pan + tilt), plus colour, beam-width, etc. — per-light **vector** state, not just colour, and a different draw (a cone/ray, not a disc). The static-positions-`0x03` + colour-`0x02` split can't express "this fixture's beam now points here." The industry-standard model is **DMX/GDTF fixtures**: a fixture has a position *and* a set of typed attributes (color, pan, tilt, beam). The preview becomes a fixture renderer (disc for a pixel, cone for a beam); this is also the "make Preview a general-purpose module, not light-specific" goal. A domain-model change (the fixture/attribute model), not just transport. Plan when moving heads are actually on the bench. + +### Extract the resumable backpressure transport as a domain-neutral channel (long term) + +The preview's transport — resumable cross-tick send from a stable buffer + newest-wins backpressure drop + adaptive graceful degradation (see [architecture.md § graceful degradation under transport backpressure](../architecture.md)) — is **payload-agnostic**: any bulky throttled stream (a future MJPEG/video preview, fixture-state streams, fleet telemetry) could ride it. The *payload* model (count/stride/RGB) is light-specific; the *byte-pump* is not. When a second consumer for this transport appears, promote the pump into a domain-neutral core primitive (a `ThrottledChannel`-style sink) that PreviewDriver becomes *a* producer on, rather than owning the protocol. Concrete-first: extract on the second use, not before — until then the seam stays inside HttpServerModule/PreviewDriver. + --- ## Testing diff --git a/docs/building.md b/docs/building.md index 7359ab8..0a8dcff 100644 --- a/docs/building.md +++ b/docs/building.md @@ -188,7 +188,7 @@ If a firmware *key* changes its feature set (e.g. the classic `esp32` collapse t Each ESP32-S3 SKU has its own firmware key because the sdkconfig fragment encodes flash size, partition table, and PSRAM mode — flashing an `n16r8` binary onto a different module (e.g. N8R2) either misaligns the partition table (boot loop) or fails PSRAM init. New SKUs become new keys (e.g. `esp32s3-n8r8`); there is no generic `esp32s3` shortcut. -The Ethernet PHY type and pin map are runtime config, not baked into the build: each firmware carries the driver(s) its chip can host (RMII EMAC for classic/P4, W5500 SPI for S3), and `deviceModels.json` supplies the per-board PHY/pins (pushed into NetworkModule's eth controls at provision). The Olimex pins are the classic chip default, so a board with the same LAN8720 PHY but different pinout (e.g. WT32-ETH01 with reset on GPIO 16) just needs a different `deviceModels.json` entry — no rebuild. +The Ethernet PHY type and pin map are runtime config, not baked into the build: each firmware carries the driver(s) its chip can host (RMII EMAC for classic/P4, W5500 SPI for S3), and `deviceModels.json` supplies the per-board PHY/pins (pushed into NetworkModule's eth controls at provision). The classic chip default is the common LAN8720 RMII wiring (reset GPIO 5, MDIO addr 0, clock GPIO 17 — e.g. the Olimex ESP32-Gateway), so a board with the same PHY but a different pinout (e.g. WT32-ETH01 with reset on GPIO 16) just needs a different `deviceModels.json` entry — no rebuild. `--profile` is accepted one release for migration: `--profile default` → `--firmware esp32`, `--profile eth-only` → `--firmware esp32-eth`. The legacy `build_esp32_ethonly.py` wrapper still works (it now forwards `--firmware esp32-eth`). diff --git a/docs/history/decisions.md b/docs/history/decisions.md index 318f3f8..c5f5ae6 100644 --- a/docs/history/decisions.md +++ b/docs/history/decisions.md @@ -697,3 +697,7 @@ The installer was reworked so a board catalog ([`boards.json`](../install/boards **`deviceName` (identity) vs `deviceModel` (product) vs board (bare PCB) — one term was doing three jobs.** "Board" had been overloaded to mean the per-unit network identity, the hardware product/catalog key, AND the bare PCB. Untangling it: `deviceName` is the **per-unit identity** — one string that drives mDNS (`.local`), the SoftAP name, and the DHCP hostname, so the device shows up under one name everywhere; it's RFC-1123-coerced (`sanitizeHostname`) because it becomes a hostname. `deviceModel` is the **hardware product** (the `deviceModels.json` catalog key, e.g. "projectMM testbench S3") — display-form, spaces allowed, never a hostname. "Device" is the umbrella noun; "board" now means **only the bare PCB**. This drove the BoardModule→SystemModule fold (the identity is core unit state, not a separate module), the `board`→`deviceModel` rename across catalog/installer/Improv (SET_BOARD→SET_DEVICE_MODEL, byte 0xFE unchanged), and the eth pin-map clarification (driver = firmware, pin map = firmware-seeded but **deviceModel-authoritative** so an Olimex entry can override). Lesson: when one noun answers three different questions ("what do I call this unit on the network?", "what product is it?", "what's the bare board?"), that's a naming smell — split it into the qualified terms, pick one umbrella word, and make the split visible in every layer (control names, RPC symbols, catalog keys, docs) so the three concepts can't re-merge. **"Improv = REST over serial" — one apply-core, two transports, and the testability that follows from extracting the hard part.** The deployed HTTPS installer couldn't configure a flashed device: a browser blocks an HTTPS page from POSTing to an `http://` device (mixed-content), and the `?deviceModel=` pull/handoff that replaced it only ran if the user opened that exact link. The fix reframed the problem — the installer already owns the USB serial port during provisioning, so push the config over it as the *same REST operations the HTTP API runs*: a new `APPLY_OP` (0xFC) Improv vendor RPC whose payload is `{"op":"add|set|clearChildren",…}`, the same JSON a `POST /api/modules`/`/api/control` body carries. On the device the op routes to **one transport-free apply-core** (`HttpServerModule::applyAddModule/applySetControl/applyClearChildren/applyOp`) the HTTP handlers also call, so a network REST call and a serial APPLY_OP execute identical code; the handlers became thin `switch(applyX())` → status-code mappers. This **deleted** the whole browser handoff (device-side catalog fetch, `?deviceModel=` decoration, the inject button) — a net subtraction — and works on Ethernet-only firmware once the Improv listener is decoupled from WiFi (the vendor RPCs compile in unconditionally; only `WIFI_SETTINGS`/`GET_WIFI_NETWORKS` stay `#ifndef MM_NO_WIFI`). Lesson 1: when a push is blocked by the *medium* (mixed-content on HTTPS), look for a medium you already control (the serial port mid-flash) instead of bolting on a fragile pull. Lesson 2 (the one with legs): the way to make it *provable* was to **extract the hard part into a pure core primitive** — the chunk reassembly + out-of-order/duplicate sequence guard moved from the ESP32-only handler into `src/core/ImprovOpReassembler.h` (header-only state machine, returns `Continue/Ready/Error`), and the JS frame builders into `docs/install/improv-frame.js` so `node:test` imports them without the orchestrator's browser deps. Both are *Complexity lives in core; domain modules stay simple* applied for testability: the device handler keeps only its serial I/O, the algorithm gets unit-tested on the desktop, and a format implemented three times (device C++, Python, JS) is pinned by **one shared golden vector** asserted in `test/python` + `test/js` — a contract test is the right answer to *forced* duplication no shared compilation target can remove. The reflex worth keeping: a hard mechanism buried in a platform `.cpp` that "can only be tested on hardware" is a smell — extract its pure core, and "rock solid proven" becomes a unit test instead of a bench session. + +**A periodic re-broadcast to let late joiners "catch up" is a hack wearing a keepalive costume.** The 3D preview sends a coordinate table (positions) once, then per-frame colour. The original implementation re-sent the *whole table every ~1 second* "so a client that connected after the last rebuild catches it." It looked fine — a fresh page recovered within a second — so it shipped and sat there. But it's a workaround, not the mechanism: it rebuilt the full table from the layout **every tick-second forever**, on the hot path, whether or not anyone had connected and whether or not anything changed — and it papered over a missing request/response with polling. The correct construct is event-driven: send the table **when it actually changes** (`onBuildState` — grid/layout/LUT rebuild) and **when a client asks** (a new WS connection bumps `BinaryBroadcaster::clientGeneration()`, which `PreviewDriver::loop()` watches and re-sends on change). That's strictly *less* code than the timer and zero idle cost. How it sneaked past review: the workaround *worked* in casual testing and its cost was invisible until a later change made each rebuild heavier and the per-module tick was profiled. Lessons: (1) "re-send periodically so it eventually syncs" is the polling-instead-of-events smell — ask "what's the event that should trigger this?" and trigger on *that*; (2) a recurring rebuild on the hot path must justify itself every tick, so "every second, just in case" fails *Data over objects / fastest hot path* on sight; (3) this is *Continuous refactor, no hacks* — the fix isn't a scheduled cleanup, it's "the moment you see a keepalive timer standing in for a request, replace it." The guard is a test that advances the clock several seconds with no client change and asserts the table is **not** re-sent (the old timer would have). + +**Don't hold a vendor library's async handle across your own event loop — it races the library's internal timers.** A UI refresh intermittently crashed the device (`assert failed: xQueueSemaphoreTake queue.c:1709 (( pxQueue ))` — a null FreeRTOS queue — inside the espressif mDNS component's `mdns_query_async_get_results`, plus an `Interrupt wdt timeout`). The mDNS *browse* (discovering peers for the "Your devices" list — distinct from mDNS *advertise*, which serves `.local` and was never the problem) used the async API: `mdns_query_async_new` returns a handle that `DevicesModule` held across ticks, polling it each `loop1s` with a 0 ms timeout. The trap: the mDNS component's **own task** owns that handle's queue and **frees it when the query's window (3 s) expires** — so a poll landing in the gap after expiry asserts on a freed queue. It was intermittent and grid-size-sensitive (a bigger grid lengthens the tick, widening the gap) and looked like "refresh crashes it" only because a refresh's activity coincided with the poll. **First fix attempt was wrong:** I assumed a *service-table mutation* (live rename re-registering `_http._tcp`) tore the handle down and added a cancel-before-mutate guard — it didn't fix it, because the freeing party is the component's expiry timer, not our code. **Real fix:** stop using the async-handle API entirely — replace the start/poll/stop trio with one synchronous `mdnsBrowse()` (`mdns_query_ptr`) that queries, delivers results, and frees everything in a single call, holding **no handle across ticks**, so the race window can't exist. The catch that synchronous introduced: `mdns_query_ptr` blocks the *full* timeout (it waits the whole window for late responders, no early return) and `loop1s` is charged to the tick — an 80 ms query tanked the tick. So **throttle**: browse one service type every ~8th tick with a ~60 ms timeout — one brief hiccup every ~8 s, invisible for discovery, FPS untouched in between. Lessons: (1) a library's async/iterator handle is only valid between *its* lifecycle events — if you can't see/where those fire (here, an internal expiry timer on another task), don't hold the handle across your loop; prefer a self-contained synchronous call that owns the whole lifecycle. (2) An *intermittent, load-dependent* crash whose backtrace sits in a vendor component is a **lifecycle race**, not a component bug — but find the *actual* concurrent actor before "fixing" (my first guess at the actor was wrong and the fix did nothing). (3) Trading async for synchronous trades a race for a blocking cost — budget it (throttle + bound the timeout) so the cure isn't a tick-killer. (4) Desktop stubs these mDNS calls to no-ops, so it's a hardware-only fix the unit suite can't reach; the reproduction (concurrent WS churn at a large grid → crash before, stable after, uptime climbing) is the proof, in the commit, not a desktop test. diff --git a/docs/history/plans/Plan-20260622 - Non-blocking preview send.md b/docs/history/plans/Plan-20260622 - Non-blocking preview send.md new file mode 100644 index 0000000..f77fbcb --- /dev/null +++ b/docs/history/plans/Plan-20260622 - Non-blocking preview send.md @@ -0,0 +1,76 @@ +# Plan — Non-blocking preview send: high-resolution preview without stalling the render task + +## Context + +**The problem.** On a large grid (128² = 16K LEDs) the WebSocket preview shows nothing or kills the connection. The goal: the preview runs fluently at large sizes, reaching **16K on a no-PSRAM classic ESP32** and higher on PSRAM boards. + +**What the measurements proved (P4, 2026-06-22) — our own diagnosis.** The wall is **not** RAM, **not** bandwidth (72 KB/s at 128²), **not** the render tick (343 µs, 8× headroom). The wall is the *send mechanism*: a preview frame is one **synchronous `writev` on the shared HTTP/render task**. +- A frame over ~5 KB (≈1700 points) can't go out in one `lwip_writev`; it partial-writes → `broadcastBinary` closes the connection (preview vanishes above ~48²). Bisected live: 32² streams, 48² drops. +- Raising the drain budget made it **worse** — a 60 ms drain at 128² blocked the shared task long enough to starve HTTP accept; the WS handshake itself failed. Confirmed live. + +**Root cause, named.** We're doing a **blocking, all-or-nothing send on the render thread**. The textbook fix is the standard one for any producer that must hand bulk data to a slower consumer over a socket: **a non-blocking bounded queue with backpressure** — the producer copies the frame and returns; a separate drain step writes it out in slices as the socket accepts them; if the producer outruns the consumer, the newest frame is **dropped, not blocked** (backpressure). This is the same producer/consumer discipline the render pipeline already uses (effects produce, drivers consume); here the consumer is the socket. + +**Why this reaches 16K without downsampling (the analysis).** Once the send is enqueue-and-drain, the per-frame cost is no longer "what fits one `writev`." The remaining limits are concrete and addressable: +- **16-bit WS frame length** (`broadcastBinary` max 65535 B = 21843 pts) → extend to the RFC 6455 **64-bit length form** (~10 lines, localized). Ceiling gone. +- **`count` field is `u16`** in the frame header (max 65535 pts) → widen to `u32` (the system already supports >65K lights: `nrOfLightsType` is `u32` on PSRAM boards). +- **Staging-buffer RAM** (the queued frame, `pts*3`): 16K = **48 KB**, 64K = 192 KB. `platform::alloc` prefers PSRAM with internal-RAM fallback, so a classic board fits ~48 KB in internal RAM and PSRAM boards fit far larger. This *is* the "16K easily, above that needs PSRAM" boundary — derived, not guessed. +- **Drain latency** (not blocking): a 48 KB frame drains over ~10 `loop20ms` ticks ≈ 200 ms, so the preview frame-rate **adapts down** (~5 fps at 16K) — fine for a preview, the tick never stalls. + +**Conclusion: raise the cap empirically, keep downsampling as the tested fallback — do NOT remove it.** The enqueue model lets the cap rise well past 1800, but we don't declare a number from a spreadsheet: per the test-first principle, **raise it and measure where it actually breaks on each board** (classic internal-RAM limit, PSRAM headroom, drain-latency floor). The spatial-lattice downsample (already built) **stays** as the deliberate fallback beyond the tested cap — bigger panels will hit limits, so the graceful-degrade path must remain. The cap is **RAM-derived with a measured safety margin**. + +**>65K lights is a real target (big ArtNet HUB75 walls).** The system already supports it (`nrOfLightsType` u32 on PSRAM; GridLayout anticipates `512×512 > 65535`), but the preview wire `count` is `u16` — a contradiction. Widening to `u32` is in scope so a >65K panel can be previewed (downsampled to whatever staging RAM allows, but the *count* is no longer capped at 65535). + +**Forward-compatible with the producer/consumer two-task split (architecture.md §145).** The two-core design lands soon. This enqueue/drain model **is that shape**: `broadcastBinary` enqueue = producer handoff; `drainWsSends()` = consumer transmit. Today both run on the one Scheduler thread; when §145 lands, **`drainWsSends()` moves to the consumer/network task unchanged** — the queue *is* the handoff boundary. A down-payment on §145, not a single-task hack. + +## Approach + +Three seams, all in core transport + the driver. No new task yet (it arrives with §145), no new module. + +### 1. Non-blocking send queue with backpressure (`HttpServerModule`) + +- **One staging buffer for the live-preview client**, sized to the RAM-derived point cap, allocated once via `platform::alloc` (PSRAM-preferred; classic falls back to internal RAM). Single live client (§2) → one buffer. +- `broadcastBinary` → **non-blocking enqueue**: **backpressure gate first** — if the live client still has unsent bytes from the previous frame, **drop this frame** (newest-wins). Else copy WS header + payload into the staging buffer, set `len`, `sent=0`, return. Never blocks. +- New **`HttpServerModule::drainWsSends()`** called from `loop20ms()`: flush the staging buffer with the **non-blocking** `writeSome` — send what the socket takes now, advance `sent`, leave the rest for the next tick. Mid-frame partial is expected (we own the offset); only a real socket error closes. The exact function §145 later hosts on the consumer task. (As implemented, the drain runs before the accept so a connection burst can't strand it.) +- **Extend `broadcastBinary`'s WS header to the 64-bit length form** so a >65535-byte frame is legal (replaces the current `else { return; }`). + +### 2. Single live-preview client (bound the memory) + +The preview is a *live view* — one viewer at a time is the real use case, and it bounds the staging buffer to one instance instead of `MAX_WS_CLIENTS`×48 KB. Target the **most-recently-connected** WS client (`wsClientGeneration_` already tracks new connections; PreviewDriver re-sends its coord table on a generation bump). State-JSON pushes still go to all clients — only the binary preview is single-target. + +### 3. PreviewDriver: raise the cap empirically, widen count to u32, keep downsample fallback + +- Replace fixed `MAX_PREVIEW_POINTS = 1800` with a **RAM-derived cap** (`platform::hasPsram`/`freeInternalHeap()`/`maxAllocBlock()` with a margin for stack/HTTP/WiFi), **tuned by measurement**. The spatial-lattice downsample **stays** and engages beyond the cap. +- **Widen the frame `count` field `u16 → u32`** (both 0x02 colour and 0x03 coord headers) on device and browser. +- **Does NOT touch the `rgb_`/`coords_` build buffers** — only how the built frame is *sent* and the count width. Zero-copy producer-buffer reuse + channelsPerLight/offset wire model are a separate deferred step. + +## Files + +**Core transport (the enqueue + drain — the §145-ready seam):** +1. **Edit** `src/core/HttpServerModule.h` — staging buffer (`wsPreviewBuf_`, `wsPreviewCap_/Len_/Sent_`, target client index + generation), `drainWsSends()` decl, free in `teardown()`. +2. **Edit** `src/core/HttpServerModule.cpp` — rewrite `broadcastBinary` as non-blocking enqueue with the backpressure gate; add the 64-bit WS length branch; add `drainWsSends()`; call it from `loop20ms()` after the accept early-return. Lazy-alloc staging via `platform::alloc`. + +**Driver + wire format (RAM-derived cap, u32 count):** +3. **Edit** `src/light/drivers/PreviewDriver.h` — `MAX_PREVIEW_POINTS` → RAM-derived cap, tuned by measurement; keep the lattice fallback. Widen the 0x02/0x03 header `count` to `u32`. +4. **Edit** `src/ui/preview3d.js` — read `count` as `u32` (`getUint32`) in `renderPreviewFrame` (0x02) and `parsePreviewCoords` (0x03); adjust header offsets. + +**Tests + docs:** +5. **Edit** `test/unit/light/unit_PreviewDriver.cpp` — count fits the RAM-derived cap + lattice-regularity; a grid past the cap still downsamples (fallback intact); a `u32`-count round-trip for a >65535-point grid. Add a `unit_HttpServerModule` case: a second `broadcastBinary` while the first is undrained is **dropped** (backpressure); `drainWsSends()` makes partial progress; the 64-bit WS length header is emitted for a >65535 B frame. +6. **Edit** `docs/moonmodules/light/drivers/PreviewDriver.md` + `docs/moonmodules/core/HttpServerModule.md` — non-blocking enqueue + backpressure-drop + RAM-derived cap + u32 count + 64-bit frames; update the wire-contract layout. Update `docs/architecture.md`: preview send is enqueue-on-produce + drain-on-transport-poll, never synchronous on the render tick; the drain is the consumer-side step §145 will host. + +## Verification + +- **Host:** `cmake --build build` (-Werror), `ctest`, `uv run scripts/scenario/run_scenario.py`, `check_specs.py`, `check_platform_boundary.py`. +- **ESP32 build** (`esp32p4-eth` + `esp32s3-n16r8` + classic `esp32`). +- **Live — find where it breaks (test-first):** websockets probe — sweep Grid 48²→64²→128²→195²→256²→512², recording at each size: WS open?, frame point-count (full vs downsampled), `/api/system` tick, preview fps. Locate the real break point **per board** and set the cap from that. Assert the WS never closes and the tick never stalls. +- **Classic board (the key test):** sweep toward 16K+, find where internal RAM / drain-latency forces downsampling, confirm the device degrades-never-crashes. Sets the classic-tier cap. +- **>65K (u32 count):** a grid above 65535 lights previews (downsampled) with a correct count — the HUB75-wall path. + +## Risks / notes + +- **Memory:** one staging buffer, RAM-derived; single live client keeps it ×1. Classic internal-RAM headroom is the binding constraint. +- **Drain latency, not blocking:** preview fps adapts down at big sizes; the tick never stalls. §145 consumer task can later drain continuously for smoother large previews. +- **`broadcastBinary` is preview-only** (only PreviewDriver calls it), so the contract change is safe. +- **Two-task forward-compat:** `drainWsSends()` is a standalone entry point §145 moves to the consumer task without a rewrite. +- **Downsampling stays** — raised, not removed; the lattice fallback is the tested graceful-degrade path. +- **Deferred (next commit):** zero-copy producer-buffer reuse + channelsPerLight/offset wire model. +- The diagnostic `writeChunks`/`maxDrainMs` machinery was removed entirely; the transport primitive is the non-blocking `writeSome`. diff --git a/docs/history/plans/Plan-20260622 - Responsive split-pane preview with draggable PiP.md b/docs/history/plans/Plan-20260622 - Responsive split-pane preview with draggable PiP.md new file mode 100644 index 0000000..9ac6695 --- /dev/null +++ b/docs/history/plans/Plan-20260622 - Responsive split-pane preview with draggable PiP.md @@ -0,0 +1,35 @@ +# Plan — Responsive preview: docked split-pane (wide) ↔ draggable PiP (narrow) + +## Problem + +The 3D preview and the module cards stack **vertically** inside `.main-area`: a sticky `.preview-wrap` (aspect 1/1, `max-height: 50vh`, scroll-shrinks to ~25vh) sits above `#main` (cards, capped 500px, centered). On short or small screens the preview eats most of the viewport height even when configuring a module unrelated to the 3D view (e.g. Network/SSID), leaving the cards crammed into a narrow column far down the page; on wide screens there's large empty space *beside* the 500px card column while the preview hogs vertical space *above* it. The vertical stack is the worst fit for short/wide screens. + +## Model (product-owner decisions) + +One canvas, two modes, switched by width with a manual override. "Always visible, sometimes as a small popup." + +- **Mode A — docked split-pane** (wide, ≥ ~960px): `.content` is a 3-column row — nav (200px) · preview (flex:1, sticky, fills its pane height) · cards (**fixed ~480px**, own `overflow-y:auto`, full height). The scroll-shrink hack is removed: the preview is stable, only the cards column scrolls. Industry standard: editor+canvas (Blender / Figma / VS Code). +- **Mode B — floating PiP** (narrow < ~960px, OR docked-preview manually dismissed): the **same** canvas moves into a fixed-position, **draggable, corner-snapping** card (~160px), with a drag handle + expand + close (×). Cards take the full content width. Industry standard: YouTube-mobile PiP. +- **Switching**: a `ResizeObserver` / matchMedia listener toggles a class on `.content` (`mode-docked` ↔ `mode-pip`); CSS does the layout, and the preview's existing `resize` handler (preview3d.js:180, renders at `clientWidth/clientHeight`) re-fits the canvas — so a dynamic window resize pops in/out smoothly with no reload or state loss. +- **PiP trigger**: auto on narrow + a manual toggle on wide (pop the preview out to reclaim card space). +- **PiP dismiss**: × fully hides it; a small "show preview" affordance (status-bar icon or floating pill) brings it back. + +## Files + +- **`src/ui/index.html`** — restructure `.content`: keep `#nav`; wrap preview + cards so they're siblings in a row (`.workspace` flex: `.preview-pane` + `#main`). Add the PiP chrome (drag handle, expand/close buttons, the re-show pill). The `` stays one element — it's *reparented* (or its wrapper is restyled) between modes, never duplicated (one WebGL context). +- **`src/ui/style.css`** — `.content` row layout; `.preview-pane` (sticky, flex:1) + `#main` (fixed 480px, `overflow-y:auto`, `height: calc(100vh - 44px)`) for docked. `.mode-pip` rules: preview becomes `position:fixed`, small, draggable; `#main` goes full-width. The `<820px` block + a new `~960px` breakpoint drive the auto-switch. Remove `.preview-wrap` sticky-scroll styling + the `max-height:50vh`. +- **`src/ui/preview3d.js`** — replace `setupShrink` (scroll-shrink) with `setupLayout`: the mode toggle (matchMedia/ResizeObserver) + PiP drag/snap + dismiss/re-show. Keep the existing `resize` re-fit. Drag = pointer events, clamp to viewport, snap to nearest corner on release; persist PiP corner + dismissed state in localStorage (hostile-storage guarded, like the other UI prefs). +- **`src/ui/app.js`** — `setupShrink()` call site → `setupLayout()`. + +## Verification + +- `node --check` the JS; manual responsive sweep: wide (docked split), drag-narrow (auto-pops to PiP, canvas re-fits), drag-back-wide (re-docks), PiP drag + corner-snap, ×-dismiss + re-show, mobile (<820px nav drawer still works with PiP). On a real device (S3 UI) at phone width. +- No backend change → no ctest/scenario/ESP32 impact; the commit gates that fire are spec (none — no control names change) + the build only if `src/ui` compiles into the binary (it's embedded via `embed_ui.cmake`, so a desktop build confirms the embed). +- Confirm the preview still renders binary frames in both modes (one canvas, one WebGL context throughout). + +## Risks / notes + +- **One WebGL context**: the canvas must never be duplicated — reparent or restyle in place, or the context is lost. Test the dock↔PiP transition keeps rendering. +- **Drag vs. orbit**: the PiP's drag handle must be a separate element from the canvas, or dragging the window fights the camera-orbit pointer handler (preview3d.js owns `touch-action:none` on the canvas). +- **Cards column height**: `height: calc(100vh - 44px)` with its own scroll means the page itself no longer scrolls in docked mode — verify the status bar + nav still behave. +- Pure front-end, UI-only; no protocol/control/spec change. diff --git a/docs/history/plans/Plan-20260623 - Stream preview from buffers, zero preview buffers.md b/docs/history/plans/Plan-20260623 - Stream preview from buffers, zero preview buffers.md new file mode 100644 index 0000000..6e705ba --- /dev/null +++ b/docs/history/plans/Plan-20260623 - Stream preview from buffers, zero preview buffers.md @@ -0,0 +1,58 @@ +# Plan — Stream the preview from the producer buffer; eliminate all preview-side buffers + +## Context + +The non-blocking preview rework (committed 1e48e92) made the WebSocket preview stream without stalling the render tick, but it introduced/retained **frame-sized buffers** in the preview path: `coords_` (~49 KB packed positions), `rgb_` (per-frame colour copy), `sampledIdx_` (the lattice index map), and the HttpServer **staging buffer** (~49 KB). On a no-PSRAM classic ESP32 these compete for scarce *contiguous* internal RAM: at 128² (16384 lights) the render buffer (49 KB) and a preview buffer (49 KB) can't both find a contiguous block once the heap fragments from grid-resize churn — so the preview (and sometimes the render) fails to allocate. Measured on the bench: a clean boot has a 108 KB contiguous block (128² fits); after resize churn it collapses to ~20–40 KB (128² fails). + +The principle this violates is CLAUDE.md's **minimal memory / data-over-objects / hot-path** rules, applied to the preview: the colours already live in the **producer/consumer buffer** (the Layer's logical buffer, or the blend buffer for multi-layer/non-identity mapping). The preview should **stream that buffer to the client**, holding no frame-sized copy of its own. Positions are communicated **once** (event-based: on a layout/modifier change, or when a client connects/refreshes); after that the per-frame stream is just the buffer, 1:1, and the client already knows where each light goes. + +architecture.md describes the preview *mechanism* (a one-time coord table + per-frame RGB, §"Output stage"/§UI) but does **not** state the memory model — that the per-frame stream is the producer buffer with no intermediate buffer, and downsampling is "send every Nth light." This plan implements that, and the doc gets updated to capture it. + +## The model (settled with the product owner) + +- **Coordinates are sent once** (0x03), on geometry change or client (re)connect. They need positions → built from `Layouts::forEachCoord` (a cold/rate-limited path, never the LED render hot path — verified: forEachCoord callers are LUT build, status, and the preview coord build only). +- **Colours are streamed per frame** (0x02) straight from the producer/consumer buffer the driver already holds (`sourceBuffer_`), **1:1, no copy**. The client places colour[i] at coord[i] from the table it already has. +- **Downsampling = send every Nth light**, applied identically to the coord table and the colour frame so they match by construction. Two regimes: + - **Full resolution (the common case, stride 1):** colour frame is a pure 1:1 buffer stream — no `forEachCoord`, no skip, no buffers. + - **Downsampled (rare: grid > cap, or link too slow):** to avoid the diagonal moiré that flat `i % N` striding causes on a 2D grid, both passes use the **spatial-lattice** skip (`x%s && y%s && z%s`) via `forEachCoord`. This walks positions (cheap integer loop, rate-limited, off the LED hot path) but still streams — no stored index map. +- **No preview-side frame buffers at all**: streaming via the broadcaster's begin/push/end means neither pass ever holds a frame-sized buffer. `coords_`, `rgb_`, `sampledIdx_`, and the HttpServer staging buffer are all removed. + +## Approach + +### 1. Broadcaster: streaming begin/push/end (already implemented) +`BinaryBroadcaster` gains `beginBinaryFrame(totalLen)` / `pushBinaryFrame(data,len)` / `endBinaryFrame()`. HttpServerModule sends the WS header on begin, fans each pushed slice to every client via the non-blocking `sendAllOrClose` (close a client that can't keep up — it reconnects), and reports all-sent on end. No frame-sized staging buffer. + +### 2. PreviewDriver: stream both passes, drop the buffers +- **Coord table** (`buildAndSendCoordTable` → `streamCoordTable`): compute the per-axis lattice step `s` (1 = full res; >1 when the light count exceeds the cap or adaptive downscale raised it). `beginBinaryFrame(coordCount*3 + …)`, then walk `forEachCoord` pushing scaled (x,y,z) for lights on the lattice (`x%s && y%s && z%s`); `endBinaryFrame`. No `coords_`. +- **Colour frame** (`sendFrame`): + - **stride 1:** `beginBinaryFrame(n*3)`, then push the producer buffer directly. If `cpl==3` it's one push of `sourceBuffer_->data()`; if `cpl!=3` (RGBW) push per-light 3 bytes through a tiny stack temp. No `rgb_`. + - **stride > 1:** walk `forEachCoord`; for each light on the lattice push its 3 colour bytes from `sourceBuffer_[idx]`. Same predicate as the coord table → same subset/order. No `sampledIdx_`. +- Remove members: `coords_`, `rgb_`, `sampledIdx_`/`sampledIdxCap_`, and the `~PreviewDriver` delete. +- Keep: the **cap** (now just "downsample above N points" — bounds the per-frame work/wire size, not a buffer), the **adaptive downscale** (latency + pending-drop driven), `coordPending_` retry, the u32 count, the browser count/stride guard. + +### 3. HttpServerModule: remove the staging machinery +With both passes streamed, the staging buffer + `wsPreviewBuf_/Cap_/Len_/Sent_[]`, the stage-vs-DIRECT branch in `broadcastBinary`, `drainWsSends`, `directBroadcast`, and the drain-tick/stuck-client guard are no longer used by the preview. `broadcastBinary` (the chunk-array form) and `lastDrainTicks` may become unused → remove what's dead. (Adaptive downscale now keys off `endBinaryFrame()` returning false / the coord-pending retry, not `lastDrainTicks` — confirm and simplify.) + +### 4. Tests + docs +- `unit_PreviewDriver`: update the `CaptureBroadcaster` mock to implement begin/push/end (accumulate pushed bytes into `lastCoord`/`lastFrame`). Keep the assertions (count, header sizes, lattice regularity, full-res-not-downsampled, coord-pending retry). Add: colour frame at stride 1 equals the source buffer (1:1, no copy). +- `docs/moonmodules/light/drivers/PreviewDriver.md` + `core/HttpServerModule.md`: rewrite to the streamed model (no buffers; positions once; colours 1:1 from the producer buffer; every-Nth downsample; begin/push/end wire). +- `docs/architecture.md`: add the preview **memory model** to the output-stage/UI section — the preview streams the producer buffer with no intermediate copy; this is the data-over-objects / minimal-memory principle applied to the preview. + +## Files +- `src/core/BinaryBroadcaster.h` — begin/push/end (done); remove `broadcastBinary` chunk-form + `lastDrainTicks` if dead. +- `src/core/HttpServerModule.h/.cpp` — begin/push/end impl (done); remove staging buffer + drain machinery + stage/DIRECT. +- `src/light/drivers/PreviewDriver.h` — stream both passes; drop `coords_`/`rgb_`/`sampledIdx_`; keep cap + adaptive. +- `test/unit/light/unit_PreviewDriver.cpp` — mock + assertions for the streamed model. +- docs: PreviewDriver.md, HttpServerModule.md, architecture.md. + +## Verification +- Host: build (-Werror), ctest, scenarios, spec, platform-boundary. +- ESP32: S3 + classic build. +- **Classic 128² (the target):** with `coords_`/`rgb_`/`sampledIdx_`/staging gone, confirm the render buffer allocates AND the preview streams at 128² without those competing 49 KB blocks; measure `freeInternal`/`maxBlock` to confirm the contiguous-RAM pressure is relieved. Confirm full-res streams 1:1 (no moiré) and a grid past the cap downsamples cleanly (no moiré, matched colour/coord counts). +- S3/P4: confirm no regression (full-res 128²+ still streams; adaptive downscale still engages on a slow link). + +## Risks / notes +- **Streaming is synchronous on the preview loop** (rate-limited ≤ fps, off the LED render tick). A slow client is closed (bounded), never an unbounded tick stall. The adaptive downscale shrinks frames on slow links so per-tick send stays small. +- **Multi-client**: each pushed slice fans to all clients in order; a forward-only producer (forEachCoord / buffer walk) is walked once per frame, slices sent to all. Fine for the handful of WS clients. +- **cpl≠3 (RGBW)** stays a per-light 3-byte push (no buffer); cpl==3 is the bulk 1:1 push. +- This is a net **subtraction**: removes ~3 buffers + the staging/drain code; the colour hot path becomes "stream the buffer." diff --git a/docs/install/README.md b/docs/install/README.md index e581939..fb3fc5d 100644 --- a/docs/install/README.md +++ b/docs/install/README.md @@ -7,7 +7,7 @@ This directory holds the source for the **custom installer page** (driven by End users land here, pick a channel + device, click Install. The browser flashes the device over USB (Web Serial → ESP32), runs Improv-Serial provisioning, then pushes the picked device-model's whole config over the **same serial port** as REST -operations (**"Improv = REST over serial"**: SET_DEVICE_MODEL + APPLY_OP) — all from +operations (**"Improv = REST over serial"**: `APPLY_OP` frames, one per control/module) — all from the same orchestrator, no ESP Web Tools dependency. Pushing over serial (not HTTP) is what makes the deployed HTTPS installer work: a browser blocks an HTTPS page from POSTing to a plain-`http://` device (mixed-content), so the old HTTP fan-out + the diff --git a/docs/install/firmwares.json b/docs/install/firmwares.json index 7384da9..f9e6904 100644 --- a/docs/install/firmwares.json +++ b/docs/install/firmwares.json @@ -5,7 +5,7 @@ "chip": "esp32", "eth_only": false, "ships": true, - "description": "ESP32 classic — WiFi + Ethernet (RMII; per-board pins/PHY from deviceModels.json, Olimex defaults)." + "description": "ESP32 classic — WiFi + Ethernet (RMII; per-board pins/PHY from deviceModels.json, default LAN8720 pins)." }, { "name": "esp32-16mb", diff --git a/docs/install/improv-frame.js b/docs/install/improv-frame.js index 409ca0d..9051df8 100644 --- a/docs/install/improv-frame.js +++ b/docs/install/improv-frame.js @@ -12,11 +12,6 @@ // [I][M][P][R][O][V][version=1][type][length][payload×length][checksum] // checksum = sum-mod-256 of the first 9+length bytes. -// SET_DEVICE_MODEL vendor RPC command ID. High end of the conventional 0x80-0xFE -// vendor extension range. Matches the device-side handler at -// src/platform/esp32/platform_esp32_improv.cpp. -export const IMPROV_CMD_SET_DEVICE_MODEL = 0xFE; - // SET_TX_POWER vendor RPC command ID — the pre-association TX-power cap for boards // whose LDO browns out at full power. Sent BEFORE provisioning so the very first // association runs capped. Matches the device-side handler. diff --git a/docs/install/index.html b/docs/install/index.html index fc13db4..65a70fc 100644 --- a/docs/install/index.html +++ b/docs/install/index.html @@ -692,7 +692,7 @@

projectMM Installer