From 86fea06b28c1ddae7339a067f4929560164d0be1 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Sat, 30 May 2026 16:08:29 +0200 Subject: [PATCH 01/72] Plan a new scheduler + dataflow DAG. DAG = Directed [Edge] Acyclic Graph --- WIP.md | 2 + builder/PLAN-scheduler.md | 1479 +++++++++++++++++++++++++++++++++++++ builder/PLAN.md | 8 + 3 files changed, 1489 insertions(+) create mode 100644 builder/PLAN-scheduler.md diff --git a/WIP.md b/WIP.md index e96b2986..58285de7 100644 --- a/WIP.md +++ b/WIP.md @@ -432,6 +432,8 @@ Python scripts are reserved for non-render concerns: one-off content conversion The site builds via [builder/](builder/), a custom Node.js static site generator (`tbdocs`). See [builder/PLAN.md](builder/PLAN.md) for the architecture overview, [builder/README.md](builder/README.md) for the quickstart, and the [tbdocs Internals](docs/Documentation/Builder.md) site page for the high-level tour. +A planned task-graph scheduler / parallelisation pass is designed in [builder/PLAN-scheduler.md](builder/PLAN-scheduler.md) (not yet implemented). Hand a fresh session that file and `Phase 0` as the entry point. + Historical engineering notes from the Jekyll era --- the original build pipeline, the HTML-compress plugin, the per-phase optimisation passes that preceded the JS port, the migration notes, and the Phase 11 parity-update retrospective --- live in [WIP.OldJekyll.md](WIP.OldJekyll.md). ## Build / preview diff --git a/builder/PLAN-scheduler.md b/builder/PLAN-scheduler.md new file mode 100644 index 00000000..f107a4af --- /dev/null +++ b/builder/PLAN-scheduler.md @@ -0,0 +1,1479 @@ +# Task-graph scheduler -- design sketch + +## Current state + +The build pipeline lives in `builder/`. The orchestrator is +`tbdocs.mjs`'s `runBuild()`, which today is a mostly-linear sequence of +awaited async calls on the main thread, with a sprinkling of +cooperative concurrency (`Promise.all` barriers and one background +`buildInfoPromise`). There are **no worker threads**; every CPU-bound +phase blocks the main thread. + +An earlier round of parallelization work (a `CheckPool` of persistent +link-checker workers and a bespoke `render-worker.mjs` driven by a +hand-coded message protocol) was reverted; this plan replaces both +with one task-graph abstraction. + +### Key files + +| File | Role | +|---|---| +| `tbdocs.mjs` | Orchestrator: `runBuild()`, CLI parsing, summary output | +| `template.mjs` | `templatePhase()`, internal `buildInit()` | +| `render.mjs` | `renderPhase()`, `createMarkdownIt()`, `buildLinkTables()` | +| `highlight.mjs` | `initHighlighter()` -- Shiki WASM init | +| `discover.mjs` | `discover()` -- walk source tree, parse frontmatter | +| `nav.mjs` | `computeNav()` -- build nav tree from pages | +| `seo.mjs` | `precomputeSeo()` -- derive SEO titles/URLs | +| `book.mjs` | `resolveBookChapters()` -- resolve book chapter list | +| `build-info.mjs` | `captureBuildInfo()` -- git rev-parse/log | +| `scss.mjs` | `compileScss()` -- sass compilation | +| `mermaid.mjs` | `regenerateMermaid()` -- stale SVG regen via puppeteer | +| `data.mjs` | `loadData()` -- load `_book.yml` | +| `write.mjs` | `writePhase()` -- write pages + static files to `_site/` | +| `redirects.mjs` | `deriveRedirectStubs()`, `writeRedirects()` | +| `sitemap.mjs` | `deriveSitemapUrls()`, `writeSitemap()` | +| `search.mjs` | `writeSearchData()` | +| `offline.mjs` | `writeOffline()` -- produce `_site-offline/` | +| `pdf.mjs` | `writePdf()` -- produce `_site-pdf/` | +| `serve.mjs` | `runServe()` -- dev server with watcher + rebuild | + +There is no `render-worker.mjs`, no `cpu-worker.mjs`, no `CheckPool`, +no `createRenderPool()`. The build is single-threaded except for I/O. + +### Current dataflow + +`runBuild()` reads top-to-bottom; the only off-main-thread work is the +git shell-outs inside `captureBuildInfo()` (launched as a background +promise and awaited later). Approximate wall-clock numbers from a +recent clean build are noted in parentheses: + +``` +mermaid (~2 ms; ~150 ms when SVGs regenerate) + ↓ +scss (~700 ms — CPU-bound on main thread) + ↓ +load _config.yml + apply CLI overrides + ↓ +buildInfoPromise = captureBuildInfo() (background: git shell-outs) + ↓ +discover (~135 ms — fs traversal + frontmatter parse) + ↓ +nav (~8 ms) + ↓ +initHighlighter (~50–100 ms — Shiki WASM init; overlaps with buildInfo) + ↓ +buildLinkTables + createMarkdownIt + precomputeSeo + loadData ++ resolveBookChapters (~110 ms together — "markdown-init" + "seo" + "book") + ↓ +await buildInfoPromise (~0 ms; usually already settled) + ↓ +renderPhase (~2700 ms — CPU-bound; cooperative Promise.all over pages) + ↓ +templatePhase (~800 ms — CPU-bound; same shape) + ↓ +writePhase + ├─ Promise.all { writePages | copyTheme | copyStaticFiles } + └─ writeGeneratedAssets (~625 ms total) + ↓ +Promise.all { writeRedirects, writeSitemap, writeSearchData } (~200 ms) + ↓ +writeOffline (~1100 ms) + ↓ +writePdf (~240 ms) +``` + +Wall-clock total is roughly **6.7 seconds**. The dominant terms are +`render` (~2.7 s), `writeOffline` (~1.1 s), `template` (~0.8 s), +`scss` (~0.7 s), and `write` (~0.6 s). + +Visible idle time on the main thread: + +- `scss` and `mermaid` run before `discover`, both serially. Neither + depends on `discover`'s output. +- `buildInfo`'s git shell-outs already overlap with the + discover/nav/markdown-init/seo chain, but everything *else* in that + chain runs serially even though `discover` → `nav` → + `markdown-init` → `seo` → `book` is the only true dependency edge. +- `writeOffline` and `writePdf` are independent of each other; both + read `_site/` (already written by `writePhase`) and write into + independent output trees. They run sequentially today. +- `renderPhase` and `templatePhase` are CPU-bound and block the main + thread completely. `Promise.all(pages.map(...))` is cooperative + concurrency only -- it interleaves on a single thread. + +### Data shapes and in-place mutation + +The two large structures that flow through the pipeline: + +**`pages[]`** -- array of ~857 page objects. After discover, each has: +``` +{ srcPath, srcRel, ext, frontmatter, rawContent, permalink, destPath, + layoutDefault, imageScope } +``` +Later phases mutate **in place**, adding: `navPath`, `breadcrumbs`, +`children`, `navLevels` (nav); `seoTitle`, `seoFullTitle`, +`seoCanonical`, `seoIsHome` (seo); `renderedContent` (render); +`html` (template). The current pipeline relies on this in-place +enrichment -- every consumer assumes the same page object accumulates +fields as it flows through phases. + +**`staticFiles[]`** -- array of ~214 static file descriptors: +``` +{ srcPath, srcRel, destRel, size } +``` + +**`config`** -- the parsed `_config.yml` object. Small, ~30 keys. +Read-only after initial CLI override merges. + +**`navTree`** -- array of `NavNode` objects (recursive tree). ~857 +nodes total. Only consumed by `buildInit()` → `renderSidebar()`. + +The mutation-in-place pattern matters for the scheduler design: +mutations performed on a worker's structured-clone copy do not reach +the main-thread master unless explicitly merged. See §Page deltas +below. + +## Setup + +No new runtime dependencies. The scheduler, the worker pool, and the +worker dispatcher are all in-tree code -- ~150 LOC for the scheduler, +~50 LOC for `WorkerPool`, ~30 LOC for the worker dispatcher and its +handler table. A general-purpose pool library like **piscina** was +considered but the project's use is narrow enough (fixed pool size, +one task per worker at a time, no recycling, no dynamic scaling, no +abort signals) that the dependency cost outweighs the saved code, and +an added dep widens the supply-chain attack surface. + +## Model + +The build is a DAG of **tasks**. Each task has a unique string ID, +takes an input map `{ [predecessorId]: output }`, produces an immutable +output, and declares which downstream tasks receive (slices of) that +output. + +The **scheduler** lives on the main thread. It tracks task +dependencies, decides what's ready, and dispatches. The worker pool +(`WorkerPool` -- a ~50 LOC in-tree class wrapping +`node:worker_threads`) handles everything below the task-graph layer: +spawning workers at construction, named dispatch, idle/busy +bookkeeping, lifecycle. + +Each task carries a `runOnMain: true` flag if it must execute on the +main thread -- for tasks that own the master `pages[]` merge, mutate +state in place, or do I/O that coordinates with main-thread state. +All other tasks run on a worker, dispatched by name. + +``` +┌─────────────────────────────────────────────────┐ +│ Main thread (scheduler) │ +│ │ +│ tasks: Map │ +│ pending: Map │ +│ ready: TaskDef[] │ +│ results: Map │ +│ state: SharedState (§Shared state) │ +│ │ +│ on task complete: │ +│ store result, run task.submit() to route │ +│ output to downstream tasks, check newly │ +│ ready, dispatch each to pool or main │ +└────────────┬────────────────────────────────────┘ + │ + ▼ WorkerPool ── named handlers in cpu-worker.mjs +``` + +## Task placement (main vs worker) + +Mutating a worker's local `pages[]` doesn't reach `state.pages` unless +the mutation is explicitly shipped back as a delta. The current code +mutates pages in place across nav / seo / render / template, so every +mutating step must be modeled deliberately. The cheap way out is: +**keep small mutating steps on the main thread** and only ship pages +across the boundary when there's a real CPU win to amortize the copy. + +The split: + +| Task | Placement | Why | +|---|---|---| +| `config` | M | Trivial fs read, no benefit to round-trip | +| `discover` | M | ~135 ms; output mutates pages[] in place | +| `nav` | M | ~8 ms; mutates pages with navPath/navLevels/breadcrumbs/children | +| `markdownInit` | M | ~63 ms; produces an `md` instance (NOT serializable -- can't cross boundary) | +| `seo` | M | ~34 ms; mutates pages with seoXxx fields | +| `loadData` | M | Trivial fs read | +| `resolveBookChapters` | M | Mutates `state.site.bookData._chapters` with refs into `state.pages` -- identity-critical | +| `buildInit` | M | Tiny; consumes navTree, produces ~50 KB of html strings | +| `deriveRedirects` | M | Pure compute, ~ms | +| `deriveSitemap` | M | Pure compute, ~ms | +| `dispatch` | M | Slices `state.pages` into render chunks; no benefit on a worker | +| `renderJoin` | M | Pure barrier | +| `write` | M | Owns `state.pages` + `state.staticFiles` reads; I/O dominated | +| `searchData` | M | ~40 ms; owns pages read | +| `writeAux` | M | Owns pages + bookData reads | +| `writeOffline` | M | I/O dominated (~1.1 s); see §Post-write tasks for the workerize-or-not call | +| `writePdf` | M | I/O dominated (~240 ms); ditto | +| `buildInfo` | **W** | Free overlap with the main spine | +| `scss` | **W** | ~700 ms -- the biggest seed-task parallelism win | +| `mermaid` | **W** | ~2 ms idle, ~150 ms when SVGs regen; runs concurrently with discover | +| `render:i` | **W** | The big win -- ~2.7 s of CPU work fans out across N cores | + +The worker side ships with four handlers: `scss`, `mermaid`, +`buildInfo`, `render`. Plus the parentPort dispatcher (~15 LOC). +Everything else is plain main-thread code wrapped in a task envelope. + +This keeps `pages[]` from crossing the worker boundary except for the +render fan-out (which only ships per-page slices, not the master). +The seo / nav / markdown-init mutations stay on the main-thread +master directly -- no delta merge needed. + +## Target dataflow + +`[W]` = pool worker; `[M]` = main thread (`runOnMain: true`). + +``` +Seeds (workers, concurrent): + buildInfo [W] ──────────────────────────────────────────┐ + scss [W] ──────────────────────────────────────────┤ + mermaid [W] ──────────────────────────────────────────┤ + │ +Main spine (sequential, on M): │ + config │ + └─→ discover ──┬─→ deriveRedirects ─────────────────┐ │ + ├─→ deriveSitemap ─────────────────┤ │ + └─→ nav │ │ + └─→ markdownInit │ │ + ├─→ seo │ │ + │ └─→ loadData │ │ + │ └─→ resolveBookChapters + │ ↓ │ │ + └─→ buildInit │ │ │ + ↓ │ │ │ + └──────────┴─→ dispatch ◄── buildInfo joins here + │ +Render fan-out (workers, concurrent): │ + ┌────────────────────────────────────────────────────┘ + │ + render:0 [W] render:1 [W] ... render:N-1 [W] + │ + ▼ + renderJoin [M] ◄── waits for all render:i + │ +Write fence: │ scss [W], mermaid [W] join here too + ▼ + write [M] ◄── reads state.pages, state.staticFiles + │ + ▼ + searchData [M] + │ + ▼ + writeAux [M] ◄── derived redirects + sitemap join here + │ │ + ┌──────────┘ └──────────┐ + ▼ ▼ + writeOffline [M] writePdf [M] + │ │ + └─────────────┬─────────────┘ + ▼ + done +``` + +Edges into `dispatch`: `buildInit`, `resolveBookChapters`, +`buildInfo`. +Edges into `write`: `renderJoin`, `scss`, `mermaid`. +Edges into `writeAux`: `searchData`, `deriveRedirects`, `deriveSitemap`. + +Three structural wins over the serial baseline: + +1. **`scss`, `mermaid`, `buildInfo` overlap with the main spine.** The + main spine (discover → nav → markdownInit → seo → loadData → + resolveBookChapters + buildInit) takes ~250 ms total. `scss` takes + ~700 ms. The overlap saves ~250 ms of `scss` from the critical path + (not the full ~700 ms -- after the spine finishes, ~450 ms of scss + is still on the critical path until `write` runs). +2. **`render:0..N` fans out across CPUs.** Today's ~2.7 s of cooperative + render + ~0.8 s template = ~3.5 s of CPU work, all on one thread. + Across N workers this compresses to ~`3500 / N + dispatch overhead`. + On a 4-core box, ~875 ms wall-clock (saving ~2.6 s). On an 8-core + box, ~440 ms (saving ~3 s). Dispatch overhead is ~50 ms. +3. **`writeOffline` and `writePdf` overlap on async I/O.** Both stay + `runOnMain` initially -- they share the main thread for their CPU + sections but `await fs.writeFile`-style I/O windows interleave. The + gain is the shorter of their two CPU sections (~240 ms). See + §Post-write tasks for the case to workerize one of them later. + +**Mermaid → staticFiles ordering.** Under the scheduler `mermaid` and +`discover` run in parallel, so freshly-emitted SVGs aren't in +`state.staticFiles` after discover. The mermaid task's `execute()` +(running on a worker) does the full `fs.stat` for each managed SVG +and returns a list of `{ srcPath, srcRel, destRel, size }` descriptors; +the `submit()` does only a **synchronous** push into +`state.staticFiles`. Putting the stat in `submit()` would race with +downstream consumers since `submit` is called synchronously by the +scheduler and cannot await. + +## Page deltas (mutation merge pattern) + +The render fan-out is the only place where pages cross the worker +boundary. The pattern: + +- Each `render:i` task receives a chunk of pages (worker's clone). +- The worker mutates its local pages with `renderedContent` and `html`. +- The task returns a **delta**: an array of + `[{ destPath, renderedContent, html }]` -- only the changed fields, + keyed by `destPath`. +- `render:i.submit()` walks the delta on the main thread, looks up + each page via `state.pageByDest`, and assigns the fields onto the + master page object. + +The full pages array never crosses back across the boundary; only the +output deltas do. `state.pageByDest` is built once in +`discover.submit()`: + +```js +discover.submit(out, emit, state) { + state.pages = out.pages; + state.staticFiles = out.staticFiles; + state.site.config = out.config; + for (const p of out.pages) state.pageByDest.set(p.destPath, p); + emit("nav", out); + emit("deriveRedirects", out); + emit("deriveSitemap", out); +} +``` + +After this initial assignment, `state.pages` is mutated in place; +no task ever replaces it. `state.pageByDest` stays valid for the +whole build. + +For tasks that run on `[M]` (nav, seo, etc.), the mutation is direct +on `state.pages` -- no delta needed. + +## Task definition + +Each task is a plain object: + +- **`expected`**: array of predecessor task IDs. The scheduler runs + the task only when every expected ID has submitted its output. An + empty array means a seed task (dispatchable immediately). +- **`handler`** *(optional, worker tasks)*: the worker dispatcher's + named handler. Defaults to the task's own ID. Used so multiple task + IDs can share one worker function + (e.g. `render:0`, `render:1`, ... → `"render"`). +- **`runOnMain: true`** *(optional)*: execute on the main thread + instead of dispatching to the pool. The `execute()` function + receives `(inputs, ctx, state)` -- where `state` is the + `SharedState` instance -- and may mutate it. +- **`execute(inputs, ctx [, state])`**: runs the task body. On a + worker, runs as the dispatch table's named handler. On main, runs + synchronously through the scheduler. Returns an output value. +- **`submit(output, emit [, state, scheduler])`**: runs **synchronously** + on the main thread after `execute` resolves. Calls + `emit(targetTaskId, dataSlice)` to route (slices of) the output to + downstream tasks. May mutate `state`. May not perform async work -- + see §Scheduler core. The optional `scheduler` arg is used only by + tasks that dynamically register downstream tasks (see `dispatch`). + +Representative task defs: + +```js +const TASKS = { + config: { + expected: [], + runOnMain: true, + async execute(_, ctx) { + const text = await fs.readFile( + path.join(ctx.srcRoot, "_config.yml"), "utf8"); + const config = yaml.load(text); + if (ctx.opts.baseurl != null) config.baseurl = ctx.opts.baseurl; + if (ctx.opts.url != null) config.url = ctx.opts.url; + return { config }; + }, + submit(out, emit) { emit("discover", out); }, + }, + + buildInfo: { + expected: [], + async execute() { return { buildInfo: await captureBuildInfo() }; }, + submit(out, emit) { emit("dispatch", out); }, + }, + + scss: { + expected: [], + async execute(_, ctx) { return { scssResult: await compileScss(ctx.srcRoot) }; }, + submit(out, emit) { emit("write", out); }, + }, + + mermaid: { + expected: [], + async execute(_, ctx) { + // The worker stats every managed SVG and returns full descriptors. + // Stat-in-submit on main would race with downstream readers. + const stats = await regenerateMermaid(ctx.srcRoot); + // stats.svgFiles: [{ srcPath, srcRel, destRel, size }, ...] + return { mermaidStats: stats }; + }, + submit(out, emit, state) { + const known = new Set(state.staticFiles.map((f) => f.srcRel)); + for (const f of out.mermaidStats.svgFiles ?? []) { + if (!known.has(f.srcRel)) state.staticFiles.push(f); + } + emit("write", out); + }, + }, + + discover: { + expected: ["config"], + runOnMain: true, + async execute({ config: { config } }, ctx) { + const { pages, staticFiles } = await discover( + ctx.srcRoot, config.exclude ?? []); + return { pages, staticFiles, config }; + }, + submit(out, emit, state) { + state.pages = out.pages; + state.staticFiles = out.staticFiles; + state.site.config = out.config; + for (const p of out.pages) state.pageByDest.set(p.destPath, p); + emit("nav", out); + emit("deriveRedirects", out); + emit("deriveSitemap", out); + }, + }, + + nav: { + expected: ["discover"], + runOnMain: true, + execute(_, ctx, state) { + const { navTree } = computeNav(state.pages, state.site.config); + state.site.navTree = navTree; + return {}; // mutates state in place + }, + submit(_, emit) { + emit("markdownInit", {}); + emit("buildInit", {}); + }, + }, + + buildInit: { + expected: ["nav"], + runOnMain: true, + execute(_, ctx, state) { + // buildInit() takes site.config + site.navTree; returns the + // ~50 KB of pre-rendered sidebar + header + svg-sprite HTML used + // by templatePhase. + return { initData: buildInitFn(state.site) }; + }, + submit(out, emit) { emit("dispatch", out); }, + }, + + markdownInit: { + expected: ["nav"], + runOnMain: true, + async execute(_, ctx, state) { + // Main's own initHighlighter cache -- workers maintain theirs + // independently. Both call paths converge on the same Shiki + // initialisation work, but the singletons are per-thread. + const highlighter = await initHighlighter(); + const linkTables = buildLinkTables(state.pages); + const baseurl = String(state.site.config.baseurl || ""); + const staticFileSet = new Set(state.staticFiles.map(s => s.srcRel)); + state.site.highlighter = highlighter; // write reads .themeCss from here + state.site.markdown = createMarkdownIt({ + highlighter, linkTables, baseurl, staticFiles: staticFileSet, + }); + // linkTables travels to render workers as a serialized payload. + state.site.linkTablesSerialized = serializeLinkTables(linkTables); + return {}; + }, + submit(_, emit) { + emit("seo", {}); + emit("loadData", {}); + }, + }, + + seo: { + expected: ["markdownInit"], + runOnMain: true, + execute(_, ctx, state) { + const { seoSiteTitle, seoLogoUrl } = precomputeSeo( + state.pages, state.site.config, state.site.markdown); + state.site.seoSiteTitle = seoSiteTitle; + state.site.seoLogoUrl = seoLogoUrl; + return {}; + }, + submit(_, emit) { emit("resolveBookChapters", {}); }, + }, + + loadData: { + expected: ["markdownInit"], + runOnMain: true, + async execute(_, ctx, state) { + const data = await loadData(ctx.srcRoot); + state.site.data = data; + state.site.bookData = data.book ?? null; + return {}; + }, + submit(_, emit) { emit("resolveBookChapters", {}); }, + }, + + resolveBookChapters: { + expected: ["seo", "loadData"], + runOnMain: true, + execute(_, ctx, state) { + // Mutates state.site.bookData with _chapters arrays whose + // entries are refs into state.pages. Identity-critical: + // render:i.submit() merges renderedContent into those same + // page objects, so writePdf later sees the rendered bodies + // via bookData._chapters[i].renderedContent. + resolveBookChapters(state.site.bookData, state.pages); + return {}; + }, + submit(_, emit) { emit("dispatch", {}); }, + }, + + deriveRedirects: { + expected: ["discover"], + runOnMain: true, + execute(_, ctx, state) { + // redirects.mjs's deriveRedirectStubs uses a layout-based + // filter (layout !== "book-combined") rather than checking + // page.html, so the derive can run before template. + return { stubs: deriveRedirectStubs(state.pages, state.site) }; + }, + submit(out, emit) { emit("writeAux", out); }, + }, + + deriveSitemap: { + expected: ["discover"], + runOnMain: true, + execute(_, ctx, state) { + return { urls: deriveSitemapUrls(state.pages, state.site) }; + }, + submit(out, emit) { emit("writeAux", out); }, + }, + + dispatch: { + expected: ["buildInit", "resolveBookChapters", "buildInfo"], + runOnMain: true, + execute({ buildInit: { initData }, buildInfo: { buildInfo } }, ctx, state) { + // Read pages directly from state.pages -- main-thread access, + // no need to ship them through the input map. + const chunks = chunkPages(state.pages, ctx.workerCount); + const siteData = { + config: state.site.config, + seoSiteTitle: state.site.seoSiteTitle, + seoLogoUrl: state.site.seoLogoUrl, + }; + return { + chunks, siteData, initData, buildInfo, + linkTablesData: state.site.linkTablesSerialized, + staticFilesArr: state.staticFiles.map(f => f.srcRel), + baseurl: String(state.site.config.baseurl || ""), + }; + }, + submit(out, emit, _state, scheduler) { + const N = out.chunks.length; + + // Register the barrier with the dynamic predecessor count. + // write declares "renderJoin" statically; emit() looks up + // pending entries by id, not by source, so the static + // declaration is satisfied as soon as renderJoin submits. + scheduler.register("renderJoin", { + expected: Array.from({ length: N }, (_, i) => `render:${i}`), + runOnMain: true, + execute() { return {}; }, + submit(_, emit) { emit("write", {}); }, + }); + + for (let i = 0; i < N; i++) { + const id = `render:${i}`; + scheduler.register(id, { + expected: [], + handler: "render", + submit(renderOut, emit, state) { + for (const r of renderOut) { + const p = state.pageByDest.get(r.destPath); + if (!p) continue; + p.renderedContent = r.renderedContent; + if (r.html !== undefined) p.html = r.html; + } + emit("renderJoin", renderOut); + }, + }); + scheduler.seed(id, { + siteData: out.siteData, + initData: out.initData, + linkTablesData: out.linkTablesData, + staticFilesArr: out.staticFilesArr, + baseurl: out.baseurl, + buildInfo: out.buildInfo, + chunk: out.chunks[i], + }); + } + }, + }, +}; +``` + +`chunkPages` rounds up to keep all chunks non-empty when there are +fewer pages than workers (e.g. dry-run paths or future incremental +builds): + +```js +function chunkPages(pages, workers) { + const n = Math.min(workers, pages.length); // never more chunks than pages + if (n === 0) return []; + const size = Math.ceil(pages.length / n); + const chunks = []; + for (let i = 0; i < pages.length; i += size) chunks.push(pages.slice(i, i + size)); + return chunks; +} +``` + +Two non-obvious bits in `dispatch.submit`: + +1. **Dynamic registration.** `dispatch` doesn't know N at definition + time, so it calls `scheduler.register(taskId, def)` per chunk plus + one for `renderJoin`. +2. **Why `renderJoin` exists at all.** Each `render:i.submit()` already + emits into the page-deltas merge, and could emit directly to + `write`. But `write.expected` is declared statically with + `["renderJoin", "scss", "mermaid"]` -- mutating it from + `dispatch.submit` to add the N dynamic render predecessors would be + awkward. The barrier is the cleaner expression: register it once + with the right count, let write keep its static `expected`. + +## Shared state + +```js +class SharedState { + pages = []; // master copy; mutated in place by [M] tasks and by render delta merges + staticFiles = []; // master copy; mermaid.submit appends new SVG descriptors + site = {}; // config, navTree, seoSiteTitle, seoLogoUrl, bookData, data, markdown, ... + pageByDest = new Map(); // destPath → page; built once in discover.submit +} +``` + +**After the initial discover.submit assignment, `state.pages` is +never replaced** -- only mutated in place. Every phase that adds +fields to pages does so on the same object identities, which is what +keeps `bookData._chapters` refs (set by `resolveBookChapters`) +pointing at the rendered pages by the time `writePdf` walks them. + +Worker tasks receive structured-clone snapshots of whatever input they +need -- they cannot see the master and cannot mutate it. Their +`submit()` runs on the main thread, where it merges the worker's +output (a delta keyed by `destPath` for page mutations) into `state`. + +This is the explicit form of what today's `runBuild` does implicitly +through closure mutation. Making it explicit lets `serve.mjs` re-use +the scheduler across rebuilds without leaking state, and gives +post-write tasks a clean read path. + +## Scheduler core + +The scheduler is a thin coordinator. The pool is constructed externally +and passed in. + +```js +class Scheduler { + constructor({ pool, tasks }) { + this.pool = pool; // WorkerPool instance + this.tasks = new Map(Object.entries(tasks)); + this.pending = new Map(); + this.ready = []; + this.results = new Map(); + this.timings = new Map(); + this.state = new SharedState(); + this.inFlight = 0; + [this._doneP, this._doneResolve, this._doneReject] = deferred(); + for (const [id, def] of this.tasks) this._initPending(id, def); + } + + _initPending(id, def) { + this.pending.set(id, { expected: def.expected.length, received: new Map() }); + } + + register(id, def) { this.tasks.set(id, def); this._initPending(id, def); } + + // Seed a freshly-registered task directly (used by dispatch.submit + // to feed each render:i its chunk without going through emit()). + seed(id, inputs) { + const def = this.tasks.get(id); + this.pending.delete(id); + this.ready.push({ id, def, inputs }); + this._flush(); + } + + emit(targetId, data, sourceId) { + const entry = this.pending.get(targetId); + if (!entry) throw new Error(`unknown or already-dispatched task: ${targetId}`); + entry.received.set(sourceId, data); + if (entry.received.size === entry.expected) { + this.pending.delete(targetId); + const def = this.tasks.get(targetId); + this.ready.push({ id: targetId, def, inputs: Object.fromEntries(entry.received) }); + this._flush(); + } + } + + async start(ctx) { + this._ctx = ctx; + for (const [id, def] of this.tasks) { + if (def.expected.length === 0) this.ready.push({ id, def, inputs: {} }); + } + this._flush(); + return this._doneP; + } + + _flush() { + while (this.ready.length > 0) this._run(this.ready.shift()); + } + + _run(task) { + const start = Date.now(); + this.inFlight++; + const p = task.def.runOnMain + ? Promise.resolve(task.def.execute(task.inputs, this._ctx, this.state)) + : this.pool.run({ inputs: task.inputs, ctx: this._ctx }, + { name: task.def.handler ?? task.id }); + p.then( + (output) => this._onDone(task, output, start), + (err) => this._onError(task, err), + ); + } + + _onDone(task, output, start) { + this.timings.set(task.id, { start, end: Date.now() }); + this.results.set(task.id, output); + this.inFlight--; + // submit() is invoked synchronously. It must not return a Promise + // (or, if it does, must not race with the emits it makes). Async + // work belongs in execute(). + task.def.submit( + output, + (tgt, data) => this.emit(tgt, data, task.id), + this.state, + this, + ); + if (this.inFlight === 0 && this.ready.length === 0 && this.pending.size === 0) { + this._doneResolve(this.results); + } + } + + _onError(task, err) { + this._doneReject(new Error(`task ${task.id} failed`, { cause: err })); + } + + summary() { + return [...this.timings.entries()] + .sort((a, b) => a[1].start - b[1].start) + .map(([id, { start, end }]) => `${id}=${end - start}ms`) + .join(" "); + } +} + +function deferred() { + let res, rej; + const p = new Promise((r1, r2) => { res = r1; rej = r2; }); + return [p, res, rej]; +} +``` + +The `WorkerPool` instance is constructed by `runBuild()` and injected +into the scheduler; the scheduler never sees `worker_threads` +directly. + +## Worker pool + +A minimal pool over `node:worker_threads`. One file, ~50 LOC. Spawns +`size` workers eagerly at construction (so WASM warmup overlaps with +seed-task work; see §Boot sequence), routes named tasks to whichever +worker is idle, queues the rest. No dynamic scaling, no recycling. + +```js +// builder/worker-pool.mjs + +import { Worker } from "node:worker_threads"; + +export class WorkerPool { + constructor(size, workerUrl) { + this._workerUrl = workerUrl; + this._idle = []; // Worker[] + this._busy = new Map(); // Worker → { resolve, reject } + this._queue = []; // pending { message, transferList, resolve, reject } + this._workers = Array.from({ length: size }, () => this._spawn()); + } + + _spawn() { + const w = new Worker(this._workerUrl); + w.on("message", (msg) => { + const entry = this._busy.get(w); + if (!entry) return; // ignore late messages + this._busy.delete(w); + this._idle.push(w); + if (msg.error) entry.reject(Object.assign(new Error(msg.error), { stack: msg.stack })); + else entry.resolve(msg.result); + this._drain(); + }); + w.on("error", (err) => { + // Worker crash: reject the in-flight task. The dead worker + // stays in this._workers (won't respawn -- see §Worker death + // policy) so the pool degrades to size-1 for the rest of the + // run. For a one-shot build, the resulting task rejection + // aborts via the scheduler's _onError path. + const entry = this._busy.get(w); + if (entry) { this._busy.delete(w); entry.reject(err); } + }); + this._idle.push(w); + return w; + } + + run(payload, { name, transferList } = {}) { + return new Promise((resolve, reject) => { + this._queue.push({ + message: { name, ...payload }, + transferList, + resolve, reject, + }); + this._drain(); + }); + } + + _drain() { + while (this._queue.length && this._idle.length) { + const w = this._idle.shift(); + const { message, transferList, resolve, reject } = this._queue.shift(); + this._busy.set(w, { resolve, reject }); + w.postMessage(message, transferList); + } + } + + destroy() { + return Promise.all(this._workers.map(w => w.terminate())); + } +} +``` + +What we explicitly do **not** support, vs. a general-purpose pool: +dynamic resizing, per-worker concurrency above 1, worker recycling +after N tasks, abort signals, task-priority queues, utilization +histograms. Each is real complexity we don't need. + +### Worker death policy + +If a worker crashes mid-task, `w.on("error")` rejects the in-flight +task and removes it from `_busy`. The crashed worker is NOT +respawned; `_workers[]` still lists it for `destroy()` (terminate is +idempotent on a dead worker), but it never returns to `_idle`. The +pool effectively shrinks by one for the remainder of the run. + +For a one-shot `runBuild`, the rejected task surfaces through the +scheduler's `_onError` → `_doneP.reject` and `runBuild` aborts. Fine +as-is. + +For `serve` mode, a crash permanently degrades the long-lived pool. +The current policy is "tell the user to restart serve"; respawn-on- +error is a follow-up if it ever happens in practice. + +### Worker spawn cost + +`new Worker(url)` costs ~50-100 ms per worker. We spawn +`os.availableParallelism()` of them at construction; they spawn +concurrently but Node's process model adds contention -- realistically +~100-200 ms before any task can `postMessage` to a free worker. This +is a one-shot cost in `runBuild`; for `serve` mode it amortizes to +zero across rebuilds. It's worth noting against Phase 1's expected +savings (~250 ms scss overlap is partly eaten by the ~100-200 ms +worker boot). + +## Worker + +Single file with named handlers in a dispatch table. The pool sends +`{ name, ...payload }`; the worker routes to the right handler and +posts back `{ result }` or `{ error, stack }`. Four handlers total +(`scss`, `mermaid`, `buildInfo`, `render`) plus the ~15 LOC dispatcher. + +```js +// builder/cpu-worker.mjs + +import { parentPort } from "node:worker_threads"; + +import { initHighlighter } from "./highlight.mjs"; +import { compileScss } from "./scss.mjs"; +import { regenerateMermaid } from "./mermaid.mjs"; +import { captureBuildInfo } from "./build-info.mjs"; +import { + createMarkdownIt, + buildLinkTables, + renderPhase, +} from "./render.mjs"; +import { templatePhase } from "./template.mjs"; + +// Start WASM init immediately, do NOT await. The module finishes +// loading synchronously so the parentPort.on('message') dispatcher is +// installed before the pool sends any work. Only the `render` handler +// awaits highlighterP. +const highlighterP = initHighlighter(); + +const handlers = { + async scss({ ctx }) { + return { scssResult: await compileScss(ctx.srcRoot) }; + }, + + async mermaid({ ctx }) { + return { mermaidStats: await regenerateMermaid(ctx.srcRoot) }; + }, + + async buildInfo() { + return { buildInfo: await captureBuildInfo() }; + }, + + async render({ inputs }) { + const { siteData, initData, linkTablesData, staticFilesArr, + baseurl, buildInfo, chunk } = inputs; + + const highlighter = await highlighterP; + const linkTables = reconstructLinkTables(linkTablesData); + const staticFiles = new Set(staticFilesArr); + const markdown = createMarkdownIt({ highlighter, linkTables, baseurl, staticFiles }); + + const site = { ...siteData, markdown, buildInfo }; + await renderPhase(chunk, site); + await templatePhase(chunk, site, initData); + + // book-combined pages have renderedContent but no html (Phase 8 + // handles them from renderedContent); send html: undefined for those. + return chunk.map(p => ({ + destPath: p.destPath, + renderedContent: p.renderedContent, + html: p.html, + })); + }, +}; + +parentPort.on("message", async (msg) => { + const { name, ...payload } = msg; + const handler = handlers[name]; + if (!handler) { + parentPort.postMessage({ error: `unknown task: ${name}` }); + return; + } + try { + const result = await handler(payload); + parentPort.postMessage({ result }); + } catch (err) { + parentPort.postMessage({ error: err.message, stack: err.stack }); + } +}); + +// linkTables values are page objects in the main pipeline, but +// resolveLink() in the relative-links plugin only reads .permalink. +// The serialized form ships [key, permalink] pairs; we reconstruct +// minimal { permalink } stubs in the worker. +function reconstructLinkTables({ byPath, byUrl, byRedirect }) { + const make = (pairs) => new Map(pairs.map(([k, pl]) => [k, { permalink: pl }])); + return { byPath: make(byPath), byUrl: make(byUrl), byRedirect: make(byRedirect) }; +} +``` + +The matching `serializeLinkTables` lives in `render.mjs` next to +`buildLinkTables` and is called from `markdownInit.execute()` on main: + +```js +// builder/render.mjs +export function serializeLinkTables(lt) { + const pairs = (m) => [...m.entries()].map(([k, p]) => [k, p.permalink]); + return { byPath: pairs(lt.byPath), byUrl: pairs(lt.byUrl), byRedirect: pairs(lt.byRedirect) }; +} +``` + +**`buildInit` export from template.mjs.** The `markdownInit` / +`buildInit` tasks both run on main; they call `template.mjs`'s +internal `buildInit()` helper, which is currently file-local. Phase 0 +of the migration re-exports it as `buildInitFn` (renamed to avoid +shadowing the task ID inside `tbdocs.mjs`). Today's `templatePhase()` +still calls the local function directly; the export adds no overhead. + +## Boot sequence (WASM init) + +Two independent `initHighlighter()` invocations: + +- **Main thread.** `markdownInit.execute()` awaits `initHighlighter()` + to build the shared markdown-it instance used by seo and (indirectly) + by `book.mjs`'s subtitle/intro rendering during `assembleBook`. The + cached singleton lives in `highlight.mjs`'s `cached` module-level + variable on the main thread. +- **Each worker.** `cpu-worker.mjs` calls `initHighlighter()` at module + scope without awaiting. Module evaluation finishes synchronously, so + `parentPort.on("message")` is installed before the pool dispatches. + Only the `render` handler awaits the promise. The `scss`, `mermaid`, + `buildInfo` handlers don't need a highlighter -- their workers can + service tasks while Shiki is still loading. + +The two contexts each have their own cached singleton. Total WASM +init cost is paid once per thread (main + N workers), all in parallel, +overlapping with worker spawn and the main spine. + +## Data transfer strategy + +### Small outputs (config, navTree, initData, buildInfo, scssResult) +Structured clone via `postMessage`. Negligible cost (< 1 ms). + +### `linkTables` (medium, ~857 entries × ~3 keys) +Serialized once on main inside `markdownInit.execute()` to +`linkTablesData` ([key, permalink] pairs, ~50 KB). Shipped to each +render worker via `dispatch`'s output; each worker reconstructs +minimal `{ permalink }` stubs. + +### Render chunk (medium-large, ~857/N pages including `rawContent`) +The biggest single transfer. The `dispatch` output's `chunks[i]` +contains roughly `pages.length / N` page objects with `rawContent` +attached. On a 4-worker box: ~215 pages × ~4 KB raw + frontmatter = +~860 KB per chunk × 4 workers = ~3.4 MB total ship-out. The deltas +returned are ~30 KB per worker (just destPath + renderedContent + +html). Two crossings per chunk; one-way ~3.4 MB total at chunk send, +much smaller at delta return. + +### Mutations that stay on the main thread +nav / seo / loadData / resolveBookChapters / buildInit run on main +against `state.pages` directly. No marshalling, no delta merge -- +mutations are immediately visible to downstream main-thread tasks. + +### SharedArrayBuffer broadcast (Phase 4, optional) +For the render fan-out, serialize `siteData + initData + linkTables` +once into a SharedArrayBuffer and pass the wrapper to all render tasks +via the pool's `transferList`. The SAB itself is shared memory, not +cloned. Each worker deserializes its slice. The SAB is read-only by +convention. Measure first -- likely modest savings vs. the N +structured-clone fan-outs, only worth doing if profiling shows +dispatch overhead is meaningful. + +## Error handling + +Three severity levels: + +1. **Fatal** (task throws): `_onError` rejects `_doneP`. The + orchestrator catches, prints, exits 1. Matches today's behavior + for nav integrity failures, unsupported layouts, redirect + collisions. The pool's outstanding work is implicitly cancelled + when the orchestrator calls `pool.destroy()` during shutdown. + +2. **Degraded** (task sets a flag): the task returns normally with a + `{ failed: true }` field in its output. Downstream tasks receive + the output (write still needs `scssResult` even if compilation + failed -- it just skips emitting the generated asset). After + `_doneP` resolves, the orchestrator checks results for degraded + flags and sets `process.exitCode`. Matches today's mermaid / scss + behavior. Applies symmetrically to both seed tasks: a sass + compile error sets `scssResult.failed`, a mermaid render error + sets `mermaidStats.failed`. + +3. **Setup skip** (puppeteer / sass missing): task returns + `{ setupSkipped: true }`. Downstream tasks see existing-on-disk + artifacts (mermaid: prior SVGs; scss: nothing emitted, but the + theme tree's hand-extracted CSS still applies). Not an error. + +4. **Worker death** (worker crashes, OOM, native segfault): the + pool's `w.on("error")` rejects the in-flight task; the rejection + surfaces through `_onError` as Fatal above. The dead worker is + not respawned (see §Worker death policy). + +## Serve / watch mode + +The pool is constructed in `serve.mjs`'s long-lived process and +re-used across rebuilds; only the `Scheduler` instance (and its +`SharedState`) is fresh per rebuild. Workers stay warm -- WASM, JIT, +module cache all survive. The worker spawn cost is paid once, at +`runServe()` startup. + +```js +// serve.mjs (sketch) +const pool = new WorkerPool(os.availableParallelism(), CPU_WORKER_URL); + +watcher.on("change", debounce(async () => { + const scheduler = new Scheduler({ pool, tasks: TASKS }); + await scheduler.start(ctx); +}, 100)); +``` + +Incremental invalidation (rebuild only changed tasks) is a much later +phase; defer. + +## Link checks (out of scope) + +The link-checker passes (`scripts/check_links.mjs`) currently run +**outside** the build, via `check.bat`. The earlier `CheckPool` +worker_threads integration in `tbdocs.mjs` has been removed; this plan +inherits that decision and does **not** re-integrate link-checking +into the scheduler. + +The scheduler design accommodates checks cleanly as `runOnMain: true` +tasks that delegate to a `CheckPool` instance passed in via `ctx`, so +the integration can be re-added as a follow-up phase if desired. The +shape would be: + +```js +checkOnline: { + expected: ["writeAux"], // _site/ must be fully written + runOnMain: true, + async execute(_, ctx) { + const r = await ctx.checkPool.run(buildCheckArgv("online", ctx.destRoot, ...)); + return { name: "online", ...r }; + }, + submit() { /* terminal */ }, +}, +``` + +But the initial scheduler migration treats `check.bat` as the +canonical post-build verifier and lands without touching it. + +## Post-write tasks + +The DAG nodes downstream of `render` and `scss`/`mermaid`. All +`write`-family tasks are `runOnMain: true` -- they own the master +`pages[]` and `state.site` reads; their CPU sections are short +relative to their I/O. + +```js +write: { + expected: ["renderJoin", "scss", "mermaid"], + runOnMain: true, + async execute({ scss: { scssResult }, mermaid: { mermaidStats } }, ctx, state) { + // render delta merges already happened in each render:i.submit(). + // mermaid.submit() already appended new SVG descriptors to + // state.staticFiles synchronously. + const generatedAssets = []; + if (state.site.highlighter?.themeCss) generatedAssets.push(/* tb-highlight.css */); + if (scssResult.compiled) generatedAssets.push(/* just-the-docs-combined.css */); + return writePhase(state.pages, state.staticFiles, { + destRoot: ctx.destRoot, + generatedAssets, + baseurl: String(state.site.config.baseurl || ""), + dryRun: ctx.opts.dryRun, + }); + }, + submit(out, emit) { emit("searchData", out); }, +}, + +searchData: { + expected: ["write"], + runOnMain: true, + async execute(_, ctx, state) { + return writeSearchData(state.pages, state.site, ctx.destRoot); + }, + submit(out, emit) { emit("writeAux", out); }, +}, + +writeAux: { + expected: ["searchData", "deriveRedirects", "deriveSitemap"], + runOnMain: true, + async execute({ deriveRedirects, deriveSitemap }, ctx, state) { + await Promise.all([ + writeRedirects(state.pages, state.site, ctx.destRoot, deriveRedirects.stubs), + writeSitemap (state.pages, state.site, ctx.destRoot, deriveSitemap.urls), + ]); + }, + submit(out, emit) { + emit("writeOffline", out); + emit("writePdf", out); + }, +}, + +writeOffline: { + expected: ["writeAux"], + runOnMain: true, + async execute(_, ctx, state) { + return writeOffline(state.pages, state.staticFiles, state.site, + ctx.destRoot, { /* ... */ }); + }, + submit() { /* terminal */ }, +}, + +// writePdf -- mirrors writeOffline. +``` + +**Restoring the derive-time exports.** `redirects.mjs` / +`sitemap.mjs` regain the `precomputedStubs` / `precomputedUrls` +passthrough parameters so the derive tasks' outputs can flow into the +write calls without re-deriving. + +**`deriveRedirectStubs` filter change.** The current filter +(`p.html !== undefined`) is set after template runs. Under the +scheduler, `deriveRedirects` runs concurrently with the main spine +and well before render+template. Change the filter in `redirects.mjs` +to `p.frontmatter.layout !== "book-combined"` (the property that +determines whether `html` will be set; known after discover) so the +derive can run at any point after discover. + +**Workerizing writeOffline or writePdf (deferred).** Both phases have +non-trivial CPU sections: writeOffline rewrites URLs across all 856 +HTML files (~700 ms CPU); writePdf assembles `book.html` via +`assembleBook` (~150 ms CPU). On `runOnMain` both serialize their CPU +work; only the async I/O windows interleave. The expected wall-clock +win from workerizing one of them (move offline to a worker, keep pdf +on main) is ~250 ms but at the cost of shipping `pages[]` and +`staticFiles[]` across the boundary. Profile first, then decide -- the +shipping cost may eat the gain. This is a Phase 3-follow-up choice, +not a Phase 3 requirement. + +## Timing / profiling + +The scheduler records `{ start, end }` per task. The summary is +formatted to match the current `t.lap()` output style: + +``` +config=1ms discover=98ms scss=1041ms mermaid=2ms buildInfo=8ms +nav=9ms buildInit=1ms markdownInit=63ms seo=34ms loadData=4ms +resolveBookChapters=7ms render:0=312ms render:1=298ms ... +write=542ms searchData=41ms writeAux=12ms +writeOffline=1210ms writePdf=352ms +``` + +## Entry point + +```js +// tbdocs.mjs + +import os from "node:os"; +import { WorkerPool } from "./worker-pool.mjs"; +import { Scheduler } from "./scheduler.mjs"; + +const CPU_WORKER_URL = new URL("./cpu-worker.mjs", import.meta.url); + +export async function runBuild(opts) { + const srcRoot = path.resolve(process.cwd(), opts.src); + const destRoot = path.resolve(opts.dest ?? path.join(srcRoot, "_site")); + + const workerCount = os.availableParallelism(); + const pool = new WorkerPool(workerCount, CPU_WORKER_URL); + + const scheduler = new Scheduler({ pool, tasks: TASKS }); + const ctx = { srcRoot, destRoot, opts, workerCount }; + + try { + const results = await scheduler.start(ctx); + console.log(scheduler.summary()); + // ... existing summary output using results + scheduler.state ... + return { pages: scheduler.state.pages, + staticFiles: scheduler.state.staticFiles, + site: scheduler.state.site, + destRoot }; + } finally { + await pool.destroy(); + } +} +``` + +## Migration path + +The current code is the simple serial baseline -- there is no +scaffolding to delete, only new pieces to add. Each phase keeps the +build working end-to-end and produces byte-identical output. + +**Where to start.** Implement the phases in order, beginning with +Phase 0. Commit after each phase. The verification gate at the end +of each phase block below is the done-signal -- if it doesn't pass +cleanly, the phase isn't done. + +**Historical context (cited inline below).** Some refactors restore +small carriers (function parameters, return-value fields) that +existed in an earlier WIP iteration of the parallelisation work -- +commit `5736fee4` ("WIP dataflow parallelization work") -- and were +reverted along with the threading shape they served. Where this plan +refers to "restoring" something, `git show 5736fee4 -- ` shows +the prior form for reference; do not re-introduce the whole reverted +shape, only the small pieces noted. + +### Suggested Claude model per phase + +To reduce session cost, the phases are labelled with the model that +fits each phase's nature. Sonnet is preferred when the spec is +precise enough that the implementation is a translation; Opus is +preferred when correctness depends on reasoning about concurrency, +data lifetime across the worker boundary, or low-level serialisation. + +| Phase | Model | Why | +|---|---|---| +| 0. Skeleton + small refactors | Sonnet | Code is given verbatim in §Worker pool / §Scheduler core. The five refactors are precisely-bounded edits to existing modules. No design judgement needed. | +| 1. Seeds + main-thread spine | Sonnet | Each task body is a thin wrapper around an existing phase function. The scheduler core is copy-from-plan. All mutation is on the main thread (no cross-worker identity yet). | +| 2. Render fan-out | Opus | Cross-thread structured-clone semantics, module-scope initialisation order, dynamic task registration, per-page delta-merge identity. Debugging concurrency bugs needs depth. | +| 3. Post-write tasks | Sonnet | All `runOnMain`; thin wrappers around existing write functions. No new concurrency surface beyond the already-built scheduler. | +| 3-follow-up. Workerize writeOffline | Opus to decide, Sonnet to implement | Profile-driven judgement call. If the move is chosen, the port itself is mechanical. | +| 4. SAB broadcast | Opus | Manual serialisation into shared memory; layout / endianness / varint choices need reasoning about memory ordering. | + +Escalate to Opus mid-phase if a Sonnet session hits a debugging block +it can't reason through. + +### Verification gate (applies to every phase) + +After each phase, run: + +```sh +build.bat && check.bat +``` + +The phase is done iff: + +- `build.bat` exits 0; no new warnings vs. the prior phase. +- The summary line shows `pages.length` at the current baseline + (857 today; the drift guard in `tbdocs.mjs` warns if it slips + below 836). +- All three `check.bat` passes return `0 issue(s)` for online and + offline, and the PDF pass's pre-existing broken-link count + matches the baseline (8 today; unrelated to the scheduler work). +- The scheduler's timing summary shows the **expected concurrency + pattern** for the phase (each phase block notes what to look for). + +The site is the regression bar; there is no separate unit-test +harness for builder/ to satisfy. + +### Phase 0: skeleton + small refactors + +**Suggested model:** Sonnet. + +Create three new modules: + +- `builder/worker-pool.mjs` -- the `WorkerPool` class (~50 LOC, see + §Worker pool). +- `builder/scheduler.mjs` -- the `Scheduler` class + `SharedState` + (~150 LOC, see §Scheduler core). +- `builder/cpu-worker.mjs` -- the worker harness (`parentPort` message + dispatcher + empty handlers map for now, ~15 LOC). + +Refactors needed (each is small and lands cleanly under today's serial +`runBuild`): + +- Re-export `buildInit` from `template.mjs` (currently file-local) as + `buildInitFn`. Today's `templatePhase()` still calls the local + function; the export is for the upcoming main-thread `buildInit` + task. +- Export `serializeLinkTables` from `render.mjs`. Today nothing calls + it; Phase 2 wires it up. +- Change the filter in `redirects.mjs`'s `deriveRedirectStubs` from + `p.html !== undefined` to `p.frontmatter.layout !== "book-combined"`. + Behaviour is identical under the current serial pipeline (template + has already run by the time writeRedirects fires), but the new + filter lets the derive step run before template under the scheduler. +- Restore the `precomputedStubs` / `precomputedUrls` passthrough + parameters on `writeRedirects` / `writeSitemap` (these were in commit + `5736fee4` and removed in the revert). +- Add `svgFiles: [{ srcPath, srcRel, destRel, size }, ...]` to + `regenerateMermaid`'s return value (this field was in `5736fee4` + and removed; the stat happens inside the existing + `regenerateMermaid` call). + +The build still runs from the existing `runBuild()` exactly as today. +No new runtime dependencies. + +**Deliverable:** new modules compile, refactors land under the serial +pipeline, build output unchanged. + +**Verification.** `build.bat && check.bat` clean. Output and timing +summary unchanged vs. before Phase 0 -- no scheduler is wired up +yet, so the build must look identical. + +### Phase 1: Seeds + main-thread spine + +**Suggested model:** Sonnet. + +Wire `runBuild()` to construct the pool, instantiate the scheduler, +and call `scheduler.start(ctx)`. Port: + +- **Worker seeds:** `config`, `buildInfo`, `scss`, `mermaid`. +- **Main-thread spine:** `discover`, `nav`, `markdownInit`, `seo`, + `loadData`, `resolveBookChapters`, `buildInit`, `deriveRedirects`, + `deriveSitemap`. + +The existing `renderPhase` → `templatePhase` → `writePhase` → +post-write code stays in `runBuild()` as a trailing block that +consumes `scheduler.state` after `scheduler.start()` resolves. + +`config` runs on main as a `runOnMain` task; `discover` ditto for +identity reasons (it builds `state.pages` and `state.pageByDest`). + +**`serve.mjs` is untouched.** `serve.mjs` imports `runBuild` from +`tbdocs.mjs` and calls it per change. As long as `runBuild`'s +signature stays stable (it does), `serve.mjs` needs no changes +through Phases 0-3. The dev-server flow keeps working end-to-end +without scheduler-aware code in `serve.mjs`. + +**Expected savings:** ~150 ms wall-clock. The main spine takes +~250 ms; `scss` (~700 ms) overlaps roughly half of it. Worker spawn +(~100-200 ms one-shot) eats a chunk of that. Honest end-to-end +estimate: 6.7 s → ~6.55 s. Most of the value here is structural -- +the DAG is now explicit -- not raw wall-clock. + +**Verification.** `build.bat && check.bat` clean. The timing summary +should show `scss=...ms` starting at t=0 alongside the main-thread +spine entries (`discover`, `nav`, ...), not after them. `render` / +`template` / `write` / `writeOffline` / `writePdf` still appear in +the summary at roughly their current durations -- they haven't been +moved to the scheduler yet. + +### Phase 2: Render fan-out + +**Suggested model:** Opus. + +Wire up the `render` named handler in `cpu-worker.mjs` (`renderPhase` ++ `templatePhase` over a chunk, return per-page deltas). Add the +`dispatch` task + dynamic `render:0..N` registration. Drop the serial +`renderPhase` / `templatePhase` calls from `runBuild()`. + +This is the largest single win: ~3.5 s of CPU compresses to +~`3500 / N` ms. + +**Expected savings:** on 4 cores, ~2.6 s saved (6.55 s → ~4.0 s). +On 8 cores, ~3 s saved (~3.6 s). Dispatch overhead is ~50 ms. + +**Verification.** `build.bat && check.bat` clean. The timing summary +should show N `render:i` entries (one per worker) whose individual +durations sum to roughly today's combined `render` + `template` time, +not today's per-page renders. Wall-clock drop should match the +expected savings above within ±20%. + +### Phase 3: Post-write tasks + +**Suggested model:** Sonnet. + +Port `write`, `searchData`, `writeAux`, `writeOffline`, `writePdf` as +`runOnMain` tasks. `writeOffline` and `writePdf` run in parallel on +the main thread; the gain is the shorter of their two CPU sections +(~240 ms) plus interleaved I/O. + +`runBuild()` shrinks to: pool construction + `scheduler.start(ctx)` + +summary output + `pool.destroy()`. + +**Expected savings:** ~240 ms (4.0 s → ~3.75 s on 4 cores). + +**Verification.** `build.bat && check.bat` clean. `writeOffline` and +`writePdf` should overlap in the timing summary -- their `start` +timestamps should be within a few ms of each other. + +### Phase 3-follow-up: workerize writeOffline (optional) + +**Suggested model:** Opus for the measure-and-decide call; Sonnet for the port if the call comes back yes. + +Move `writeOffline` to a worker handler if profiling shows its CPU +section blocks `writePdf` meaningfully. The shipping cost is +non-trivial -- `state.pages` and `state.staticFiles` cross the +boundary -- so measure first. + +**Verification.** `build.bat && check.bat` clean. Wall-clock should +drop by the measured difference; revert if it doesn't. + +### Phase 4: SharedArrayBuffer broadcast (optional) + +**Suggested model:** Opus. + +For the render fan-out, serialize `siteData + initData + linkTables` +once into a SharedArrayBuffer and pass it to all render tasks via +the pool's `transferList`. Measure first -- likely modest savings +vs. the N structured-clone fan-outs. + +**Deliverable:** measured reduction in render-dispatch overhead. + +**Verification.** `build.bat && check.bat` clean. Render dispatch +overhead (the gap between `dispatch` end and the earliest `render:i` +start in the timing summary) should decrease vs. Phase 2. diff --git a/builder/PLAN.md b/builder/PLAN.md index b12ff397..e068e0ac 100644 --- a/builder/PLAN.md +++ b/builder/PLAN.md @@ -62,6 +62,14 @@ live-reloads the browser via SSE. Renames `--serving` to static server; `docs/serve.bat` becomes a one-line `--serve` shim. Closes the PLAN-10 §7.D4 and §7.D11 watch-mode deferrals. +A **task-graph scheduler** for the build pipeline is designed in +[PLAN-scheduler.md](PLAN-scheduler.md) -- not yet implemented. The +plan covers a thin in-tree scheduler + `WorkerPool` over +`node:worker_threads`, moves CPU-bound seed tasks (`scss`, `mermaid`, +`buildInfo`) onto workers, and fans out `renderPhase` + `templatePhase` +across CPUs. Wall-clock target: ~6.7 s → ~3.6 s on 8 cores. Start +with §Migration path Phase 0. + Open follow-ups (deferred enhancements, divergence investigations) live in [FUTURE-WORK.md](FUTURE-WORK.md). From c4670c8e2e090b28ebd0dc6467997eb91526f9cd Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Sat, 30 May 2026 16:15:03 +0200 Subject: [PATCH 02/72] Scheduler Phase 0: skeleton + small refactors - Add worker-pool.mjs (WorkerPool class) - Add scheduler.mjs (Scheduler + SharedState) - Add cpu-worker.mjs (dispatcher skeleton, empty handlers) - Re-export buildInit from template.mjs as buildInitFn - Export serializeLinkTables from render.mjs - Change deriveRedirectStubs filter to frontmatter.layout check - Add precomputedStubs/precomputedUrls passthrough params - Add svgFiles to regenerateMermaid return value --- builder/cpu-worker.mjs | 26 ++++++++++ builder/mermaid.mjs | 31 +++++++++-- builder/redirects.mjs | 8 +-- builder/render.mjs | 8 +++ builder/scheduler.mjs | 112 ++++++++++++++++++++++++++++++++++++++++ builder/sitemap.mjs | 4 +- builder/template.mjs | 2 + builder/worker-pool.mjs | 54 +++++++++++++++++++ 8 files changed, 235 insertions(+), 10 deletions(-) create mode 100644 builder/cpu-worker.mjs create mode 100644 builder/scheduler.mjs create mode 100644 builder/worker-pool.mjs diff --git a/builder/cpu-worker.mjs b/builder/cpu-worker.mjs new file mode 100644 index 00000000..12d7709b --- /dev/null +++ b/builder/cpu-worker.mjs @@ -0,0 +1,26 @@ +// Worker harness for the tbdocs build pipeline. Routes named tasks to the +// appropriate handler and posts back { result } or { error, stack }. +// See PLAN-scheduler.md §Worker for the full handler set; Phase 0 ships +// the dispatcher skeleton only -- handlers are filled in per phase. + +import { parentPort } from "node:worker_threads"; + +const handlers = { + // Phase 1: scss, mermaid, buildInfo + // Phase 2: render +}; + +parentPort.on("message", async (msg) => { + const { name, ...payload } = msg; + const handler = handlers[name]; + if (!handler) { + parentPort.postMessage({ error: `unknown task: ${name}` }); + return; + } + try { + const result = await handler(payload); + parentPort.postMessage({ result }); + } catch (err) { + parentPort.postMessage({ error: err.message, stack: err.stack }); + } +}); diff --git a/builder/mermaid.mjs b/builder/mermaid.mjs index 483f2395..85517263 100644 --- a/builder/mermaid.mjs +++ b/builder/mermaid.mjs @@ -59,7 +59,7 @@ export async function regenerateMermaid(srcRoot) { const mmdRoot = path.join(srcRoot, MMD_REL_DIR); const sources = await listMermaidSources(mmdRoot); if (sources.length === 0) { - return { processed: 0, regenerated: 0 }; + return { processed: 0, regenerated: 0, svgFiles: [] }; } const stale = []; @@ -68,7 +68,8 @@ export async function regenerateMermaid(srcRoot) { if (!(await isUpToDate(svg, src))) stale.push({ src, svg }); } if (stale.length === 0) { - return { processed: sources.length, regenerated: 0 }; + return { processed: sources.length, regenerated: 0, + svgFiles: await statSvgFiles(sources, srcRoot) }; } // Lazy-load puppeteer + resolve the mermaid dist directory. Either @@ -85,7 +86,8 @@ export async function regenerateMermaid(srcRoot) { console.warn( `mermaid: skipped batch (${explainLoadFailure(err)}); existing SVGs retained`, ); - return { processed: sources.length, regenerated: 0, failed: 0, setupSkipped: true }; + return { processed: sources.length, regenerated: 0, failed: 0, setupSkipped: true, + svgFiles: await statSvgFiles(sources, srcRoot) }; } let browser; @@ -103,7 +105,8 @@ export async function regenerateMermaid(srcRoot) { console.warn( `mermaid: skipped batch (${explainLaunchFailure(err)}); existing SVGs retained`, ); - return { processed: sources.length, regenerated: 0, failed: 0, setupSkipped: true }; + return { processed: sources.length, regenerated: 0, failed: 0, setupSkipped: true, + svgFiles: await statSvgFiles(sources, srcRoot) }; } // CONTENT failures (one diagram throws) don't abort the batch -- the @@ -122,7 +125,25 @@ export async function regenerateMermaid(srcRoot) { } finally { await browser.close().catch(() => {}); } - return { processed: sources.length, regenerated, failed }; + return { processed: sources.length, regenerated, failed, + svgFiles: await statSvgFiles(sources, srcRoot) }; +} + +// Stat all managed SVG files and return their static-file descriptors. +// Called after rendering so freshly-written SVGs are included. +async function statSvgFiles(sources, srcRoot) { + const results = []; + for (const src of sources) { + const svgPath = svgFor(src); + try { + const stat = await fs.stat(svgPath); + const srcRel = path.relative(srcRoot, svgPath).replace(/\\/g, "/"); + results.push({ srcPath: svgPath, srcRel, destRel: srcRel, size: stat.size }); + } catch { + // SVG not on disk (render failed or never generated); skip. + } + } + return results; } async function listMermaidSources(mmdRoot) { diff --git a/builder/redirects.mjs b/builder/redirects.mjs index 14b248bd..18bec4a9 100644 --- a/builder/redirects.mjs +++ b/builder/redirects.mjs @@ -23,8 +23,8 @@ import { permalinkToDestPath } from "./paths.mjs"; import { absoluteUrl } from "./seo.mjs"; import { runLimited, writeFileMkdirp, WRITE_LIMIT } from "./write.mjs"; -export async function writeRedirects(pages, site, destRoot) { - const stubs = deriveRedirectStubs(pages, site); +export async function writeRedirects(pages, site, destRoot, precomputedStubs) { + const stubs = precomputedStubs ?? deriveRedirectStubs(pages, site); await runLimited(stubs, WRITE_LIMIT, async (s) => { await writeFileMkdirp(path.join(destRoot, s.destPath), s.html); }); @@ -44,9 +44,11 @@ export function deriveRedirectStubs(pages, site) { // §7.D2: build the set of every real page's on-disk path so a bad // redirect_from entry that would overwrite a page surfaces with a // clear error rather than silently clobbering. + // Filter uses frontmatter.layout rather than p.html so deriveRedirectStubs + // can run before templatePhase under the scheduler. const pageDestPaths = new Map(); for (const p of pages) { - if (p.html !== undefined) pageDestPaths.set(p.destPath, p); + if (p.frontmatter.layout !== "book-combined") pageDestPaths.set(p.destPath, p); } const stubs = []; diff --git a/builder/render.mjs b/builder/render.mjs index 4d6e209f..f6c97e2e 100644 --- a/builder/render.mjs +++ b/builder/render.mjs @@ -1232,6 +1232,14 @@ export function buildLinkTables(pages) { return { byPath, byUrl, byRedirect }; } +// Serialize linkTables for cross-thread transfer. resolveLink() only reads +// .permalink from each page, so shipping [key, permalink] pairs is sufficient. +// Workers reconstruct minimal { permalink } stubs via reconstructLinkTables. +export function serializeLinkTables(lt) { + const pairs = (m) => [...m.entries()].map(([k, p]) => [k, p.permalink]); + return { byPath: pairs(lt.byPath), byUrl: pairs(lt.byUrl), byRedirect: pairs(lt.byRedirect) }; +} + function putOnce(map, key, value) { if (!map.has(key)) map.set(key, value); } diff --git a/builder/scheduler.mjs b/builder/scheduler.mjs new file mode 100644 index 00000000..b718ff26 --- /dev/null +++ b/builder/scheduler.mjs @@ -0,0 +1,112 @@ +// Task-graph scheduler for the tbdocs build pipeline. See PLAN-scheduler.md +// for the full design, data-flow diagram, and task placement rationale. + +export class SharedState { + pages = []; // master copy; mutated in place by [M] tasks and render delta merges + staticFiles = []; // master copy; mermaid.submit appends new SVG descriptors + site = {}; // config, navTree, seoSiteTitle, seoLogoUrl, bookData, data, markdown, … + pageByDest = new Map(); // destPath → page; built once in discover.submit +} + +export class Scheduler { + constructor({ pool, tasks }) { + this.pool = pool; + this.tasks = new Map(Object.entries(tasks)); + this.pending = new Map(); + this.ready = []; + this.results = new Map(); + this.timings = new Map(); + this.state = new SharedState(); + this.inFlight = 0; + [this._doneP, this._doneResolve, this._doneReject] = deferred(); + for (const [id, def] of this.tasks) this._initPending(id, def); + } + + _initPending(id, def) { + this.pending.set(id, { expected: def.expected.length, received: new Map() }); + } + + register(id, def) { + this.tasks.set(id, def); + this._initPending(id, def); + } + + // Seed a freshly-registered task directly (used by dispatch.submit to feed + // each render:i its chunk without going through emit()). + seed(id, inputs) { + this.pending.delete(id); + this.ready.push({ id, def: this.tasks.get(id), inputs }); + this._flush(); + } + + emit(targetId, data, sourceId) { + const entry = this.pending.get(targetId); + if (!entry) throw new Error(`unknown or already-dispatched task: ${targetId}`); + entry.received.set(sourceId, data); + if (entry.received.size === entry.expected) { + this.pending.delete(targetId); + const def = this.tasks.get(targetId); + this.ready.push({ id: targetId, def, inputs: Object.fromEntries(entry.received) }); + this._flush(); + } + } + + async start(ctx) { + this._ctx = ctx; + for (const [id, def] of this.tasks) { + if (def.expected.length === 0) this.ready.push({ id, def, inputs: {} }); + } + this._flush(); + return this._doneP; + } + + _flush() { + while (this.ready.length > 0) this._run(this.ready.shift()); + } + + _run(task) { + const start = Date.now(); + this.inFlight++; + const p = task.def.runOnMain + ? Promise.resolve(task.def.execute(task.inputs, this._ctx, this.state)) + : this.pool.run({ inputs: task.inputs, ctx: this._ctx }, + { name: task.def.handler ?? task.id }); + p.then( + (output) => this._onDone(task, output, start), + (err) => this._onError(task, err), + ); + } + + _onDone(task, output, start) { + this.timings.set(task.id, { start, end: Date.now() }); + this.results.set(task.id, output); + this.inFlight--; + // submit() must be synchronous; async work belongs in execute(). + task.def.submit( + output, + (tgt, data) => this.emit(tgt, data, task.id), + this.state, + this, + ); + if (this.inFlight === 0 && this.ready.length === 0 && this.pending.size === 0) { + this._doneResolve(this.results); + } + } + + _onError(task, err) { + this._doneReject(new Error(`task ${task.id} failed`, { cause: err })); + } + + summary() { + return [...this.timings.entries()] + .sort((a, b) => a[1].start - b[1].start) + .map(([id, { start, end }]) => `${id}=${end - start}ms`) + .join(" "); + } +} + +function deferred() { + let res, rej; + const p = new Promise((r1, r2) => { res = r1; rej = r2; }); + return [p, res, rej]; +} diff --git a/builder/sitemap.mjs b/builder/sitemap.mjs index 4649f0bf..8feb2c41 100644 --- a/builder/sitemap.mjs +++ b/builder/sitemap.mjs @@ -13,10 +13,10 @@ import path from "node:path"; import { absoluteUrl } from "./seo.mjs"; import { writeFileMkdirp } from "./write.mjs"; -export async function writeSitemap(pages, site, destRoot) { +export async function writeSitemap(pages, site, destRoot, precomputedUrls) { const config = site.config; - const sitemapUrls = [...deriveSitemapUrls(pages, site)].sort(); + const sitemapUrls = [...(precomputedUrls ?? deriveSitemapUrls(pages, site))].sort(); const xml = renderSitemapXml(sitemapUrls); diff --git a/builder/template.mjs b/builder/template.mjs index 76491a77..8a8008cf 100644 --- a/builder/template.mjs +++ b/builder/template.mjs @@ -37,6 +37,8 @@ export async function templatePhase(pages, site) { // One-time per-build constants: pre-rendered SVG sprite, sidebar HTML, // header static parts, aux-nav, search-footer, mermaid script, favicon // link, GA snippet. Per §4 init order. +// Exported as buildInitFn for the scheduler's main-thread buildInit task. +export { buildInit as buildInitFn }; function buildInit(site) { return { svgSprites: buildSvgSprites(site.config), diff --git a/builder/worker-pool.mjs b/builder/worker-pool.mjs new file mode 100644 index 00000000..ebfec929 --- /dev/null +++ b/builder/worker-pool.mjs @@ -0,0 +1,54 @@ +// Worker pool over node:worker_threads. Spawns `size` workers eagerly at +// construction, routes named tasks to whichever worker is idle, queues the +// rest. No dynamic scaling, no recycling, no abort signals. + +import { Worker } from "node:worker_threads"; + +export class WorkerPool { + constructor(size, workerUrl) { + this._workerUrl = workerUrl; + this._idle = []; // Worker[] + this._busy = new Map(); // Worker → { resolve, reject } + this._queue = []; // pending { message, transferList, resolve, reject } + this._workers = Array.from({ length: size }, () => this._spawn()); + } + + _spawn() { + const w = new Worker(this._workerUrl); + w.on("message", (msg) => { + const entry = this._busy.get(w); + if (!entry) return; + this._busy.delete(w); + this._idle.push(w); + if (msg.error) entry.reject(Object.assign(new Error(msg.error), { stack: msg.stack })); + else entry.resolve(msg.result); + this._drain(); + }); + w.on("error", (err) => { + const entry = this._busy.get(w); + if (entry) { this._busy.delete(w); entry.reject(err); } + }); + this._idle.push(w); + return w; + } + + run(payload, { name, transferList } = {}) { + return new Promise((resolve, reject) => { + this._queue.push({ message: { name, ...payload }, transferList, resolve, reject }); + this._drain(); + }); + } + + _drain() { + while (this._queue.length && this._idle.length) { + const w = this._idle.shift(); + const { message, transferList, resolve, reject } = this._queue.shift(); + this._busy.set(w, { resolve, reject }); + w.postMessage(message, transferList); + } + } + + destroy() { + return Promise.all(this._workers.map(w => w.terminate())); + } +} From 88b1f880d9cf3c491c5bcd7fa5e201c43516c329 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Sat, 30 May 2026 16:45:58 +0200 Subject: [PATCH 03/72] Scheduler Phase 1: seeds + main-thread spine --- builder/cpu-worker.mjs | 29 +++- builder/scheduler.mjs | 5 +- builder/tbdocs.mjs | 313 +++++++++++++++++++++++++++++++---------- 3 files changed, 270 insertions(+), 77 deletions(-) diff --git a/builder/cpu-worker.mjs b/builder/cpu-worker.mjs index 12d7709b..3422a47d 100644 --- a/builder/cpu-worker.mjs +++ b/builder/cpu-worker.mjs @@ -1,13 +1,34 @@ // Worker harness for the tbdocs build pipeline. Routes named tasks to the // appropriate handler and posts back { result } or { error, stack }. -// See PLAN-scheduler.md §Worker for the full handler set; Phase 0 ships -// the dispatcher skeleton only -- handlers are filled in per phase. +// See PLAN-scheduler.md §Worker for the full handler set. import { parentPort } from "node:worker_threads"; +import { compileScss } from "./scss.mjs"; +import { regenerateMermaid } from "./mermaid.mjs"; +import { captureBuildInfo } from "./build-info.mjs"; + +// Phase 2: uncomment when render fan-out is wired up. +// import { initHighlighter } from "./highlight.mjs"; +// import { createMarkdownIt, buildLinkTables, +// renderPhase } from "./render.mjs"; +// import { templatePhase } from "./template.mjs"; +// const highlighterP = initHighlighter(); + const handlers = { - // Phase 1: scss, mermaid, buildInfo - // Phase 2: render + async scss({ ctx }) { + return { scssResult: await compileScss(ctx.srcRoot) }; + }, + + async mermaid({ ctx }) { + return { mermaidStats: await regenerateMermaid(ctx.srcRoot) }; + }, + + async buildInfo() { + return { buildInfo: await captureBuildInfo() }; + }, + + // Phase 2: render handler (renderPhase + templatePhase over a chunk). }; parentPort.on("message", async (msg) => { diff --git a/builder/scheduler.mjs b/builder/scheduler.mjs index b718ff26..dd96ae10 100644 --- a/builder/scheduler.mjs +++ b/builder/scheduler.mjs @@ -54,7 +54,10 @@ export class Scheduler { async start(ctx) { this._ctx = ctx; for (const [id, def] of this.tasks) { - if (def.expected.length === 0) this.ready.push({ id, def, inputs: {} }); + if (def.expected.length === 0) { + this.pending.delete(id); // seeds have no inputs; remove from pending before dispatching + this.ready.push({ id, def, inputs: {} }); + } } this._flush(); return this._doneP; diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index dbe45699..726a23a6 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -14,28 +14,34 @@ // the actual deployment instead of the configured production host). import { promises as fs } from "node:fs"; +import os from "node:os"; import path from "node:path"; import process from "node:process"; import { fileURLToPath } from "node:url"; import yaml from "js-yaml"; +import { WorkerPool } from "./worker-pool.mjs"; +import { Scheduler } from "./scheduler.mjs"; + import { discover } from "./discover.mjs"; -import { regenerateMermaid } from "./mermaid.mjs"; -import { compileScss } from "./scss.mjs"; import { computeNav } from "./nav.mjs"; import { precomputeSeo } from "./seo.mjs"; import { resolveBookChapters } from "./book.mjs"; -import { captureBuildInfo } from "./build-info.mjs"; import { loadData } from "./data.mjs"; -import { renderPhase, createMarkdownIt, initHighlighter, buildLinkTables } from "./render.mjs"; -import { templatePhase } from "./template.mjs"; +import { + renderPhase, createMarkdownIt, initHighlighter, + buildLinkTables, serializeLinkTables, +} from "./render.mjs"; +import { templatePhase, buildInitFn } from "./template.mjs"; import { writePhase } from "./write.mjs"; -import { writeRedirects } from "./redirects.mjs"; -import { writeSitemap } from "./sitemap.mjs"; +import { writeRedirects, deriveRedirectStubs } from "./redirects.mjs"; +import { writeSitemap, deriveSitemapUrls } from "./sitemap.mjs"; import { writeSearchData } from "./search.mjs"; import { writeOffline } from "./offline.mjs"; import { writePdf } from "./pdf.mjs"; +const CPU_WORKER_URL = new URL("./cpu-worker.mjs", import.meta.url); + function parseArgs(argv) { const args = { src: "docs", @@ -106,72 +112,235 @@ export function makeTimer() { }; } +// ── Task graph ──────────────────────────────────────────────────────────────── +// +// Phase 1 wires the seeds (config, buildInfo, scss, mermaid) and the main- +// thread spine (discover → nav → markdownInit / buildInit → seo / loadData → +// resolveBookChapters + deriveRedirects / deriveSitemap). +// +// Render, template, write, and post-write tasks are still the trailing serial +// block below scheduler.start(); they graduate to scheduler tasks in Phases 2–3. + +const TASKS = { + // ── Seeds ───────────────────────────────────────────────────────────────── + + // Reads and merges _config.yml + CLI overrides. Seed on main because the + // output object flows directly into discover (identity matters, no worker + // boundary crossing needed, and it's a trivial I/O read). + config: { + expected: [], + runOnMain: true, + async execute(_, ctx) { + const text = await fs.readFile(path.join(ctx.srcRoot, "_config.yml"), "utf8"); + const config = yaml.load(text); + if (ctx.opts.baseurl != null) config.baseurl = ctx.opts.baseurl; + if (ctx.opts.url != null) config.url = ctx.opts.url; + return { config }; + }, + submit(out, emit) { emit("discover", out); }, + }, + + // Git rev-parse / log shell-outs. Worker so they overlap with the main spine. + buildInfo: { + expected: [], + // execute() runs in cpu-worker.mjs as the "buildInfo" handler. + submit() { /* Phase 2: emit("dispatch", out) */ }, + }, + + // Sass compilation (~700 ms CPU). Worker so it overlaps with the main spine. + scss: { + expected: [], + // execute() runs in cpu-worker.mjs as the "scss" handler. + submit() { /* Phase 3: emit("write", out) */ }, + }, + + // Stale mermaid SVG regeneration. Worker for the same reason. + mermaid: { + expected: [], + // execute() runs in cpu-worker.mjs as the "mermaid" handler. + submit(out, emit, state) { + // Append any freshly-generated SVG descriptors that discover didn't see + // (because mermaid and discover run concurrently). Dedup by srcRel so + // SVGs already on disk at discover time aren't double-counted. + const known = new Set(state.staticFiles.map((f) => f.srcRel)); + for (const f of out.mermaidStats.svgFiles ?? []) { + if (!known.has(f.srcRel)) state.staticFiles.push(f); + } + /* Phase 3: emit("write", out) */ + }, + }, + + // ── Main-thread spine ───────────────────────────────────────────────────── + + discover: { + expected: ["config"], + runOnMain: true, + async execute({ config: { config } }, ctx) { + const { pages, staticFiles } = await discover(ctx.srcRoot, config.exclude ?? []); + return { pages, staticFiles, config }; + }, + submit(out, emit, state) { + state.pages = out.pages; + state.staticFiles = out.staticFiles; + state.site.config = out.config; + for (const p of out.pages) state.pageByDest.set(p.destPath, p); + emit("nav", out); + emit("deriveRedirects", out); + emit("deriveSitemap", out); + }, + }, + + nav: { + expected: ["discover"], + runOnMain: true, + execute(_, ctx, state) { + const { navTree } = computeNav(state.pages, state.site.config); + state.site.navTree = navTree; + return {}; + }, + submit(_, emit) { + emit("markdownInit", {}); + emit("buildInit", {}); + }, + }, + + // Pre-renders the sidebar/header/svg-sprite HTML used by templatePhase. + // Depends only on nav so it can start while markdownInit is in flight. + buildInit: { + expected: ["nav"], + runOnMain: true, + execute(_, ctx, state) { + return { initData: buildInitFn(state.site) }; + }, + submit() { /* Phase 2: emit("dispatch", out) */ }, + }, + + // Shiki WASM init + link-table build + markdown-it instance creation. + // Not serializable (Shiki's highlighter is a live object), so it stays + // on main. Workers initialize their own independent highlighter instances. + markdownInit: { + expected: ["nav"], + runOnMain: true, + async execute(_, ctx, state) { + const highlighter = await initHighlighter(); + const linkTables = buildLinkTables(state.pages); + const baseurl = String(state.site.config.baseurl || ""); + const staticFileSet = new Set(state.staticFiles.map(s => s.srcRel)); + state.site.highlighter = highlighter; + state.site.markdown = createMarkdownIt({ + highlighter, linkTables, baseurl, staticFiles: staticFileSet, + }); + state.site.linkTablesSerialized = serializeLinkTables(linkTables); + return {}; + }, + submit(_, emit) { + emit("seo", {}); + emit("loadData", {}); + }, + }, + + seo: { + expected: ["markdownInit"], + runOnMain: true, + execute(_, ctx, state) { + const { seoSiteTitle, seoLogoUrl } = precomputeSeo( + state.pages, state.site.config, state.site.markdown); + state.site.seoSiteTitle = seoSiteTitle; + state.site.seoLogoUrl = seoLogoUrl; + return {}; + }, + submit(_, emit) { emit("resolveBookChapters", {}); }, + }, + + loadData: { + expected: ["markdownInit"], + runOnMain: true, + async execute(_, ctx, state) { + const data = await loadData(ctx.srcRoot); + state.site.data = data; + state.site.bookData = data.book ?? null; + return {}; + }, + submit(_, emit) { emit("resolveBookChapters", {}); }, + }, + + // Mutates bookData._chapters with refs into state.pages. Identity-critical: + // the same page objects must be read by writePdf later (after renderPhase + // fills in renderedContent on those same objects). + resolveBookChapters: { + expected: ["seo", "loadData"], + runOnMain: true, + execute(_, ctx, state) { + resolveBookChapters(state.site.bookData, state.pages); + return {}; + }, + submit() { /* Phase 2: emit("dispatch", {}) */ }, + }, + + // Can run in parallel with nav/markdownInit -- only needs pages + config, + // both available after discover. The layout-based filter (not p.html) + // lets this run before templatePhase. + deriveRedirects: { + expected: ["discover"], + runOnMain: true, + execute(_, ctx, state) { + return { stubs: deriveRedirectStubs(state.pages, state.site) }; + }, + submit() { /* Phase 3: emit("writeAux", out) */ }, + }, + + deriveSitemap: { + expected: ["discover"], + runOnMain: true, + execute(_, ctx, state) { + return { urls: deriveSitemapUrls(state.pages, state.site) }; + }, + submit() { /* Phase 3: emit("writeAux", out) */ }, + }, +}; + +// ── Build entry point ───────────────────────────────────────────────────────── + export async function runBuild(opts) { const { src, dest, dryRun, tolerateMissingImages, profileOffline } = opts; const srcRoot = path.resolve(process.cwd(), src); const destRoot = path.resolve(dest ?? path.join(srcRoot, "_site")); + const workerCount = os.availableParallelism(); + const pool = new WorkerPool(workerCount, CPU_WORKER_URL); + const scheduler = new Scheduler({ pool, tasks: TASKS }); + const ctx = { srcRoot, destRoot, opts, workerCount }; + + // Run the scheduler (seeds + main-thread spine). Render/template/write + // tasks are still the trailing serial block below; they graduate to + // scheduler tasks in Phases 2–3. + let results; + try { + results = await scheduler.start(ctx); + } finally { + await pool.destroy(); + } + + // ── Trailing serial block (Phases 2–3 will absorb these) ───────────────── + const t = makeTimer(); + const { pages, staticFiles } = scheduler.state; + const site = scheduler.state.site; + + // Wire in the three worker-task outputs that the serial block needs. + const { mermaidStats } = results.get("mermaid"); + const { scssResult } = results.get("scss"); + site.buildInfo = results.get("buildInfo").buildInfo; - // Phase 11 (B1) preprocess: regenerate stale mermaid SVGs before - // discover walks the tree so freshly-emitted siblings land in - // staticFiles[] on this same build. - const mermaidStats = await regenerateMermaid(srcRoot); - t.lap("mermaid"); if (mermaidStats.regenerated > 0 || mermaidStats.failed > 0) { const parts = [`regenerated ${mermaidStats.regenerated}`]; if (mermaidStats.failed > 0) parts.push(`failed ${mermaidStats.failed}`); console.log(`mermaid: ${parts.join(", ")} of ${mermaidStats.processed} SVG(s)`); } - // Content failures (broken .mmd, render exception) flip the build - // exit code so CI catches them; setup failures (missing puppeteer / - // Chrome) only warn and leave the existing SVGs in place. - if (mermaidStats.failed > 0) { - process.exitCode = 1; - } - - const scssResult = await compileScss(srcRoot); - t.lap("scss"); - if (scssResult.failed) { - process.exitCode = 1; - } - - const config = yaml.load(await fs.readFile(path.join(srcRoot, "_config.yml"), "utf8")); - if (opts.baseurl != null) config.baseurl = opts.baseurl; - if (opts.url != null) config.url = opts.url; - - // Issue build-info immediately so the git shell-outs overlap with the - // CPU-bound nav work. - const buildInfoPromise = captureBuildInfo(); - - const { pages, staticFiles } = await discover(srcRoot, config.exclude ?? []); - t.lap("discover"); - - const { navTree } = computeNav(pages, config); - t.lap("nav"); - - // Build the shared markdown-it instance up front so Phase 2's SEO - // pass and Phase 3's body renderer use the same configured renderer. - // initHighlighter overlaps with the running git shell-outs above. - const highlighter = await initHighlighter(); - const linkTables = buildLinkTables(pages); - const baseurl = String(config.baseurl || ""); - const staticFileSet = new Set(staticFiles.map((s) => s.srcRel)); - const markdown = createMarkdownIt({ highlighter, linkTables, baseurl, staticFiles: staticFileSet }); - t.lap("markdown-init"); - - const { seoSiteTitle, seoLogoUrl } = precomputeSeo(pages, config, markdown); - t.lap("seo"); - - const data = await loadData(srcRoot); - const bookData = data.book ?? null; - resolveBookChapters(bookData, pages); - t.lap("book"); - - const buildInfo = await buildInfoPromise; - t.lap("buildInfo"); + if (mermaidStats.failed > 0) process.exitCode = 1; + if (scssResult.failed) process.exitCode = 1; - const site = { config, navTree, seoSiteTitle, seoLogoUrl, buildInfo, bookData, data, markdown }; + const baseurl = String(site.config.baseurl || ""); await renderPhase(pages, site, staticFiles); t.lap("render"); @@ -180,8 +349,8 @@ export async function runBuild(opts) { t.lap("template"); const generatedAssets = []; - if (highlighter.themeCss) { - generatedAssets.push({ rel: "assets/css/tb-highlight.css", content: highlighter.themeCss }); + if (site.highlighter?.themeCss) { + generatedAssets.push({ rel: "assets/css/tb-highlight.css", content: site.highlighter.themeCss }); } if (scssResult.compiled) { generatedAssets.push({ rel: "assets/css/just-the-docs-combined.css", content: scssResult.css }); @@ -196,22 +365,22 @@ export async function runBuild(opts) { let auxStats = null; if (!dryRun) { + // Use the pre-derived stubs/urls from the scheduler rather than re-deriving. + const stubs = results.get("deriveRedirects").stubs; + const urls = results.get("deriveSitemap").urls; const [redirectStats, sitemapStats, searchStats] = await Promise.all([ - writeRedirects(pages, site, destRoot), - writeSitemap(pages, site, destRoot), + writeRedirects(pages, site, destRoot, stubs), + writeSitemap(pages, site, destRoot, urls), writeSearchData(pages, site, destRoot), ]); auxStats = { redirects: redirectStats, sitemap: sitemapStats, search: searchStats }; } t.lap("auxiliaries"); - // CLI flag takes precedence (PLAN-9 §7.D4); fall back to the - // `also_build_offline` / `also_build_pdf` config knobs Jekyll uses - // when the flag isn't passed. - const skipOffline = opts.skipOffline - ?? (config.also_build_offline === false); - const skipPdf = opts.skipPdf - ?? (config.also_build_pdf === false); + // CLI flag takes precedence; fall back to the `also_build_offline` / + // `also_build_pdf` config knobs. + const skipOffline = opts.skipOffline ?? (site.config.also_build_offline === false); + const skipPdf = opts.skipPdf ?? (site.config.also_build_pdf === false); let offlineStats = null; let offlineTimer = null; @@ -252,6 +421,7 @@ export async function runBuild(opts) { console.log(` pdf: book.html (${mb} MB), ${pdfStats.css} CSS, ` + `${pdfStats.images} images${missingClause} -> ${destRoot}-pdf`); } + console.log(scheduler.summary()); console.log(t.summary()); if (offlineTimer) { console.log(` offline: ${offlineTimer.summary()}`); @@ -263,7 +433,6 @@ export async function runBuild(opts) { process.exitCode = 1; } - // Phase 8+ chains in here. return { pages, staticFiles, site, destRoot }; } From ee2f52d3b81fd307e3f3dbd49d140b7f04b05577 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Sat, 30 May 2026 17:45:21 +0200 Subject: [PATCH 04/72] Scheduler Phase 2: render fan-out across worker threads --- builder/cpu-worker.mjs | 46 +++++++++++++++--- builder/tbdocs.mjs | 107 +++++++++++++++++++++++++++++++++-------- builder/template.mjs | 4 +- 3 files changed, 127 insertions(+), 30 deletions(-) diff --git a/builder/cpu-worker.mjs b/builder/cpu-worker.mjs index 3422a47d..c028e5a0 100644 --- a/builder/cpu-worker.mjs +++ b/builder/cpu-worker.mjs @@ -8,12 +8,15 @@ import { compileScss } from "./scss.mjs"; import { regenerateMermaid } from "./mermaid.mjs"; import { captureBuildInfo } from "./build-info.mjs"; -// Phase 2: uncomment when render fan-out is wired up. -// import { initHighlighter } from "./highlight.mjs"; -// import { createMarkdownIt, buildLinkTables, -// renderPhase } from "./render.mjs"; -// import { templatePhase } from "./template.mjs"; -// const highlighterP = initHighlighter(); +import { initHighlighter } from "./highlight.mjs"; +import { createMarkdownIt, buildLinkTables, + renderPhase } from "./render.mjs"; +import { templatePhase } from "./template.mjs"; + +// Start WASM init immediately, do NOT await. Module evaluation finishes +// synchronously so the parentPort.on('message') dispatcher is installed +// before the pool sends any work. Only the `render` handler awaits. +const highlighterP = initHighlighter(); const handlers = { async scss({ ctx }) { @@ -28,7 +31,27 @@ const handlers = { return { buildInfo: await captureBuildInfo() }; }, - // Phase 2: render handler (renderPhase + templatePhase over a chunk). + async render({ inputs }) { + const { siteData, initData, linkTablesData, staticFilesArr, + baseurl, buildInfo, chunk } = inputs; + + const highlighter = await highlighterP; + const linkTables = reconstructLinkTables(linkTablesData); + const staticFiles = new Set(staticFilesArr); + const markdown = createMarkdownIt({ highlighter, linkTables, baseurl, staticFiles }); + + const site = { ...siteData, markdown, buildInfo }; + await renderPhase(chunk, site); + await templatePhase(chunk, site, initData); + + // book-combined pages have renderedContent but no html (Phase 8 + // handles them from renderedContent); send html: undefined for those. + return chunk.map(p => ({ + destPath: p.destPath, + renderedContent: p.renderedContent, + html: p.html, + })); + }, }; parentPort.on("message", async (msg) => { @@ -45,3 +68,12 @@ parentPort.on("message", async (msg) => { parentPort.postMessage({ error: err.message, stack: err.stack }); } }); + +// linkTables values are page objects in the main pipeline, but +// resolveLink() in the relative-links plugin only reads .permalink. +// The serialized form ships [key, permalink] pairs; we reconstruct +// minimal { permalink } stubs in the worker. +function reconstructLinkTables({ byPath, byUrl, byRedirect }) { + const make = (pairs) => new Map(pairs.map(([k, pl]) => [k, { permalink: pl }])); + return { byPath: make(byPath), byUrl: make(byUrl), byRedirect: make(byRedirect) }; +} diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 726a23a6..159e1cb9 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -29,10 +29,10 @@ import { precomputeSeo } from "./seo.mjs"; import { resolveBookChapters } from "./book.mjs"; import { loadData } from "./data.mjs"; import { - renderPhase, createMarkdownIt, initHighlighter, + createMarkdownIt, initHighlighter, buildLinkTables, serializeLinkTables, } from "./render.mjs"; -import { templatePhase, buildInitFn } from "./template.mjs"; +import { buildInitFn } from "./template.mjs"; import { writePhase } from "./write.mjs"; import { writeRedirects, deriveRedirectStubs } from "./redirects.mjs"; import { writeSitemap, deriveSitemapUrls } from "./sitemap.mjs"; @@ -114,12 +114,13 @@ export function makeTimer() { // ── Task graph ──────────────────────────────────────────────────────────────── // -// Phase 1 wires the seeds (config, buildInfo, scss, mermaid) and the main- -// thread spine (discover → nav → markdownInit / buildInit → seo / loadData → -// resolveBookChapters + deriveRedirects / deriveSitemap). +// Seeds (config, buildInfo, scss, mermaid), the main-thread spine (discover → +// nav → markdownInit / buildInit → seo / loadData → resolveBookChapters + +// deriveRedirects / deriveSitemap), and the render fan-out (dispatch → +// render:0..N → renderJoin) are all scheduler tasks. // -// Render, template, write, and post-write tasks are still the trailing serial -// block below scheduler.start(); they graduate to scheduler tasks in Phases 2–3. +// Write and post-write tasks are still the trailing serial block below +// scheduler.start(); they graduate to scheduler tasks in Phase 3. const TASKS = { // ── Seeds ───────────────────────────────────────────────────────────────── @@ -144,7 +145,7 @@ const TASKS = { buildInfo: { expected: [], // execute() runs in cpu-worker.mjs as the "buildInfo" handler. - submit() { /* Phase 2: emit("dispatch", out) */ }, + submit(out, emit) { emit("dispatch", out); }, }, // Sass compilation (~700 ms CPU). Worker so it overlaps with the main spine. @@ -212,7 +213,7 @@ const TASKS = { execute(_, ctx, state) { return { initData: buildInitFn(state.site) }; }, - submit() { /* Phase 2: emit("dispatch", out) */ }, + submit(out, emit) { emit("dispatch", out); }, }, // Shiki WASM init + link-table build + markdown-it instance creation. @@ -274,7 +275,7 @@ const TASKS = { resolveBookChapters(state.site.bookData, state.pages); return {}; }, - submit() { /* Phase 2: emit("dispatch", {}) */ }, + submit(_, emit) { emit("dispatch", {}); }, }, // Can run in parallel with nav/markdownInit -- only needs pages + config, @@ -297,8 +298,78 @@ const TASKS = { }, submit() { /* Phase 3: emit("writeAux", out) */ }, }, + + // ── Render fan-out ───────────────────────────────────────────────────────── + + // Slices state.pages into chunks and dynamically registers render:0..N + // worker tasks plus a renderJoin barrier. Waits for buildInit (template + // chrome), resolveBookChapters (identity-critical page refs), and + // buildInfo (git metadata for the footer). + dispatch: { + expected: ["buildInit", "resolveBookChapters", "buildInfo"], + runOnMain: true, + execute({ buildInit: { initData }, buildInfo: { buildInfo } }, ctx, state) { + const chunks = chunkPages(state.pages, ctx.workerCount); + const siteData = { + config: state.site.config, + seoSiteTitle: state.site.seoSiteTitle, + seoLogoUrl: state.site.seoLogoUrl, + }; + return { + chunks, siteData, initData, buildInfo, + linkTablesData: state.site.linkTablesSerialized, + staticFilesArr: state.staticFiles.map(f => f.srcRel), + baseurl: String(state.site.config.baseurl || ""), + }; + }, + submit(out, emit, _state, scheduler) { + const N = out.chunks.length; + + scheduler.register("renderJoin", { + expected: Array.from({ length: N }, (_, i) => `render:${i}`), + runOnMain: true, + execute() { return {}; }, + submit() { /* Phase 3: emit("write", {}) */ }, + }); + + for (let i = 0; i < N; i++) { + const id = `render:${i}`; + scheduler.register(id, { + expected: [], + handler: "render", + submit(renderOut, emit, state) { + for (const r of renderOut) { + const p = state.pageByDest.get(r.destPath); + if (!p) continue; + p.renderedContent = r.renderedContent; + if (r.html !== undefined) p.html = r.html; + } + emit("renderJoin", renderOut); + }, + }); + scheduler.seed(id, { + siteData: out.siteData, + initData: out.initData, + linkTablesData: out.linkTablesData, + staticFilesArr: out.staticFilesArr, + baseurl: out.baseurl, + buildInfo: out.buildInfo, + chunk: out.chunks[i], + }); + } + }, + }, }; +function chunkPages(pages, workers) { + const n = Math.min(workers, pages.length); + if (n === 0) return []; + const size = Math.ceil(pages.length / n); + const chunks = []; + for (let i = 0; i < pages.length; i += size) chunks.push(pages.slice(i, i + size)); + return chunks; +} + // ── Build entry point ───────────────────────────────────────────────────────── export async function runBuild(opts) { @@ -311,9 +382,9 @@ export async function runBuild(opts) { const scheduler = new Scheduler({ pool, tasks: TASKS }); const ctx = { srcRoot, destRoot, opts, workerCount }; - // Run the scheduler (seeds + main-thread spine). Render/template/write - // tasks are still the trailing serial block below; they graduate to - // scheduler tasks in Phases 2–3. + // Run the scheduler (seeds + main-thread spine + render fan-out). + // Write and post-write tasks are still the trailing serial block below; + // they graduate to scheduler tasks in Phase 3. let results; try { results = await scheduler.start(ctx); @@ -321,13 +392,13 @@ export async function runBuild(opts) { await pool.destroy(); } - // ── Trailing serial block (Phases 2–3 will absorb these) ───────────────── + // ── Trailing serial block (Phase 3 will absorb these) ──────────────────── const t = makeTimer(); const { pages, staticFiles } = scheduler.state; const site = scheduler.state.site; - // Wire in the three worker-task outputs that the serial block needs. + // Wire in worker-task outputs that the serial block needs. const { mermaidStats } = results.get("mermaid"); const { scssResult } = results.get("scss"); site.buildInfo = results.get("buildInfo").buildInfo; @@ -342,12 +413,6 @@ export async function runBuild(opts) { const baseurl = String(site.config.baseurl || ""); - await renderPhase(pages, site, staticFiles); - t.lap("render"); - - await templatePhase(pages, site); - t.lap("template"); - const generatedAssets = []; if (site.highlighter?.themeCss) { generatedAssets.push({ rel: "assets/css/tb-highlight.css", content: site.highlighter.themeCss }); diff --git a/builder/template.mjs b/builder/template.mjs index 8a8008cf..1c1f5a14 100644 --- a/builder/template.mjs +++ b/builder/template.mjs @@ -18,7 +18,7 @@ import { compressHtml } from "./compress.mjs"; -export async function templatePhase(pages, site) { +export async function templatePhase(pages, site, initData) { if (site.config.just_the_docs?.collections) { throw new Error( "site.config.just_the_docs.collections is set; Phase 4 (and Phase 2) " @@ -26,7 +26,7 @@ export async function templatePhase(pages, site) { ); } - const init = buildInit(site); + const init = initData ?? buildInit(site); await Promise.all(pages.map(async (page) => { if (page.frontmatter.layout === "book-combined") return; From f3e8d9ff05f128b3c1dc6f2b0855456f3d0484cb Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Sat, 30 May 2026 18:51:35 +0200 Subject: [PATCH 05/72] Scheduler Phase 3: write + post-write tasks --- builder/PLAN-scheduler.md | 46 ++++---- builder/tbdocs.mjs | 218 ++++++++++++++++++++++---------------- 2 files changed, 154 insertions(+), 110 deletions(-) diff --git a/builder/PLAN-scheduler.md b/builder/PLAN-scheduler.md index f107a4af..02f1410e 100644 --- a/builder/PLAN-scheduler.md +++ b/builder/PLAN-scheduler.md @@ -1207,16 +1207,15 @@ to `p.frontmatter.layout !== "book-combined"` (the property that determines whether `html` will be set; known after discover) so the derive can run at any point after discover. -**Workerizing writeOffline or writePdf (deferred).** Both phases have -non-trivial CPU sections: writeOffline rewrites URLs across all 856 -HTML files (~700 ms CPU); writePdf assembles `book.html` via -`assembleBook` (~150 ms CPU). On `runOnMain` both serialize their CPU -work; only the async I/O windows interleave. The expected wall-clock -win from workerizing one of them (move offline to a worker, keep pdf -on main) is ~250 ms but at the cost of shipping `pages[]` and -`staticFiles[]` across the boundary. Profile first, then decide -- the -shipping cost may eat the gain. This is a Phase 3-follow-up choice, -not a Phase 3 requirement. +**Workerizing writeOffline or writePdf (measured, declined).** Both +phases have non-trivial CPU sections: writeOffline rewrites URLs +across all 856 HTML files; writePdf assembles `book.html` via +`assembleBook`. Profiling shows the cooperative async concurrency +already achieves zero-contention overlap — `writePdf` runs entirely +within `writeOffline`'s I/O await gaps, and the combined wall-clock +equals `max(writeOffline, writePdf)`. The structured-clone cost of +shipping `pages[]` across the worker boundary (~37–65 ms) would be +pure overhead. See §Phase 3-follow-up for the full measurement. ## Timing / profiling @@ -1300,7 +1299,7 @@ data lifetime across the worker boundary, or low-level serialisation. | 1. Seeds + main-thread spine | Sonnet | Each task body is a thin wrapper around an existing phase function. The scheduler core is copy-from-plan. All mutation is on the main thread (no cross-worker identity yet). | | 2. Render fan-out | Opus | Cross-thread structured-clone semantics, module-scope initialisation order, dynamic task registration, per-page delta-merge identity. Debugging concurrency bugs needs depth. | | 3. Post-write tasks | Sonnet | All `runOnMain`; thin wrappers around existing write functions. No new concurrency surface beyond the already-built scheduler. | -| 3-follow-up. Workerize writeOffline | Opus to decide, Sonnet to implement | Profile-driven judgement call. If the move is chosen, the port itself is mechanical. | +| 3-follow-up. Workerize writeOffline | Opus (decided) | Profiled: zero CPU contention; cooperative async overlap is already optimal. Declined — no implementation needed. | | 4. SAB broadcast | Opus | Manual serialisation into shared memory; layout / endianness / varint choices need reasoning about memory ordering. | Escalate to Opus mid-phase if a Sonnet session hits a debugging block @@ -1451,17 +1450,26 @@ summary output + `pool.destroy()`. `writePdf` should overlap in the timing summary -- their `start` timestamps should be within a few ms of each other. -### Phase 3-follow-up: workerize writeOffline (optional) +### Phase 3-follow-up: workerize writeOffline (optional) — DECLINED -**Suggested model:** Opus for the measure-and-decide call; Sonnet for the port if the call comes back yes. +**Decision:** do not workerize. Profiling shows zero CPU contention. -Move `writeOffline` to a worker handler if profiling shows its CPU -section blocks `writePdf` meaningfully. The shipping cost is -non-trivial -- `state.pages` and `state.staticFiles` cross the -boundary -- so measure first. +Two independent measurement runs confirmed that `writeOffline` and +`writePdf` already achieve perfect overlap via cooperative async +concurrency on the main thread. In both runs the combined wall-clock +equalled `max(writeOffline, writePdf)` — the "wasted (CPU contention)" +metric was 0 ms. `writePdf`'s `assembleBook` synchronous section +(~150 ms) runs entirely inside `writeOffline`'s I/O await gaps. -**Verification.** `build.bat && check.bat` clean. Wall-clock should -drop by the measured difference; revert if it doesn't. +Structured-clone cost for shipping `pages[]` across the worker +boundary was measured at ~37–65 ms (depending on per-page HTML size), +which would be pure overhead against a 0 ms contention baseline. +Adding the `offline` handler to `cpu-worker.mjs` would also increase +worker spawn time (acorn import), complicate the worker's module +surface, and add a result-merge path — all for no measurable gain. + +The overlap already saves ~260 ms vs. sequential execution (the full +duration of `writePdf`). No further action needed. ### Phase 4: SharedArrayBuffer broadcast (optional) diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 159e1cb9..45d40bf7 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -116,11 +116,11 @@ export function makeTimer() { // // Seeds (config, buildInfo, scss, mermaid), the main-thread spine (discover → // nav → markdownInit / buildInit → seo / loadData → resolveBookChapters + -// deriveRedirects / deriveSitemap), and the render fan-out (dispatch → -// render:0..N → renderJoin) are all scheduler tasks. -// -// Write and post-write tasks are still the trailing serial block below -// scheduler.start(); they graduate to scheduler tasks in Phase 3. +// deriveRedirects / deriveSitemap), the render fan-out (dispatch → +// render:0..N → renderJoin), and all write/post-write tasks (write → +// searchData → writeAux → writeOffline / writePdf) are scheduler tasks. +// runBuild() constructs the pool + scheduler, awaits start(), logs the +// summary, and returns. const TASKS = { // ── Seeds ───────────────────────────────────────────────────────────────── @@ -145,14 +145,17 @@ const TASKS = { buildInfo: { expected: [], // execute() runs in cpu-worker.mjs as the "buildInfo" handler. - submit(out, emit) { emit("dispatch", out); }, + submit(out, emit, state) { + state.site.buildInfo = out.buildInfo; + emit("dispatch", out); + }, }, // Sass compilation (~700 ms CPU). Worker so it overlaps with the main spine. scss: { expected: [], // execute() runs in cpu-worker.mjs as the "scss" handler. - submit() { /* Phase 3: emit("write", out) */ }, + submit(out, emit) { emit("write", out); }, }, // Stale mermaid SVG regeneration. Worker for the same reason. @@ -167,7 +170,7 @@ const TASKS = { for (const f of out.mermaidStats.svgFiles ?? []) { if (!known.has(f.srcRel)) state.staticFiles.push(f); } - /* Phase 3: emit("write", out) */ + emit("write", out); }, }, @@ -287,7 +290,7 @@ const TASKS = { execute(_, ctx, state) { return { stubs: deriveRedirectStubs(state.pages, state.site) }; }, - submit() { /* Phase 3: emit("writeAux", out) */ }, + submit(out, emit) { emit("writeAux", out); }, }, deriveSitemap: { @@ -296,7 +299,7 @@ const TASKS = { execute(_, ctx, state) { return { urls: deriveSitemapUrls(state.pages, state.site) }; }, - submit() { /* Phase 3: emit("writeAux", out) */ }, + submit(out, emit) { emit("writeAux", out); }, }, // ── Render fan-out ───────────────────────────────────────────────────────── @@ -329,7 +332,7 @@ const TASKS = { expected: Array.from({ length: N }, (_, i) => `render:${i}`), runOnMain: true, execute() { return {}; }, - submit() { /* Phase 3: emit("write", {}) */ }, + submit(_, emit) { emit("write", {}); }, }); for (let i = 0; i < N; i++) { @@ -359,6 +362,96 @@ const TASKS = { } }, }, + + // ── Write and post-write tasks ───────────────────────────────────────────── + + // Materialise pages + static files + generated CSS to _site/. + // Waits for renderJoin (pages rendered + templated), scss (generated CSS), + // and mermaid (SVG descriptors appended to state.staticFiles by mermaid.submit). + write: { + expected: ["renderJoin", "scss", "mermaid"], + runOnMain: true, + async execute({ scss: { scssResult }, mermaid: { mermaidStats } }, ctx, state) { + void mermaidStats; // dependency signal only; append already happened in mermaid.submit + const generatedAssets = []; + if (state.site.highlighter?.themeCss) { + generatedAssets.push({ rel: "assets/css/tb-highlight.css", content: state.site.highlighter.themeCss }); + } + if (scssResult.compiled) { + generatedAssets.push({ rel: "assets/css/just-the-docs-combined.css", content: scssResult.css }); + } + return writePhase(state.pages, state.staticFiles, { + destRoot: ctx.destRoot, + dryRun: ctx.opts.dryRun, + generatedAssets, + baseurl: String(state.site.config.baseurl || ""), + }); + }, + submit(out, emit) { emit("searchData", out); }, + }, + + // Write search-data.json. Skipped on dry-run; result passes through to + // writeAux so its search.json field reaches writeOffline. + searchData: { + expected: ["write"], + runOnMain: true, + async execute(_, ctx, state) { + if (ctx.opts.dryRun) return { entries: 0, json: "" }; + return writeSearchData(state.pages, state.site, ctx.destRoot); + }, + submit(out, emit) { emit("writeAux", out); }, + }, + + // Write redirect stubs + sitemap/robots. Waits for searchData (sequencing), + // deriveRedirects, and deriveSitemap (pre-computed stubs/urls). + // Passes searchStats through to writeOffline (for search-data.js). + writeAux: { + expected: ["searchData", "deriveRedirects", "deriveSitemap"], + runOnMain: true, + async execute({ searchData: searchStats, deriveRedirects: { stubs }, deriveSitemap: { urls } }, ctx, state) { + if (ctx.opts.dryRun) return { redirectStats: null, sitemapStats: null, searchStats }; + const [redirectStats, sitemapStats] = await Promise.all([ + writeRedirects(state.pages, state.site, ctx.destRoot, stubs), + writeSitemap(state.pages, state.site, ctx.destRoot, urls), + ]); + return { redirectStats, sitemapStats, searchStats }; + }, + submit(out, emit) { + emit("writeOffline", out); + emit("writePdf", out); + }, + }, + + // Produce _site-offline/. Runs in parallel with writePdf on the main thread; + // the gain is interleaved async I/O windows. + writeOffline: { + expected: ["writeAux"], + runOnMain: true, + async execute({ writeAux: { redirectStats, sitemapStats, searchStats } }, ctx, state) { + const skipOffline = ctx.opts.skipOffline ?? (state.site.config.also_build_offline === false); + if (ctx.opts.dryRun || skipOffline) return null; + const auxStats = { redirects: redirectStats, sitemap: sitemapStats, search: searchStats }; + return writeOffline(state.pages, state.staticFiles, state.site, ctx.destRoot, { + auxStats, + profileOffline: ctx.opts.profileOffline, + }); + }, + submit() { /* terminal */ }, + }, + + // Produce _site-pdf/. Runs in parallel with writeOffline. + writePdf: { + expected: ["writeAux"], + runOnMain: true, + async execute(_, ctx, state) { + const skipPdf = ctx.opts.skipPdf ?? (state.site.config.also_build_pdf === false); + if (ctx.opts.dryRun || skipPdf) return null; + return writePdf(state.pages, state.staticFiles, state.site, ctx.destRoot, { + tolerateMissingImages: ctx.opts.tolerateMissingImages, + }); + }, + submit() { /* terminal */ }, + }, }; function chunkPages(pages, workers) { @@ -373,7 +466,7 @@ function chunkPages(pages, workers) { // ── Build entry point ───────────────────────────────────────────────────────── export async function runBuild(opts) { - const { src, dest, dryRun, tolerateMissingImages, profileOffline } = opts; + const { src, dest } = opts; const srcRoot = path.resolve(process.cwd(), src); const destRoot = path.resolve(dest ?? path.join(srcRoot, "_site")); @@ -382,9 +475,6 @@ export async function runBuild(opts) { const scheduler = new Scheduler({ pool, tasks: TASKS }); const ctx = { srcRoot, destRoot, opts, workerCount }; - // Run the scheduler (seeds + main-thread spine + render fan-out). - // Write and post-write tasks are still the trailing serial block below; - // they graduate to scheduler tasks in Phase 3. let results; try { results = await scheduler.start(ctx); @@ -392,16 +482,11 @@ export async function runBuild(opts) { await pool.destroy(); } - // ── Trailing serial block (Phase 3 will absorb these) ──────────────────── - - const t = makeTimer(); const { pages, staticFiles } = scheduler.state; const site = scheduler.state.site; - // Wire in worker-task outputs that the serial block needs. const { mermaidStats } = results.get("mermaid"); const { scssResult } = results.get("scss"); - site.buildInfo = results.get("buildInfo").buildInfo; if (mermaidStats.regenerated > 0 || mermaidStats.failed > 0) { const parts = [`regenerated ${mermaidStats.regenerated}`]; @@ -411,86 +496,37 @@ export async function runBuild(opts) { if (mermaidStats.failed > 0) process.exitCode = 1; if (scssResult.failed) process.exitCode = 1; - const baseurl = String(site.config.baseurl || ""); - - const generatedAssets = []; - if (site.highlighter?.themeCss) { - generatedAssets.push({ rel: "assets/css/tb-highlight.css", content: site.highlighter.themeCss }); - } - if (scssResult.compiled) { - generatedAssets.push({ rel: "assets/css/just-the-docs-combined.css", content: scssResult.css }); - } - const writeStats = await writePhase(pages, staticFiles, { - destRoot, - dryRun, - generatedAssets, - baseurl, - }); - t.lap("write"); - - let auxStats = null; - if (!dryRun) { - // Use the pre-derived stubs/urls from the scheduler rather than re-deriving. - const stubs = results.get("deriveRedirects").stubs; - const urls = results.get("deriveSitemap").urls; - const [redirectStats, sitemapStats, searchStats] = await Promise.all([ - writeRedirects(pages, site, destRoot, stubs), - writeSitemap(pages, site, destRoot, urls), - writeSearchData(pages, site, destRoot), - ]); - auxStats = { redirects: redirectStats, sitemap: sitemapStats, search: searchStats }; - } - t.lap("auxiliaries"); - - // CLI flag takes precedence; fall back to the `also_build_offline` / - // `also_build_pdf` config knobs. - const skipOffline = opts.skipOffline ?? (site.config.also_build_offline === false); - const skipPdf = opts.skipPdf ?? (site.config.also_build_pdf === false); - - let offlineStats = null; - let offlineTimer = null; - if (!dryRun && !skipOffline) { - offlineStats = await writeOffline(pages, staticFiles, site, destRoot, { - auxStats, - profileOffline, - }); - if (profileOffline) offlineTimer = offlineStats.subT ?? null; - } - t.lap(skipOffline ? "offline:skipped" : "offline"); - - let pdfStats = null; - if (!dryRun && !skipPdf) { - pdfStats = await writePdf(pages, staticFiles, site, destRoot, { tolerateMissingImages }); - } - t.lap(skipPdf ? "pdf:skipped" : "pdf"); + const writeStats = results.get("write"); + const auxResult = results.get("writeAux"); + const offlineResult = results.get("writeOffline"); + const pdfResult = results.get("writePdf"); console.log(`Phase 1+2+3+4+5+6+7+8 done: ${pages.length} pages, ${staticFiles.length} static files`); console.log(` wrote: ${writeStats.pages.written} pages (${writeStats.pages.skipped} skipped), ` + `${writeStats.theme.copied} theme assets, ${writeStats.staticFiles.copied} static files ` + `-> ${destRoot}`); - if (auxStats) { - console.log(` aux: ${auxStats.redirects.written} redirect stubs, ` + - `${auxStats.sitemap.entries} sitemap entries, ` + - `${auxStats.search.entries} search-index entries`); + if (auxResult?.redirectStats) { + console.log(` aux: ${auxResult.redirectStats.written} redirect stubs, ` + + `${auxResult.sitemapStats.entries} sitemap entries, ` + + `${auxResult.searchStats.entries} search-index entries`); } - if (offlineStats) { - console.log(` offline: ${offlineStats.html} HTML, ${offlineStats.css} CSS, ` + - `${offlineStats.redirects} redirect stubs, ` + - `${offlineStats.statics + offlineStats.assets} assets, ` + - `${offlineStats.excluded} excluded ` + - `(${offlineStats.unresolved} unresolved) -> ${destRoot}-offline`); + if (offlineResult) { + console.log(` offline: ${offlineResult.html} HTML, ${offlineResult.css} CSS, ` + + `${offlineResult.redirects} redirect stubs, ` + + `${offlineResult.statics + offlineResult.assets} assets, ` + + `${offlineResult.excluded} excluded ` + + `(${offlineResult.unresolved} unresolved) -> ${destRoot}-offline`); + if (opts.profileOffline && offlineResult.subT) { + console.log(` offline: ${offlineResult.subT.summary()}`); + } } - if (pdfStats) { - const mb = (pdfStats.bookBytes / (1024 * 1024)).toFixed(1); - const missingClause = pdfStats.missing > 0 ? ` (${pdfStats.missing} missing)` : ""; - console.log(` pdf: book.html (${mb} MB), ${pdfStats.css} CSS, ` + - `${pdfStats.images} images${missingClause} -> ${destRoot}-pdf`); + if (pdfResult) { + const mb = (pdfResult.bookBytes / (1024 * 1024)).toFixed(1); + const missingClause = pdfResult.missing > 0 ? ` (${pdfResult.missing} missing)` : ""; + console.log(` pdf: book.html (${mb} MB), ${pdfResult.css} CSS, ` + + `${pdfResult.images} images${missingClause} -> ${destRoot}-pdf`); } console.log(scheduler.summary()); - console.log(t.summary()); - if (offlineTimer) { - console.log(` offline: ${offlineTimer.summary()}`); - } // Drift guard from PLAN-1.md §1. if (pages.length < 836) { From 9a4d6798c8ba450ba7ca6966987e2b024833c0c3 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Sat, 30 May 2026 19:23:56 +0200 Subject: [PATCH 06/72] Scheduler Phase 4: SAB broadcast for render fan-out --- builder/PLAN-scheduler.md | 97 +++++++++++++++++++++++++-------------- builder/cpu-worker.mjs | 4 +- builder/sab-broadcast.mjs | 17 +++++++ builder/tbdocs.mjs | 27 ++++++----- 4 files changed, 95 insertions(+), 50 deletions(-) create mode 100644 builder/sab-broadcast.mjs diff --git a/builder/PLAN-scheduler.md b/builder/PLAN-scheduler.md index 02f1410e..da8dab02 100644 --- a/builder/PLAN-scheduler.md +++ b/builder/PLAN-scheduler.md @@ -560,17 +560,22 @@ const TASKS = { // Read pages directly from state.pages -- main-thread access, // no need to ship them through the input map. const chunks = chunkPages(state.pages, ctx.workerCount); - const siteData = { - config: state.site.config, - seoSiteTitle: state.site.seoSiteTitle, - seoLogoUrl: state.site.seoLogoUrl, - }; - return { - chunks, siteData, initData, buildInfo, + const shared = { + siteData: { + config: state.site.config, + seoSiteTitle: state.site.seoSiteTitle, + seoLogoUrl: state.site.seoLogoUrl, + }, + initData, buildInfo, linkTablesData: state.site.linkTablesSerialized, staticFilesArr: state.staticFiles.map(f => f.srcRel), baseurl: String(state.site.config.baseurl || ""), }; + // Pack the shared payload into a SharedArrayBuffer so each + // postMessage sends a SAB reference (shared memory) instead of + // structured-cloning ~286 KB per worker. + const sharedSAB = packShared(shared); + return { chunks, sharedSAB }; }, submit(out, emit, _state, scheduler) { const N = out.chunks.length; @@ -602,13 +607,8 @@ const TASKS = { }, }); scheduler.seed(id, { - siteData: out.siteData, - initData: out.initData, - linkTablesData: out.linkTablesData, - staticFilesArr: out.staticFilesArr, - baseurl: out.baseurl, - buildInfo: out.buildInfo, - chunk: out.chunks[i], + sharedSAB: out.sharedSAB, + chunk: out.chunks[i], }); } }, @@ -910,6 +910,7 @@ import { renderPhase, } from "./render.mjs"; import { templatePhase } from "./template.mjs"; +import { unpackShared } from "./sab-broadcast.mjs"; // Start WASM init immediately, do NOT await. The module finishes // loading synchronously so the parentPort.on('message') dispatcher is @@ -931,8 +932,9 @@ const handlers = { }, async render({ inputs }) { + const { sharedSAB, chunk } = inputs; const { siteData, initData, linkTablesData, staticFilesArr, - baseurl, buildInfo, chunk } = inputs; + baseurl, buildInfo } = unpackShared(sharedSAB); const highlighter = await highlighterP; const linkTables = reconstructLinkTables(linkTablesData); @@ -1041,14 +1043,15 @@ nav / seo / loadData / resolveBookChapters / buildInit run on main against `state.pages` directly. No marshalling, no delta merge -- mutations are immediately visible to downstream main-thread tasks. -### SharedArrayBuffer broadcast (Phase 4, optional) -For the render fan-out, serialize `siteData + initData + linkTables` -once into a SharedArrayBuffer and pass the wrapper to all render tasks -via the pool's `transferList`. The SAB itself is shared memory, not -cloned. Each worker deserializes its slice. The SAB is read-only by -convention. Measure first -- likely modest savings vs. the N -structured-clone fan-outs, only worth doing if profiling shows -dispatch overhead is meaningful. +### SharedArrayBuffer broadcast (Phase 4) +The render fan-out's shared payload (`siteData + initData + +linkTablesData + staticFilesArr + baseurl + buildInfo`, ~286 KB) is +JSON-serialized once on the main thread into a SharedArrayBuffer via +`sab-broadcast.mjs`'s `packShared()`. Each render task receives the +SAB reference (shared memory, not cloned) alongside its per-worker +chunk. Workers call `unpackShared()` to deserialize independently and +in parallel. Measured saving: ~8 ms per build (fan-out drops from +~19 ms to ~9 ms). ## Error handling @@ -1300,7 +1303,7 @@ data lifetime across the worker boundary, or low-level serialisation. | 2. Render fan-out | Opus | Cross-thread structured-clone semantics, module-scope initialisation order, dynamic task registration, per-page delta-merge identity. Debugging concurrency bugs needs depth. | | 3. Post-write tasks | Sonnet | All `runOnMain`; thin wrappers around existing write functions. No new concurrency surface beyond the already-built scheduler. | | 3-follow-up. Workerize writeOffline | Opus (decided) | Profiled: zero CPU contention; cooperative async overlap is already optimal. Declined — no implementation needed. | -| 4. SAB broadcast | Opus | Manual serialisation into shared memory; layout / endianness / varint choices need reasoning about memory ordering. | +| 4. SAB broadcast | Opus | JSON + SAB approach; measured ~55% fan-out reduction (~8 ms saving). | Escalate to Opus mid-phase if a Sonnet session hits a debugging block it can't reason through. @@ -1471,17 +1474,41 @@ surface, and add a result-merge path — all for no measurable gain. The overlap already saves ~260 ms vs. sequential execution (the full duration of `writePdf`). No further action needed. -### Phase 4: SharedArrayBuffer broadcast (optional) +### Phase 4: SharedArrayBuffer broadcast **Suggested model:** Opus. -For the render fan-out, serialize `siteData + initData + linkTables` -once into a SharedArrayBuffer and pass it to all render tasks via -the pool's `transferList`. Measure first -- likely modest savings -vs. the N structured-clone fan-outs. - -**Deliverable:** measured reduction in render-dispatch overhead. - -**Verification.** `build.bat && check.bat` clean. Render dispatch -overhead (the gap between `dispatch` end and the earliest `render:i` -start in the timing summary) should decrease vs. Phase 2. +For the render fan-out, serialize `siteData + initData + linkTables + +staticFilesArr + baseurl + buildInfo` once into a SharedArrayBuffer +and pass it to all render tasks. The SAB is shared memory --- each +worker deserializes its own copy from the same buffer instead of the +main thread structured-cloning the ~286 KB shared payload 16 times. + +**Implementation.** Three files: + +- `builder/sab-broadcast.mjs` (~15 LOC): `packShared(obj)` serializes + an object to JSON, encodes to UTF-8, and copies into a SAB; + `unpackShared(sab)` reverses the process. +- `tbdocs.mjs` `dispatch.execute()`: packs the shared payload into a + SAB and returns `{ chunks, sharedSAB }` instead of the flat fields. +- `cpu-worker.mjs` `render` handler: calls `unpackShared(sharedSAB)` + to reconstruct the shared fields before rendering. + +**Measurements** (16 workers, ~286 KB shared payload, 857 pages): + +| | Run 1 | Run 2 | Run 3 | Median | +|---|---|---|---|---| +| Baseline (structured-clone) | 18.1 ms | 35.0 ms | 19.5 ms | ~19 ms | +| SAB broadcast | 9.2 ms | 7.3 ms | 10.2 ms | ~9 ms | + +SAB packing cost (in `dispatch.execute()`): ~2 ms (visible as +`dispatch=2ms` vs. prior `dispatch=0ms`). Net saving: ~8 ms per build, +a ~55% reduction in fan-out overhead. The saving is modest in absolute +terms (~0.2% of a ~4 s build) but the implementation is small and the +pattern moves redundant serialization work off the main thread --- +each worker independently deserializes from shared memory in parallel +instead of the main thread serializing 16 identical copies +sequentially. + +**Verification.** `build.bat && check.bat` clean. `dispatch` now +shows ~2 ms (SAB packing) vs. 0 ms before. diff --git a/builder/cpu-worker.mjs b/builder/cpu-worker.mjs index c028e5a0..50377266 100644 --- a/builder/cpu-worker.mjs +++ b/builder/cpu-worker.mjs @@ -12,6 +12,7 @@ import { initHighlighter } from "./highlight.mjs"; import { createMarkdownIt, buildLinkTables, renderPhase } from "./render.mjs"; import { templatePhase } from "./template.mjs"; +import { unpackShared } from "./sab-broadcast.mjs"; // Start WASM init immediately, do NOT await. Module evaluation finishes // synchronously so the parentPort.on('message') dispatcher is installed @@ -32,8 +33,9 @@ const handlers = { }, async render({ inputs }) { + const { sharedSAB, chunk } = inputs; const { siteData, initData, linkTablesData, staticFilesArr, - baseurl, buildInfo, chunk } = inputs; + baseurl, buildInfo } = unpackShared(sharedSAB); const highlighter = await highlighterP; const linkTables = reconstructLinkTables(linkTablesData); diff --git a/builder/sab-broadcast.mjs b/builder/sab-broadcast.mjs new file mode 100644 index 00000000..54ca17a9 --- /dev/null +++ b/builder/sab-broadcast.mjs @@ -0,0 +1,17 @@ +// SharedArrayBuffer broadcast for the render fan-out. Serializes an object +// once into a SAB so all workers share the same memory instead of each +// receiving an independent structured-clone copy. + +const encoder = new TextEncoder(); +const decoder = new TextDecoder(); + +export function packShared(obj) { + const bytes = encoder.encode(JSON.stringify(obj)); + const sab = new SharedArrayBuffer(bytes.byteLength); + new Uint8Array(sab).set(bytes); + return sab; +} + +export function unpackShared(sab) { + return JSON.parse(decoder.decode(new Uint8Array(sab))); +} diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 45d40bf7..8c31938b 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -39,6 +39,7 @@ import { writeSitemap, deriveSitemapUrls } from "./sitemap.mjs"; import { writeSearchData } from "./search.mjs"; import { writeOffline } from "./offline.mjs"; import { writePdf } from "./pdf.mjs"; +import { packShared } from "./sab-broadcast.mjs"; const CPU_WORKER_URL = new URL("./cpu-worker.mjs", import.meta.url); @@ -313,17 +314,20 @@ const TASKS = { runOnMain: true, execute({ buildInit: { initData }, buildInfo: { buildInfo } }, ctx, state) { const chunks = chunkPages(state.pages, ctx.workerCount); - const siteData = { - config: state.site.config, - seoSiteTitle: state.site.seoSiteTitle, - seoLogoUrl: state.site.seoLogoUrl, - }; - return { - chunks, siteData, initData, buildInfo, + const shared = { + siteData: { + config: state.site.config, + seoSiteTitle: state.site.seoSiteTitle, + seoLogoUrl: state.site.seoLogoUrl, + }, + initData, + buildInfo, linkTablesData: state.site.linkTablesSerialized, staticFilesArr: state.staticFiles.map(f => f.srcRel), baseurl: String(state.site.config.baseurl || ""), }; + const sharedSAB = packShared(shared); + return { chunks, sharedSAB }; }, submit(out, emit, _state, scheduler) { const N = out.chunks.length; @@ -351,13 +355,8 @@ const TASKS = { }, }); scheduler.seed(id, { - siteData: out.siteData, - initData: out.initData, - linkTablesData: out.linkTablesData, - staticFilesArr: out.staticFilesArr, - baseurl: out.baseurl, - buildInfo: out.buildInfo, - chunk: out.chunks[i], + sharedSAB: out.sharedSAB, + chunk: out.chunks[i], }); } }, From 6b7f1ba6199473f14b53eb50aa74aad8ae3d6e83 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Sat, 30 May 2026 19:27:25 +0200 Subject: [PATCH 07/72] Close out the scheduler work. --- WIP.md | 2 +- builder/PLAN.md | 7 +++---- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/WIP.md b/WIP.md index 58285de7..0df1030c 100644 --- a/WIP.md +++ b/WIP.md @@ -432,7 +432,7 @@ Python scripts are reserved for non-render concerns: one-off content conversion The site builds via [builder/](builder/), a custom Node.js static site generator (`tbdocs`). See [builder/PLAN.md](builder/PLAN.md) for the architecture overview, [builder/README.md](builder/README.md) for the quickstart, and the [tbdocs Internals](docs/Documentation/Builder.md) site page for the high-level tour. -A planned task-graph scheduler / parallelisation pass is designed in [builder/PLAN-scheduler.md](builder/PLAN-scheduler.md) (not yet implemented). Hand a fresh session that file and `Phase 0` as the entry point. +A task-graph scheduler / parallelisation pass is designed in [builder/PLAN-scheduler.md](builder/PLAN-scheduler.md) and has been implemented (Phases 0--4). Historical engineering notes from the Jekyll era --- the original build pipeline, the HTML-compress plugin, the per-phase optimisation passes that preceded the JS port, the migration notes, and the Phase 11 parity-update retrospective --- live in [WIP.OldJekyll.md](WIP.OldJekyll.md). diff --git a/builder/PLAN.md b/builder/PLAN.md index e068e0ac..86dfc62a 100644 --- a/builder/PLAN.md +++ b/builder/PLAN.md @@ -63,12 +63,11 @@ static server; `docs/serve.bat` becomes a one-line `--serve` shim. Closes the PLAN-10 §7.D4 and §7.D11 watch-mode deferrals. A **task-graph scheduler** for the build pipeline is designed in -[PLAN-scheduler.md](PLAN-scheduler.md) -- not yet implemented. The -plan covers a thin in-tree scheduler + `WorkerPool` over +[PLAN-scheduler.md](PLAN-scheduler.md) and has been implemented +(Phases 0--4). It covers a thin in-tree scheduler + `WorkerPool` over `node:worker_threads`, moves CPU-bound seed tasks (`scss`, `mermaid`, `buildInfo`) onto workers, and fans out `renderPhase` + `templatePhase` -across CPUs. Wall-clock target: ~6.7 s → ~3.6 s on 8 cores. Start -with §Migration path Phase 0. +across CPUs via SAB broadcast. Open follow-ups (deferred enhancements, divergence investigations) live in [FUTURE-WORK.md](FUTURE-WORK.md). From 699ec79c3cf7d3778e9be983c0bc6544a1505c07 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Sat, 30 May 2026 19:44:29 +0200 Subject: [PATCH 08/72] Update the scheduler documentation. --- docs/Documentation/Builder.md | 75 +++++++++++++++- docs/Documentation/Extending.md | 21 +++-- docs/Documentation/Pipeline-Stages.md | 107 +++++++++++++++++------ docs/Documentation/index.md | 4 +- docs/assets/images/mmd/scheduler-dag.mmd | 55 ++++++++++++ docs/assets/images/mmd/scheduler-dag.svg | 1 + 6 files changed, 224 insertions(+), 39 deletions(-) create mode 100644 docs/assets/images/mmd/scheduler-dag.mmd create mode 100644 docs/assets/images/mmd/scheduler-dag.svg diff --git a/docs/Documentation/Builder.md b/docs/Documentation/Builder.md index ad46c598..1ae34bac 100644 --- a/docs/Documentation/Builder.md +++ b/docs/Documentation/Builder.md @@ -60,6 +60,10 @@ One entry point, ~17 production modules. The content model is fixed (markdown + | [`search.mjs`](https://github.com/twinbasic/documentation/blob/main/builder/search.mjs) | Phase 6 Lunr index emitter (`search-data.json`). | | [`offline.mjs`](https://github.com/twinbasic/documentation/blob/main/builder/offline.mjs) | Phase 7 offline tree: URL rewriting, JS patching for `file://` browsing. | | [`pdf.mjs`](https://github.com/twinbasic/documentation/blob/main/builder/pdf.mjs) | Phase 8 sparse PDF source tree. | +| [`scheduler.mjs`](https://github.com/twinbasic/documentation/blob/main/builder/scheduler.mjs) | Task-graph scheduler: `Scheduler` class + `SharedState`. Tracks dependencies, dispatches tasks to the main thread or worker pool. | +| [`worker-pool.mjs`](https://github.com/twinbasic/documentation/blob/main/builder/worker-pool.mjs) | `WorkerPool` --- minimal `node:worker_threads` pool with named dispatch and idle/busy bookkeeping. | +| [`cpu-worker.mjs`](https://github.com/twinbasic/documentation/blob/main/builder/cpu-worker.mjs) | Worker harness: `parentPort` message dispatcher + four named handlers (`scss`, `mermaid`, `buildInfo`, `render`). | +| [`sab-broadcast.mjs`](https://github.com/twinbasic/documentation/blob/main/builder/sab-broadcast.mjs) | SharedArrayBuffer broadcast: `packShared` / `unpackShared` for the render fan-out's shared payload. | `builder/` lives at the repo root (not under `docs/`) so it is not part of the Jekyll source tree the legacy renderer reads. The `build.bat` path writes to `docs/_site/`, `docs/_site-offline/`, and `docs/_site-pdf/` --- the same destinations Jekyll used, so deployment tooling stays unchanged. The `serve.bat` path writes to a separate `docs/_serve/` tree so a one-off `build.bat` run (refreshing the PDF, for example) never clobbers a running serve session's output. @@ -78,6 +82,34 @@ One entry point, ~17 production modules. The content model is fixed (markdown + Phases 9, 10, and 11 are historical: Phase 9 was a no-output QoL pass, Phase 10 retired Jekyll, Phase 11 introduces the output-changing parity updates. None adds a runtime step. Phase 12 adds the `--serve` dev-server mode (a separate lifecycle, not a build phase; writes to `docs/_serve/` and skips the offline + PDF passes by default so the rebuild loop stays under one second). The per-phase `PLAN-N.md` files retain the implementation history. +## Scheduler and worker threads + +The build pipeline is driven by a task-graph **scheduler** that models the pipeline as a DAG (directed acyclic graph) of tasks. Each task declares its predecessor task IDs, runs an `execute()` body, and routes its output to downstream tasks via a synchronous `submit()` callback. The scheduler dispatches tasks to either the main thread or a **worker pool** of `node:worker_threads`, depending on the task definition's `runOnMain` flag. + +![Scheduler task DAG](/assets/images/mmd/scheduler-dag.svg) + +**[M]** = runs on main thread; **[W]** = dispatched to a worker thread. Worker tasks overlap with the main-thread spine and with each other; main-thread tasks run serially within the main thread but can overlap with worker tasks. + +Three structural wins over the earlier serial baseline: + +1. **Seed tasks overlap with the main spine.** `scss` (~700 ms), `mermaid`, and `buildInfo` run on workers concurrently with the main-thread spine (discover → nav → markdownInit → seo → loadData → resolveBookChapters + buildInit). The spine takes ~250 ms total, so ~250 ms of `scss` hides behind it. +2. **Render + template fans out across CPUs.** The single-threaded render + template work (~3.5 s combined) splits into N page chunks, each dispatched to a worker that runs both `renderPhase` and `templatePhase` on its slice. On a 4-core machine this compresses to ~875 ms; on 8 cores, ~440 ms. +3. **`writeOffline` and `writePdf` overlap.** Both are `runOnMain` but their I/O-dominated `await` gaps interleave via cooperative async concurrency. The combined wall-clock equals `max(writeOffline, writePdf)` rather than their sum. + +### Shared state and page deltas + +The scheduler owns a `SharedState` instance that carries the master `pages[]`, `staticFiles[]`, `site` object, and a `pageByDest` lookup map. Main-thread tasks mutate `state.pages` directly --- no marshalling, no delta merge. Worker tasks receive structured-clone copies of their inputs and return **page deltas** (arrays of `{ destPath, renderedContent, html }`); the task's `submit()` runs on the main thread and merges each delta entry into the master page via `state.pageByDest`. + +After `discover.submit()` builds the master `pages[]`, no task ever replaces the array --- only mutates it in place. This preserves object identity across phases, which is critical for `resolveBookChapters` (stores `Page` references into `bookData._chapters`) and `writePdf` (reads `renderedContent` from those same objects after the render fan-out fills them in). + +### Render fan-out + +The `dispatch` task slices `state.pages` into N chunks (one per available CPU), packs the shared per-build payload (site config, SEO metadata, pre-rendered sidebar HTML, serialized link tables, static file set, build info) into a **SharedArrayBuffer** via `sab-broadcast.mjs`'s `packShared()`, and dynamically registers N `render:i` worker tasks plus a `renderJoin` barrier. Each `render:i` receives the SAB reference (shared memory, not cloned) alongside its per-worker page chunk, deserializes the shared payload independently, initializes its own markdown-it + Shiki instance, runs `renderPhase` + `templatePhase` over its slice, and returns the per-page deltas. The `renderJoin` barrier waits for all N render tasks before unblocking `write`. + +### Data transfer strategy + +Small outputs (config, navTree, initData, buildInfo, scssResult) cross the worker boundary via structured clone at negligible cost. The link tables (~50 KB) are serialized once on main as `[key, permalink]` pairs and shipped to workers, which reconstruct minimal `{ permalink }` stubs. The render chunks (~860 KB per worker on a 4-core box) are the largest single transfer; the deltas returned are ~30 KB per worker. The SharedArrayBuffer broadcast avoids structured-cloning the ~286 KB shared payload once per worker, reducing fan-out overhead by ~55%. + ## Dependencies A single `package.json` at the repo root carries everything --- the static site generator's deps, the PDF renderer's deps, and the few packages both consume. There is no per-`builder/` `package.json` (an earlier split was consolidated; the previous arrangement required `npm ci --prefix builder` in CI and ended up dragging in a duplicate puppeteer-core via `@mermaid-js/mermaid-cli`): @@ -118,12 +150,18 @@ Each subsection covers the design rationale and implementation details for one m ### [tbdocs.mjs](https://github.com/twinbasic/documentation/blob/main/builder/tbdocs.mjs) --- entry point and orchestrator -`_config.yml` is loaded first so its `exclude:` list can be passed to `discover()`. `captureBuildInfo()` is launched as a promise immediately after the config load so the two `git` shell-outs overlap with the I/O-bound discover and the CPU-bound nav computation that follows; the result is `await`ed only once Phase 2's other substeps are done. The shared markdown-it instance is built once via `initHighlighter` + `createMarkdownIt` and stored on `site.markdown` so Phase 2's SEO precompute and Phase 3's body renderer use the same configured pipeline --- titles run through the same dash, quote, and footnote-stripping rules as page body text. +`runBuild()` constructs a `WorkerPool` of `os.availableParallelism()` workers, instantiates a `Scheduler` with the static `TASKS` graph, and calls `scheduler.start(ctx)`. The scheduler dispatches all seed tasks immediately (`config`, `buildInfo`, `scss`, `mermaid`), chains the main-thread spine as each task's `submit()` emits to the next, dynamically registers the `render:0..N` fan-out from `dispatch.submit()`, and resolves when the two terminal tasks (`writeOffline`, `writePdf`) complete. After `start()` resolves, `runBuild()` reads results from the scheduler's `results` map, logs the build summary, and destroys the pool. + +The `TASKS` object defines every task in the DAG as a plain object with `expected` (predecessor IDs), `execute()` (task body), and `submit()` (synchronous output router). Tasks without `runOnMain: true` are dispatched to the worker pool by handler name; the four worker handlers (`scss`, `mermaid`, `buildInfo`, `render`) live in `cpu-worker.mjs`. + +The shared markdown-it instance is built once during the `markdownInit` task via `initHighlighter` + `createMarkdownIt` and stored on `state.site.markdown` so the `seo` task and each render worker use the same configured pipeline --- titles run through the same dash, quote, and footnote-stripping rules as page body text. Workers build their own independent markdown-it + Shiki instances from the serialized link tables and shared payload. The drift guard at the end (`if (pages.length < 836)`) sets `process.exitCode = 1` when discover loses pages --- a discovery-rule regression that silently drops content appears as a non-zero exit even though the build itself "succeeded". ### [serve.mjs](https://github.com/twinbasic/documentation/blob/main/builder/serve.mjs) --- Phase 12 dev server +The worker pool is constructed once at `runServe()` startup and re-used across rebuilds; only the `Scheduler` instance (and its `SharedState`) is fresh per rebuild. Workers stay warm --- WASM caches, JIT state, and module caches all survive. The worker spawn cost (~100--200 ms) is paid once and amortizes to zero across rebuilds. + The 300 ms debounce coalesces rapid file changes into a single rebuild. A lightweight inject middleware splices the SSE client script before `` at HTTP-response time so the on-disk `_serve/` stays byte-identical to what `runBuild --dest docs/_serve` would have produced outside of serve mode. `shouldRebuild` filters watcher events along three axes: prefixes (`_site/`, `_site-offline/`, `_site-pdf/`, `_serve/`, `_pdf/`, `node_modules/`, `.git/`), basename patterns (dotfiles, editor swap files, the `4913` sentinel vim writes), and the specific `assets/images/mmd/*.svg` path. The last bit deserves a callout: those SVGs are emitted by the mermaid pre-phase back under `srcRoot`, so without the filter every `.mmd` edit fires the watcher twice (once on the `.mmd` save, once on the `.svg` write mid-rebuild) and the queued second rebuild is a no-op that triggers a redundant browser reload ~3 s later. The filter treats the `.mmd` as the source of truth and the `.svg` as a build artifact, matching how `_site/` writes are already excluded. @@ -170,7 +208,7 @@ Each part and chapter divider page contains the entry's title as an H1/H2 headin ### [build-info.mjs](https://github.com/twinbasic/documentation/blob/main/builder/build-info.mjs) --- Phase 2 git capture -Both git shell-outs fall back to `"unknown"` on failure so a tarball install or a sparse checkout never aborts the build. +Runs on a worker thread as a seed task so the two `git` shell-outs overlap with the main-thread spine. Both fall back to `"unknown"` on failure so a tarball install or a sparse checkout never aborts the build. ### [data.mjs](https://github.com/twinbasic/documentation/blob/main/builder/data.mjs) --- Phase 2 data loader @@ -178,6 +216,8 @@ Replaces the book-specific YAML load that originally lived in [`book.mjs`](https ### [mermaid.mjs](https://github.com/twinbasic/documentation/blob/main/builder/mermaid.mjs) --- Phase 11 (B1) preprocessor +Runs on a worker thread as a seed task, concurrently with `discover` and the rest of the main-thread spine. Because the two tasks overlap, freshly-emitted SVGs may not appear in `discover`'s `staticFiles[]`; the `mermaid` task's `submit()` appends any new SVG descriptors (with full `{ srcPath, srcRel, destRel, size }` metadata, stat'd on the worker) into `state.staticFiles` synchronously on the main thread. + Drives `puppeteer` + the in-tree `mermaid` package directly. Earlier this module shelled out to `@mermaid-js/mermaid-cli` via `npx mmdc`, which forked a fresh node + Chrome process per diagram and shipped its own bundled puppeteer-core (forcing a duplicate Chrome download); the direct path collapses both costs into one browser launch for the whole batch and one entry in the dependency tree. The render runs in a single `page.evaluate` that dynamic-imports `mermaid.esm.mjs` and calls `mermaid.render('my-svg', definition, container)`, then serialises the resulting `` via `XMLSerializer`. The SVG id matches mermaid-cli's default so any previously-committed SVG diffs cleanly against the new output. The bare HTML page is a `data:text/html` URL with one `
`; nothing else is loaded. @@ -191,6 +231,8 @@ The render runs in a single `page.evaluate` that dynamic-imports `mermaid.esm.mj ### [scss.mjs](https://github.com/twinbasic/documentation/blob/main/builder/scss.mjs) --- Phase 11 (B3) SCSS compiler +Runs on a worker thread as a seed task so the ~700 ms Sass compilation overlaps with the main-thread spine. The compiled CSS result flows to the `write` task via `scss.submit()`. + Runs Dart Sass (the [`sass`](https://www.npmjs.com/package/sass) npm package) over `docs/assets/css/just-the-docs-combined.scss` and pushes the result onto `generatedAssets` as `assets/css/just-the-docs-combined.css`. Replaces the Jekyll-era pre-compiled CSS that used to live under `builder/assets/`; editing any SCSS partial now reflects on the next build instead of requiring a re-extraction. Load paths are stacked, searched in order: `docs/_sass/` first (our customizations under `custom/`), then `builder/vendor/just-the-docs/_sass/` (the gem at v0.10.1). The same shadowing Jekyll relied on still applies --- `@import "custom/custom"` resolves to our `docs/_sass/custom/custom.scss` because the load-path order puts our `_sass/` first. @@ -206,7 +248,7 @@ Upstream Dart Sass emits deprecation warnings against several gem-vendored const ### [render.mjs](https://github.com/twinbasic/documentation/blob/main/builder/render.mjs) --- Phase 3 markdown pipeline -The largest single module (~1,580 lines) and the runtime hot path --- this is what dominates the ~1--2 s build time. +The largest single module (~1,580 lines) and the runtime hot path. Under the scheduler, the render phase fans out across N worker threads (one `render:i` task per available CPU); each worker builds its own markdown-it + Shiki instance from the serialized link tables and shared payload, then runs `renderPhase` + `templatePhase` over its page chunk. The single-threaded ~3.5 s of combined render + template work compresses to ~3500/N ms wall-clock. `createMarkdownIt(ctx)` is the configuration heart. The base options (`html: true`, `xhtmlOut: true`, `breaks: false`, `linkify: false`, `typographer: true`, `quotes: "“”‘’"`) match kramdown's defaults. Plugins layer on: `markdown-it-attrs` with the `{:` / `}` delimiters that kramdown uses, `markdown-it-deflist`, `markdown-it-footnote` with the kramdown render rules (`fnref:N` / `reversefootnote` / `
` shapes; see `configureFootnotes`), plus a stack of in-tree plugins: @@ -287,6 +329,33 @@ The image-path collector folds into `assembleBook`'s per-chapter emit (Phase 9 `reportMissingImages` implements pdfify.rb's strict mode: per-path error log, then throw if `!tolerateMissingImages`. Every Phase 8 invocation runs in strict mode by default --- a missing image in the assembled book is a build-fail rather than a warning, since the alternative is a PDF with broken-image placeholders nobody notices until publication. The `--tolerate-missing-images` flag (renamed from `--serving` in Phase 12) downgrades the throw to a warning for iterative work. +### [scheduler.mjs](https://github.com/twinbasic/documentation/blob/main/builder/scheduler.mjs) --- task-graph scheduler + +~110 lines. Two exports: `Scheduler` and `SharedState`. + +`SharedState` carries the master `pages[]`, `staticFiles[]`, `site` object, and the `pageByDest` lookup map. The `Scheduler` constructor takes a `WorkerPool` instance and a `tasks` map; `start(ctx)` queues all seed tasks (those with `expected: []`) and returns a promise that resolves when every task has completed. The scheduler is a thin coordinator --- it tracks pending dependency counts, maintains a ready queue, and dispatches tasks to either the main thread (via `Promise.resolve(task.execute(...))`) or the pool (via `pool.run()`). Task completion invokes the task's synchronous `submit()` callback, which calls `emit(targetId, data)` to route output slices to downstream tasks. + +`register(id, def)` and `seed(id, inputs)` support dynamic task creation: `dispatch.submit()` calls `register` for each `render:i` task and the `renderJoin` barrier at fan-out time, then `seed` to immediately enqueue each render task with its chunk. `summary()` returns the per-task timing string (e.g. `config=1ms discover=98ms scss=1041ms ...`). + +### [worker-pool.mjs](https://github.com/twinbasic/documentation/blob/main/builder/worker-pool.mjs) --- worker thread pool + +~55 lines. Spawns `size` workers eagerly at construction (so Shiki WASM warmup overlaps with seed-task work) and routes named tasks to whichever worker is idle. No dynamic scaling, no recycling, no abort signals --- each feature is real complexity the project does not need. The pool's only public methods are `run(payload, { name })` (returns a promise that resolves with the worker's result) and `destroy()` (terminates all workers). Worker crashes are not recovered --- the dead worker stays out of the idle list and the pool degrades by one for the rest of the run; for a one-shot `runBuild`, the resulting task rejection aborts via the scheduler's error path. + +### [cpu-worker.mjs](https://github.com/twinbasic/documentation/blob/main/builder/cpu-worker.mjs) --- worker harness + +~80 lines. A `parentPort.on("message")` dispatcher routes `{ name, ...payload }` messages to one of four named handlers: + +- **`scss`** --- calls `compileScss(ctx.srcRoot)`. +- **`mermaid`** --- calls `regenerateMermaid(ctx.srcRoot)`. +- **`buildInfo`** --- calls `captureBuildInfo()`. +- **`render`** --- the render + template hot path. Deserializes the shared payload from the SharedArrayBuffer, awaits the module-scope Shiki WASM init (started eagerly at import time, before any message arrives), builds a fresh markdown-it instance from the serialized link tables, runs `renderPhase` + `templatePhase` over the page chunk, and returns per-page deltas (`{ destPath, renderedContent, html }`). + +Each worker starts its own `initHighlighter()` call at module scope without awaiting it, so the WASM init overlaps with worker spawn and the main-thread spine. Only the `render` handler awaits the result; the other three handlers service tasks while Shiki is still loading. + +### [sab-broadcast.mjs](https://github.com/twinbasic/documentation/blob/main/builder/sab-broadcast.mjs) --- SharedArrayBuffer broadcast + +~15 lines. `packShared(obj)` JSON-serializes an object, encodes to UTF-8, and copies into a `SharedArrayBuffer`. `unpackShared(sab)` reverses the process. The render fan-out's shared payload (~286 KB of site config, pre-rendered sidebar HTML, serialized link tables, static file set, and build info) is packed once on the main thread; each worker receives the SAB reference (shared memory, not cloned) and deserializes independently. Measured saving: ~55% reduction in fan-out overhead (~8 ms per build). + ## Asset layout The site's `/assets/` tree at deploy time is assembled from three sources: diff --git a/docs/Documentation/Extending.md b/docs/Documentation/Extending.md index cd6cc2d4..534aed27 100644 --- a/docs/Documentation/Extending.md +++ b/docs/Documentation/Extending.md @@ -16,7 +16,7 @@ How to add a new pipeline stage or a custom markdown-it plugin to `tbdocs`. This ## Two extension points -**New pipeline stage** --- a new `.mjs` module that reads from the `pages` array or `site` object and writes output to disk or to page fields. The module exports one async function. The orchestrator in `tbdocs.mjs` calls it at the right point in the fixed sequence. No plugin registry or hook system is involved. +**New pipeline stage** --- a new `.mjs` module that reads from the `pages` array or `site` object and writes output to disk or to page fields. The module exports one async function. The orchestrator in `tbdocs.mjs` registers it as a task in the scheduler's DAG, with declared predecessor dependencies that determine when it runs. No plugin registry or hook system is involved. **New markdown-it plugin** --- a function that configures the shared markdown-it instance with additional parsing or rendering rules. Registered in `createMarkdownIt` inside `render.mjs`. Both Phase 2's SEO title extraction and Phase 3's body render use the same instance, so the plugin runs on every page. @@ -65,21 +65,24 @@ Add an import at the top of `builder/tbdocs.mjs`: import { myStage } from "./my-stage.mjs"; ``` -Then call the stage in `runBuild` at the right position in the sequence. Most auxiliary stages belong after Phase 5 (write) and before Phase 7 (offline), so the online tree is complete when they run: +Then add a task definition in the `TASKS` object. Each task declares its predecessor IDs in `expected`, runs its work in `execute()`, and routes its output to downstream tasks in `submit()`. Most auxiliary stages depend on `write` (so the online tree is complete) and emit into `writeAux` or a similar downstream task: ```js -const myStats = await myStage(pages, site, destRoot); -t.lap("my-stage"); -if (myStats) { - console.log(` my-stage: ${myStats.entries} entries`); -} +myStage: { + expected: ["write"], + runOnMain: true, + async execute(_, ctx, state) { + return myStage(state.pages, state.site, ctx.destRoot); + }, + submit(out, emit) { emit("writeAux", out); }, +}, ``` -`t.lap("my-stage")` records wall-clock time for the step; the label appears in the timing summary line at the end of the build. +Set `runOnMain: true` for stages that read from `state.pages` or `state.site` directly. The scheduler records per-task wall-clock timings automatically; the label in the timing summary is the task's key in the `TASKS` object. ### 3. Handle the `dryRun` flag -When `dryRun` is `true`, the stage should log what it would do without touching the filesystem: +When `dryRun` is `true`, the stage should log what it would do without touching the filesystem. The flag is available as `ctx.opts.dryRun` inside `execute()`: ```js export async function myStage(pages, site, destRoot, { dryRun = false } = {}) { diff --git a/docs/Documentation/Pipeline-Stages.md b/docs/Documentation/Pipeline-Stages.md index b15b80dd..1d367118 100644 --- a/docs/Documentation/Pipeline-Stages.md +++ b/docs/Documentation/Pipeline-Stages.md @@ -48,22 +48,24 @@ The pipeline passes two mutable data structures through every stage. ### Site object (`site`) -Built at the end of Phase 2 and passed unchanged to every subsequent phase. +Populated progressively by scheduler tasks on the main thread. Each task's `execute()` or `submit()` stores its output on `state.site`; downstream tasks read from it directly. -| Field | Type | Description | -|---|---|---| -| `config` | `object` | Parsed `_config.yml`, with CLI overrides (`--baseurl`, `--url`) already applied. | -| `navTree` | `object` | Top-level nav hierarchy produced by `nav.mjs`. | -| `seoSiteTitle` | `string` | Rendered site title from `config.title`. | -| `seoLogoUrl` | `string` | Absolute URL of the site logo. | -| `buildInfo` | `object` | `{ commit: string, commitDate: string }` from git. Both fall back to `"unknown"` outside a git repository. | -| `bookData` | `object\|null` | Parsed `_book.yml` with chapter selectors resolved to `Page` references. `null` when the file is absent. See [Book Configuration](Book-Configuration). | -| `data` | `object` | `_book.yml` loaded as `{ book: … }`, or `{}` when absent. | -| `markdown` | `MarkdownIt` | Shared markdown-it instance, built once during Phase 2 setup and reused by Phase 2's SEO pass and Phase 3's render pass. | +| Field | Type | Set by | Description | +|---|---|---|---| +| `config` | `object` | `discover` | Parsed `_config.yml`, with CLI overrides (`--baseurl`, `--url`) already applied. | +| `navTree` | `object` | `nav` | Top-level nav hierarchy produced by `nav.mjs`. | +| `seoSiteTitle` | `string` | `seo` | Rendered site title from `config.title`. | +| `seoLogoUrl` | `string` | `seo` | Absolute URL of the site logo. | +| `buildInfo` | `object` | `buildInfo` | `{ commit: string, commitDate: string }` from git. Both fall back to `"unknown"` outside a git repository. | +| `bookData` | `object\|null` | `loadData` | Parsed `_book.yml` with chapter selectors resolved to `Page` references. `null` when the file is absent. See [Book Configuration](Book-Configuration). | +| `data` | `object` | `loadData` | `_book.yml` loaded as `{ book: … }`, or `{}` when absent. | +| `markdown` | `MarkdownIt` | `markdownInit` | Shared markdown-it instance, built once and reused by the `seo` task and Phase 3's render pass. Workers build their own independent instances. | +| `highlighter` | `object` | `markdownInit` | Shiki highlighter instance. `render(code, lang)` returns highlighted HTML; `themeCss` is the generated `tb-highlight.css` string or `null`. | +| `linkTablesSerialized` | `object` | `markdownInit` | Serialized `[key, permalink]` pair arrays for structured-clone transfer to render workers. | ### Static files (`staticFiles[]`) -Also produced by Phase 1. Every file that is not a page --- images, fonts, prebuilt CSS/JS, and any `.md`/`.html` file without frontmatter --- becomes a static file object. This array does not grow after Phase 1. +Also produced by Phase 1. Every file that is not a page --- images, fonts, prebuilt CSS/JS, and any `.md`/`.html` file without frontmatter --- becomes a static file object. The `mermaid` task's `submit()` may append additional SVG descriptors after Phase 1 (because `mermaid` and `discover` run concurrently under the scheduler, freshly-emitted SVGs can miss Phase 1's traversal). | Field | Type | Description | |---|---|---| @@ -86,6 +88,7 @@ regenerateMermaid(srcRoot: string): Promise<{ regenerated: number, failed: number, setupSkipped?: true, + svgFiles: { srcPath: string, srcRel: string, destRel: string, size: number }[], }> ``` @@ -105,7 +108,7 @@ Two failure-mode distinctions: | Symbol | Signature | Description | |---|---|---| -| `regenerateMermaid` | `(srcRoot) → Promise<{ processed, regenerated, failed, setupSkipped? }>` | Main entry point. | +| `regenerateMermaid` | `(srcRoot) → Promise<{ processed, regenerated, failed, setupSkipped?, svgFiles }>` | Main entry point. `svgFiles` is an array of `{ srcPath, srcRel, destRel, size }` descriptors for every managed SVG, stat'd during the call. | --- @@ -134,7 +137,7 @@ Runs a single `fast-glob` call over `srcRoot` with the `exclude:` list read from ## Phase 2: COMPUTE -Phase 2 runs several modules in sequence (with `captureBuildInfo` running in parallel). Together they build the `site` object and add nav, SEO, and book-chapter data to each page. +Phase 2 runs several scheduler tasks on the main thread. Together they build the `site` object and add nav, SEO, and book-chapter data to each page. `captureBuildInfo` runs on a worker thread concurrently with the main spine; `deriveRedirects` and `deriveSitemap` also branch off after `discover` and run in parallel with the nav/seo chain. ### `nav.mjs` @@ -227,7 +230,7 @@ Captures git commit hash and date for the PDF title page. captureBuildInfo(): Promise<{ commit: string, commitDate: string }> ``` -Issues two parallel `git` shell-outs (`rev-parse --short HEAD` and `log -1 --format=%cs`). Falls back to `"unknown"` on any failure, so the build never aborts outside a git repository. The orchestrator launches this promise immediately after Phase 1 so the shell-outs overlap with the CPU-bound nav computation. +Issues two parallel `git` shell-outs (`rev-parse --short HEAD` and `log -1 --format=%cs`). Falls back to `"unknown"` on any failure, so the build never aborts outside a git repository. Runs on a worker thread as a seed task so the shell-outs overlap with the entire main-thread spine. **Reads:** local git repository state. **Writes:** nothing to pages (result returned directly to the orchestrator). @@ -265,7 +268,7 @@ Returns `{ book: }`, or `{}` when the file is absent. The orchestr ### Phase 2 setup --- shared markdown-it instance -Before Phase 2 completes, the orchestrator builds the shared markdown-it instance that both Phase 2's SEO pass and Phase 3's render pass reuse. Three functions from `render.mjs` are called in sequence: +The `markdownInit` scheduler task builds the shared markdown-it instance that both the `seo` task and each render worker reuse. Three functions from `render.mjs` are called in sequence: ```js const highlighter = await initHighlighter(); @@ -273,7 +276,9 @@ const linkTables = buildLinkTables(pages); const markdown = createMarkdownIt({ highlighter, linkTables, baseurl, staticFiles }); ``` -These functions are documented under [Phase 3](#phase-3-rendermjs) below since they are defined in `render.mjs`. They are called here during Phase 2 only to allow the SEO pass to share the same configured pipeline. +The main-thread instance is stored on `state.site.markdown`; the link tables are also serialized via `serializeLinkTables()` and stored on `state.site.linkTablesSerialized` so the render fan-out can ship them to workers (each worker reconstructs minimal `{ permalink }` stubs). + +These functions are documented under [Phase 3](#phase-3-rendermjs) below since they are defined in `render.mjs`. --- @@ -302,6 +307,7 @@ Renders each page's `rawContent` to `page.renderedContent` using the shared `sit | `buildLinkTables` | `(pages: Page[]) → { byPath, byUrl, byRedirect }` | Builds lookup tables keyed by `srcRel`, `permalink`, and `redirect_from` entries. Used by the relative-links plugin to resolve in-source `[X](Y.md)` links to absolute URLs at render time. | | `kramdownSlug` | `(text: string) → string` | Converts heading text to a kramdown-compatible anchor slug: lowercase, strip non-word characters, deduplicate with `-1`, `-2`, and so on. | | `rewriteAdmonitions` | `(src: string) → string` | Pre-render text pass: converts GFM `> [!NOTE]` / `[!IMPORTANT]` / `[!WARNING]` / `[!TIP]` / `[!CAUTION]` blocks to the `markdown-alert markdown-alert-` class structure. | +| `serializeLinkTables` | `(lt: { byPath, byUrl, byRedirect }) → { byPath, byUrl, byRedirect }` | Serializes the link-table Maps to `[key, permalink]` pair arrays for structured-clone transfer to render workers. Called by the `markdownInit` scheduler task. | --- @@ -326,7 +332,8 @@ Pre-computes the per-build static sidebar HTML once, then wraps each page's `ren | Symbol | Signature | Description | |---|---|---| -| `templatePhase` | `(pages, site) → Promise` | Main entry point. | +| `templatePhase` | `(pages, site, initData?) → Promise` | Main entry point. When `initData` is passed (on workers), skips the internal `buildInit()` call and uses the pre-rendered sidebar/header/svg-sprite HTML directly. | +| `buildInitFn` | `(site: object) → object` | Pre-renders the ~50 KB of sidebar, header, and svg-sprite HTML used by `templatePhase`. Called by the `buildInit` scheduler task on the main thread; the result is shipped to render workers as part of the shared payload. | | `navActivationCss` | `(page: Page) → string` | Generates the per-page `

render:0
[W]

render:1
[W]

render:N-1
[W]

buildInfo
[W]

scss
[W]

mermaid
[W]

config
[M]

discover
[M]

nav
[M]

deriveRedirects
[M]

deriveSitemap
[M]

markdownInit
[M]

buildInit
[M]

seo
[M]

loadData
[M]

resolveBook-
Chapters
[M]

dispatch
[M]

renderJoin
[M]

write
[M]

searchData
[M]

writeAux
[M]

writeOffline
[M]

writePdf
[M]

 \ No newline at end of file From 5a3a8187ef9f0ce44b059b8bb08c7dd2a60b3220 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Sat, 30 May 2026 20:24:16 +0200 Subject: [PATCH 09/72] Factor out prepDest and start it immediately. Factor out writePdf. --- builder/PLAN-5.md | 61 +++++++++++++----------- builder/PLAN-scheduler.md | 53 +++++++++++++------- builder/PLAN.md | 5 +- builder/pdf.mjs | 33 ++++++++----- builder/tbdocs.mjs | 43 ++++++++++++----- builder/write.mjs | 3 +- docs/Documentation/Builder.md | 8 ++-- docs/Documentation/Pipeline-Stages.md | 9 ++-- docs/assets/images/mmd/scheduler-dag.mmd | 21 ++++---- docs/assets/images/mmd/scheduler-dag.svg | 2 +- 10 files changed, 143 insertions(+), 95 deletions(-) diff --git a/builder/PLAN-5.md b/builder/PLAN-5.md index 68fe6284..51ed1f59 100644 --- a/builder/PLAN-5.md +++ b/builder/PLAN-5.md @@ -307,21 +307,22 @@ size. `fs.writeFile(path, buffer)` is the right primitive. ▼ [1] assertNoDestinationCollisions(pages, staticFiles) ← §6.4 (throws on any page.destPath == staticFile.destRel; - runs BEFORE prepareDestination so a collision aborts - without wiping the previous destination) + a collision aborts the build without any destructive I/O) │ ▼ - [2] prepareDestination(destRoot, dryRun) ← §5.1 - (delete + recreate, or skip if dry-run) + NOTE: prepareDestination (§5.1) is now a separate scheduler + task (`prepDest`) that runs as a seed -- it overlaps with + the entire main-thread spine and joins only at `write`. + writePhase assumes the destination is already prepared. │ ▼ - [3] In parallel (Promise.all): + [2] In parallel (Promise.all): writePages(pages, destRoot, limit) ← §5.2 copyTheme(builderAssetsRoot, destRoot, limit) ← §5.3 copyStaticFiles(staticFiles, destRoot, limit) ← §5.4 │ ▼ - [4] summarise(totals) ← §5.5 + [3] summarise(totals) ← §5.5 (file counts, byte counts, timing; one log line) ``` @@ -356,23 +357,22 @@ constrained systems; on the dev machine, no cap at all also works If profiling shows the cap is too low (write throughput < expected), bump it. The arg lives at the top of `write.mjs` as a constant. -### Why prepare-destination is sequential before the parallel writes +### Why prepare-destination is a separate seed task -Two reasons: +`prepareDestination` (§5.1) deletes and recreates the destination +directory. It now runs as the `prepDest` scheduler seed --- no +dependencies, so it overlaps with the entire main-thread spine and +worker seeds. The `write` task joins on `prepDest` alongside +`renderJoin`, `scss`, and `mermaid`, guaranteeing the destination is +clean before any file is written. -1. **Correctness.** The clean step deletes the existing tree. The - parallel writers would race the delete if it ran concurrently -- - a page write could land before the matching directory is removed, - then the delete would either fail (`ENOTEMPTY`) or destroy the - freshly-written file. -2. **Predictability.** A user-facing error from `prepareDestination` - (e.g. "destination is locked by another process") has a clean - single-source point. If it raced with writes, the error message - would be one of dozens of `EBUSY`s with no obvious culprit. +The same two invariants from the original design hold: -The prepare step is ~50 ms (recursive delete of a tree with ~1,080 -files + recreate). Sequencing it costs that 50 ms; parallelising -would save it but risk the failure modes above. +1. **Correctness.** The DAG edge `prepDest → write` ensures the + clean step finishes before the parallel writers start. +2. **Predictability.** A `prepareDestination` failure (e.g. locked + directory) surfaces as a single clear error before any write + begins. ### Phase 5 init order (one-time) @@ -886,12 +886,12 @@ function assertNoDestinationCollisions(pages, staticFiles) { } ``` -Called from `writePhase` **before** `prepareDestination`. The order -matters: a collision detected after the clean step would have -already wiped the previous `_site-new/` contents, leaving the user -no way to investigate the previous state. Running the assertion -first means a collision aborts the build without any destructive -I/O. Fast (set membership check over ~1,080 entries; <1 ms). +Called from `writePhase` before any file writes. Because +`prepareDestination` now runs as a separate seed task (`prepDest`), +it may have already cleaned the destination by the time the +collision check runs --- but the check still fires before `write` +begins writing any pages, so a collision still aborts cleanly. +Fast (set membership check over ~1,080 entries; <1 ms). --- @@ -1614,10 +1614,13 @@ async function main() { `writePhase(pages, staticFiles, { destRoot, dryRun })`: 1. Calls `assertNoDestinationCollisions(pages, staticFiles)` (§6.4). -2. Calls `prepareDestination(destRoot, dryRun)` (§5.1). -3. `Promise.all`-fans `writePages`, `copyTheme`, `copyStaticFiles` +2. `Promise.all`-fans `writePages`, `copyTheme`, `copyStaticFiles` (§5.2 / §5.3 / §5.4). -4. Returns `{ pages: {written, skipped}, theme: {copied}, staticFiles: {copied} }`. +3. Returns `{ pages: {written, skipped}, theme: {copied}, staticFiles: {copied} }`. + +Note: `prepareDestination` (§5.1) is no longer called inside +`writePhase`. The scheduler's `prepDest` seed task handles it +before `write` runs. The orchestrator's return value gains `destRoot` so Phase 6 / Phase 7 / Phase 8 know where to write or read. diff --git a/builder/PLAN-scheduler.md b/builder/PLAN-scheduler.md index da8dab02..81de7298 100644 --- a/builder/PLAN-scheduler.md +++ b/builder/PLAN-scheduler.md @@ -232,10 +232,11 @@ master directly -- no delta merge needed. `[W]` = pool worker; `[M]` = main thread (`runOnMain: true`). ``` -Seeds (workers, concurrent): +Seeds (concurrent): buildInfo [W] ──────────────────────────────────────────┐ scss [W] ──────────────────────────────────────────┤ mermaid [W] ──────────────────────────────────────────┤ + prepDest [M] ──────────────────────────────────────────┤ │ Main spine (sequential, on M): │ config │ @@ -259,7 +260,7 @@ Render fan-out (workers, concurrent): │ ▼ renderJoin [M] ◄── waits for all render:i │ -Write fence: │ scss [W], mermaid [W] join here too +Write fence: │ scss [W], mermaid [W], prepDest [M] join here too ▼ write [M] ◄── reads state.pages, state.staticFiles │ @@ -268,10 +269,16 @@ Write fence: │ scss [W], mermaid [W] join here too │ ▼ writeAux [M] ◄── derived redirects + sitemap join here - │ │ - ┌──────────┘ └──────────┐ - ▼ ▼ - writeOffline [M] writePdf [M] + │ + ▼ + writeOffline [M] + + (in parallel with write → ... → writeOffline:) + + renderJoin + mermaid + │ + ▼ + writePdf [M] │ │ └─────────────┬─────────────┘ ▼ @@ -280,7 +287,9 @@ Write fence: │ scss [W], mermaid [W] join here too Edges into `dispatch`: `buildInit`, `resolveBookChapters`, `buildInfo`. -Edges into `write`: `renderJoin`, `scss`, `mermaid`. +Edges into `write`: `renderJoin`, `scss`, `mermaid`, `prepDest`. +Edges out of `write`: `searchData`. +Edges into `writePdf`: `renderJoin`, `mermaid`. Edges into `writeAux`: `searchData`, `deriveRedirects`, `deriveSitemap`. Three structural wins over the serial baseline: @@ -639,7 +648,7 @@ Two non-obvious bits in `dispatch.submit`: 2. **Why `renderJoin` exists at all.** Each `render:i.submit()` already emits into the page-deltas merge, and could emit directly to `write`. But `write.expected` is declared statically with - `["renderJoin", "scss", "mermaid"]` -- mutating it from + `["renderJoin", "scss", "mermaid", "prepDest"]` -- mutating it from `dispatch.submit` to add the N dynamic render predecessors would be awkward. The barrier is the cleaner expression: register it once with the right count, let write keep its static `expected`. @@ -1140,8 +1149,15 @@ The DAG nodes downstream of `render` and `scss`/`mermaid`. All relative to their I/O. ```js +writePdf: { + expected: ["renderJoin", "mermaid"], + runOnMain: true, + // Sources CSS directly: tb-highlight.css from state.site.highlighter, + // print.css from staticFiles. No dependency on write or _site/. +}, + write: { - expected: ["renderJoin", "scss", "mermaid"], + expected: ["renderJoin", "scss", "mermaid", "prepDest"], runOnMain: true, async execute({ scss: { scssResult }, mermaid: { mermaidStats } }, ctx, state) { // render delta merges already happened in each render:i.submit(). @@ -1180,7 +1196,6 @@ writeAux: { }, submit(out, emit) { emit("writeOffline", out); - emit("writePdf", out); }, }, @@ -1194,7 +1209,8 @@ writeOffline: { submit() { /* terminal */ }, }, -// writePdf -- mirrors writeOffline. +// writePdf depends on renderJoin + mermaid (CSS sourced directly). +// It runs in parallel with write → searchData → writeAux → writeOffline. ``` **Restoring the derive-time exports.** `redirects.mjs` / @@ -1213,12 +1229,13 @@ derive can run at any point after discover. **Workerizing writeOffline or writePdf (measured, declined).** Both phases have non-trivial CPU sections: writeOffline rewrites URLs across all 856 HTML files; writePdf assembles `book.html` via -`assembleBook`. Profiling shows the cooperative async concurrency -already achieves zero-contention overlap — `writePdf` runs entirely -within `writeOffline`'s I/O await gaps, and the combined wall-clock -equals `max(writeOffline, writePdf)`. The structured-clone cost of -shipping `pages[]` across the worker boundary (~37–65 ms) would be -pure overhead. See §Phase 3-follow-up for the full measurement. +`assembleBook`. `writePdf` now depends only on `renderJoin` + `mermaid` +(CSS is sourced directly, not from `_site/`), so it runs in parallel +with the entire `write → searchData → writeAux → writeOffline` chain. +Cooperative async concurrency on the main thread interleaves their I/O +gaps. The structured-clone cost of shipping `pages[]` across the worker +boundary (~37–65 ms) would be pure overhead. See §Phase 3-follow-up +for the full measurement. ## Timing / profiling @@ -1383,7 +1400,7 @@ yet, so the build must look identical. Wire `runBuild()` to construct the pool, instantiate the scheduler, and call `scheduler.start(ctx)`. Port: -- **Worker seeds:** `config`, `buildInfo`, `scss`, `mermaid`. +- **Seeds:** `config`, `buildInfo`, `scss`, `mermaid`, `prepDest`. - **Main-thread spine:** `discover`, `nav`, `markdownInit`, `seo`, `loadData`, `resolveBookChapters`, `buildInit`, `deriveRedirects`, `deriveSitemap`. diff --git a/builder/PLAN.md b/builder/PLAN.md index 86dfc62a..4215f6f9 100644 --- a/builder/PLAN.md +++ b/builder/PLAN.md @@ -66,8 +66,9 @@ A **task-graph scheduler** for the build pipeline is designed in [PLAN-scheduler.md](PLAN-scheduler.md) and has been implemented (Phases 0--4). It covers a thin in-tree scheduler + `WorkerPool` over `node:worker_threads`, moves CPU-bound seed tasks (`scss`, `mermaid`, -`buildInfo`) onto workers, and fans out `renderPhase` + `templatePhase` -across CPUs via SAB broadcast. +`buildInfo`) onto workers, runs `prepDest` (destination clean/recreate) +as a main-thread seed in parallel with the spine, and fans out +`renderPhase` + `templatePhase` across CPUs via SAB broadcast. Open follow-ups (deferred enhancements, divergence investigations) live in [FUTURE-WORK.md](FUTURE-WORK.md). diff --git a/builder/pdf.mjs b/builder/pdf.mjs index c832b6f8..6178fa81 100644 --- a/builder/pdf.mjs +++ b/builder/pdf.mjs @@ -18,7 +18,7 @@ // §F Missing-image reporting (port of pdfify.rb's strict mode) import { promises as fs } from "node:fs"; -import { existsSync } from "node:fs"; + import path from "node:path"; import { assembleBook } from "./book.mjs"; @@ -39,7 +39,7 @@ const LIMIT = WRITE_LIMIT; // §A Top-level orchestration // --------------------------------------------------------------------------- -export async function writePdf(pages, staticFiles, site, destRoot, { tolerateMissingImages = false } = {}) { +export async function writePdf(pages, staticFiles, site, destRoot, { tolerateMissingImages = false, highlightCss = null } = {}) { if (!destRoot) { throw new Error("writePdf requires a destRoot"); } @@ -59,7 +59,7 @@ export async function writePdf(pages, staticFiles, site, destRoot, { tolerateMis await Promise.all([ writePdfBook(bookHtml, pdfRoot, counters), - copyPdfCss(destRoot, pdfRoot, counters), + copyPdfCss(staticByDestRel, highlightCss, pdfRoot, counters), copyPdfImages(imagePaths, staticByDestRel, pdfRoot, counters, missingPaths), ]); @@ -156,20 +156,27 @@ async function writePdfBook(bookHtml, pdfRoot, counters) { return counters.bookBytes; } -// PLAN-8 §5.6: copy `print.css` and `rouge.css` from /assets/ -// css/ to /assets/css/. Missing files become warnings (not -// fatal); the strict-mode throw only applies to image references. -async function copyPdfCss(destRoot, pdfRoot, counters) { +// Copy the two required CSS files into /assets/css/. +// tb-highlight.css is written from the in-memory highlightCss string +// (generated by highlight-theme.mjs during markdownInit); print.css is +// copied from its source path via the staticFiles inventory. Neither +// requires _site/ to exist, so writePdf can run before the write task. +async function copyPdfCss(staticByDestRel, highlightCss, pdfRoot, counters) { const warnings = []; await runLimited(REQUIRED_CSS, LIMIT, async (rel) => { - const src = path.join(destRoot, rel); const dest = path.join(pdfRoot, rel); - if (!existsSync(src)) { - warnings.push(`missing required asset ${rel}; pagedjs render may break`); - return; - } await mkdirRec(path.dirname(dest)); - await safeWrite(dest, () => fs.copyFile(src, dest)); + const key = rel.replaceAll("\\", "/"); + if (key === "assets/css/tb-highlight.css" && highlightCss) { + await safeWrite(dest, () => fs.writeFile(dest, highlightCss, "utf8")); + } else { + const sf = staticByDestRel.get(key); + if (!sf) { + warnings.push(`missing required asset ${rel}; pagedjs render may break`); + return; + } + await safeWrite(dest, () => fs.copyFile(sf.srcPath, dest)); + } counters.css++; }); for (const w of warnings) console.warn(`pdf: ${w}`); diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 8c31938b..3849c87f 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -33,7 +33,7 @@ import { buildLinkTables, serializeLinkTables, } from "./render.mjs"; import { buildInitFn } from "./template.mjs"; -import { writePhase } from "./write.mjs"; +import { writePhase, prepareDestination } from "./write.mjs"; import { writeRedirects, deriveRedirectStubs } from "./redirects.mjs"; import { writeSitemap, deriveSitemapUrls } from "./sitemap.mjs"; import { writeSearchData } from "./search.mjs"; @@ -115,11 +115,12 @@ export function makeTimer() { // ── Task graph ──────────────────────────────────────────────────────────────── // -// Seeds (config, buildInfo, scss, mermaid), the main-thread spine (discover → +// Seeds (config, buildInfo, scss, mermaid, prepDest), the main-thread spine (discover → // nav → markdownInit / buildInit → seo / loadData → resolveBookChapters + // deriveRedirects / deriveSitemap), the render fan-out (dispatch → -// render:0..N → renderJoin), and all write/post-write tasks (write → -// searchData → writeAux → writeOffline / writePdf) are scheduler tasks. +// render:0..N → renderJoin), and write/post-write tasks (write → +// searchData → writeAux → writeOffline; renderJoin + mermaid → writePdf) +// are scheduler tasks. // runBuild() constructs the pool + scheduler, awaits start(), logs the // summary, and returns. @@ -171,10 +172,23 @@ const TASKS = { for (const f of out.mermaidStats.svgFiles ?? []) { if (!known.has(f.srcRel)) state.staticFiles.push(f); } - emit("write", out); + emit("write", out); + emit("writePdf", out); }, }, + // Clean and recreate _site/. No dependencies -- overlaps with the entire + // main-thread spine and worker seeds, joined only by `write`. + prepDest: { + expected: [], + runOnMain: true, + async execute(_, ctx) { + await prepareDestination(ctx.destRoot, ctx.opts.dryRun); + return {}; + }, + submit(_, emit) { emit("write", {}); }, + }, + // ── Main-thread spine ───────────────────────────────────────────────────── discover: { @@ -336,7 +350,10 @@ const TASKS = { expected: Array.from({ length: N }, (_, i) => `render:${i}`), runOnMain: true, execute() { return {}; }, - submit(_, emit) { emit("write", {}); }, + submit(_, emit) { + emit("write", {}); + emit("writePdf", {}); + }, }); for (let i = 0; i < N; i++) { @@ -366,9 +383,10 @@ const TASKS = { // Materialise pages + static files + generated CSS to _site/. // Waits for renderJoin (pages rendered + templated), scss (generated CSS), - // and mermaid (SVG descriptors appended to state.staticFiles by mermaid.submit). + // mermaid (SVG descriptors appended to state.staticFiles by mermaid.submit), + // and prepDest (_site/ cleaned and recreated). write: { - expected: ["renderJoin", "scss", "mermaid"], + expected: ["renderJoin", "scss", "mermaid", "prepDest"], runOnMain: true, async execute({ scss: { scssResult }, mermaid: { mermaidStats } }, ctx, state) { void mermaidStats; // dependency signal only; append already happened in mermaid.submit @@ -417,7 +435,6 @@ const TASKS = { }, submit(out, emit) { emit("writeOffline", out); - emit("writePdf", out); }, }, @@ -438,15 +455,19 @@ const TASKS = { submit() { /* terminal */ }, }, - // Produce _site-pdf/. Runs in parallel with writeOffline. + // Produce _site-pdf/. Depends on renderJoin (pages have renderedContent) + // and mermaid (SVG descriptors in staticFiles). Sources CSS directly: + // tb-highlight.css from state.site.highlighter, print.css from staticFiles. + // Runs in parallel with write → searchData → writeAux → writeOffline. writePdf: { - expected: ["writeAux"], + expected: ["renderJoin", "mermaid"], runOnMain: true, async execute(_, ctx, state) { const skipPdf = ctx.opts.skipPdf ?? (state.site.config.also_build_pdf === false); if (ctx.opts.dryRun || skipPdf) return null; return writePdf(state.pages, state.staticFiles, state.site, ctx.destRoot, { tolerateMissingImages: ctx.opts.tolerateMissingImages, + highlightCss: state.site.highlighter?.themeCss, }); }, submit() { /* terminal */ }, diff --git a/builder/write.mjs b/builder/write.mjs index 2c5c2f6d..579bf689 100644 --- a/builder/write.mjs +++ b/builder/write.mjs @@ -44,7 +44,6 @@ export async function writePhase(pages, staticFiles, { destRoot, dryRun = false, mkdirInflight.clear(); assertNoDestinationCollisions(pages, staticFiles); - await prepareDestination(destRoot, dryRun); if (dryRun) { const pagesToWrite = pages.filter(p => p.html !== undefined).length; @@ -96,7 +95,7 @@ async function writeGeneratedAssets(assets, destRoot, limit, baseurl) { // ---------- §5.1 prepareDestination ------------------------------------- -async function prepareDestination(destRoot, dryRun) { +export async function prepareDestination(destRoot, dryRun) { if (dryRun) { console.log(`[dry-run] would clean ${destRoot}`); return; diff --git a/docs/Documentation/Builder.md b/docs/Documentation/Builder.md index 1ae34bac..ba962881 100644 --- a/docs/Documentation/Builder.md +++ b/docs/Documentation/Builder.md @@ -92,9 +92,9 @@ The build pipeline is driven by a task-graph **scheduler** that models the pipel Three structural wins over the earlier serial baseline: -1. **Seed tasks overlap with the main spine.** `scss` (~700 ms), `mermaid`, and `buildInfo` run on workers concurrently with the main-thread spine (discover → nav → markdownInit → seo → loadData → resolveBookChapters + buildInit). The spine takes ~250 ms total, so ~250 ms of `scss` hides behind it. +1. **Seed tasks overlap with the main spine.** `scss` (~700 ms), `mermaid`, `buildInfo`, and `prepDest` (destination clean + recreate) run concurrently with the main-thread spine (discover → nav → markdownInit → seo → loadData → resolveBookChapters + buildInit). The spine takes ~250 ms total, so ~250 ms of `scss` hides behind it. 2. **Render + template fans out across CPUs.** The single-threaded render + template work (~3.5 s combined) splits into N page chunks, each dispatched to a worker that runs both `renderPhase` and `templatePhase` on its slice. On a 4-core machine this compresses to ~875 ms; on 8 cores, ~440 ms. -3. **`writeOffline` and `writePdf` overlap.** Both are `runOnMain` but their I/O-dominated `await` gaps interleave via cooperative async concurrency. The combined wall-clock equals `max(writeOffline, writePdf)` rather than their sum. +3. **`writePdf` overlaps with the entire write chain.** `writePdf` depends on `renderJoin` + `mermaid` --- it sources CSS directly (highlight CSS from `state.site.highlighter`, `print.css` from the static-file inventory) and never reads from `_site/`. It starts as soon as pages are rendered and runs in parallel with `write → searchData → writeAux → writeOffline`. All five are `runOnMain` but their I/O-dominated `await` gaps interleave via cooperative async concurrency. ### Shared state and page deltas @@ -150,9 +150,9 @@ Each subsection covers the design rationale and implementation details for one m ### [tbdocs.mjs](https://github.com/twinbasic/documentation/blob/main/builder/tbdocs.mjs) --- entry point and orchestrator -`runBuild()` constructs a `WorkerPool` of `os.availableParallelism()` workers, instantiates a `Scheduler` with the static `TASKS` graph, and calls `scheduler.start(ctx)`. The scheduler dispatches all seed tasks immediately (`config`, `buildInfo`, `scss`, `mermaid`), chains the main-thread spine as each task's `submit()` emits to the next, dynamically registers the `render:0..N` fan-out from `dispatch.submit()`, and resolves when the two terminal tasks (`writeOffline`, `writePdf`) complete. After `start()` resolves, `runBuild()` reads results from the scheduler's `results` map, logs the build summary, and destroys the pool. +`runBuild()` constructs a `WorkerPool` of `os.availableParallelism()` workers, instantiates a `Scheduler` with the static `TASKS` graph, and calls `scheduler.start(ctx)`. The scheduler dispatches all seed tasks immediately (`config`, `buildInfo`, `scss`, `mermaid`, `prepDest`), chains the main-thread spine as each task's `submit()` emits to the next, dynamically registers the `render:0..N` fan-out from `dispatch.submit()`, and resolves when the two terminal tasks (`writeOffline`, `writePdf`) complete. After `start()` resolves, `runBuild()` reads results from the scheduler's `results` map, logs the build summary, and destroys the pool. -The `TASKS` object defines every task in the DAG as a plain object with `expected` (predecessor IDs), `execute()` (task body), and `submit()` (synchronous output router). Tasks without `runOnMain: true` are dispatched to the worker pool by handler name; the four worker handlers (`scss`, `mermaid`, `buildInfo`, `render`) live in `cpu-worker.mjs`. +The `TASKS` object defines every task in the DAG as a plain object with `expected` (predecessor IDs), `execute()` (task body), and `submit()` (synchronous output router). Most seed tasks without `runOnMain: true` are dispatched to the worker pool by handler name; the four worker handlers (`scss`, `mermaid`, `buildInfo`, `render`) live in `cpu-worker.mjs`. `prepDest` is the exception --- a main-thread seed that cleans and recreates the destination directory, overlapping with the spine and joining only at `write`. The shared markdown-it instance is built once during the `markdownInit` task via `initHighlighter` + `createMarkdownIt` and stored on `state.site.markdown` so the `seo` task and each render worker use the same configured pipeline --- titles run through the same dash, quote, and footnote-stripping rules as page body text. Workers build their own independent markdown-it + Shiki instances from the serialized link tables and shared payload. diff --git a/docs/Documentation/Pipeline-Stages.md b/docs/Documentation/Pipeline-Stages.md index 1d367118..cbcd5b10 100644 --- a/docs/Documentation/Pipeline-Stages.md +++ b/docs/Documentation/Pipeline-Stages.md @@ -378,7 +378,7 @@ writePhase( ): Promise<{ pages: { written, skipped }, theme: { copied }, staticFiles: { copied } }> ``` -Clears then recreates `destRoot`, then runs three operations in parallel: writes each `page.html` to its `destPath`; copies the vendored just-the-docs JS from `builder/vendor/just-the-docs/assets/` to `/assets/`; and copies every `staticFiles[]` entry (which includes the project-owned theme files now living under `docs/assets/`). A CSS `url()` baseurl rewrite runs over both copy paths and over generated CSS assets so root-absolute `url("/path")` references resolve correctly under non-empty baseurls. After the parallel batch, `writeGeneratedAssets` writes `generatedAssets[]` (the SCSS-compiled CSS and the highlight theme CSS) sequentially so they win any rel-path collision. Skips pages where `page.html` is `undefined`. +Assumes the destination directory has already been prepared (cleaned and recreated) by the `prepDest` scheduler task. Runs three operations in parallel: writes each `page.html` to its `destPath`; copies the vendored just-the-docs JS from `builder/vendor/just-the-docs/assets/` to `/assets/`; and copies every `staticFiles[]` entry (which includes the project-owned theme files now living under `docs/assets/`). A CSS `url()` baseurl rewrite runs over both copy paths and over generated CSS assets so root-absolute `url("/path")` references resolve correctly under non-empty baseurls. After the parallel batch, `writeGeneratedAssets` writes `generatedAssets[]` (the SCSS-compiled CSS and the highlight theme CSS) sequentially so they win any rel-path collision. Skips pages where `page.html` is `undefined`. **Reads:** `page.html`, `page.destPath`, `staticFile.srcPath`, `staticFile.destRel`. **Writes:** `/**` (the online tree). @@ -387,7 +387,8 @@ Clears then recreates `destRoot`, then runs three operations in parallel: writes | Symbol | Signature | Description | |---|---|---| -| `writePhase` | `(pages, staticFiles, opts) → Promise` | Main entry point. | +| `writePhase` | `(pages, staticFiles, opts) → Promise` | Main entry point. Assumes destination is already prepared. | +| `prepareDestination` | `(destRoot: string, dryRun: boolean) → Promise` | Deletes and recreates `destRoot`. Called by the `prepDest` scheduler task. | | `WRITE_LIMIT` | `64` | Concurrency ceiling for `runLimited`. Phases 6, 7, and 8 pass this value to their own `runLimited` calls for consistent I/O throttling. | | `isUnderProject` | `(destRoot: string) → boolean` | Returns `true` only when `destRoot` is a descendant of the project root. Used by Phases 7 and 8 as a guard against destructive `--dest` values. | | `mkdirRec` | `(dir: string) → Promise` | Recursive `mkdir` with an in-flight deduplication cache. Shared by Phases 6, 7, and 8. | @@ -516,11 +517,11 @@ writePdf( staticFiles: StaticFile[], site: object, destRoot: string, - { tolerateMissingImages?: boolean } + { tolerateMissingImages?: boolean, highlightCss?: string } ): Promise<{ bookBytes, css, images, missing }> ``` -Calls `book.mjs`'s `assembleBook(site, pages)` to produce `book.html`, copies `print.css` and `tb-highlight.css`, and collects every image referenced in `book.html`. Reports missing images as build errors by default; `--tolerate-missing-images` downgrades them to warnings. +Calls `book.mjs`'s `assembleBook(site, pages)` to produce `book.html`, writes `tb-highlight.css` from the `highlightCss` string (generated by `highlight-theme.mjs` during `markdownInit`), copies `print.css` from its source path via the `staticFiles` inventory, and collects every image referenced in `book.html`. Does not read from `_site/`. Reports missing images as build errors by default; `--tolerate-missing-images` downgrades them to warnings. **Reads:** `site.bookData` (with chapter selectors resolved by Phase 2's `resolveBookChapters`), all pages' `page.html`, `staticFiles`. **Writes:** `-pdf/book.html`, `-pdf/*.css`, image copies in `-pdf/`. diff --git a/docs/assets/images/mmd/scheduler-dag.mmd b/docs/assets/images/mmd/scheduler-dag.mmd index 73646a75..5cc2f533 100644 --- a/docs/assets/images/mmd/scheduler-dag.mmd +++ b/docs/assets/images/mmd/scheduler-dag.mmd @@ -1,11 +1,12 @@ %%{init: {"themeVariables": {"clusterBkg": "transparent", "clusterBorder": "#7a8090", "lineColor": "#7a8090"}}}%% flowchart TB - subgraph seeds [" "] - direction LR + %%subgraph seeds [" "] + %% direction LR BI["buildInfo
[W]"] SC["scss
[W]"] MM["mermaid
[W]"] - end + PD["prepDest
[M]"] + %%end CF["config
[M]"] --> DI["discover
[M]"] DI --> NV["nav
[M]"] @@ -29,17 +30,14 @@ flowchart TB RN["render:N-1
[W]"] end - DP --> R0 - DP --> R1 - DP --> RN + DP --> fanout - R0 --> RJ["renderJoin
[M]"] - R1 --> RJ - RN --> RJ + fanout --> RJ["renderJoin
[M]"] RJ --> WR["write
[M]"] SC --> WR MM --> WR + PD --> WR WR --> SD["searchData
[M]"] SD --> WA["writeAux
[M]"] @@ -47,9 +45,10 @@ flowchart TB DS --> WA WA --> WO["writeOffline
[M]"] - WA --> WP["writePdf
[M]"] + RJ --> WP["writePdf
[M]"] + MM --> WP classDef worker fill:#e8f0fe,stroke:#4285f4,color:#1a1a2e classDef main fill:#fef7e0,stroke:#f9ab00,color:#1a1a2e class BI,SC,MM,R0,R1,RN worker - class CF,DI,NV,MI,BU,SE,LD,RC,DP,RJ,WR,SD,WA,DR,DS,WO,WP main + class CF,DI,NV,MI,BU,SE,LD,RC,DP,RJ,WR,SD,WA,DR,DS,WO,WP,PD main diff --git a/docs/assets/images/mmd/scheduler-dag.svg b/docs/assets/images/mmd/scheduler-dag.svg index 7df7fa7f..a0d5322a 100644 --- a/docs/assets/images/mmd/scheduler-dag.svg +++ b/docs/assets/images/mmd/scheduler-dag.svg @@ -1 +1 @@ -

render:0
[W]

render:1
[W]

render:N-1
[W]

buildInfo
[W]

scss
[W]

mermaid
[W]

config
[M]

discover
[M]

nav
[M]

deriveRedirects
[M]

deriveSitemap
[M]

markdownInit
[M]

buildInit
[M]

seo
[M]

loadData
[M]

resolveBook-
Chapters
[M]

dispatch
[M]

renderJoin
[M]

write
[M]

searchData
[M]

writeAux
[M]

writeOffline
[M]

writePdf
[M]

 \ No newline at end of file +

render:0
[W]

render:1
[W]

render:N-1
[W]

buildInfo
[W]

scss
[W]

mermaid
[W]

prepDest
[M]

config
[M]

discover
[M]

nav
[M]

deriveRedirects
[M]

deriveSitemap
[M]

markdownInit
[M]

buildInit
[M]

seo
[M]

loadData
[M]

resolveBook-
Chapters
[M]

dispatch
[M]

renderJoin
[M]

write
[M]

searchData
[M]

writeAux
[M]

writeOffline
[M]

writePdf
[M]

 \ No newline at end of file From 6968a2a3d067a2085b878e22dc5a43fff7b4d353 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Sat, 30 May 2026 20:31:22 +0200 Subject: [PATCH 10/72] SearchData really requires renderJoin and prepDest, not Write. --- builder/PLAN-scheduler.md | 27 ++++++++++++------- builder/tbdocs.mjs | 33 ++++++++++++++---------- docs/Documentation/Builder.md | 2 +- docs/assets/images/mmd/scheduler-dag.mmd | 6 +++-- docs/assets/images/mmd/scheduler-dag.svg | 2 +- 5 files changed, 42 insertions(+), 28 deletions(-) diff --git a/builder/PLAN-scheduler.md b/builder/PLAN-scheduler.md index 81de7298..25eb0f2b 100644 --- a/builder/PLAN-scheduler.md +++ b/builder/PLAN-scheduler.md @@ -264,11 +264,16 @@ Write fence: │ scss [W], mermaid [W], prepDest [M] join here ▼ write [M] ◄── reads state.pages, state.staticFiles │ - ▼ - searchData [M] - │ - ▼ - writeAux [M] ◄── derived redirects + sitemap join here + (in parallel with write:) │ + │ + renderJoin + prepDest │ + │ │ + ▼ │ + searchData [M] │ + │ │ + └──────────────────────────────────┤ + │ + writeAux [M] ◄── derived redirects + sitemap join here too │ ▼ writeOffline [M] @@ -288,9 +293,9 @@ Write fence: │ scss [W], mermaid [W], prepDest [M] join here Edges into `dispatch`: `buildInit`, `resolveBookChapters`, `buildInfo`. Edges into `write`: `renderJoin`, `scss`, `mermaid`, `prepDest`. -Edges out of `write`: `searchData`. +Edges into `searchData`: `renderJoin`, `prepDest`. Edges into `writePdf`: `renderJoin`, `mermaid`. -Edges into `writeAux`: `searchData`, `deriveRedirects`, `deriveSitemap`. +Edges into `writeAux`: `write`, `searchData`, `deriveRedirects`, `deriveSitemap`. Three structural wins over the serial baseline: @@ -1173,12 +1178,14 @@ write: { dryRun: ctx.opts.dryRun, }); }, - submit(out, emit) { emit("searchData", out); }, + submit(out, emit) { emit("writeAux", out); }, }, searchData: { - expected: ["write"], + expected: ["renderJoin", "prepDest"], runOnMain: true, + // Reads only in-memory renderedContent; writes search-data.json + // into _site/ (needs prepDest). Runs in parallel with write. async execute(_, ctx, state) { return writeSearchData(state.pages, state.site, ctx.destRoot); }, @@ -1186,7 +1193,7 @@ searchData: { }, writeAux: { - expected: ["searchData", "deriveRedirects", "deriveSitemap"], + expected: ["write", "searchData", "deriveRedirects", "deriveSitemap"], runOnMain: true, async execute({ deriveRedirects, deriveSitemap }, ctx, state) { await Promise.all([ diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 3849c87f..8c91ec46 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -118,9 +118,9 @@ export function makeTimer() { // Seeds (config, buildInfo, scss, mermaid, prepDest), the main-thread spine (discover → // nav → markdownInit / buildInit → seo / loadData → resolveBookChapters + // deriveRedirects / deriveSitemap), the render fan-out (dispatch → -// render:0..N → renderJoin), and write/post-write tasks (write → -// searchData → writeAux → writeOffline; renderJoin + mermaid → writePdf) -// are scheduler tasks. +// render:0..N → renderJoin), and write/post-write tasks +// (renderJoin + prepDest → searchData; write + searchData → writeAux → +// writeOffline; renderJoin + mermaid → writePdf) are scheduler tasks. // runBuild() constructs the pool + scheduler, awaits start(), logs the // summary, and returns. @@ -178,7 +178,7 @@ const TASKS = { }, // Clean and recreate _site/. No dependencies -- overlaps with the entire - // main-thread spine and worker seeds, joined only by `write`. + // main-thread spine and worker seeds. Joined by write and searchData. prepDest: { expected: [], runOnMain: true, @@ -186,7 +186,10 @@ const TASKS = { await prepareDestination(ctx.destRoot, ctx.opts.dryRun); return {}; }, - submit(_, emit) { emit("write", {}); }, + submit(_, emit) { + emit("write", {}); + emit("searchData", {}); + }, }, // ── Main-thread spine ───────────────────────────────────────────────────── @@ -351,8 +354,9 @@ const TASKS = { runOnMain: true, execute() { return {}; }, submit(_, emit) { - emit("write", {}); - emit("writePdf", {}); + emit("write", {}); + emit("writePdf", {}); + emit("searchData", {}); }, }); @@ -404,13 +408,14 @@ const TASKS = { baseurl: String(state.site.config.baseurl || ""), }); }, - submit(out, emit) { emit("searchData", out); }, + submit(out, emit) { emit("writeAux", out); }, }, - // Write search-data.json. Skipped on dry-run; result passes through to - // writeAux so its search.json field reaches writeOffline. + // Write search-data.json. Depends on renderJoin (pages have + // renderedContent) and prepDest (_site/ exists). Result passes + // through to writeAux so its search.json field reaches writeOffline. searchData: { - expected: ["write"], + expected: ["renderJoin", "prepDest"], runOnMain: true, async execute(_, ctx, state) { if (ctx.opts.dryRun) return { entries: 0, json: "" }; @@ -419,11 +424,11 @@ const TASKS = { submit(out, emit) { emit("writeAux", out); }, }, - // Write redirect stubs + sitemap/robots. Waits for searchData (sequencing), - // deriveRedirects, and deriveSitemap (pre-computed stubs/urls). + // Write redirect stubs + sitemap/robots. Waits for write (pages on disk), + // searchData, deriveRedirects, and deriveSitemap. // Passes searchStats through to writeOffline (for search-data.js). writeAux: { - expected: ["searchData", "deriveRedirects", "deriveSitemap"], + expected: ["write", "searchData", "deriveRedirects", "deriveSitemap"], runOnMain: true, async execute({ searchData: searchStats, deriveRedirects: { stubs }, deriveSitemap: { urls } }, ctx, state) { if (ctx.opts.dryRun) return { redirectStats: null, sitemapStats: null, searchStats }; diff --git a/docs/Documentation/Builder.md b/docs/Documentation/Builder.md index ba962881..2f32d083 100644 --- a/docs/Documentation/Builder.md +++ b/docs/Documentation/Builder.md @@ -94,7 +94,7 @@ Three structural wins over the earlier serial baseline: 1. **Seed tasks overlap with the main spine.** `scss` (~700 ms), `mermaid`, `buildInfo`, and `prepDest` (destination clean + recreate) run concurrently with the main-thread spine (discover → nav → markdownInit → seo → loadData → resolveBookChapters + buildInit). The spine takes ~250 ms total, so ~250 ms of `scss` hides behind it. 2. **Render + template fans out across CPUs.** The single-threaded render + template work (~3.5 s combined) splits into N page chunks, each dispatched to a worker that runs both `renderPhase` and `templatePhase` on its slice. On a 4-core machine this compresses to ~875 ms; on 8 cores, ~440 ms. -3. **`writePdf` overlaps with the entire write chain.** `writePdf` depends on `renderJoin` + `mermaid` --- it sources CSS directly (highlight CSS from `state.site.highlighter`, `print.css` from the static-file inventory) and never reads from `_site/`. It starts as soon as pages are rendered and runs in parallel with `write → searchData → writeAux → writeOffline`. All five are `runOnMain` but their I/O-dominated `await` gaps interleave via cooperative async concurrency. +3. **`writePdf` and `searchData` overlap with the write chain.** Both depend on `renderJoin` rather than `write`, so they start as soon as pages are rendered. `writePdf` (+ `mermaid`) sources CSS directly from in-memory state and the static-file inventory --- it never reads from `_site/`. `searchData` (+ `prepDest`) reads only in-memory `renderedContent`. Both run in parallel with `write`; `writeAux` joins `write` + `searchData` before starting. All are `runOnMain` but their I/O-dominated `await` gaps interleave via cooperative async concurrency. ### Shared state and page deltas diff --git a/docs/assets/images/mmd/scheduler-dag.mmd b/docs/assets/images/mmd/scheduler-dag.mmd index 5cc2f533..48a08563 100644 --- a/docs/assets/images/mmd/scheduler-dag.mmd +++ b/docs/assets/images/mmd/scheduler-dag.mmd @@ -39,8 +39,10 @@ flowchart TB MM --> WR PD --> WR - WR --> SD["searchData
[M]"] - SD --> WA["writeAux
[M]"] + RJ --> SD["searchData
[M]"] + PD --> SD + WR --> WA["writeAux
[M]"] + SD --> WA DR --> WA DS --> WA diff --git a/docs/assets/images/mmd/scheduler-dag.svg b/docs/assets/images/mmd/scheduler-dag.svg index a0d5322a..7065be09 100644 --- a/docs/assets/images/mmd/scheduler-dag.svg +++ b/docs/assets/images/mmd/scheduler-dag.svg @@ -1 +1 @@ -

render:0
[W]

render:1
[W]

render:N-1
[W]

buildInfo
[W]

scss
[W]

mermaid
[W]

prepDest
[M]

config
[M]

discover
[M]

nav
[M]

deriveRedirects
[M]

deriveSitemap
[M]

markdownInit
[M]

buildInit
[M]

seo
[M]

loadData
[M]

resolveBook-
Chapters
[M]

dispatch
[M]

renderJoin
[M]

write
[M]

searchData
[M]

writeAux
[M]

writeOffline
[M]

writePdf
[M]

 \ No newline at end of file +

render:0
[W]

render:1
[W]

render:N-1
[W]

buildInfo
[W]

scss
[W]

mermaid
[W]

prepDest
[M]

config
[M]

discover
[M]

nav
[M]

deriveRedirects
[M]

deriveSitemap
[M]

markdownInit
[M]

buildInit
[M]

seo
[M]

loadData
[M]

resolveBook-
Chapters
[M]

dispatch
[M]

renderJoin
[M]

write
[M]

searchData
[M]

writeAux
[M]

writeOffline
[M]

writePdf
[M]

 \ No newline at end of file From 2b55c5c8c30f181eef826ebd47cfb3669d98edc1 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Sat, 30 May 2026 21:22:02 +0200 Subject: [PATCH 11/72] Add the plan to move the offline rewriter to workers. --- builder/PLAN-scheduler-offline.md | 320 ++++++++++++++++++++++++++++++ 1 file changed, 320 insertions(+) create mode 100644 builder/PLAN-scheduler-offline.md diff --git a/builder/PLAN-scheduler-offline.md b/builder/PLAN-scheduler-offline.md new file mode 100644 index 00000000..48cd6df2 --- /dev/null +++ b/builder/PLAN-scheduler-offline.md @@ -0,0 +1,320 @@ +# Move offline HTML rewrite into render workers + +Companion to [PLAN-scheduler.md](PLAN-scheduler.md). Covers moving +the CPU-bound per-page offline URL rewrite from `writeOffline` (main +thread, ~700 ms) into the render worker fan-out, so it parallelises +across all CPUs and `writeOffline` becomes I/O-only (~200 ms). + +## Motivation + +`writeOffline` is the longest single task after `write`. Profiling +shows the time is dominated by `deriveOfflinePage` — a pure-compute +function that strips SEO metadata, rewrites every URL from absolute +to page-relative, and injects the offline search setup script. The +function reads only `page.html`, `page.destPath`, a `sitePaths` Set, +resolution caches, and `baseurl`. All of these can be made available +to workers before dispatch, without reading from `_site/`. + +The `sitePaths` set's current dependency on `_site/assets/` (theme +file enumeration in `buildSitePaths`) is artificial — the files come +from `builder/vendor/just-the-docs/assets/` (known statically) plus +two generated CSS paths that are deterministic. + +## Model assignments + +| Phase | Model | Rationale | +|-------|--------|-----------| +| I | Sonnet | Mechanical move-and-re-export. | +| II | Sonnet | Straightforward wiring with clear instructions. | +| III | Opus | Most judgement: SAB state reconstruction, nav-cache pre-pass, render handler integration. | +| IV | Sonnet | Small, well-specified signature changes. | +| V | Sonnet | Documentation updates. | + +## Phase I: Extract `offline-rewrite.mjs` + +Create `builder/offline-rewrite.mjs` with all pure-compute rewrite +functions extracted from `offline.mjs`. This isolates worker-safe +code from the I/O + acorn-dependent code that stays on main. + +### Exports from `offline-rewrite.mjs` + +- `deriveOfflinePage`, `deriveOfflinePageCached`, `sliceNavBlock` +- `deriveOfflineCss` (used by `copyOfflineThemeAssets` on main) +- `deriveOfflineRedirect` (used by `writeOfflineRedirects` on main) +- `normalizeBaseurl`, `posixDirname`, `fileDirSegsFromRel` +- `offlineExcluded`, `fnmatchPathname` +- All internal helpers: `stripSeo`, `rewriteHtml`, + `injectSearchSetup`, `rewriteCss`, `computeRelative`, + `resolveRaw`, `buildSegs`, `decode`, `computeRelUrl`, + `getPageCache`, `escapeRegExp`, regex constants + +### New function: `buildSitePathsSync` + +Synchronous version of `buildSitePaths` that takes an explicit +`themeAssetRels` array instead of walking `_site/assets/`. + +```js +function buildSitePathsSync(pages, staticFiles, excludePatterns, stubs, themeAssetRels) { + const paths = new Set(); + for (const p of pages) { + if (p.frontmatter?.layout === "book-combined") continue; + const rel = p.destPath.replaceAll("\\", "/"); + if (offlineExcluded(rel, excludePatterns)) continue; + paths.add("/" + rel); + } + for (const s of staticFiles) { + const rel = s.destRel.replaceAll("\\", "/"); + if (offlineExcluded(rel, excludePatterns)) continue; + paths.add("/" + rel); + } + for (const stub of stubs) { + const rel = stub.destPath.replaceAll("\\", "/"); + if (offlineExcluded(rel, excludePatterns)) continue; + paths.add("/" + rel); + } + for (const rel of themeAssetRels) { + if (offlineExcluded(rel, excludePatterns)) continue; + paths.add("/" + rel); + } + return paths; +} +``` + +### New function: `enumerateVendoredThemeAssets` + +Sync `readdirSync` walk of `builder/vendor/just-the-docs/assets/`. +Returns paths like `["assets/js/just-the-docs.js", +"assets/js/vendor/lunr.min.js"]`. Lives in `offline.mjs` (not +`offline-rewrite.mjs`) to keep the worker-imported module free of +`node:fs` dependencies. `dispatch.execute` imports it from +`offline.mjs`. + +### `offline.mjs` changes + +- Remove moved functions, import and re-export from + `offline-rewrite.mjs`. +- Keep all I/O functions: `writeOffline`, `buildOfflineState`, + `writeOfflinePages`, `writeOfflineRedirects`, + `copyOfflineStatics`, `copyOfflineThemeAssets`, + `setupOfflineDest`, `patchJustTheDocsJs`, `writeSearchDataJs`, + `collectThemeFiles`. +- Keep `buildSitePaths` (async, for diff-tool backward compat). + +### Verification + +`build.bat && check.bat` — byte-identical output, no behaviour +change. + +--- + +## Phase II: Expand `dispatch` and SAB payload + +### `dispatch.expected` + +Add `"mermaid"` and `"deriveRedirects"`. + +- `mermaid` ensures `state.staticFiles` includes freshly-generated + SVGs (appended in `mermaid.submit`). +- `deriveRedirects` provides the redirect stubs for `sitePaths`. +- Neither adds latency — both complete well before + `resolveBookChapters` (~600 ms into the build). + +### Emitter updates + +The scheduler requires every expected predecessor to emit to the +waiting task: + +- `mermaid.submit`: add `emit("dispatch", out)` +- `deriveRedirects.submit`: add `emit("dispatch", out)` + +### `dispatch.execute` + +After receiving `deriveRedirects: { stubs }`: + +1. Enumerate vendored theme assets via + `enumerateVendoredThemeAssets()`. +2. Append the two known generated-CSS paths + (`assets/css/tb-highlight.css`, + `assets/css/just-the-docs-combined.css`). +3. Call `buildSitePathsSync(state.pages, state.staticFiles, + excludePatterns, stubs, themeAssetRels)`. +4. Stash `sitePaths` on `state` for later use by `writeOffline`. +5. Compute `skipOffline` from config / CLI opts. + +### SAB payload + +Three new fields in the `shared` object: + +```js +{ + ...existing, + sitePathsArr: [...sitePaths], // ~1080 strings, ~30-50 KB + offlineExcludePatterns: [...], // from config + skipOffline: Boolean, // from --no-offline / config +} +``` + +### Verification + +Build succeeds, workers receive the expanded SAB, offline output +unchanged (workers don't use the new data yet). + +--- + +## Phase III: Worker offline rewrite + +### `cpu-worker.mjs` render handler + +Import from `offline-rewrite.mjs`: `deriveOfflinePage`, +`deriveOfflinePageCached`, `sliceNavBlock`, `normalizeBaseurl`, +`posixDirname`. + +After `templatePhase`, if `!skipOffline`: + +1. Build per-worker offline state: `new Set(sitePathsArr)`, fresh + caches, normalized baseurl. +2. Run the nav-cache pre-pass (group chunk pages by dest dir, derive + the first page per dir, cache nav block slices) — same logic as + current `writeOfflinePages` lines 207-223. +3. Set `offlineState.navCache = navCache` so + `deriveOfflinePageCached` can find it via `deps.navCache`. +4. Call `deriveOfflinePageCached` per writable page, storing + `offlineHtml` and `offlineMisses` on the page object. + +Return delta gains two fields: + +```js +{ destPath, renderedContent, html, offlineHtml, offlineMisses } +``` + +When `skipOffline` is true, the entire offline pass is skipped — no +Set construction, no rewriting, `offlineHtml` is `undefined`. + +### `render:i.submit` in `tbdocs.mjs` + +Merge `offlineHtml` and `offlineMisses` onto master pages alongside +the existing fields. + +### Nav-cache and cross-chunk dedup cost + +Works per-chunk. Pages in the same directory within a chunk share the +cache. Cross-chunk directories build their cache independently — +correct but slightly less efficient. The cache is an optimization, +not a correctness dependency. + +Two cache systems are affected by per-worker isolation: + +**Nav-cache** (per-directory sidebar substitution). The sidebar nav +block is ~80 KB, byte-identical across every page before rewrite. +The nav-cache runs `deriveOfflinePage` on the first page per +destination directory, stashes the pre/post-rewrite nav block, and +substitutes it directly for subsequent pages — avoiding re-running +the regex over 80 KB per page. With per-worker caches, a directory +that spans a chunk boundary gets its first-page rewrite done +independently in both chunks. Cost per extra rewrite: ~0.24 ms +(~200 ms / 837 pages, from the comment at `offline.mjs:189`). With +16 workers there are 15 chunk boundaries; worst case 15 directories +are split — 15 extra nav-block rewrites at 0.24 ms = ~4 ms total. +There are ~200 unique destination directories. Current single- +threaded nav-cache: 200 full rewrites. Per-worker: 200 + 15 = 215. + +**URL resolution caches** (`rawResolution`, `seg`, `result`). These +cache the resolved form of each unique URL so it isn't re-resolved +for a later page. With per-worker caches, each worker resolves URLs +independently. But the nav-cache already eliminates the dominant +source of shared URLs — the ~800 sidebar links are cached as a +block, not resolved individually. The remaining per-page body URLs +are ~5-20 links per page, many unique to that page. The common ones +(links to frequently-referenced symbols) might total ~2,000 unique +URLs across the site. Each resolution is a Set lookup + string +manipulation — ~1 us. Even if every worker re-resolves all 2,000: +16 workers x 2,000 x 1 us = ~32 ms total, spread across workers +running in parallel. + +**Net impact:** ~35 ms of redundant work total, spread across 16 +parallel workers — ~2 ms added wall-clock. Noise against the +~500 ms saved by parallelisation. + +### Transfer cost + +Roughly doubles the render delta size (adding `offlineHtml` per +page). Estimated +40 ms per worker at structured-clone throughput. +Bounded and acceptable. + +### Verification + +`page.offlineHtml` is populated on all pages. Existing `writeOffline` +still runs its own CPU path (redundant but correct). Output +identical. + +--- + +## Phase IV: Switch `writeOffline` to pre-computed HTML + +### `writeOfflinePages` in `offline.mjs` + +Add a `precomputed` option: + +- When true: skip `deriveOfflinePage` / nav-cache entirely, write + `page.offlineHtml` directly (I/O only). +- When false: existing CPU-bound path (kept for diff tools). + +### `buildOfflineState` + +Add optional `sitePaths` parameter: + +- When provided: skip the async `buildSitePaths` call (avoids the + `_site/assets/` walk). +- When absent: existing async path (for diff tools). + +### `writeOffline` task in `tbdocs.mjs` + +Pass both options: + +```js +return writeOffline(state.pages, state.staticFiles, state.site, ctx.destRoot, { + auxStats, + precomputed: true, + sitePaths: state.sitePaths, +}); +``` + +### Verification + +`build.bat && check.bat` — byte-identical offline output. + +Compare `_site-offline/` output byte-for-byte against a baseline +built before Phase I. The offline tree must be identical. + +Timing: `writeOffline` should drop from ~700 ms to ~200-300 ms. The +render worker times will increase modestly (~50-100 ms each) to +absorb the rewrite work. + +--- + +## Phase V: Documentation + +Update: + +- `builder/PLAN-scheduler.md` — dispatch dependencies, dataflow + diagram, render delta shape. +- `docs/Documentation/Builder.md` — offline build timing, structural + win description. +- `docs/Documentation/Pipeline-Stages.md` — `writeOffline` signature, + `offline-rewrite.mjs` exports. +- `docs/assets/images/mmd/scheduler-dag.mmd` — edges from + `mermaid` / `deriveRedirects` to `dispatch`. +- `offline.mjs` header comment — note the extraction to + `offline-rewrite.mjs`. + +--- + +## Files to modify + +| File | Changes | +|------|---------| +| `builder/offline-rewrite.mjs` | **New.** Pure-compute rewrite functions + `buildSitePathsSync` + `enumerateVendoredThemeAssets`. | +| `builder/offline.mjs` | Remove moved functions, re-export from `offline-rewrite.mjs`. Add `precomputed` path to `writeOfflinePages`. Add `sitePaths` option to `buildOfflineState`. | +| `builder/cpu-worker.mjs` | Import from `offline-rewrite.mjs`. Add offline rewrite pass after `templatePhase`. Expand return delta. | +| `builder/tbdocs.mjs` | `dispatch`: add deps, compute sitePaths, expand SAB. `render:i.submit`: merge offlineHtml. `writeOffline` task: pass `precomputed` + `sitePaths`. Emitter updates for `mermaid.submit` and `deriveRedirects.submit`. | +| `builder/sab-broadcast.mjs` | No changes — existing JSON serialize/deserialize handles the expanded payload. | From 7623c36177891a7a3db951cb1ebaf22bbe5e221d Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Sat, 30 May 2026 21:20:18 +0200 Subject: [PATCH 12/72] Phase I: extract pure-compute offline rewrite into offline-rewrite.mjs --- builder/offline-rewrite.mjs | 452 ++++++++++++++++++++++++++++++++++ builder/offline.mjs | 477 ++++++------------------------------ 2 files changed, 527 insertions(+), 402 deletions(-) create mode 100644 builder/offline-rewrite.mjs diff --git a/builder/offline-rewrite.mjs b/builder/offline-rewrite.mjs new file mode 100644 index 00000000..f8442c3c --- /dev/null +++ b/builder/offline-rewrite.mjs @@ -0,0 +1,452 @@ +// Pure-compute offline URL-rewrite helpers --- no node:fs, worker-safe. +// Extracted from offline.mjs (Phase I of PLAN-scheduler-offline.md). +// cpu-worker.mjs (Phase III) will import these to run the per-page +// offline rewrite inside render workers, in parallel across CPUs. +// +// Sections: +// +// §B Site-paths set (buildSitePathsSync, offlineExcluded, +// fnmatchPathname) +// §C URL resolution (computeRelative, resolveRaw, computeRelUrl, +// buildSegs, decode, fileDirSegsFromRel, +// posixDirname, normalizeBaseurl, escapeRegExp, +// getPageCache) +// §D HTML rewrite (stripSeo, rewriteHtml, injectSearchSetup, +// sliceNavBlock, NAV_OPEN_RE, NAV_CLOSE, +// NAV_PLACEHOLDER, deriveOfflinePageCached, +// deriveOfflinePage, SEO_BLOCK_RE, TITLE_RE, +// HTML_COMBINED_RE, JTD_SCRIPT_TAG_RE) +// §E CSS rewrite (rewriteCss, CSS_URL_RE, deriveOfflineCss) +// §F Redirect-stub (deriveOfflineRedirect) + +// --------------------------------------------------------------------------- +// §B Site-paths set +// --------------------------------------------------------------------------- + +// Synchronous version of buildSitePaths. Takes an explicit themeAssetRels +// array (from enumerateVendoredThemeAssets in offline.mjs) instead of +// walking _site/assets/ --- so it can run before the output tree exists. +export function buildSitePathsSync(pages, staticFiles, excludePatterns, stubs, themeAssetRels) { + const paths = new Set(); + for (const p of pages) { + if (p.frontmatter?.layout === "book-combined") continue; + const rel = p.destPath.replaceAll("\\", "/"); + if (offlineExcluded(rel, excludePatterns)) continue; + paths.add("/" + rel); + } + for (const s of staticFiles) { + const rel = s.destRel.replaceAll("\\", "/"); + if (offlineExcluded(rel, excludePatterns)) continue; + paths.add("/" + rel); + } + for (const stub of stubs) { + const rel = stub.destPath.replaceAll("\\", "/"); + if (offlineExcluded(rel, excludePatterns)) continue; + paths.add("/" + rel); + } + for (const rel of themeAssetRels) { + if (offlineExcluded(rel, excludePatterns)) continue; + paths.add("/" + rel); + } + return paths; +} + +// §6.5 offlineExcluded -- File.fnmatch(..., FNM_PATHNAME) semantics: +// `*` does NOT cross `/`, `**` does. +export function offlineExcluded(rel, patterns) { + if (!patterns.length) return false; + return patterns.some(pat => fnmatchPathname(pat, rel)); +} + +export function fnmatchPathname(pattern, str) { + let re = "^"; + for (let i = 0; i < pattern.length; i++) { + const c = pattern[i]; + if (c === "*") { + if (pattern[i + 1] === "*") { re += ".*"; i++; } + else { re += "[^/]*"; } + } else if (c === "?") { + re += "[^/]"; + } else if (".+^$()|[]{}\\".includes(c)) { + re += "\\" + c; + } else { + re += c; + } + } + re += "$"; + return new RegExp(re).test(str); +} + +// --------------------------------------------------------------------------- +// §C URL resolution +// --------------------------------------------------------------------------- + +// Chars safe in a URL path segment (RFC 3986 unreserved + sub-delims that +// don't need encoding in a path). +export const PATH_SAFE_RE = /[^A-Za-z0-9\-_.~!$&'()*+,;=:@]/; +export const PATH_SAFE_CHAR_RE = /^[A-Za-z0-9\-_.~!$&'()*+,;=:@]$/; + +// §6.3 computeRelative -- absolute URL → page-relative URL. +export function computeRelative(raw, fileSegs, sitePaths, caches, baseurl) { + let resolved = caches.rawResolution.get(raw); + if (resolved === undefined) { + resolved = resolveRaw(raw, sitePaths, baseurl); + caches.rawResolution.set(raw, resolved); + } + const [sep, tail, sitePath] = resolved; + if (sitePath === null) return null; + + let segCacheEntry = caches.seg.get(sitePath); + if (segCacheEntry === undefined) { + segCacheEntry = buildSegs(sitePath); + caches.seg.set(sitePath, segCacheEntry); + } + const [decodedSegs, encodedSegs] = segCacheEntry; + + let common = 0; + const fsLen = fileSegs.length; + const tsLen = decodedSegs.length; + while (common < fsLen && common < tsLen && fileSegs[common] === decodedSegs[common]) { + common++; + } + + const ascend = "../".repeat(fsLen - common); + const descend = encodedSegs.slice(common).join("/"); + let rel = ascend + descend; + if (rel === "") rel = "./"; + return rel + sep + tail; +} + +// File-dir-independent half of computeRelative. +export function resolveRaw(raw, sitePaths, baseurl) { + const splitIdx = raw.search(/[?#]/); + const pathPart = splitIdx === -1 ? raw : raw.slice(0, splitIdx); + const sep = splitIdx === -1 ? "" : raw[splitIdx]; + const tail = splitIdx === -1 ? "" : raw.slice(splitIdx + 1); + let fsPath = decode(pathPart); + + if (baseurl) { + if (fsPath === baseurl) fsPath = "/"; + else if (fsPath.startsWith(baseurl + "/")) fsPath = fsPath.slice(baseurl.length); + } + + let candidates; + if (fsPath.endsWith("/")) { + candidates = [fsPath, fsPath + "index.html"]; + } else if (fsPath.includes(".")) { + candidates = [fsPath, fsPath + "/index.html"]; + } else { + candidates = [fsPath, fsPath + ".html", fsPath + "/index.html"]; + } + let sitePath = null; + for (const c of candidates) { + if (sitePaths.has(c)) { sitePath = c; break; } + } + return [sep, tail, sitePath]; +} + +// §6.4 computeRelUrl -- page-relative URL → page-relative URL with the +// .html / /index.html / "" suffix that makes it resolve under file://. +export function computeRelUrl(raw, fileSegs, sitePaths) { + const splitIdx = raw.search(/[?#]/); + const pathPart = splitIdx === -1 ? raw : raw.slice(0, splitIdx); + const sep = splitIdx === -1 ? "" : raw[splitIdx]; + const tail = splitIdx === -1 ? "" : raw.slice(splitIdx + 1); + if (pathPart === "") return null; + + const decoded = decode(pathPart); + const trailingSlash = decoded.endsWith("/"); + const stack = [...fileSegs]; + for (const seg of decoded.split("/")) { + if (seg === "" || seg === ".") continue; + if (seg === "..") stack.pop(); + else stack.push(seg); + } + + let probePath = "/" + stack.join("/"); + if (trailingSlash && !probePath.endsWith("/")) probePath += "/"; + + let candidates; + if (probePath.endsWith("/")) { + candidates = [["", probePath], ["index.html", probePath + "index.html"]]; + } else if (probePath.includes(".")) { + candidates = [["", probePath], ["/index.html", probePath + "/index.html"]]; + } else { + candidates = [["", probePath], [".html", probePath + ".html"], ["/index.html", probePath + "/index.html"]]; + } + + for (const [suffix, full] of candidates) { + if (sitePaths.has(full)) return pathPart + suffix + sep + tail; + } + return null; +} + +// Cached decoded/encoded segments for a site-rooted path. +export function buildSegs(sitePath) { + const decoded = sitePath.slice(1).split("/"); + const encoded = decoded.map(seg => { + if (!PATH_SAFE_RE.test(seg)) return seg; + // Encode per UTF-8 byte so non-ASCII characters in future content + // round-trip correctly. + const bytes = new TextEncoder().encode(seg); + let out = ""; + for (const b of bytes) { + if (b < 0x80 && PATH_SAFE_CHAR_RE.test(String.fromCharCode(b))) { + out += String.fromCharCode(b); + } else { + out += "%" + b.toString(16).toUpperCase().padStart(2, "0"); + } + } + return out; + }); + return [decoded, encoded]; +} + +// Percent-decode a URL path (sequences of %XX bytes interpreted as UTF-8). +export function decode(s) { + return s.replace(/(?:%[0-9A-Fa-f]{2})+/g, (m) => { + const bytes = new Uint8Array(m.length / 3); + for (let i = 0; i < bytes.length; i++) { + bytes[i] = parseInt(m.slice(i * 3 + 1, i * 3 + 3), 16); + } + return new TextDecoder("utf-8").decode(bytes); + }); +} + +// §6.11 fileDirSegsFromRel +export function fileDirSegsFromRel(rel) { + const normalised = rel.replaceAll("\\", "/"); + const dir = posixDirname(normalised); + if (dir === "." || dir === "") return []; + return dir.split("/"); +} + +export function posixDirname(rel) { + const normalised = rel.replaceAll("\\", "/"); + const idx = normalised.lastIndexOf("/"); + return idx === -1 ? "." : normalised.slice(0, idx); +} + +// §6.12 normalizeBaseurl +export function normalizeBaseurl(raw) { + let baseurl = String(raw ?? "").replace(/\/+$/, ""); + if (baseurl && !baseurl.startsWith("/")) baseurl = "/" + baseurl; + return baseurl; +} + +// §6.13 escapeRegExp +export function escapeRegExp(s) { + return s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); +} + +// Hoist the per-file-dir inner cache so the per-match cost is one +// map lookup. +export function getPageCache(resultCache, fileDir) { + let pageCache = resultCache.get(fileDir); + if (!pageCache) { + pageCache = new Map(); + resultCache.set(fileDir, pageCache); + } + return pageCache; +} + +// --------------------------------------------------------------------------- +// §D HTML rewrite pipeline +// --------------------------------------------------------------------------- + +export const SEO_BLOCK_RE = //s; +export const TITLE_RE = /.*?<\/title>/s; + +// §6.2 stripSeo -- drop the jekyll-seo-tag block, keep its <title>. +export function stripSeo(html) { + if (!html.includes("<!-- Begin Jekyll SEO tag")) return html; + return html.replace(SEO_BLOCK_RE, (block) => { + const titleMatch = block.match(TITLE_RE); + return titleMatch ? titleMatch[0] : ""; + }); +} + +// Combined regex: three top-level alternatives -- <code> block, <pre> +// block, or a real href|src attribute carrying either an absolute or +// page-relative URL. The code/pre alternatives consume their bodies +// atomically so href/src matches inside code samples are skipped. +export const HTML_COMBINED_RE = /<code\b[^>]*>[\s\S]*?<\/code>|<pre\b[^>]*>[\s\S]*?<\/pre>|\b(href|src)=(["'])(\/(?!\/)[^"']*|(?![#/]|[a-zA-Z][a-zA-Z0-9+.\-]*:)[^"']+)\2/g; + +// §6.6 rewriteHtml -- single regex pass over the HTML. +export function rewriteHtml(html, fileDir, fileSegs, sitePaths, caches, baseurl) { + let misses = 0; + const pageCache = getPageCache(caches.result, fileDir); + + const rewritten = html.replace(HTML_COMBINED_RE, (match, attrName, quote, rawUrl) => { + if (attrName === undefined) { + // <code> or <pre> block; leave verbatim. + return match; + } + let rel = pageCache.get(rawUrl); + if (rel === undefined) { + rel = rawUrl.startsWith("/") + ? computeRelative(rawUrl, fileSegs, sitePaths, caches, baseurl) + : computeRelUrl(rawUrl, fileSegs, sitePaths); + pageCache.set(rawUrl, rel); + } + if (rel === null) { + misses++; + return match; + } + if (rel === rawUrl) { + // File already correct at the relative path (rare). + return match; + } + return `${attrName}=${quote}${rel}${quote}`; + }); + + return { rewritten, misses }; +} + +export const JTD_SCRIPT_TAG_RE = /<script\s+src="([^"]*)just-the-docs\.js"/; + +// §6.8 injectSearchSetup -- two <script> tags before the just-the-docs.js +// tag carry the per-page relative site-root and the lunr-index data load. +export function injectSearchSetup(html, fileSegs) { + return html.replace(JTD_SCRIPT_TAG_RE, (match, prefix) => { + const siteRoot = fileSegs.length === 0 ? "" : "../".repeat(fileSegs.length); + return `<script>window.OFFLINE_SITE_ROOT="${siteRoot}";</script>\n` + + `<script src="${prefix}search-data.js"></script>` + + match; + }); +} + +export const NAV_OPEN_RE = /<nav aria-label="Main" id="site-nav"[^>]*>/; +export const NAV_CLOSE = "</nav>"; + +// Slice the sidebar nav block out of an HTML page. Returns the literal +// `<nav ...>...</nav>` substring, or null if the page doesn't carry +// the expected sidebar shape (in which case the cache entry is skipped +// for the source dir and subsequent pages fall back to the full path). +export function sliceNavBlock(html) { + const m = html.match(NAV_OPEN_RE); + if (!m) return null; + const start = m.index; + const end = html.indexOf(NAV_CLOSE, start); + if (end === -1) return null; + return html.slice(start, end + NAV_CLOSE.length); +} + +// Placeholder spliced in place of the cached input nav while +// deriveOfflinePage runs. An HTML comment so it never collides with +// the three alternatives in HTML_COMBINED_RE (<code> / <pre> / +// href|src=), the SEO-block regex (different prefix), the JTD script +// tag regex (different prefix), or any other rewrite step. +export const NAV_PLACEHOLDER = "<!--TBDOCS_NAV_CACHE_-->"; + +// Cache-consulting wrapper around deriveOfflinePage. On hit: +// substitutes the cached input slice with a placeholder, runs the +// rewrite over the ~80kB-smaller string, splices the cached output +// back in. On miss (no cache entry for the source dir OR the input +// slice doesn't match byte-for-byte): falls back to the full +// rewrite with a warning. +export function deriveOfflinePageCached(page, deps) { + const destDir = posixDirname(page.destPath); + const cached = deps.navCache?.get(destDir); + if (!cached) return deriveOfflinePage(page, deps); + + const idx = page.html.indexOf(cached.input); + if (idx === -1) { + console.warn( + `offline nav cache miss for ${page.srcRel}: ` + + `nav block doesn't match first page in ${destDir}; ` + + `falling back to full rewrite`, + ); + return deriveOfflinePage(page, deps); + } + + const stubbed = page.html.slice(0, idx) + NAV_PLACEHOLDER + + page.html.slice(idx + cached.input.length); + const stubbedPage = { ...page, html: stubbed }; + const { html: stubbedOut, misses } = deriveOfflinePage(stubbedPage, deps); + const out = stubbedOut.replace(NAV_PLACEHOLDER, cached.output); + return { html: out, misses }; +} + +// Pure-compute: apply strip + URL rewrite + script injection to a +// single rendered page. Returns `{ html, misses }`. The page must +// have `page.html !== undefined`. The state's caches are mutated for +// per-build reuse; pass a fresh state if cache pollution across pages +// is a concern (see _diff.mjs's per-call buildOfflineState). +export function deriveOfflinePage(page, state) { + const { sitePaths, caches, baseurl } = state; + const fileDir = posixDirname(page.destPath); + const fileSegs = fileDirSegsFromRel(page.destPath); + let html = page.html; + html = stripSeo(html); + const { rewritten, misses } = rewriteHtml(html, fileDir, fileSegs, sitePaths, caches, baseurl); + html = rewritten; + html = injectSearchSetup(html, fileSegs); + return { html, misses }; +} + +// --------------------------------------------------------------------------- +// §E CSS rewrite pipeline +// --------------------------------------------------------------------------- + +export const CSS_URL_RE = /url\(\s*(["']?)(\/(?!\/)[^"'()\s]*)\1\s*\)/g; + +// §6.7 rewriteCss -- url(/...) → page-relative. +export function rewriteCss(css, fileDir, fileSegs, sitePaths, caches, baseurl) { + let misses = 0; + const pageCache = getPageCache(caches.result, fileDir); + + const rewritten = css.replace(CSS_URL_RE, (match, quote, rawUrl) => { + let rel = pageCache.get(rawUrl); + if (rel === undefined) { + rel = computeRelative(rawUrl, fileSegs, sitePaths, caches, baseurl); + pageCache.set(rawUrl, rel); + } + if (rel === null) { + misses++; + return match; + } + return `url(${quote}${rel}${quote})`; + }); + + return { rewritten, misses }; +} + +// Pure-compute: rewrite `url(/...)` references in a single CSS file. +// `themeRel` is the file's path relative to <destRoot>/ (e.g. +// "assets/css/just-the-docs-combined.css"). Returns `{ css, misses }`. +export function deriveOfflineCss(cssIn, themeRel, state) { + const { sitePaths, caches, baseurl } = state; + const fileDir = posixDirname(themeRel); + const fileSegs = fileDirSegsFromRel(themeRel); + const { rewritten, misses } = rewriteCss(cssIn, fileDir, fileSegs, sitePaths, caches, baseurl); + return { css: rewritten, misses }; +} + +// --------------------------------------------------------------------------- +// §F Redirect-stub rewrite +// --------------------------------------------------------------------------- + +// Pure-compute: rewrite the absolute <site.url>/<path> URLs in a single +// redirect stub. Returns the rewritten HTML. With no site.url configured, +// returns the stub verbatim. +export function deriveOfflineRedirect(stub, state) { + const { sitePaths, caches, baseurl, siteUrl } = state; + if (!siteUrl) return stub.html; + + const siteUrlEsc = escapeRegExp(siteUrl); + const prefixRe = new RegExp(`${siteUrlEsc}(/[^"' >]*)`, "g"); + + const fileDir = posixDirname(stub.destPath); + const fileSegs = fileDirSegsFromRel(stub.destPath); + const pageCache = getPageCache(caches.result, fileDir); + + return stub.html.replace(prefixRe, (match, raw) => { + let rel = pageCache.get(raw); + if (rel === undefined) { + rel = computeRelative(raw, fileSegs, sitePaths, caches, baseurl); + pageCache.set(raw, rel); + } + return rel ?? match; + }); +} diff --git a/builder/offline.mjs b/builder/offline.mjs index bf3f0264..a26c516b 100644 --- a/builder/offline.mjs +++ b/builder/offline.mjs @@ -4,27 +4,22 @@ // and docs/_plugins/offlinify.rb for the canonical Jekyll reference. // // One entry point: writeOffline(pages, staticFiles, site, destRoot, -// { auxStats }). Pure-compute derive helpers (buildOfflineState + -// derive*) are also exported for `_diff.mjs` / `_triage.mjs` to reuse -// without writing anything to disk. +// { auxStats }). Pure-compute derive helpers are in offline-rewrite.mjs +// and re-exported from here for `_diff.mjs` / `_triage.mjs` backward +// compatibility. // // Internal sections: // // §A Top-level orchestration -// §B Site-paths set -// §C URL resolution (computeRelative, computeRelUrl, -// resolveRaw, buildSegs, decode) -// §D HTML rewrite pipeline (stripSeo, rewriteHtml, -// injectSearchSetup) -// §E CSS rewrite pipeline (rewriteCss) -// §F Redirect-stub rewrite +// §B Site-paths set (buildSitePaths async + enumerateVendoredThemeAssets) // §G just-the-docs.js patches + search-data.js wrapper // §H Static-file pass + theme-asset pass -// §I Pure-compute derive helpers (re-export surface for diff tools) +// §I Re-export surface for diff tools (from offline-rewrite.mjs) import { promises as fs } from "node:fs"; -import { existsSync } from "node:fs"; +import { existsSync, readdirSync } from "node:fs"; import path from "node:path"; +import { fileURLToPath } from "node:url"; import * as acorn from "acorn"; import * as acornWalk from "acorn-walk"; @@ -38,9 +33,61 @@ import { writeFileMkdirp, } from "./write.mjs"; +import { + offlineExcluded, + normalizeBaseurl, + posixDirname, + sliceNavBlock, + deriveOfflinePage, + deriveOfflinePageCached, + deriveOfflineCss, + deriveOfflineRedirect, +} from "./offline-rewrite.mjs"; + +// --------------------------------------------------------------------------- +// §I Re-export surface for diff tools (from offline-rewrite.mjs) +// --------------------------------------------------------------------------- + +export { + buildSitePathsSync, + offlineExcluded, + fnmatchPathname, + normalizeBaseurl, + posixDirname, + fileDirSegsFromRel, + sliceNavBlock, + NAV_OPEN_RE, + NAV_CLOSE, + NAV_PLACEHOLDER, + deriveOfflinePageCached, + deriveOfflinePage, + deriveOfflineCss, + deriveOfflineRedirect, + stripSeo, + rewriteHtml, + injectSearchSetup, + rewriteCss, + computeRelative, + resolveRaw, + buildSegs, + decode, + computeRelUrl, + getPageCache, + escapeRegExp, + PATH_SAFE_RE, + PATH_SAFE_CHAR_RE, + SEO_BLOCK_RE, + TITLE_RE, + HTML_COMBINED_RE, + JTD_SCRIPT_TAG_RE, + CSS_URL_RE, +} from "./offline-rewrite.mjs"; + const OFFLINE_SUFFIX = "-offline"; const LIMIT = WRITE_LIMIT; +const _builderDir = path.dirname(fileURLToPath(import.meta.url)); + // Local copy of tbdocs.mjs's makeTimer (PLAN-9 §7.D13). Avoids a // cyclic import; the verify harnesses and diff tools import // offline.mjs without going through tbdocs.mjs's main() side effects. @@ -232,75 +279,6 @@ async function writeOfflinePages(pages, deps) { }); } -const NAV_OPEN_RE = /<nav aria-label="Main" id="site-nav"[^>]*>/; -const NAV_CLOSE = "</nav>"; - -// Slice the sidebar nav block out of an HTML page. Returns the literal -// `<nav ...>...</nav>` substring, or null if the page doesn't carry -// the expected sidebar shape (in which case the cache entry is skipped -// for the source dir and subsequent pages fall back to the full path). -function sliceNavBlock(html) { - const m = html.match(NAV_OPEN_RE); - if (!m) return null; - const start = m.index; - const end = html.indexOf(NAV_CLOSE, start); - if (end === -1) return null; - return html.slice(start, end + NAV_CLOSE.length); -} - -// Placeholder spliced in place of the cached input nav while -// deriveOfflinePage runs. An HTML comment so it never collides with -// the three alternatives in HTML_COMBINED_RE (<code> / <pre> / -// href|src=), the SEO-block regex (different prefix), the JTD script -// tag regex (different prefix), or any other rewrite step. -const NAV_PLACEHOLDER = "<!--TBDOCS_NAV_CACHE_-->"; - -// Cache-consulting wrapper around deriveOfflinePage. On hit: -// substitutes the cached input slice with a placeholder, runs the -// rewrite over the ~80kB-smaller string, splices the cached output -// back in. On miss (no cache entry for the source dir OR the input -// slice doesn't match byte-for-byte): falls back to the full -// rewrite with a warning. -function deriveOfflinePageCached(page, deps) { - const destDir = posixDirname(page.destPath); - const cached = deps.navCache?.get(destDir); - if (!cached) return deriveOfflinePage(page, deps); - - const idx = page.html.indexOf(cached.input); - if (idx === -1) { - console.warn( - `offline nav cache miss for ${page.srcRel}: ` + - `nav block doesn't match first page in ${destDir}; ` + - `falling back to full rewrite`, - ); - return deriveOfflinePage(page, deps); - } - - const stubbed = page.html.slice(0, idx) + NAV_PLACEHOLDER + - page.html.slice(idx + cached.input.length); - const stubbedPage = { ...page, html: stubbed }; - const { html: stubbedOut, misses } = deriveOfflinePage(stubbedPage, deps); - const out = stubbedOut.replace(NAV_PLACEHOLDER, cached.output); - return { html: out, misses }; -} - -// Pure-compute: apply strip + URL rewrite + script injection to a -// single rendered page. Returns `{ html, misses }`. The page must -// have `page.html !== undefined`. The state's caches are mutated for -// per-build reuse; pass a fresh state if cache pollution across pages -// is a concern (see _diff.mjs's per-call buildOfflineState). -export function deriveOfflinePage(page, state) { - const { sitePaths, caches, baseurl } = state; - const fileDir = posixDirname(page.destPath); - const fileSegs = fileDirSegsFromRel(page.destPath); - let html = page.html; - html = stripSeo(html); - const { rewritten, misses } = rewriteHtml(html, fileDir, fileSegs, sitePaths, caches, baseurl); - html = rewritten; - html = injectSearchSetup(html, fileSegs); - return { html, misses }; -} - // §5.3 writeOfflineRedirects -- rewrite the four <site.url><path> // occurrences in each stub. async function writeOfflineRedirects(stubs, deps) { @@ -312,30 +290,6 @@ async function writeOfflineRedirects(stubs, deps) { }); } -// Pure-compute: rewrite the absolute <site.url>/<path> URLs in a single -// redirect stub. Returns the rewritten HTML. With no site.url configured, -// returns the stub verbatim. -export function deriveOfflineRedirect(stub, state) { - const { sitePaths, caches, baseurl, siteUrl } = state; - if (!siteUrl) return stub.html; - - const siteUrlEsc = escapeRegExp(siteUrl); - const prefixRe = new RegExp(`${siteUrlEsc}(/[^"' >]*)`, "g"); - - const fileDir = posixDirname(stub.destPath); - const fileSegs = fileDirSegsFromRel(stub.destPath); - const pageCache = getPageCache(caches.result, fileDir); - - return stub.html.replace(prefixRe, (match, raw) => { - let rel = pageCache.get(raw); - if (rel === undefined) { - rel = computeRelative(raw, fileSegs, sitePaths, caches, baseurl); - pageCache.set(raw, rel); - } - return rel ?? match; - }); -} - // §5.4 copyOfflineStatics -- mirror staticFiles[] minus offline_exclude. async function copyOfflineStatics(staticFiles, deps) { const { offlineRoot, excludePatterns, counters } = deps; @@ -381,17 +335,6 @@ async function copyOfflineThemeAssets(deps) { }); } -// Pure-compute: rewrite `url(/...)` references in a single CSS file. -// `themeRel` is the file's path relative to <destRoot>/ (e.g. -// "assets/css/just-the-docs-combined.css"). Returns `{ css, misses }`. -export function deriveOfflineCss(cssIn, themeRel, state) { - const { sitePaths, caches, baseurl } = state; - const fileDir = posixDirname(themeRel); - const fileSegs = fileDirSegsFromRel(themeRel); - const { rewritten, misses } = rewriteCss(cssIn, fileDir, fileSegs, sitePaths, caches, baseurl); - return { css: rewritten, misses }; -} - // --------------------------------------------------------------------------- // §B Site-paths set // --------------------------------------------------------------------------- @@ -433,296 +376,26 @@ async function buildSitePaths(pages, staticFiles, destRoot, excludePatterns, stu return paths; } -// §6.5 offlineExcluded -- File.fnmatch(..., FNM_PATHNAME) semantics: -// `*` does NOT cross `/`, `**` does. -function offlineExcluded(rel, patterns) { - if (!patterns.length) return false; - return patterns.some(pat => fnmatchPathname(pat, rel)); -} - -function fnmatchPathname(pattern, str) { - let re = "^"; - for (let i = 0; i < pattern.length; i++) { - const c = pattern[i]; - if (c === "*") { - if (pattern[i + 1] === "*") { re += ".*"; i++; } - else { re += "[^/]*"; } - } else if (c === "?") { - re += "[^/]"; - } else if (".+^$()|[]{}\\".includes(c)) { - re += "\\" + c; - } else { - re += c; - } - } - re += "$"; - return new RegExp(re).test(str); -} - -// --------------------------------------------------------------------------- -// §C URL resolution -// --------------------------------------------------------------------------- - -// Chars safe in a URL path segment (RFC 3986 unreserved + sub-delims that -// don't need encoding in a path). -const PATH_SAFE_RE = /[^A-Za-z0-9\-_.~!$&'()*+,;=:@]/; -const PATH_SAFE_CHAR_RE = /^[A-Za-z0-9\-_.~!$&'()*+,;=:@]$/; - -// §6.3 computeRelative -- absolute URL → page-relative URL. -function computeRelative(raw, fileSegs, sitePaths, caches, baseurl) { - let resolved = caches.rawResolution.get(raw); - if (resolved === undefined) { - resolved = resolveRaw(raw, sitePaths, baseurl); - caches.rawResolution.set(raw, resolved); - } - const [sep, tail, sitePath] = resolved; - if (sitePath === null) return null; - - let segCacheEntry = caches.seg.get(sitePath); - if (segCacheEntry === undefined) { - segCacheEntry = buildSegs(sitePath); - caches.seg.set(sitePath, segCacheEntry); - } - const [decodedSegs, encodedSegs] = segCacheEntry; - - let common = 0; - const fsLen = fileSegs.length; - const tsLen = decodedSegs.length; - while (common < fsLen && common < tsLen && fileSegs[common] === decodedSegs[common]) { - common++; - } - - const ascend = "../".repeat(fsLen - common); - const descend = encodedSegs.slice(common).join("/"); - let rel = ascend + descend; - if (rel === "") rel = "./"; - return rel + sep + tail; -} - -// File-dir-independent half of computeRelative. -function resolveRaw(raw, sitePaths, baseurl) { - const splitIdx = raw.search(/[?#]/); - const pathPart = splitIdx === -1 ? raw : raw.slice(0, splitIdx); - const sep = splitIdx === -1 ? "" : raw[splitIdx]; - const tail = splitIdx === -1 ? "" : raw.slice(splitIdx + 1); - let fsPath = decode(pathPart); - - if (baseurl) { - if (fsPath === baseurl) fsPath = "/"; - else if (fsPath.startsWith(baseurl + "/")) fsPath = fsPath.slice(baseurl.length); - } - - let candidates; - if (fsPath.endsWith("/")) { - candidates = [fsPath, fsPath + "index.html"]; - } else if (fsPath.includes(".")) { - candidates = [fsPath, fsPath + "/index.html"]; - } else { - candidates = [fsPath, fsPath + ".html", fsPath + "/index.html"]; - } - let sitePath = null; - for (const c of candidates) { - if (sitePaths.has(c)) { sitePath = c; break; } - } - return [sep, tail, sitePath]; -} - -// §6.4 computeRelUrl -- page-relative URL → page-relative URL with the -// .html / /index.html / "" suffix that makes it resolve under file://. -function computeRelUrl(raw, fileSegs, sitePaths) { - const splitIdx = raw.search(/[?#]/); - const pathPart = splitIdx === -1 ? raw : raw.slice(0, splitIdx); - const sep = splitIdx === -1 ? "" : raw[splitIdx]; - const tail = splitIdx === -1 ? "" : raw.slice(splitIdx + 1); - if (pathPart === "") return null; - - const decoded = decode(pathPart); - const trailingSlash = decoded.endsWith("/"); - const stack = [...fileSegs]; - for (const seg of decoded.split("/")) { - if (seg === "" || seg === ".") continue; - if (seg === "..") stack.pop(); - else stack.push(seg); - } - - let probePath = "/" + stack.join("/"); - if (trailingSlash && !probePath.endsWith("/")) probePath += "/"; - - let candidates; - if (probePath.endsWith("/")) { - candidates = [["", probePath], ["index.html", probePath + "index.html"]]; - } else if (probePath.includes(".")) { - candidates = [["", probePath], ["/index.html", probePath + "/index.html"]]; - } else { - candidates = [["", probePath], [".html", probePath + ".html"], ["/index.html", probePath + "/index.html"]]; - } - - for (const [suffix, full] of candidates) { - if (sitePaths.has(full)) return pathPart + suffix + sep + tail; - } - return null; -} - -// Cached decoded/encoded segments for a site-rooted path. -function buildSegs(sitePath) { - const decoded = sitePath.slice(1).split("/"); - const encoded = decoded.map(seg => { - if (!PATH_SAFE_RE.test(seg)) return seg; - // Encode per UTF-8 byte so non-ASCII characters in future content - // round-trip correctly. - const bytes = new TextEncoder().encode(seg); - let out = ""; - for (const b of bytes) { - if (b < 0x80 && PATH_SAFE_CHAR_RE.test(String.fromCharCode(b))) { - out += String.fromCharCode(b); +// Synchronous walk of builder/vendor/just-the-docs/assets/. Returns +// paths like ["assets/js/just-the-docs.js", +// "assets/js/vendor/lunr.min.js"] for use as the themeAssetRels +// argument to buildSitePathsSync. Lives here (not offline-rewrite.mjs) +// to keep the worker-imported module free of node:fs dependencies. +export function enumerateVendoredThemeAssets() { + const assetsRoot = path.join(_builderDir, "vendor/just-the-docs/assets"); + const out = []; + function walk(rel) { + for (const entry of readdirSync(path.join(assetsRoot, rel), { withFileTypes: true })) { + const childRel = rel === "" ? entry.name : path.posix.join(rel, entry.name); + if (entry.isDirectory()) { + walk(childRel); } else { - out += "%" + b.toString(16).toUpperCase().padStart(2, "0"); + out.push("assets/" + childRel); } } - return out; - }); - return [decoded, encoded]; -} - -// Percent-decode a URL path (sequences of %XX bytes interpreted as UTF-8). -function decode(s) { - return s.replace(/(?:%[0-9A-Fa-f]{2})+/g, (m) => { - const bytes = new Uint8Array(m.length / 3); - for (let i = 0; i < bytes.length; i++) { - bytes[i] = parseInt(m.slice(i * 3 + 1, i * 3 + 3), 16); - } - return new TextDecoder("utf-8").decode(bytes); - }); -} - -// §6.11 fileDirSegsFromRel -function fileDirSegsFromRel(rel) { - const normalised = rel.replaceAll("\\", "/"); - const dir = posixDirname(normalised); - if (dir === "." || dir === "") return []; - return dir.split("/"); -} - -function posixDirname(rel) { - const normalised = rel.replaceAll("\\", "/"); - const idx = normalised.lastIndexOf("/"); - return idx === -1 ? "." : normalised.slice(0, idx); -} - -// §6.12 normalizeBaseurl -function normalizeBaseurl(raw) { - let baseurl = String(raw ?? "").replace(/\/+$/, ""); - if (baseurl && !baseurl.startsWith("/")) baseurl = "/" + baseurl; - return baseurl; -} - -// §6.13 escapeRegExp -function escapeRegExp(s) { - return s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); -} - -// Hoist the per-file-dir inner cache so the per-match cost is one -// map lookup. -function getPageCache(resultCache, fileDir) { - let pageCache = resultCache.get(fileDir); - if (!pageCache) { - pageCache = new Map(); - resultCache.set(fileDir, pageCache); } - return pageCache; -} - -// --------------------------------------------------------------------------- -// §D HTML rewrite pipeline -// --------------------------------------------------------------------------- - -const SEO_BLOCK_RE = /<!-- Begin Jekyll SEO tag.*?<!-- End Jekyll SEO tag -->/s; -const TITLE_RE = /<title>.*?<\/title>/s; - -// §6.2 stripSeo -- drop the jekyll-seo-tag block, keep its <title>. -function stripSeo(html) { - if (!html.includes("<!-- Begin Jekyll SEO tag")) return html; - return html.replace(SEO_BLOCK_RE, (block) => { - const titleMatch = block.match(TITLE_RE); - return titleMatch ? titleMatch[0] : ""; - }); -} - -// Combined regex: three top-level alternatives -- <code> block, <pre> -// block, or a real href|src attribute carrying either an absolute or -// page-relative URL. The code/pre alternatives consume their bodies -// atomically so href/src matches inside code samples are skipped. -const HTML_COMBINED_RE = /<code\b[^>]*>[\s\S]*?<\/code>|<pre\b[^>]*>[\s\S]*?<\/pre>|\b(href|src)=(["'])(\/(?!\/)[^"']*|(?![#/]|[a-zA-Z][a-zA-Z0-9+.\-]*:)[^"']+)\2/g; - -// §6.6 rewriteHtml -- single regex pass over the HTML. -function rewriteHtml(html, fileDir, fileSegs, sitePaths, caches, baseurl) { - let misses = 0; - const pageCache = getPageCache(caches.result, fileDir); - - const rewritten = html.replace(HTML_COMBINED_RE, (match, attrName, quote, rawUrl) => { - if (attrName === undefined) { - // <code> or <pre> block; leave verbatim. - return match; - } - let rel = pageCache.get(rawUrl); - if (rel === undefined) { - rel = rawUrl.startsWith("/") - ? computeRelative(rawUrl, fileSegs, sitePaths, caches, baseurl) - : computeRelUrl(rawUrl, fileSegs, sitePaths); - pageCache.set(rawUrl, rel); - } - if (rel === null) { - misses++; - return match; - } - if (rel === rawUrl) { - // File already correct at the relative path (rare). - return match; - } - return `${attrName}=${quote}${rel}${quote}`; - }); - - return { rewritten, misses }; -} - -const JTD_SCRIPT_TAG_RE = /<script\s+src="([^"]*)just-the-docs\.js"/; - -// §6.8 injectSearchSetup -- two <script> tags before the just-the-docs.js -// tag carry the per-page relative site-root and the lunr-index data load. -function injectSearchSetup(html, fileSegs) { - return html.replace(JTD_SCRIPT_TAG_RE, (match, prefix) => { - const siteRoot = fileSegs.length === 0 ? "" : "../".repeat(fileSegs.length); - return `<script>window.OFFLINE_SITE_ROOT="${siteRoot}";</script>\n` + - `<script src="${prefix}search-data.js"></script>` + - match; - }); -} - -// --------------------------------------------------------------------------- -// §E CSS rewrite pipeline -// --------------------------------------------------------------------------- - -const CSS_URL_RE = /url\(\s*(["']?)(\/(?!\/)[^"'()\s]*)\1\s*\)/g; - -// §6.7 rewriteCss -- url(/...) → page-relative. -function rewriteCss(css, fileDir, fileSegs, sitePaths, caches, baseurl) { - let misses = 0; - const pageCache = getPageCache(caches.result, fileDir); - - const rewritten = css.replace(CSS_URL_RE, (match, quote, rawUrl) => { - let rel = pageCache.get(rawUrl); - if (rel === undefined) { - rel = computeRelative(rawUrl, fileSegs, sitePaths, caches, baseurl); - pageCache.set(rawUrl, rel); - } - if (rel === null) { - misses++; - return match; - } - return `url(${quote}${rel}${quote})`; - }); - - return { rewritten, misses }; + walk(""); + return out; } // --------------------------------------------------------------------------- From 855b8eb5f8eccf648145b2d5abea25b0eb8c6b03 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sat, 30 May 2026 21:31:07 +0200 Subject: [PATCH 13/72] Phase II: expand dispatch deps, compute sitePaths, expand SAB payload --- builder/tbdocs.mjs | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 8c91ec46..4e65af8b 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -37,7 +37,8 @@ import { writePhase, prepareDestination } from "./write.mjs"; import { writeRedirects, deriveRedirectStubs } from "./redirects.mjs"; import { writeSitemap, deriveSitemapUrls } from "./sitemap.mjs"; import { writeSearchData } from "./search.mjs"; -import { writeOffline } from "./offline.mjs"; +import { writeOffline, enumerateVendoredThemeAssets } from "./offline.mjs"; +import { buildSitePathsSync } from "./offline-rewrite.mjs"; import { writePdf } from "./pdf.mjs"; import { packShared } from "./sab-broadcast.mjs"; @@ -174,6 +175,7 @@ const TASKS = { } emit("write", out); emit("writePdf", out); + emit("dispatch", out); }, }, @@ -308,7 +310,7 @@ const TASKS = { execute(_, ctx, state) { return { stubs: deriveRedirectStubs(state.pages, state.site) }; }, - submit(out, emit) { emit("writeAux", out); }, + submit(out, emit) { emit("writeAux", out); emit("dispatch", out); }, }, deriveSitemap: { @@ -327,10 +329,22 @@ const TASKS = { // chrome), resolveBookChapters (identity-critical page refs), and // buildInfo (git metadata for the footer). dispatch: { - expected: ["buildInit", "resolveBookChapters", "buildInfo"], + expected: ["buildInit", "resolveBookChapters", "buildInfo", "mermaid", "deriveRedirects"], runOnMain: true, - execute({ buildInit: { initData }, buildInfo: { buildInfo } }, ctx, state) { + execute({ buildInit: { initData }, buildInfo: { buildInfo }, mermaid: { mermaidStats }, deriveRedirects: { stubs } }, ctx, state) { + void mermaidStats; // dependency signal only -- static files already appended in mermaid.submit const chunks = chunkPages(state.pages, ctx.workerCount); + const excludePatterns = Array.isArray(state.site.config?.offline_exclude) + ? state.site.config.offline_exclude.map(String) + : []; + const themeAssetRels = [ + ...enumerateVendoredThemeAssets(), + "assets/css/tb-highlight.css", + "assets/css/just-the-docs-combined.css", + ]; + const sitePaths = buildSitePathsSync(state.pages, state.staticFiles, excludePatterns, stubs, themeAssetRels); + state.sitePaths = sitePaths; + const skipOffline = ctx.opts.skipOffline ?? (state.site.config.also_build_offline === false); const shared = { siteData: { config: state.site.config, @@ -342,6 +356,9 @@ const TASKS = { linkTablesData: state.site.linkTablesSerialized, staticFilesArr: state.staticFiles.map(f => f.srcRel), baseurl: String(state.site.config.baseurl || ""), + sitePathsArr: [...sitePaths], + offlineExcludePatterns: excludePatterns, + skipOffline, }; const sharedSAB = packShared(shared); return { chunks, sharedSAB }; From 831d0f3b5fed6d17fa2da19aa34d4aff3648f748 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sat, 30 May 2026 21:44:06 +0200 Subject: [PATCH 14/72] Phase III: run offline rewrite in render workers --- builder/cpu-worker.mjs | 48 +++++++++++++++++++++++++++++++++++++++++- builder/tbdocs.mjs | 2 ++ 2 files changed, 49 insertions(+), 1 deletion(-) diff --git a/builder/cpu-worker.mjs b/builder/cpu-worker.mjs index 50377266..46119f19 100644 --- a/builder/cpu-worker.mjs +++ b/builder/cpu-worker.mjs @@ -14,6 +14,10 @@ import { createMarkdownIt, buildLinkTables, import { templatePhase } from "./template.mjs"; import { unpackShared } from "./sab-broadcast.mjs"; +import { deriveOfflinePage, deriveOfflinePageCached, + sliceNavBlock, normalizeBaseurl, + posixDirname } from "./offline-rewrite.mjs"; + // Start WASM init immediately, do NOT await. Module evaluation finishes // synchronously so the parentPort.on('message') dispatcher is installed // before the pool sends any work. Only the `render` handler awaits. @@ -35,7 +39,8 @@ const handlers = { async render({ inputs }) { const { sharedSAB, chunk } = inputs; const { siteData, initData, linkTablesData, staticFilesArr, - baseurl, buildInfo } = unpackShared(sharedSAB); + baseurl, buildInfo, sitePathsArr, + skipOffline } = unpackShared(sharedSAB); const highlighter = await highlighterP; const linkTables = reconstructLinkTables(linkTablesData); @@ -46,12 +51,53 @@ const handlers = { await renderPhase(chunk, site); await templatePhase(chunk, site, initData); + // Offline rewrite pass (Phase III of PLAN-scheduler-offline.md). + // Runs the per-page URL rewrite inside the worker so it + // parallelises across CPUs. When skipOffline is true the entire + // pass is skipped — no Set construction, no rewriting. + if (!skipOffline) { + const sitePaths = new Set(sitePathsArr); + const caches = { rawResolution: new Map(), seg: new Map(), result: new Map() }; + const offlineState = { sitePaths, caches, baseurl: normalizeBaseurl(baseurl) }; + + // Nav-cache pre-pass: group chunk pages by dest dir, derive the + // first page per dir, cache nav block slices. Same logic as + // writeOfflinePages in offline.mjs. + const writable = chunk.filter(p => p.html !== undefined); + const byDir = new Map(); + for (const p of writable) { + const destDir = posixDirname(p.destPath); + let g = byDir.get(destDir); + if (!g) { g = []; byDir.set(destDir, g); } + g.push(p); + } + const navCache = new Map(); + for (const [destDir, group] of byDir) { + const first = group[0]; + const input = sliceNavBlock(first.html); + if (input === null) continue; + const { html: rendered } = deriveOfflinePage(first, offlineState); + const output = sliceNavBlock(rendered); + if (output === null) continue; + navCache.set(destDir, { input, output }); + } + offlineState.navCache = navCache; + + for (const p of writable) { + const { html, misses } = deriveOfflinePageCached(p, offlineState); + p.offlineHtml = html; + p.offlineMisses = misses; + } + } + // book-combined pages have renderedContent but no html (Phase 8 // handles them from renderedContent); send html: undefined for those. return chunk.map(p => ({ destPath: p.destPath, renderedContent: p.renderedContent, html: p.html, + offlineHtml: p.offlineHtml, + offlineMisses: p.offlineMisses, })); }, }; diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 4e65af8b..25d04ecb 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -388,6 +388,8 @@ const TASKS = { if (!p) continue; p.renderedContent = r.renderedContent; if (r.html !== undefined) p.html = r.html; + if (r.offlineHtml !== undefined) p.offlineHtml = r.offlineHtml; + if (r.offlineMisses !== undefined) p.offlineMisses = r.offlineMisses; } emit("renderJoin", renderOut); }, From 824992b900f65df0e4af1b8a23aceb9d9ab86d58 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sat, 30 May 2026 21:48:28 +0200 Subject: [PATCH 15/72] Phase IV: switch writeOffline to pre-computed HTML --- builder/offline.mjs | 26 +++++++++++++++++++------- builder/tbdocs.mjs | 2 ++ 2 files changed, 21 insertions(+), 7 deletions(-) diff --git a/builder/offline.mjs b/builder/offline.mjs index a26c516b..2289a167 100644 --- a/builder/offline.mjs +++ b/builder/offline.mjs @@ -110,13 +110,13 @@ function makeTimer() { // §A Top-level orchestration // --------------------------------------------------------------------------- -export async function writeOffline(pages, staticFiles, site, destRoot, { auxStats, profileOffline = false } = {}) { +export async function writeOffline(pages, staticFiles, site, destRoot, { auxStats, profileOffline = false, precomputed = false, sitePaths } = {}) { if (!destRoot) { throw new Error("writeOffline requires a destRoot"); } const stubs = auxStats?.redirects?.stubs ?? []; - const state = await buildOfflineState(pages, staticFiles, site, destRoot, { stubs }); + const state = await buildOfflineState(pages, staticFiles, site, destRoot, { stubs, sitePaths }); const deps = { ...state, offlineRoot: destRoot + OFFLINE_SUFFIX, @@ -161,7 +161,7 @@ export async function writeOffline(pages, staticFiles, site, destRoot, { auxStat const t0Pages = Date.now(); let dPages = 0, dRedirects = 0, dStatics = 0, dThemes = 0; const branches = [ - writeOfflinePages(pages, deps).then(() => { dPages = Date.now() - t0Pages; }), + writeOfflinePages(pages, deps, { precomputed }).then(() => { dPages = Date.now() - t0Pages; }), writeOfflineRedirects(auxStats?.redirects?.stubs ?? [], deps).then(() => { dRedirects = Date.now() - t0Pages; }), copyOfflineStatics(staticFiles, deps).then(() => { dStatics = Date.now() - t0Pages; }), copyOfflineThemeAssets(deps).then(() => { dThemes = Date.now() - t0Pages; }), @@ -174,7 +174,7 @@ export async function writeOffline(pages, staticFiles, site, destRoot, { auxStat console.log(` offline.themeAssets (concurrent): ${dThemes} ms`); } else { await Promise.all([ - writeOfflinePages(pages, deps), + writeOfflinePages(pages, deps, { precomputed }), writeOfflineRedirects(auxStats?.redirects?.stubs ?? [], deps), copyOfflineStatics(staticFiles, deps), copyOfflineThemeAssets(deps), @@ -191,13 +191,13 @@ export async function writeOffline(pages, staticFiles, site, destRoot, { auxStat // `stubs` (optional) is the redirect-stub list from Phase 6; their // destinations land in sitePaths so a page-relative link like // `LBound` resolves through the stub at `tB/Core/LBound.html`. -export async function buildOfflineState(pages, staticFiles, site, destRoot, { stubs = [] } = {}) { +export async function buildOfflineState(pages, staticFiles, site, destRoot, { stubs = [], sitePaths } = {}) { const excludePatterns = Array.isArray(site.config?.offline_exclude) ? site.config.offline_exclude.map(String) : []; return { destRoot, - sitePaths: await buildSitePaths(pages, staticFiles, destRoot, excludePatterns, stubs), + sitePaths: sitePaths ?? await buildSitePaths(pages, staticFiles, destRoot, excludePatterns, stubs), caches: { rawResolution: new Map(), seg: new Map(), @@ -244,8 +244,20 @@ async function setupOfflineDest(offlineRoot) { // its pre-rewrite nav block matches the cached `input` byte-for-byte. // On miss we fall back to the full rewrite with a warning -- the // cache is purely an optimisation, never a correctness dependency. -async function writeOfflinePages(pages, deps) { +async function writeOfflinePages(pages, deps, { precomputed = false } = {}) { const { offlineRoot } = deps; + + if (precomputed) { + const writable = pages.filter(p => p.offlineHtml !== undefined); + await runLimited(writable, LIMIT, async (page) => { + const dest = path.join(offlineRoot, page.destPath); + await writeFileMkdirp(dest, page.offlineHtml); + deps.counters.html += 1; + deps.counters.unresolved += page.offlineMisses ?? 0; + }); + return; + } + const writable = pages.filter(p => p.html !== undefined); // Pre-pass: group pages by destination dir, render the first page diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 25d04ecb..bbcc21d4 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -473,6 +473,8 @@ const TASKS = { const auxStats = { redirects: redirectStats, sitemap: sitemapStats, search: searchStats }; return writeOffline(state.pages, state.staticFiles, state.site, ctx.destRoot, { auxStats, + precomputed: true, + sitePaths: state.sitePaths, profileOffline: ctx.opts.profileOffline, }); }, From 3a6d5740f8dd5933b9c09cc7a87697d71589d4d9 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sat, 30 May 2026 22:12:50 +0200 Subject: [PATCH 16/72] Phase V: documentation for offline-rewrite-in-workers --- builder/PLAN-scheduler.md | 62 +++++++++++++++++------- builder/offline.mjs | 10 ++-- docs/Documentation/Builder.md | 13 ++--- docs/Documentation/Pipeline-Stages.md | 41 +++++++++++++--- docs/assets/images/mmd/scheduler-dag.mmd | 2 + docs/assets/images/mmd/scheduler-dag.svg | 2 +- 6 files changed, 95 insertions(+), 35 deletions(-) diff --git a/builder/PLAN-scheduler.md b/builder/PLAN-scheduler.md index 25eb0f2b..bcad6679 100644 --- a/builder/PLAN-scheduler.md +++ b/builder/PLAN-scheduler.md @@ -250,7 +250,7 @@ Main spine (sequential, on M): │ │ ↓ │ │ └─→ buildInit │ │ │ ↓ │ │ │ - └──────────┴─→ dispatch ◄── buildInfo joins here + └──────────┴─→ dispatch ◄── buildInfo, mermaid, deriveRedirects join here │ Render fan-out (workers, concurrent): │ ┌────────────────────────────────────────────────────┘ @@ -291,7 +291,7 @@ Write fence: │ scss [W], mermaid [W], prepDest [M] join here ``` Edges into `dispatch`: `buildInit`, `resolveBookChapters`, -`buildInfo`. +`buildInfo`, `mermaid`, `deriveRedirects`. Edges into `write`: `renderJoin`, `scss`, `mermaid`, `prepDest`. Edges into `searchData`: `renderJoin`, `prepDest`. Edges into `writePdf`: `renderJoin`, `mermaid`. @@ -332,9 +332,9 @@ The render fan-out is the only place where pages cross the worker boundary. The pattern: - Each `render:i` task receives a chunk of pages (worker's clone). -- The worker mutates its local pages with `renderedContent` and `html`. +- The worker mutates its local pages with `renderedContent`, `html`, and (when `!skipOffline`) `offlineHtml` + `offlineMisses`. - The task returns a **delta**: an array of - `[{ destPath, renderedContent, html }]` -- only the changed fields, + `[{ destPath, renderedContent, html, offlineHtml, offlineMisses }]` -- only the changed fields, keyed by `destPath`. - `render:i.submit()` walks the delta on the main thread, looks up each page via `state.pageByDest`, and assigns the fields onto the @@ -433,6 +433,7 @@ const TASKS = { if (!known.has(f.srcRel)) state.staticFiles.push(f); } emit("write", out); + emit("dispatch", out); }, }, @@ -555,7 +556,10 @@ const TASKS = { // page.html, so the derive can run before template. return { stubs: deriveRedirectStubs(state.pages, state.site) }; }, - submit(out, emit) { emit("writeAux", out); }, + submit(out, emit) { + emit("writeAux", out); + emit("dispatch", out); + }, }, deriveSitemap: { @@ -568,12 +572,19 @@ const TASKS = { }, dispatch: { - expected: ["buildInit", "resolveBookChapters", "buildInfo"], + expected: ["buildInit", "resolveBookChapters", "buildInfo", "mermaid", "deriveRedirects"], runOnMain: true, - execute({ buildInit: { initData }, buildInfo: { buildInfo } }, ctx, state) { + execute({ buildInit: { initData }, buildInfo: { buildInfo }, deriveRedirects: { stubs } }, ctx, state) { // Read pages directly from state.pages -- main-thread access, // no need to ship them through the input map. const chunks = chunkPages(state.pages, ctx.workerCount); + const excludePatterns = state.site.config.offline_exclude ?? []; + const skipOffline = /* from config / CLI opts */ false; + const sitePaths = buildSitePathsSync( + state.pages, state.staticFiles, excludePatterns, stubs, + enumerateVendoredThemeAssets()); + state.sitePaths = sitePaths; + const shared = { siteData: { config: state.site.config, @@ -581,13 +592,16 @@ const TASKS = { seoLogoUrl: state.site.seoLogoUrl, }, initData, buildInfo, - linkTablesData: state.site.linkTablesSerialized, - staticFilesArr: state.staticFiles.map(f => f.srcRel), - baseurl: String(state.site.config.baseurl || ""), + linkTablesData: state.site.linkTablesSerialized, + staticFilesArr: state.staticFiles.map(f => f.srcRel), + baseurl: String(state.site.config.baseurl || ""), + sitePathsArr: [...sitePaths], + offlineExcludePatterns: excludePatterns, + skipOffline: Boolean(skipOffline), }; // Pack the shared payload into a SharedArrayBuffer so each // postMessage sends a SAB reference (shared memory) instead of - // structured-cloning ~286 KB per worker. + // structured-cloning ~310--330 KB per worker. const sharedSAB = packShared(shared); return { chunks, sharedSAB }; }, @@ -615,7 +629,9 @@ const TASKS = { const p = state.pageByDest.get(r.destPath); if (!p) continue; p.renderedContent = r.renderedContent; - if (r.html !== undefined) p.html = r.html; + if (r.html !== undefined) p.html = r.html; + if (r.offlineHtml !== undefined) p.offlineHtml = r.offlineHtml; + if (r.offlineMisses !== undefined) p.offlineMisses = r.offlineMisses; } emit("renderJoin", renderOut); }, @@ -961,10 +977,13 @@ const handlers = { // book-combined pages have renderedContent but no html (Phase 8 // handles them from renderedContent); send html: undefined for those. + // offlineHtml and offlineMisses are undefined when skipOffline is true. return chunk.map(p => ({ destPath: p.destPath, renderedContent: p.renderedContent, html: p.html, + offlineHtml: p.offlineHtml, + offlineMisses: p.offlineMisses, })); }, }; @@ -1059,12 +1078,15 @@ mutations are immediately visible to downstream main-thread tasks. ### SharedArrayBuffer broadcast (Phase 4) The render fan-out's shared payload (`siteData + initData + -linkTablesData + staticFilesArr + baseurl + buildInfo`, ~286 KB) is -JSON-serialized once on the main thread into a SharedArrayBuffer via -`sab-broadcast.mjs`'s `packShared()`. Each render task receives the -SAB reference (shared memory, not cloned) alongside its per-worker +linkTablesData + staticFilesArr + baseurl + buildInfo + +sitePathsArr + offlineExcludePatterns + skipOffline`, ~310--330 KB) +is JSON-serialized once on the main thread into a SharedArrayBuffer +via `sab-broadcast.mjs`'s `packShared()`. Each render task receives +the SAB reference (shared memory, not cloned) alongside its per-worker chunk. Workers call `unpackShared()` to deserialize independently and -in parallel. Measured saving: ~8 ms per build (fan-out drops from +in parallel; each builds a `new Set(sitePathsArr)` to drive the +inline offline URL rewrite. Measured saving at Phase 4 baseline +(~286 KB, pre-offline fields): ~8 ms per build (fan-out drops from ~19 ms to ~9 ms). ## Error handling @@ -1211,7 +1233,11 @@ writeOffline: { runOnMain: true, async execute(_, ctx, state) { return writeOffline(state.pages, state.staticFiles, state.site, - ctx.destRoot, { /* ... */ }); + ctx.destRoot, { + auxStats, + precomputed: true, + sitePaths: state.sitePaths, + }); }, submit() { /* terminal */ }, }, diff --git a/builder/offline.mjs b/builder/offline.mjs index 2289a167..a1cbb365 100644 --- a/builder/offline.mjs +++ b/builder/offline.mjs @@ -4,9 +4,13 @@ // and docs/_plugins/offlinify.rb for the canonical Jekyll reference. // // One entry point: writeOffline(pages, staticFiles, site, destRoot, -// { auxStats }). Pure-compute derive helpers are in offline-rewrite.mjs -// and re-exported from here for `_diff.mjs` / `_triage.mjs` backward -// compatibility. +// { auxStats, precomputed, sitePaths }). When precomputed is true, +// per-page HTML was already derived by render workers and stored on +// page.offlineHtml; writeOfflinePages writes those directly (I/O only). +// sitePaths, when provided, skips the async _site/assets/ walk in +// buildOfflineState. Pure-compute derive helpers are in +// offline-rewrite.mjs and re-exported from here for `_diff.mjs` / +// `_triage.mjs` backward compatibility. // // Internal sections: // diff --git a/docs/Documentation/Builder.md b/docs/Documentation/Builder.md index 2f32d083..a5e82e62 100644 --- a/docs/Documentation/Builder.md +++ b/docs/Documentation/Builder.md @@ -77,7 +77,7 @@ One entry point, ~17 production modules. The content model is fixed (markdown + | 4 | `template.mjs` + `compress.mjs` | Wrap in layout, anchor headings, compress whitespace | ~200 ms | | 5 | `write.mjs` | Write `_site/` | ~400 ms | | 6 | `redirects.mjs` / `sitemap.mjs` / `search.mjs` | Redirect stubs, sitemap.xml, search-data.json, robots.txt | ~100 ms | -| 7 | `offline.mjs` | URL-rewritten copy to `_site-offline/` | ~1,000 ms | +| 7 | `offline.mjs` | URL-rewritten copy to `_site-offline/` | ~300 ms | | 8 | `pdf.mjs` + `book.mjs` | Sparse `_site-pdf/` tree (book.html + CSS + images) | ~150 ms | Phases 9, 10, and 11 are historical: Phase 9 was a no-output QoL pass, Phase 10 retired Jekyll, Phase 11 introduces the output-changing parity updates. None adds a runtime step. Phase 12 adds the `--serve` dev-server mode (a separate lifecycle, not a build phase; writes to `docs/_serve/` and skips the offline + PDF passes by default so the rebuild loop stays under one second). The per-phase `PLAN-N.md` files retain the implementation history. @@ -90,21 +90,22 @@ The build pipeline is driven by a task-graph **scheduler** that models the pipel **[M]** = runs on main thread; **[W]** = dispatched to a worker thread. Worker tasks overlap with the main-thread spine and with each other; main-thread tasks run serially within the main thread but can overlap with worker tasks. -Three structural wins over the earlier serial baseline: +Four structural wins over the earlier serial baseline: 1. **Seed tasks overlap with the main spine.** `scss` (~700 ms), `mermaid`, `buildInfo`, and `prepDest` (destination clean + recreate) run concurrently with the main-thread spine (discover → nav → markdownInit → seo → loadData → resolveBookChapters + buildInit). The spine takes ~250 ms total, so ~250 ms of `scss` hides behind it. 2. **Render + template fans out across CPUs.** The single-threaded render + template work (~3.5 s combined) splits into N page chunks, each dispatched to a worker that runs both `renderPhase` and `templatePhase` on its slice. On a 4-core machine this compresses to ~875 ms; on 8 cores, ~440 ms. 3. **`writePdf` and `searchData` overlap with the write chain.** Both depend on `renderJoin` rather than `write`, so they start as soon as pages are rendered. `writePdf` (+ `mermaid`) sources CSS directly from in-memory state and the static-file inventory --- it never reads from `_site/`. `searchData` (+ `prepDest`) reads only in-memory `renderedContent`. Both run in parallel with `write`; `writeAux` joins `write` + `searchData` before starting. All are `runOnMain` but their I/O-dominated `await` gaps interleave via cooperative async concurrency. +4. **Offline HTML rewrite runs inside the render fan-out.** Each render worker computes `offlineHtml` for its page chunk after `templatePhase`, storing the pre-computed result in the page delta. `writeOffline` (Phase 7) reads these pre-computed strings directly and drops from ~700 ms (CPU + I/O) to ~300 ms (I/O only). The `dispatch` task waits for `mermaid` and `deriveRedirects` (in addition to the spine tasks) so it has the full `sitePaths` set available to pack into the SharedArrayBuffer before dispatching workers. ### Shared state and page deltas -The scheduler owns a `SharedState` instance that carries the master `pages[]`, `staticFiles[]`, `site` object, and a `pageByDest` lookup map. Main-thread tasks mutate `state.pages` directly --- no marshalling, no delta merge. Worker tasks receive structured-clone copies of their inputs and return **page deltas** (arrays of `{ destPath, renderedContent, html }`); the task's `submit()` runs on the main thread and merges each delta entry into the master page via `state.pageByDest`. +The scheduler owns a `SharedState` instance that carries the master `pages[]`, `staticFiles[]`, `site` object, and a `pageByDest` lookup map. Main-thread tasks mutate `state.pages` directly --- no marshalling, no delta merge. Worker tasks receive structured-clone copies of their inputs and return **page deltas** (arrays of `{ destPath, renderedContent, html, offlineHtml, offlineMisses }`); the task's `submit()` runs on the main thread and merges each delta entry into the master page via `state.pageByDest`. After `discover.submit()` builds the master `pages[]`, no task ever replaces the array --- only mutates it in place. This preserves object identity across phases, which is critical for `resolveBookChapters` (stores `Page` references into `bookData._chapters`) and `writePdf` (reads `renderedContent` from those same objects after the render fan-out fills them in). ### Render fan-out -The `dispatch` task slices `state.pages` into N chunks (one per available CPU), packs the shared per-build payload (site config, SEO metadata, pre-rendered sidebar HTML, serialized link tables, static file set, build info) into a **SharedArrayBuffer** via `sab-broadcast.mjs`'s `packShared()`, and dynamically registers N `render:i` worker tasks plus a `renderJoin` barrier. Each `render:i` receives the SAB reference (shared memory, not cloned) alongside its per-worker page chunk, deserializes the shared payload independently, initializes its own markdown-it + Shiki instance, runs `renderPhase` + `templatePhase` over its slice, and returns the per-page deltas. The `renderJoin` barrier waits for all N render tasks before unblocking `write`. +The `dispatch` task waits for `buildInit`, `resolveBookChapters`, `buildInfo`, `mermaid`, and `deriveRedirects` before running. It uses the `mermaid` and `deriveRedirects` outputs to compute a `sitePaths` set (via `buildSitePathsSync` from `offline-rewrite.mjs`, using vendored theme assets rather than walking `_site/assets/`), then packs the shared per-build payload (site config, SEO metadata, pre-rendered sidebar HTML, serialized link tables, static file set, build info, site-paths array, offline exclude patterns, and skip-offline flag) into a **SharedArrayBuffer** via `sab-broadcast.mjs`'s `packShared()`, and dynamically registers N `render:i` worker tasks plus a `renderJoin` barrier. Each `render:i` receives the SAB reference (shared memory, not cloned) alongside its per-worker page chunk, deserializes the shared payload independently, initializes its own markdown-it + Shiki instance, runs `renderPhase` + `templatePhase` over its slice, then computes `offlineHtml` per page (unless `skipOffline`), and returns the per-page deltas. The `renderJoin` barrier waits for all N render tasks before unblocking `write`. ### Data transfer strategy @@ -348,13 +349,13 @@ The image-path collector folds into `assembleBook`'s per-chapter emit (Phase 9 - **`scss`** --- calls `compileScss(ctx.srcRoot)`. - **`mermaid`** --- calls `regenerateMermaid(ctx.srcRoot)`. - **`buildInfo`** --- calls `captureBuildInfo()`. -- **`render`** --- the render + template hot path. Deserializes the shared payload from the SharedArrayBuffer, awaits the module-scope Shiki WASM init (started eagerly at import time, before any message arrives), builds a fresh markdown-it instance from the serialized link tables, runs `renderPhase` + `templatePhase` over the page chunk, and returns per-page deltas (`{ destPath, renderedContent, html }`). +- **`render`** --- the render + template + offline-rewrite hot path. Deserializes the shared payload from the SharedArrayBuffer (including the site-paths array for offline rewriting), awaits the module-scope Shiki WASM init (started eagerly at import time, before any message arrives), builds a fresh markdown-it instance from the serialized link tables, runs `renderPhase` + `templatePhase` over the page chunk, then (unless `skipOffline`) computes `offlineHtml` per page via `deriveOfflinePageCached` from `offline-rewrite.mjs`, and returns per-page deltas (`{ destPath, renderedContent, html, offlineHtml, offlineMisses }`). Each worker starts its own `initHighlighter()` call at module scope without awaiting it, so the WASM init overlaps with worker spawn and the main-thread spine. Only the `render` handler awaits the result; the other three handlers service tasks while Shiki is still loading. ### [sab-broadcast.mjs](https://github.com/twinbasic/documentation/blob/main/builder/sab-broadcast.mjs) --- SharedArrayBuffer broadcast -~15 lines. `packShared(obj)` JSON-serializes an object, encodes to UTF-8, and copies into a `SharedArrayBuffer`. `unpackShared(sab)` reverses the process. The render fan-out's shared payload (~286 KB of site config, pre-rendered sidebar HTML, serialized link tables, static file set, and build info) is packed once on the main thread; each worker receives the SAB reference (shared memory, not cloned) and deserializes independently. Measured saving: ~55% reduction in fan-out overhead (~8 ms per build). +~15 lines. `packShared(obj)` JSON-serializes an object, encodes to UTF-8, and copies into a `SharedArrayBuffer`. `unpackShared(sab)` reverses the process. The render fan-out's shared payload (~310--330 KB of site config, pre-rendered sidebar HTML, serialized link tables, static file set, build info, site-paths array for offline rewriting, offline exclude patterns, and a skip-offline flag) is packed once on the main thread; each worker receives the SAB reference (shared memory, not cloned) and deserializes independently. Measured saving at Phase 4 baseline (~286 KB): ~55% reduction in fan-out overhead (~8 ms per build). ## Asset layout diff --git a/docs/Documentation/Pipeline-Stages.md b/docs/Documentation/Pipeline-Stages.md index cbcd5b10..29656cff 100644 --- a/docs/Documentation/Pipeline-Stages.md +++ b/docs/Documentation/Pipeline-Stages.md @@ -45,6 +45,8 @@ The pipeline passes two mutable data structures through every stage. | `seoIsHome` | Phase 2 (seo) | `boolean` | `true` when the page's permalink is a known home-page URL (e.g. `/`). | | `renderedContent` | Phase 3 | `string` | HTML body produced by markdown-it. Not yet wrapped in the site layout. | | `html` | Phase 4 | `string` | Complete HTML document, ready to write to disk. Absent on `layout: book-combined` pages, which Phase 8 owns. | +| `offlineHtml` | Phase 4 (render worker) | `string\|undefined` | Pre-computed offline HTML for the page, with all absolute URLs rewritten to page-relative paths. Set by the render worker after `templatePhase`. `undefined` when `skipOffline` is true or the page is `layout: book-combined`. | +| `offlineMisses` | Phase 4 (render worker) | `number\|undefined` | Count of URLs that could not be resolved during the per-page offline rewrite. `undefined` when `skipOffline` is true. Non-zero values are logged as warnings during `writeOffline`. | ### Site object (`site`) @@ -482,11 +484,16 @@ writeOffline( staticFiles: StaticFile[], site: object, destRoot: string, - { auxStats?: object, profileOffline?: boolean } + { + auxStats?: object, + profileOffline?: boolean, + precomputed?: boolean, + sitePaths?: Set<string> + } ): Promise<{ html, css, redirects, statics, assets, excluded, unresolved }> ``` -Reads every file written by Phases 5 and 6, rewrites absolute URLs to relative paths, and writes to `<destRoot>-offline/`. Patches `just-the-docs.js` via AST (acorn) to replace `navLink` and `initSearch` with offline-compatible implementations. Writes `search-data.js`, which wraps the search index as a `window.SEARCH_DATA` assignment so offline search works under `file://` (browsers block `XMLHttpRequest` there). `offline_exclude` patterns apply to pages, static files, and theme assets alike; `search-data.json` is listed in `offline_exclude` and is absent from the offline tree --- only the `.js` wrapper is present. +Reads every file written by Phases 5 and 6, rewrites absolute URLs to relative paths, and writes to `<destRoot>-offline/`. When `precomputed: true` (set by the scheduler), skips per-page CPU rewriting and writes `page.offlineHtml` directly (I/O only); when absent or false, derives offline HTML on the main thread (legacy path, used by diff tools). When `sitePaths` is provided, skips the `_site/assets/` walk in `buildOfflineState`. Patches `just-the-docs.js` via AST (acorn) to replace `navLink` and `initSearch` with offline-compatible implementations. Writes `search-data.js`, which wraps the search index as a `window.SEARCH_DATA` assignment so offline search works under `file://` (browsers block `XMLHttpRequest` there). `offline_exclude` patterns apply to pages, static files, and theme assets alike; `search-data.json` is listed in `offline_exclude` and is absent from the offline tree --- only the `.js` wrapper is present. **Reads:** all files under `<destRoot>` (online tree), `auxStats.redirects` (redirect stub list from Phase 6). **Writes:** all files to `<destRoot>-offline/`. @@ -495,14 +502,34 @@ Reads every file written by Phases 5 and 6, rewrites absolute URLs to relative p | Symbol | Signature | Description | |---|---|---| -| `writeOffline` | `(pages, staticFiles, site, destRoot, opts) → Promise<stats>` | Main entry point. | -| `buildOfflineState` | `(pages, staticFiles, site, destRoot, { stubs? }) → Promise<OfflineState>` | Constructs the state object (site-path set, resolution caches, per-directory nav caches) used by all offline derivation functions. | -| `deriveOfflinePage` | `(page: Page, state: OfflineState) → string` | Rewrites one page's HTML for offline use. | -| `deriveOfflineRedirect` | `(stub, state: OfflineState) → string` | Rewrites a redirect stub's HTML for offline use. | -| `deriveOfflineCss` | `(cssIn: string, themeRel: string, state: OfflineState) → string` | Rewrites `url()` references in a CSS file to page-relative paths. | +| `writeOffline` | `(pages, staticFiles, site, destRoot, opts) → Promise<stats>` | Main entry point. `opts.precomputed` (bool) switches to I/O-only mode using `page.offlineHtml`; `opts.sitePaths` skips the `_site/assets/` walk. | +| `buildOfflineState` | `(pages, staticFiles, site, destRoot, { stubs?, sitePaths? }) → Promise<OfflineState>` | Constructs the state object (site-path set, resolution caches, per-directory nav caches) used by all offline derivation functions. When `sitePaths` is provided, skips the async `_site/assets/` walk. | +| `deriveOfflinePage` | `(page: Page, state: OfflineState) → string` | Rewrites one page's HTML for offline use. Defined in `offline-rewrite.mjs`; re-exported for backward compatibility. | +| `deriveOfflineRedirect` | `(stub, state: OfflineState) → string` | Rewrites a redirect stub's HTML for offline use. Defined in `offline-rewrite.mjs`; re-exported for backward compatibility. | +| `deriveOfflineCss` | `(cssIn: string, themeRel: string, state: OfflineState) → string` | Rewrites `url()` references in a CSS file to page-relative paths. Defined in `offline-rewrite.mjs`; re-exported for backward compatibility. | | `deriveOfflineJtdJs` | `(src: string) → string` | Patches `just-the-docs.js` via AST: replaces `navLink` and `initSearch` with offline-compatible implementations. A parse failure at build time is a signal that re-extraction produced unreadable source. | | `deriveOfflineSearchDataJs` | `(jsonBytes: Buffer) → string` | Wraps `search-data.json` as `window.SEARCH_DATA = …` and minifies it. | +### `offline-rewrite.mjs` + +Pure-compute rewrite helpers extracted from `offline.mjs` so they can be imported by `cpu-worker.mjs` without pulling in `node:fs` or `acorn`. All symbols are also re-exported from `offline.mjs` for backward compatibility with `_diff.mjs` and `_triage.mjs`. + +**All exports** + +| Symbol | Signature | Description | +|---|---|---| +| `buildSitePathsSync` | `(pages, staticFiles, excludePatterns, stubs, themeAssetRels) → Set<string>` | Synchronous version of `buildSitePaths`. Takes an explicit `themeAssetRels` string array (from `enumerateVendoredThemeAssets`) instead of walking `_site/assets/`. Used by `dispatch.execute` on the main thread to build the `sitePaths` set before dispatching render workers. | +| `deriveOfflinePage` | `(page: Page, state: OfflineState) → string` | Rewrites one page's HTML for offline use: strips SEO metadata, rewrites every absolute URL to a page-relative path, and injects the offline search setup script. | +| `deriveOfflinePageCached` | `(page: Page, state: OfflineState) → { html: string, misses: number }` | Cached variant of `deriveOfflinePage`. Uses `state.navCache` to substitute the pre-rewritten sidebar nav block for subsequent pages in the same destination directory, avoiding a full regex pass over the ~80 KB sidebar on each page. | +| `sliceNavBlock` | `(html: string) → { before: string, nav: string, after: string }` | Splits a page's HTML into the segments before, within, and after the sidebar nav block. Used by the nav-cache pre-pass. | +| `deriveOfflineCss` | `(cssIn: string, themeRel: string, state: OfflineState) → string` | Rewrites `url()` references in a CSS file to page-relative paths. | +| `deriveOfflineRedirect` | `(stub, state: OfflineState) → string` | Rewrites a redirect stub's HTML for offline use. | +| `offlineExcluded` | `(rel: string, patterns: string[]) → boolean` | Returns `true` when a site-relative path matches any `offline_exclude` glob pattern from `_config.yml`. | +| `normalizeBaseurl` | `(baseurl: string) → string` | Normalises a baseurl string to trailing-slash form. | +| `posixDirname` | `(rel: string) → string` | Returns the POSIX directory component of a relative path. | +| `fileDirSegsFromRel` | `(rel: string) → string[]` | Splits a destination path into directory segments for use by `computeRelative`. | +| `fnmatchPathname` | `(pattern: string, path: string) → boolean` | Glob-style pathname match (`*`, `**`, `?` supported). | + --- ## Phase 8: `pdf.mjs` + `book.mjs` diff --git a/docs/assets/images/mmd/scheduler-dag.mmd b/docs/assets/images/mmd/scheduler-dag.mmd index 48a08563..cccb5ba4 100644 --- a/docs/assets/images/mmd/scheduler-dag.mmd +++ b/docs/assets/images/mmd/scheduler-dag.mmd @@ -22,6 +22,8 @@ flowchart TB BU --> DP["dispatch<br><small>[M]</small>"] RC --> DP BI --> DP + MM --> DP + DR --> DP subgraph fanout [" "] direction LR diff --git a/docs/assets/images/mmd/scheduler-dag.svg b/docs/assets/images/mmd/scheduler-dag.svg index 7065be09..ba61335f 100644 --- a/docs/assets/images/mmd/scheduler-dag.svg +++ b/docs/assets/images/mmd/scheduler-dag.svg @@ -1 +1 @@ -<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 1225.05px; background-color: transparent;" viewBox="0 0 1225.0546875 1596" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#my-svg .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#my-svg .error-icon{fill:#552222;}#my-svg .error-text{fill:#552222;stroke:#552222;}#my-svg .edge-thickness-normal{stroke-width:1px;}#my-svg .edge-thickness-thick{stroke-width:3.5px;}#my-svg .edge-pattern-solid{stroke-dasharray:0;}#my-svg .edge-thickness-invisible{stroke-width:0;fill:none;}#my-svg .edge-pattern-dashed{stroke-dasharray:3;}#my-svg .edge-pattern-dotted{stroke-dasharray:2;}#my-svg .marker{fill:#7a8090;stroke:#7a8090;}#my-svg .marker.cross{stroke:#7a8090;}#my-svg svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#my-svg p{margin:0;}#my-svg .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#my-svg .cluster-label text{fill:#333;}#my-svg .cluster-label span{color:#333;}#my-svg .cluster-label span p{background-color:transparent;}#my-svg .label text,#my-svg span{fill:#333;color:#333;}#my-svg .node rect,#my-svg .node circle,#my-svg .node ellipse,#my-svg .node polygon,#my-svg .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#my-svg .rough-node .label text,#my-svg .node .label text,#my-svg .image-shape .label,#my-svg .icon-shape .label{text-anchor:middle;}#my-svg .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#my-svg .rough-node .label,#my-svg .node .label,#my-svg .image-shape .label,#my-svg .icon-shape .label{text-align:center;}#my-svg .node.clickable{cursor:pointer;}#my-svg .root .anchor path{fill:#7a8090!important;stroke-width:0;stroke:#7a8090;}#my-svg .arrowheadPath{fill:#333333;}#my-svg .edgePath .path{stroke:#7a8090;stroke-width:1px;}#my-svg .flowchart-link{stroke:#7a8090;fill:none;}#my-svg .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#my-svg .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#my-svg .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#my-svg .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#my-svg .cluster rect{fill:transparent;stroke:#7a8090;stroke-width:1px;}#my-svg .cluster text{fill:#333;}#my-svg .cluster span{color:#333;}#my-svg div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#my-svg .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#my-svg rect.text{fill:none;stroke-width:0;}#my-svg .icon-shape,#my-svg .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#my-svg .icon-shape p,#my-svg .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#my-svg .icon-shape .label rect,#my-svg .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#my-svg .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#my-svg .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#my-svg .node .neo-node{stroke:#9370DB;}#my-svg [data-look="neo"].node rect,#my-svg [data-look="neo"].cluster rect,#my-svg [data-look="neo"].node polygon{stroke:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#my-svg [data-look="neo"].node path{stroke:#9370DB;stroke-width:1px;}#my-svg [data-look="neo"].node .outer-path{filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#my-svg [data-look="neo"].node .neo-line path{stroke:#9370DB;filter:none;}#my-svg [data-look="neo"].node circle{stroke:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#my-svg [data-look="neo"].node circle .state-start{fill:#000000;}#my-svg [data-look="neo"].icon-shape .icon{fill:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#my-svg [data-look="neo"].icon-shape .icon-neo path{stroke:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#my-svg :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}#my-svg .worker>*{fill:rgb(232, 240, 254)!important;stroke:rgb(66, 133, 244)!important;color:rgb(26, 26, 46)!important;}#my-svg .worker span{fill:rgb(232, 240, 254)!important;stroke:rgb(66, 133, 244)!important;color:rgb(26, 26, 46)!important;}#my-svg .worker tspan{fill:rgb(26, 26, 46)!important;}#my-svg .main>*{fill:rgb(254, 247, 224)!important;stroke:rgb(249, 171, 0)!important;color:rgb(26, 26, 46)!important;}#my-svg .main span{fill:rgb(254, 247, 224)!important;stroke:rgb(249, 171, 0)!important;color:rgb(26, 26, 46)!important;}#my-svg .main tspan{fill:rgb(26, 26, 46)!important;}</style><g><marker id="my-svg_flowchart-v2-pointEnd" class="marker flowchart-v2" viewBox="0 0 10 10" refX="5" refY="5" markerUnits="userSpaceOnUse" markerWidth="8" markerHeight="8" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"/></marker><marker id="my-svg_flowchart-v2-pointStart" class="marker flowchart-v2" viewBox="0 0 10 10" refX="4.5" refY="5" markerUnits="userSpaceOnUse" markerWidth="8" markerHeight="8" orient="auto"><path d="M 0 5 L 10 10 L 10 0 z" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"/></marker><marker id="my-svg_flowchart-v2-pointEnd-margin" class="marker flowchart-v2" viewBox="0 0 11.5 14" refX="11.5" refY="7" markerUnits="userSpaceOnUse" markerWidth="10.5" markerHeight="14" orient="auto"><path d="M 0 0 L 11.5 7 L 0 14 z" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"/></marker><marker id="my-svg_flowchart-v2-pointStart-margin" class="marker flowchart-v2" viewBox="0 0 11.5 14" refX="1" refY="7" markerUnits="userSpaceOnUse" markerWidth="11.5" markerHeight="14" orient="auto"><polygon points="0,7 11.5,14 11.5,0" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"/></marker><marker id="my-svg_flowchart-v2-circleEnd" class="marker flowchart-v2" viewBox="0 0 10 10" refX="11" refY="5" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"/></marker><marker id="my-svg_flowchart-v2-circleStart" class="marker flowchart-v2" viewBox="0 0 10 10" refX="-1" refY="5" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"/></marker><marker id="my-svg_flowchart-v2-circleEnd-margin" class="marker flowchart-v2" viewBox="0 0 10 10" refY="5" refX="12.25" markerUnits="userSpaceOnUse" markerWidth="14" markerHeight="14" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"/></marker><marker id="my-svg_flowchart-v2-circleStart-margin" class="marker flowchart-v2" viewBox="0 0 10 10" refX="-2" refY="5" markerUnits="userSpaceOnUse" markerWidth="14" markerHeight="14" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"/></marker><marker id="my-svg_flowchart-v2-crossEnd" class="marker cross flowchart-v2" viewBox="0 0 11 11" refX="12" refY="5.2" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><path d="M 1,1 l 9,9 M 10,1 l -9,9" class="arrowMarkerPath" style="stroke-width: 2; stroke-dasharray: 1, 0;"/></marker><marker id="my-svg_flowchart-v2-crossStart" class="marker cross flowchart-v2" viewBox="0 0 11 11" refX="-1" refY="5.2" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><path d="M 1,1 l 9,9 M 10,1 l -9,9" class="arrowMarkerPath" style="stroke-width: 2; stroke-dasharray: 1, 0;"/></marker><marker id="my-svg_flowchart-v2-crossEnd-margin" class="marker cross flowchart-v2" viewBox="0 0 15 15" refX="17.7" refY="7.5" markerUnits="userSpaceOnUse" markerWidth="12" markerHeight="12" orient="auto"><path d="M 1,1 L 14,14 M 1,14 L 14,1" class="arrowMarkerPath" style="stroke-width: 2.5;"/></marker><marker id="my-svg_flowchart-v2-crossStart-margin" class="marker cross flowchart-v2" viewBox="0 0 15 15" refX="-3.5" refY="7.5" markerUnits="userSpaceOnUse" markerWidth="12" markerHeight="12" orient="auto"><path d="M 1,1 L 14,14 M 1,14 L 14,1" class="arrowMarkerPath" style="stroke-width: 2.5; stroke-dasharray: 1, 0;"/></marker><g class="root"><g class="clusters"/><g class="edgePaths"><path d="M1009.176,86L1009.176,90.167C1009.176,94.333,1009.176,102.667,1009.176,110.333C1009.176,118,1009.176,125,1009.176,128.5L1009.176,132" id="my-svg-L_CF_DI_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_CF_DI_0" data-points="W3sieCI6MTAwOS4xNzU3ODEyNSwieSI6ODZ9LHsieCI6MTAwOS4xNzU3ODEyNSwieSI6MTExfSx7IngiOjEwMDkuMTc1NzgxMjUsInkiOjEzNn1d" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M949.551,186.046L901.91,194.872C854.27,203.697,758.988,221.349,711.348,233.674C663.707,246,663.707,253,663.707,256.5L663.707,260" id="my-svg-L_DI_NV_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_DI_NV_0" data-points="W3sieCI6OTQ5LjU1MDc4MTI1LCJ5IjoxODYuMDQ1ODYxNjAxMDg1NDh9LHsieCI6NjYzLjcwNzAzMTI1LCJ5IjoyMzl9LHsieCI6NjYzLjcwNzAzMTI1LCJ5IjoyNjR9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M953.835,214L947.922,218.167C942.009,222.333,930.184,230.667,924.272,245.5C918.359,260.333,918.359,281.667,918.359,303C918.359,324.333,918.359,345.667,918.359,367C918.359,388.333,918.359,409.667,918.359,431C918.359,452.333,918.359,473.667,918.359,495C918.359,516.333,918.359,537.667,918.359,559C918.359,580.333,918.359,601.667,918.359,625C918.359,648.333,918.359,673.667,918.359,699C918.359,724.333,918.359,749.667,918.359,773C918.359,796.333,918.359,817.667,918.359,839C918.359,860.333,918.359,881.667,918.359,908.833C918.359,936,918.359,969,918.359,1002C918.359,1035,918.359,1068,918.359,1095.167C918.359,1122.333,918.359,1143.667,918.359,1165C918.359,1186.333,918.359,1207.667,918.359,1221.833C918.359,1236,918.359,1243,918.359,1246.5L918.359,1250" id="my-svg-L_DI_DR_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_DI_DR_0" data-points="W3sieCI6OTUzLjgzNDUzMzY5MTQwNjIsInkiOjIxNH0seyJ4Ijo5MTguMzU5Mzc1LCJ5IjoyMzl9LHsieCI6OTE4LjM1OTM3NSwieSI6MzAzfSx7IngiOjkxOC4zNTkzNzUsInkiOjM2N30seyJ4Ijo5MTguMzU5Mzc1LCJ5Ijo0MzF9LHsieCI6OTE4LjM1OTM3NSwieSI6NDk1fSx7IngiOjkxOC4zNTkzNzUsInkiOjU1OX0seyJ4Ijo5MTguMzU5Mzc1LCJ5Ijo2MjN9LHsieCI6OTE4LjM1OTM3NSwieSI6Njk5fSx7IngiOjkxOC4zNTkzNzUsInkiOjc3NX0seyJ4Ijo5MTguMzU5Mzc1LCJ5Ijo4Mzl9LHsieCI6OTE4LjM1OTM3NSwieSI6OTAzfSx7IngiOjkxOC4zNTkzNzUsInkiOjEwMDJ9LHsieCI6OTE4LjM1OTM3NSwieSI6MTEwMX0seyJ4Ijo5MTguMzU5Mzc1LCJ5IjoxMTY1fSx7IngiOjkxOC4zNTkzNzUsInkiOjEyMjl9LHsieCI6OTE4LjM1OTM3NSwieSI6MTI1NH1d" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M1068.801,205.186L1079.933,210.821C1091.065,216.457,1113.329,227.729,1124.462,244.031C1135.594,260.333,1135.594,281.667,1135.594,303C1135.594,324.333,1135.594,345.667,1135.594,367C1135.594,388.333,1135.594,409.667,1135.594,431C1135.594,452.333,1135.594,473.667,1135.594,495C1135.594,516.333,1135.594,537.667,1135.594,559C1135.594,580.333,1135.594,601.667,1135.594,625C1135.594,648.333,1135.594,673.667,1135.594,699C1135.594,724.333,1135.594,749.667,1135.594,773C1135.594,796.333,1135.594,817.667,1135.594,839C1135.594,860.333,1135.594,881.667,1135.594,908.833C1135.594,936,1135.594,969,1135.594,1002C1135.594,1035,1135.594,1068,1135.594,1095.167C1135.594,1122.333,1135.594,1143.667,1135.594,1165C1135.594,1186.333,1135.594,1207.667,1135.594,1221.833C1135.594,1236,1135.594,1243,1135.594,1246.5L1135.594,1250" id="my-svg-L_DI_DS_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_DI_DS_0" data-points="W3sieCI6MTA2OC44MDA3ODEyNSwieSI6MjA1LjE4NTU4MjMwMDc3NTZ9LHsieCI6MTEzNS41OTM3NSwieSI6MjM5fSx7IngiOjExMzUuNTkzNzUsInkiOjMwM30seyJ4IjoxMTM1LjU5Mzc1LCJ5IjozNjd9LHsieCI6MTEzNS41OTM3NSwieSI6NDMxfSx7IngiOjExMzUuNTkzNzUsInkiOjQ5NX0seyJ4IjoxMTM1LjU5Mzc1LCJ5Ijo1NTl9LHsieCI6MTEzNS41OTM3NSwieSI6NjIzfSx7IngiOjExMzUuNTkzNzUsInkiOjY5OX0seyJ4IjoxMTM1LjU5Mzc1LCJ5Ijo3NzV9LHsieCI6MTEzNS41OTM3NSwieSI6ODM5fSx7IngiOjExMzUuNTkzNzUsInkiOjkwM30seyJ4IjoxMTM1LjU5Mzc1LCJ5IjoxMDAyfSx7IngiOjExMzUuNTkzNzUsInkiOjExMDF9LHsieCI6MTEzNS41OTM3NSwieSI6MTE2NX0seyJ4IjoxMTM1LjU5Mzc1LCJ5IjoxMjI5fSx7IngiOjExMzUuNTkzNzUsInkiOjEyNTR9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M621.215,334.182L613.761,339.651C606.307,345.121,591.4,356.061,583.946,365.03C576.492,374,576.492,381,576.492,384.5L576.492,388" id="my-svg-L_NV_MI_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_NV_MI_0" data-points="W3sieCI6NjIxLjIxNDg0Mzc1LCJ5IjozMzQuMTgxNjE4NjY3OTgwNX0seyJ4Ijo1NzYuNDkyMTg3NSwieSI6MzY3fSx7IngiOjU3Ni40OTIxODc1LCJ5IjozOTJ9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M706.199,323.778L720.932,330.981C735.664,338.185,765.129,352.593,779.861,370.463C794.594,388.333,794.594,409.667,794.594,431C794.594,452.333,794.594,473.667,794.594,495C794.594,516.333,794.594,537.667,794.594,559C794.594,580.333,794.594,601.667,794.594,617.833C794.594,634,794.594,645,794.594,650.5L794.594,656" id="my-svg-L_NV_BU_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_NV_BU_0" data-points="W3sieCI6NzA2LjE5OTIxODc1LCJ5IjozMjMuNzc3NTA5MTc3MTg2ODZ9LHsieCI6Nzk0LjU5Mzc1LCJ5IjozNjd9LHsieCI6Nzk0LjU5Mzc1LCJ5Ijo0MzF9LHsieCI6Nzk0LjU5Mzc1LCJ5Ijo0OTV9LHsieCI6Nzk0LjU5Mzc1LCJ5Ijo1NTl9LHsieCI6Nzk0LjU5Mzc1LCJ5Ijo2MjN9LHsieCI6Nzk0LjU5Mzc1LCJ5Ijo2NjB9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M529.663,470L524.66,474.167C519.657,478.333,509.651,486.667,504.648,494.333C499.645,502,499.645,509,499.645,512.5L499.645,516" id="my-svg-L_MI_SE_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_MI_SE_0" data-points="W3sieCI6NTI5LjY2MzE0Njk3MjY1NjIsInkiOjQ3MH0seyJ4Ijo0OTkuNjQ0NTMxMjUsInkiOjQ5NX0seyJ4Ijo0OTkuNjQ0NTMxMjUsInkiOjUyMH1d" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M623.321,470L628.324,474.167C633.327,478.333,643.334,486.667,648.337,494.333C653.34,502,653.34,509,653.34,512.5L653.34,516" id="my-svg-L_MI_LD_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_MI_LD_0" data-points="W3sieCI6NjIzLjMyMTIyODAyNzM0MzgsInkiOjQ3MH0seyJ4Ijo2NTMuMzM5ODQzNzUsInkiOjQ5NX0seyJ4Ijo2NTMuMzM5ODQzNzUsInkiOjUyMH1d" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M499.645,598L499.645,602.167C499.645,606.333,499.645,614.667,503.384,622.531C507.123,630.396,514.601,637.792,518.34,641.489L522.079,645.187" id="my-svg-L_SE_RC_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_SE_RC_0" data-points="W3sieCI6NDk5LjY0NDUzMTI1LCJ5Ijo1OTh9LHsieCI6NDk5LjY0NDUzMTI1LCJ5Ijo2MjN9LHsieCI6NTI0LjkyMzM2NTU0Mjc2MzEsInkiOjY0OH1d" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M653.34,598L653.34,602.167C653.34,606.333,653.34,614.667,649.601,622.531C645.862,630.396,638.383,637.792,634.644,641.489L630.905,645.187" id="my-svg-L_LD_RC_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_LD_RC_0" data-points="W3sieCI6NjUzLjMzOTg0Mzc1LCJ5Ijo1OTh9LHsieCI6NjUzLjMzOTg0Mzc1LCJ5Ijo2MjN9LHsieCI6NjI4LjA2MTAwOTQ1NzIzNjksInkiOjY0OH1d" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M794.594,738L794.594,744.167C794.594,750.333,794.594,762.667,767.818,776.462C741.043,790.257,687.492,805.513,660.716,813.142L633.941,820.77" id="my-svg-L_BU_DP_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_BU_DP_0" data-points="W3sieCI6Nzk0LjU5Mzc1LCJ5Ijo3Mzh9LHsieCI6Nzk0LjU5Mzc1LCJ5Ijo3NzV9LHsieCI6NjMwLjA5Mzc1LCJ5Ijo4MjEuODY1OTY2NDc0MjI5N31d" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M576.492,750L576.492,754.167C576.492,758.333,576.492,766.667,576.134,774.337C575.776,782.007,575.06,789.014,574.702,792.517L574.344,796.021" id="my-svg-L_RC_DP_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_RC_DP_0" data-points="W3sieCI6NTc2LjQ5MjE4NzUsInkiOjc1MH0seyJ4Ijo1NzYuNDkyMTg3NSwieSI6Nzc1fSx7IngiOjU3My45Mzc4NjYyMTA5Mzc1LCJ5Ijo4MDB9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M388.984,738L388.984,744.167C388.984,750.333,388.984,762.667,408.494,775.733C428.003,788.799,467.022,802.598,486.532,809.498L506.041,816.397" id="my-svg-L_BI_DP_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_BI_DP_0" data-points="W3sieCI6Mzg4Ljk4NDM3NSwieSI6NzM4fSx7IngiOjM4OC45ODQzNzUsInkiOjc3NX0seyJ4Ijo1MDkuODEyNSwieSI6ODE3LjczMTEzNDUxOTA4MTR9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M517.121,1204L511.476,1208.167C505.832,1212.333,494.543,1220.667,440.386,1233.862C386.23,1247.057,289.206,1265.114,240.694,1274.142L192.182,1283.171" id="my-svg-L_RJ_WR_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_RJ_WR_0" data-points="W3sieCI6NTE3LjEyMDc4ODU3NDIxODgsInkiOjEyMDR9LHsieCI6NDgzLjI1MzkwNjI1LCJ5IjoxMjI5fSx7IngiOjE4OC4yNSwieSI6MTI4My45MDI1Mjc0MDM4NzM0fV0=" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M51.68,1204L51.68,1208.167C51.68,1212.333,51.68,1220.667,57.609,1229.161C63.538,1237.655,75.396,1246.309,81.324,1250.637L87.253,1254.964" id="my-svg-L_SC_WR_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_SC_WR_0" data-points="W3sieCI6NTEuNjc5Njg3NSwieSI6MTIwNH0seyJ4Ijo1MS42Nzk2ODc1LCJ5IjoxMjI5fSx7IngiOjkwLjQ4NDM3NSwieSI6MTI1Ny4zMjIxNjY3ODU0NTk4fV0=" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M167.993,1204L163.222,1208.167C158.451,1212.333,148.909,1220.667,144.138,1228.333C139.367,1236,139.367,1243,139.367,1246.5L139.367,1250" id="my-svg-L_MM_WR_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_MM_WR_0" data-points="W3sieCI6MTY3Ljk5MjY3NTc4MTI1LCJ5IjoxMjA0fSx7IngiOjEzOS4zNjcxODc1LCJ5IjoxMjI5fSx7IngiOjEzOS4zNjcxODc1LCJ5IjoxMjU0fV0=" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M344.696,1204L339.925,1208.167C335.154,1212.333,325.612,1220.667,300.165,1232.322C274.717,1243.978,233.364,1258.955,212.687,1266.444L192.011,1273.933" id="my-svg-L_PD_WR_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_PD_WR_0" data-points="W3sieCI6MzQ0LjY5NTgwMDc4MTI1LCJ5IjoxMjA0fSx7IngiOjMxNi4wNzAzMTI1LCJ5IjoxMjI5fSx7IngiOjE4OC4yNSwieSI6MTI3NS4yOTUxNjMxNDQzOTgyfV0=" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M551.51,1204L549.54,1208.167C547.569,1212.333,543.628,1220.667,541.672,1228.333C539.717,1236,539.746,1243,539.76,1246.5L539.775,1250" id="my-svg-L_RJ_SD_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_RJ_SD_0" data-points="W3sieCI6NTUxLjUxMDAwOTc2NTYyNSwieSI6MTIwNH0seyJ4Ijo1MzkuNjg3NSwieSI6MTIyOX0seyJ4Ijo1MzkuNzkxMjU5NzY1NjI1LCJ5IjoxMjU0fV0=" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M434.386,1204L439.197,1208.167C444.009,1212.333,453.631,1220.667,462.924,1228.573C472.217,1236.479,481.18,1243.958,485.662,1247.698L490.143,1251.437" id="my-svg-L_PD_SD_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_PD_SD_0" data-points="W3sieCI6NDM0LjM4NTgwMzIyMjY1NjI1LCJ5IjoxMjA0fSx7IngiOjQ2My4yNTM5MDYyNSwieSI6MTIyOX0seyJ4Ijo0OTMuMjE0NTM4NTc0MjE4NzUsInkiOjEyNTR9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M139.367,1332L139.367,1336.167C139.367,1340.333,139.367,1348.667,227.751,1362.32C316.135,1375.974,492.903,1394.947,581.287,1404.434L669.671,1413.921" id="my-svg-L_WR_WA_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_WR_WA_0" data-points="W3sieCI6MTM5LjM2NzE4NzUsInkiOjEzMzJ9LHsieCI6MTM5LjM2NzE4NzUsInkiOjEzNTd9LHsieCI6NjczLjY0ODQzNzUsInkiOjE0MTQuMzQ3Njc2MjYyMTAzNX1d" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M539.953,1332L539.953,1336.167C539.953,1340.333,539.953,1348.667,561.602,1359.914C583.251,1371.162,626.549,1385.324,648.198,1392.404L669.847,1399.485" id="my-svg-L_SD_WA_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_SD_WA_0" data-points="W3sieCI6NTM5Ljk1MzEyNSwieSI6MTMzMn0seyJ4Ijo1MzkuOTUzMTI1LCJ5IjoxMzU3fSx7IngiOjY3My42NDg0Mzc1LCJ5IjoxNDAwLjcyODgxODk3MzA4OTZ9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M918.359,1332L918.359,1336.167C918.359,1340.333,918.359,1348.667,898.862,1359.662C879.365,1370.657,840.371,1384.314,820.874,1391.143L801.377,1397.971" id="my-svg-L_DR_WA_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_DR_WA_0" data-points="W3sieCI6OTE4LjM1OTM3NSwieSI6MTMzMn0seyJ4Ijo5MTguMzU5Mzc1LCJ5IjoxMzU3fSx7IngiOjc5Ny42MDE1NjI1LCJ5IjoxMzk5LjI5MzYyOTc1NjMwNn1d" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M1135.594,1332L1135.594,1336.167C1135.594,1340.333,1135.594,1348.667,1079.92,1361.742C1024.246,1374.817,912.899,1392.634,857.225,1401.542L801.551,1410.451" id="my-svg-L_DS_WA_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_DS_WA_0" data-points="W3sieCI6MTEzNS41OTM3NSwieSI6MTMzMn0seyJ4IjoxMTM1LjU5Mzc1LCJ5IjoxMzU3fSx7IngiOjc5Ny42MDE1NjI1LCJ5IjoxNDExLjA4Mjk3NTIzMjQ0fV0=" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M735.625,1460L735.625,1464.167C735.625,1468.333,735.625,1476.667,735.625,1484.333C735.625,1492,735.625,1499,735.625,1502.5L735.625,1506" id="my-svg-L_WA_WO_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_WA_WO_0" data-points="W3sieCI6NzM1LjYyNSwieSI6MTQ2MH0seyJ4Ijo3MzUuNjI1LCJ5IjoxNDg1fSx7IngiOjczNS42MjUsInkiOjE1MTB9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M638.484,1190.911L655.275,1197.259C672.065,1203.607,705.646,1216.304,669.917,1231.501C634.187,1246.699,529.148,1264.398,476.628,1273.248L424.108,1282.098" id="my-svg-L_RJ_WP_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_RJ_WP_0" data-points="W3sieCI6NjM4LjQ4NDM3NSwieSI6MTE5MC45MTA3Mzk4MzQ3NzE3fSx7IngiOjczOS4yMjY1NjI1LCJ5IjoxMjI5fSx7IngiOjQyMC4xNjQwNjI1LCJ5IjoxMjgyLjc2MjI2NDIyODU2MjF9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M257.423,1204L262.207,1208.167C266.99,1212.333,276.558,1220.667,285.61,1228.561C294.663,1236.456,303.2,1243.913,307.469,1247.641L311.738,1251.369" id="my-svg-L_MM_WP_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_MM_WP_0" data-points="W3sieCI6MjU3LjQyMzIxNzc3MzQzNzUsInkiOjEyMDR9LHsieCI6Mjg2LjEyNSwieSI6MTIyOX0seyJ4IjozMTQuNzUwNDg4MjgxMjUsInkiOjEyNTR9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M569.953,878L569.953,882.167C569.953,886.333,569.953,894.667,569.953,902.333C569.953,910,569.953,917,569.953,920.5L569.953,924" id="my-svg-L_DP_fanout_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_DP_fanout_0" data-points="W3sieCI6NTY5Ljk1MzEyNSwieSI6ODc4fSx7IngiOjU2OS45NTMxMjUsInkiOjkwM30seyJ4Ijo1NjkuOTUzMTI1LCJ5Ijo5Mjh9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M569.953,1076L569.953,1080.167C569.953,1084.333,569.953,1092.667,569.953,1100.333C569.953,1108,569.953,1115,569.953,1118.5L569.953,1122" id="my-svg-L_fanout_RJ_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_fanout_RJ_0" data-points="W3sieCI6NTY5Ljk1MzEyNSwieSI6MTA3Nn0seyJ4Ijo1NjkuOTUzMTI1LCJ5IjoxMTAxfSx7IngiOjU2OS45NTMxMjUsInkiOjExMjZ9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/></g><g class="edgeLabels"><g class="edgeLabel"><g class="label" data-id="L_CF_DI_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_DI_NV_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_DI_DR_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_DI_DS_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_NV_MI_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_NV_BU_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_MI_SE_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_MI_LD_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_SE_RC_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_LD_RC_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_BU_DP_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_RC_DP_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_BI_DP_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_RJ_WR_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_SC_WR_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_MM_WR_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_PD_WR_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_RJ_SD_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_PD_SD_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_WR_WA_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_SD_WA_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_DR_WA_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_DS_WA_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_WA_WO_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_RJ_WP_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_MM_WP_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_DP_fanout_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_fanout_RJ_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g></g><g class="nodes"><g class="root" transform="translate(258.6875, 920)"><g class="clusters"><g class="cluster" id="my-svg-fanout" data-look="classic"><rect style="" x="8" y="8" width="606.53125" height="148"/><g class="cluster-label" transform="translate(311.265625, 8)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5;"><span class="nodeLabel"></span></div></foreignObject></g></g></g><g class="edgePaths"/><g class="edgeLabels"/><g class="nodes"><g class="node default worker" id="my-svg-flowchart-R0-30" data-look="classic" transform="translate(106.40625, 82)"><rect class="basic label-container" style="fill:#e8f0fe !important;stroke:#4285f4 !important" x="-60.90625" y="-39" width="121.8125" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-30.90625, -24)"><rect/><foreignObject width="61.8125" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>render:0<br /><small>[W]</small></p></span></div></foreignObject></g></g><g class="node default worker" id="my-svg-flowchart-R1-31" data-look="classic" transform="translate(303.21875, 82)"><rect class="basic label-container" style="fill:#e8f0fe !important;stroke:#4285f4 !important" x="-60.90625" y="-39" width="121.8125" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-30.90625, -24)"><rect/><foreignObject width="61.8125" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>render:1<br /><small>[W]</small></p></span></div></foreignObject></g></g><g class="node default worker" id="my-svg-flowchart-RN-32" data-look="classic" transform="translate(508.078125, 82)"><rect class="basic label-container" style="fill:#e8f0fe !important;stroke:#4285f4 !important" x="-68.953125" y="-39" width="137.90625" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-38.953125, -24)"><rect/><foreignObject width="77.90625" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>render:N-1<br /><small>[W]</small></p></span></div></foreignObject></g></g></g></g><g class="node default worker" id="my-svg-flowchart-BI-0" data-look="classic" transform="translate(388.984375, 699)"><rect class="basic label-container" style="fill:#e8f0fe !important;stroke:#4285f4 !important" x="-61.7734375" y="-39" width="123.546875" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-31.7734375, -24)"><rect/><foreignObject width="63.546875" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>buildInfo<br /><small>[W]</small></p></span></div></foreignObject></g></g><g class="node default worker" id="my-svg-flowchart-SC-1" data-look="classic" transform="translate(51.6796875, 1165)"><rect class="basic label-container" style="fill:#e8f0fe !important;stroke:#4285f4 !important" x="-43.6796875" y="-39" width="87.359375" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-13.6796875, -24)"><rect/><foreignObject width="27.359375" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>scss<br /><small>[W]</small></p></span></div></foreignObject></g></g><g class="node default worker" id="my-svg-flowchart-MM-2" data-look="classic" transform="translate(212.6484375, 1165)"><rect class="basic label-container" style="fill:#e8f0fe !important;stroke:#4285f4 !important" x="-61.6953125" y="-39" width="123.390625" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-31.6953125, -24)"><rect/><foreignObject width="63.390625" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>mermaid<br /><small>[W]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-PD-3" data-look="classic" transform="translate(389.3515625, 1165)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-62.0703125" y="-39" width="124.140625" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-32.0703125, -24)"><rect/><foreignObject width="64.140625" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>prepDest<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-CF-4" data-look="classic" transform="translate(1009.17578125, 47)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-51.8828125" y="-39" width="103.765625" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-21.8828125, -24)"><rect/><foreignObject width="43.765625" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>config<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-DI-5" data-look="classic" transform="translate(1009.17578125, 175)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-59.625" y="-39" width="119.25" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-29.625, -24)"><rect/><foreignObject width="59.25" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>discover<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-NV-7" data-look="classic" transform="translate(663.70703125, 303)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-42.4921875" y="-39" width="84.984375" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-12.4921875, -24)"><rect/><foreignObject width="24.984375" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>nav<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-DR-9" data-look="classic" transform="translate(918.359375, 1293)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-85.7734375" y="-39" width="171.546875" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-55.7734375, -24)"><rect/><foreignObject width="111.546875" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>deriveRedirects<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-DS-11" data-look="classic" transform="translate(1135.59375, 1293)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-81.4609375" y="-39" width="162.921875" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-51.4609375, -24)"><rect/><foreignObject width="102.921875" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>deriveSitemap<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-MI-13" data-look="classic" transform="translate(576.4921875, 431)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-79.1171875" y="-39" width="158.234375" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-49.1171875, -24)"><rect/><foreignObject width="98.234375" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>markdownInit<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-BU-15" data-look="classic" transform="translate(794.59375, 699)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-59.9765625" y="-39" width="119.953125" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-29.9765625, -24)"><rect/><foreignObject width="59.953125" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>buildInit<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-SE-17" data-look="classic" transform="translate(499.64453125, 559)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-41.8984375" y="-39" width="83.796875" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-11.8984375, -24)"><rect/><foreignObject width="23.796875" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>seo<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-LD-19" data-look="classic" transform="translate(653.33984375, 559)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-61.796875" y="-39" width="123.59375" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-31.796875, -24)"><rect/><foreignObject width="63.59375" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>loadData<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-RC-21" data-look="classic" transform="translate(576.4921875, 699)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-75.734375" y="-51" width="151.46875" height="102"/><g class="label" style="color:#1a1a2e !important" transform="translate(-45.734375, -36)"><rect/><foreignObject width="91.46875" height="72"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>resolveBook-<br />Chapters<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-DP-25" data-look="classic" transform="translate(569.953125, 839)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-60.140625" y="-39" width="120.28125" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-30.140625, -24)"><rect/><foreignObject width="60.28125" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>dispatch<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-RJ-36" data-look="classic" transform="translate(569.953125, 1165)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-68.53125" y="-39" width="137.0625" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-38.53125, -24)"><rect/><foreignObject width="77.0625" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>renderJoin<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-WR-38" data-look="classic" transform="translate(139.3671875, 1293)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-48.8828125" y="-39" width="97.765625" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-18.8828125, -24)"><rect/><foreignObject width="37.765625" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>write<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-SD-46" data-look="classic" transform="translate(539.953125, 1293)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-69.734375" y="-39" width="139.46875" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-39.734375, -24)"><rect/><foreignObject width="79.46875" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>searchData<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-WA-50" data-look="classic" transform="translate(735.625, 1421)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-61.9765625" y="-39" width="123.953125" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-31.9765625, -24)"><rect/><foreignObject width="63.953125" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>writeAux<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-WO-58" data-look="classic" transform="translate(735.625, 1549)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-73.5625" y="-39" width="147.125" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-43.5625, -24)"><rect/><foreignObject width="87.125" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>writeOffline<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-WP-60" data-look="classic" transform="translate(359.40625, 1293)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-60.7578125" y="-39" width="121.515625" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-30.7578125, -24)"><rect/><foreignObject width="61.515625" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>writePdf<br /><small>[M]</small></p></span></div></foreignObject></g></g></g></g></g><defs><filter id="my-svg-drop-shadow" height="130%" width="130%"><feDropShadow dx="4" dy="4" stdDeviation="0" flood-opacity="0.06" flood-color="#000000"/></filter></defs><defs><filter id="my-svg-drop-shadow-small" height="150%" width="150%"><feDropShadow dx="2" dy="2" stdDeviation="0" flood-opacity="0.06" flood-color="#000000"/></filter></defs></svg> \ No newline at end of file +<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 1332.25px; background-color: transparent;" viewBox="0 0 1332.24609375 1596" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#my-svg .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#my-svg .error-icon{fill:#552222;}#my-svg .error-text{fill:#552222;stroke:#552222;}#my-svg .edge-thickness-normal{stroke-width:1px;}#my-svg .edge-thickness-thick{stroke-width:3.5px;}#my-svg .edge-pattern-solid{stroke-dasharray:0;}#my-svg .edge-thickness-invisible{stroke-width:0;fill:none;}#my-svg .edge-pattern-dashed{stroke-dasharray:3;}#my-svg .edge-pattern-dotted{stroke-dasharray:2;}#my-svg .marker{fill:#7a8090;stroke:#7a8090;}#my-svg .marker.cross{stroke:#7a8090;}#my-svg svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#my-svg p{margin:0;}#my-svg .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#my-svg .cluster-label text{fill:#333;}#my-svg .cluster-label span{color:#333;}#my-svg .cluster-label span p{background-color:transparent;}#my-svg .label text,#my-svg span{fill:#333;color:#333;}#my-svg .node rect,#my-svg .node circle,#my-svg .node ellipse,#my-svg .node polygon,#my-svg .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#my-svg .rough-node .label text,#my-svg .node .label text,#my-svg .image-shape .label,#my-svg .icon-shape .label{text-anchor:middle;}#my-svg .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#my-svg .rough-node .label,#my-svg .node .label,#my-svg .image-shape .label,#my-svg .icon-shape .label{text-align:center;}#my-svg .node.clickable{cursor:pointer;}#my-svg .root .anchor path{fill:#7a8090!important;stroke-width:0;stroke:#7a8090;}#my-svg .arrowheadPath{fill:#333333;}#my-svg .edgePath .path{stroke:#7a8090;stroke-width:1px;}#my-svg .flowchart-link{stroke:#7a8090;fill:none;}#my-svg .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#my-svg .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#my-svg .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#my-svg .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#my-svg .cluster rect{fill:transparent;stroke:#7a8090;stroke-width:1px;}#my-svg .cluster text{fill:#333;}#my-svg .cluster span{color:#333;}#my-svg div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#my-svg .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#my-svg rect.text{fill:none;stroke-width:0;}#my-svg .icon-shape,#my-svg .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#my-svg .icon-shape p,#my-svg .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#my-svg .icon-shape .label rect,#my-svg .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#my-svg .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#my-svg .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#my-svg .node .neo-node{stroke:#9370DB;}#my-svg [data-look="neo"].node rect,#my-svg [data-look="neo"].cluster rect,#my-svg [data-look="neo"].node polygon{stroke:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#my-svg [data-look="neo"].node path{stroke:#9370DB;stroke-width:1px;}#my-svg [data-look="neo"].node .outer-path{filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#my-svg [data-look="neo"].node .neo-line path{stroke:#9370DB;filter:none;}#my-svg [data-look="neo"].node circle{stroke:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#my-svg [data-look="neo"].node circle .state-start{fill:#000000;}#my-svg [data-look="neo"].icon-shape .icon{fill:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#my-svg [data-look="neo"].icon-shape .icon-neo path{stroke:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#my-svg :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}#my-svg .worker>*{fill:rgb(232, 240, 254)!important;stroke:rgb(66, 133, 244)!important;color:rgb(26, 26, 46)!important;}#my-svg .worker span{fill:rgb(232, 240, 254)!important;stroke:rgb(66, 133, 244)!important;color:rgb(26, 26, 46)!important;}#my-svg .worker tspan{fill:rgb(26, 26, 46)!important;}#my-svg .main>*{fill:rgb(254, 247, 224)!important;stroke:rgb(249, 171, 0)!important;color:rgb(26, 26, 46)!important;}#my-svg .main span{fill:rgb(254, 247, 224)!important;stroke:rgb(249, 171, 0)!important;color:rgb(26, 26, 46)!important;}#my-svg .main tspan{fill:rgb(26, 26, 46)!important;}</style><g><marker id="my-svg_flowchart-v2-pointEnd" class="marker flowchart-v2" viewBox="0 0 10 10" refX="5" refY="5" markerUnits="userSpaceOnUse" markerWidth="8" markerHeight="8" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"/></marker><marker id="my-svg_flowchart-v2-pointStart" class="marker flowchart-v2" viewBox="0 0 10 10" refX="4.5" refY="5" markerUnits="userSpaceOnUse" markerWidth="8" markerHeight="8" orient="auto"><path d="M 0 5 L 10 10 L 10 0 z" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"/></marker><marker id="my-svg_flowchart-v2-pointEnd-margin" class="marker flowchart-v2" viewBox="0 0 11.5 14" refX="11.5" refY="7" markerUnits="userSpaceOnUse" markerWidth="10.5" markerHeight="14" orient="auto"><path d="M 0 0 L 11.5 7 L 0 14 z" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"/></marker><marker id="my-svg_flowchart-v2-pointStart-margin" class="marker flowchart-v2" viewBox="0 0 11.5 14" refX="1" refY="7" markerUnits="userSpaceOnUse" markerWidth="11.5" markerHeight="14" orient="auto"><polygon points="0,7 11.5,14 11.5,0" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"/></marker><marker id="my-svg_flowchart-v2-circleEnd" class="marker flowchart-v2" viewBox="0 0 10 10" refX="11" refY="5" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"/></marker><marker id="my-svg_flowchart-v2-circleStart" class="marker flowchart-v2" viewBox="0 0 10 10" refX="-1" refY="5" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"/></marker><marker id="my-svg_flowchart-v2-circleEnd-margin" class="marker flowchart-v2" viewBox="0 0 10 10" refY="5" refX="12.25" markerUnits="userSpaceOnUse" markerWidth="14" markerHeight="14" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"/></marker><marker id="my-svg_flowchart-v2-circleStart-margin" class="marker flowchart-v2" viewBox="0 0 10 10" refX="-2" refY="5" markerUnits="userSpaceOnUse" markerWidth="14" markerHeight="14" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"/></marker><marker id="my-svg_flowchart-v2-crossEnd" class="marker cross flowchart-v2" viewBox="0 0 11 11" refX="12" refY="5.2" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><path d="M 1,1 l 9,9 M 10,1 l -9,9" class="arrowMarkerPath" style="stroke-width: 2; stroke-dasharray: 1, 0;"/></marker><marker id="my-svg_flowchart-v2-crossStart" class="marker cross flowchart-v2" viewBox="0 0 11 11" refX="-1" refY="5.2" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><path d="M 1,1 l 9,9 M 10,1 l -9,9" class="arrowMarkerPath" style="stroke-width: 2; stroke-dasharray: 1, 0;"/></marker><marker id="my-svg_flowchart-v2-crossEnd-margin" class="marker cross flowchart-v2" viewBox="0 0 15 15" refX="17.7" refY="7.5" markerUnits="userSpaceOnUse" markerWidth="12" markerHeight="12" orient="auto"><path d="M 1,1 L 14,14 M 1,14 L 14,1" class="arrowMarkerPath" style="stroke-width: 2.5;"/></marker><marker id="my-svg_flowchart-v2-crossStart-margin" class="marker cross flowchart-v2" viewBox="0 0 15 15" refX="-3.5" refY="7.5" markerUnits="userSpaceOnUse" markerWidth="12" markerHeight="12" orient="auto"><path d="M 1,1 L 14,14 M 1,14 L 14,1" class="arrowMarkerPath" style="stroke-width: 2.5; stroke-dasharray: 1, 0;"/></marker><g class="root"><g class="clusters"/><g class="edgePaths"><path d="M1096.293,86L1096.293,90.167C1096.293,94.333,1096.293,102.667,1096.293,110.333C1096.293,118,1096.293,125,1096.293,128.5L1096.293,132" id="my-svg-L_CF_DI_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_CF_DI_0" data-points="W3sieCI6MTA5Ni4yOTI5Njg3NSwieSI6ODZ9LHsieCI6MTA5Ni4yOTI5Njg3NSwieSI6MTExfSx7IngiOjEwOTYuMjkyOTY4NzUsInkiOjEzNn1d" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M1036.668,187.589L996.085,196.158C955.503,204.726,874.337,221.863,833.755,233.932C793.172,246,793.172,253,793.172,256.5L793.172,260" id="my-svg-L_DI_NV_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_DI_NV_0" data-points="W3sieCI6MTAzNi42Njc5Njg3NSwieSI6MTg3LjU4OTAyODIwOTEyNjR9LHsieCI6NzkzLjE3MTg3NSwieSI6MjM5fSx7IngiOjc5My4xNzE4NzUsInkiOjI2NH1d" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M1107.186,214L1108.349,218.167C1109.513,222.333,1111.84,230.667,1113.004,245.5C1114.168,260.333,1114.168,281.667,1114.168,303C1114.168,324.333,1114.168,345.667,1114.168,367C1114.168,388.333,1114.168,409.667,1114.168,431C1114.168,452.333,1114.168,473.667,1114.168,495C1114.168,516.333,1114.168,537.667,1114.168,559C1114.168,580.333,1114.168,601.667,1114.168,617.833C1114.168,634,1114.168,645,1114.168,650.5L1114.168,656" id="my-svg-L_DI_DR_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_DI_DR_0" data-points="W3sieCI6MTEwNy4xODU1NDY4NzUsInkiOjIxNH0seyJ4IjoxMTE0LjE2Nzk2ODc1LCJ5IjoyMzl9LHsieCI6MTExNC4xNjc5Njg3NSwieSI6MzAzfSx7IngiOjExMTQuMTY3OTY4NzUsInkiOjM2N30seyJ4IjoxMTE0LjE2Nzk2ODc1LCJ5Ijo0MzF9LHsieCI6MTExNC4xNjc5Njg3NSwieSI6NDk1fSx7IngiOjExMTQuMTY3OTY4NzUsInkiOjU1OX0seyJ4IjoxMTE0LjE2Nzk2ODc1LCJ5Ijo2MjN9LHsieCI6MTExNC4xNjc5Njg3NSwieSI6NjYwfV0=" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M1155.918,201.049L1170.396,207.374C1184.874,213.699,1213.829,226.35,1228.307,243.342C1242.785,260.333,1242.785,281.667,1242.785,303C1242.785,324.333,1242.785,345.667,1242.785,367C1242.785,388.333,1242.785,409.667,1242.785,431C1242.785,452.333,1242.785,473.667,1242.785,495C1242.785,516.333,1242.785,537.667,1242.785,559C1242.785,580.333,1242.785,601.667,1242.785,625C1242.785,648.333,1242.785,673.667,1242.785,699C1242.785,724.333,1242.785,749.667,1242.785,773C1242.785,796.333,1242.785,817.667,1242.785,839C1242.785,860.333,1242.785,881.667,1242.785,908.833C1242.785,936,1242.785,969,1242.785,1002C1242.785,1035,1242.785,1068,1242.785,1095.167C1242.785,1122.333,1242.785,1143.667,1242.785,1165C1242.785,1186.333,1242.785,1207.667,1242.785,1221.833C1242.785,1236,1242.785,1243,1242.785,1246.5L1242.785,1250" id="my-svg-L_DI_DS_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_DI_DS_0" data-points="W3sieCI6MTE1NS45MTc5Njg3NSwieSI6MjAxLjA0OTE3MDcxMDg5NTQyfSx7IngiOjEyNDIuNzg1MTU2MjUsInkiOjIzOX0seyJ4IjoxMjQyLjc4NTE1NjI1LCJ5IjozMDN9LHsieCI6MTI0Mi43ODUxNTYyNSwieSI6MzY3fSx7IngiOjEyNDIuNzg1MTU2MjUsInkiOjQzMX0seyJ4IjoxMjQyLjc4NTE1NjI1LCJ5Ijo0OTV9LHsieCI6MTI0Mi43ODUxNTYyNSwieSI6NTU5fSx7IngiOjEyNDIuNzg1MTU2MjUsInkiOjYyM30seyJ4IjoxMjQyLjc4NTE1NjI1LCJ5Ijo2OTl9LHsieCI6MTI0Mi43ODUxNTYyNSwieSI6Nzc1fSx7IngiOjEyNDIuNzg1MTU2MjUsInkiOjgzOX0seyJ4IjoxMjQyLjc4NTE1NjI1LCJ5Ijo5MDN9LHsieCI6MTI0Mi43ODUxNTYyNSwieSI6MTAwMn0seyJ4IjoxMjQyLjc4NTE1NjI1LCJ5IjoxMTAxfSx7IngiOjEyNDIuNzg1MTU2MjUsInkiOjExNjV9LHsieCI6MTI0Mi43ODUxNTYyNSwieSI6MTIyOX0seyJ4IjoxMjQyLjc4NTE1NjI1LCJ5IjoxMjU0fV0=" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M750.68,332.287L742.286,338.073C733.892,343.858,717.104,355.429,708.71,364.715C700.316,374,700.316,381,700.316,384.5L700.316,388" id="my-svg-L_NV_MI_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_NV_MI_0" data-points="W3sieCI6NzUwLjY3OTY4NzUsInkiOjMzMi4yODc0NTEwOTU4NzMxfSx7IngiOjcwMC4zMTY0MDYyNSwieSI6MzY3fSx7IngiOjcwMC4zMTY0MDYyNSwieSI6MzkyfV0=" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M835.664,324.713L849.456,331.761C863.249,338.809,890.833,352.904,904.626,370.619C918.418,388.333,918.418,409.667,918.418,431C918.418,452.333,918.418,473.667,918.418,495C918.418,516.333,918.418,537.667,918.418,559C918.418,580.333,918.418,601.667,918.418,617.833C918.418,634,918.418,645,918.418,650.5L918.418,656" id="my-svg-L_NV_BU_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_NV_BU_0" data-points="W3sieCI6ODM1LjY2NDA2MjUsInkiOjMyNC43MTMyNTIwMzUwNTZ9LHsieCI6OTE4LjQxNzk2ODc1LCJ5IjozNjd9LHsieCI6OTE4LjQxNzk2ODc1LCJ5Ijo0MzF9LHsieCI6OTE4LjQxNzk2ODc1LCJ5Ijo0OTV9LHsieCI6OTE4LjQxNzk2ODc1LCJ5Ijo1NTl9LHsieCI6OTE4LjQxNzk2ODc1LCJ5Ijo2MjN9LHsieCI6OTE4LjQxNzk2ODc1LCJ5Ijo2NjB9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M653.487,470L648.484,474.167C643.481,478.333,633.475,486.667,628.472,494.333C623.469,502,623.469,509,623.469,512.5L623.469,516" id="my-svg-L_MI_SE_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_MI_SE_0" data-points="W3sieCI6NjUzLjQ4NzM2NTcyMjY1NjIsInkiOjQ3MH0seyJ4Ijo2MjMuNDY4NzUsInkiOjQ5NX0seyJ4Ijo2MjMuNDY4NzUsInkiOjUyMH1d" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M747.145,470L752.149,474.167C757.152,478.333,767.158,486.667,772.161,494.333C777.164,502,777.164,509,777.164,512.5L777.164,516" id="my-svg-L_MI_LD_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_MI_LD_0" data-points="W3sieCI6NzQ3LjE0NTQ0Njc3NzM0MzgsInkiOjQ3MH0seyJ4Ijo3NzcuMTY0MDYyNSwieSI6NDk1fSx7IngiOjc3Ny4xNjQwNjI1LCJ5Ijo1MjB9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M623.469,598L623.469,602.167C623.469,606.333,623.469,614.667,627.208,622.531C630.947,630.396,638.425,637.792,642.164,641.489L645.904,645.187" id="my-svg-L_SE_RC_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_SE_RC_0" data-points="W3sieCI6NjIzLjQ2ODc1LCJ5Ijo1OTh9LHsieCI6NjIzLjQ2ODc1LCJ5Ijo2MjN9LHsieCI6NjQ4Ljc0NzU4NDI5Mjc2MzEsInkiOjY0OH1d" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M777.164,598L777.164,602.167C777.164,606.333,777.164,614.667,773.425,622.531C769.686,630.396,762.208,637.792,758.468,641.489L754.729,645.187" id="my-svg-L_LD_RC_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_LD_RC_0" data-points="W3sieCI6Nzc3LjE2NDA2MjUsInkiOjU5OH0seyJ4Ijo3NzcuMTY0MDYyNSwieSI6NjIzfSx7IngiOjc1MS44ODUyMjgyMDcyMzY5LCJ5Ijo2NDh9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M918.418,738L918.418,744.167C918.418,750.333,918.418,762.667,896.325,776.029C874.231,789.391,830.045,803.783,807.951,810.978L785.858,818.174" id="my-svg-L_BU_DP_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_BU_DP_0" data-points="W3sieCI6OTE4LjQxNzk2ODc1LCJ5Ijo3Mzh9LHsieCI6OTE4LjQxNzk2ODc1LCJ5Ijo3NzV9LHsieCI6NzgyLjA1NDY4NzUsInkiOjgxOS40MTI2MDMxMjA5NjIxfV0=" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M700.316,750L700.316,754.167C700.316,758.333,700.316,766.667,701.509,774.368C702.702,782.07,705.088,789.14,706.281,792.675L707.474,796.21" id="my-svg-L_RC_DP_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_RC_DP_0" data-points="W3sieCI6NzAwLjMxNjQwNjI1LCJ5Ijo3NTB9LHsieCI6NzAwLjMxNjQwNjI1LCJ5Ijo3NzV9LHsieCI6NzA4Ljc1Mjk5MDcyMjY1NjIsInkiOjgwMH1d" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M178.582,738L178.582,744.167C178.582,750.333,178.582,762.667,258.452,778.241C338.322,793.816,498.061,812.632,577.931,822.04L657.801,831.448" id="my-svg-L_BI_DP_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_BI_DP_0" data-points="W3sieCI6MTc4LjU4MjAzMTI1LCJ5Ijo3Mzh9LHsieCI6MTc4LjU4MjAzMTI1LCJ5Ijo3NzV9LHsieCI6NjYxLjc3MzQzNzUsInkiOjgzMS45MTU5MzM5NDM0NzY3fV0=" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M307.542,738L300.505,744.167C293.467,750.333,279.392,762.667,337.77,778.003C396.148,793.338,526.98,811.677,592.396,820.846L657.812,830.015" id="my-svg-L_MM_DP_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_MM_DP_0" data-points="W3sieCI6MzA3LjU0MjM1MTk3MzY4NDIsInkiOjczOH0seyJ4IjoyNjUuMzE2NDA2MjUsInkiOjc3NX0seyJ4Ijo2NjEuNzczNDM3NSwieSI6ODMwLjU3MDI1ODk2MzYzMjF9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M1069.074,738L1061.944,744.167C1054.814,750.333,1040.553,762.667,993.369,777.255C946.185,791.844,866.077,808.688,826.023,817.11L785.969,825.532" id="my-svg-L_DR_DP_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_DR_DP_0" data-points="W3sieCI6MTA2OS4wNzQyMTg3NSwieSI6NzM4fSx7IngiOjEwMjYuMjkyOTY4NzUsInkiOjc3NX0seyJ4Ijo3ODIuMDU0Njg3NSwieSI6ODI2LjM1NDU3NzA3MTY0OTV9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M653.383,1188.509L633.71,1195.257C614.036,1202.006,574.69,1215.503,516.558,1230.932C458.426,1246.362,381.507,1263.724,343.048,1272.405L304.589,1281.086" id="my-svg-L_RJ_WR_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_RJ_WR_0" data-points="W3sieCI6NjUzLjM4MjgxMjUsInkiOjExODguNTA4NTYzMjkyOTk0NX0seyJ4Ijo1MzUuMzQzNzUsInkiOjEyMjl9LHsieCI6MzAwLjY4NzUsInkiOjEyODEuOTY2MjQ2OTM0NjcwNn1d" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M51.68,1204L51.68,1208.167C51.68,1212.333,51.68,1220.667,76.252,1232.691C100.824,1244.716,149.968,1260.433,174.54,1268.291L199.112,1276.149" id="my-svg-L_SC_WR_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_SC_WR_0" data-points="W3sieCI6NTEuNjc5Njg3NSwieSI6MTIwNH0seyJ4Ijo1MS42Nzk2ODc1LCJ5IjoxMjI5fSx7IngiOjIwMi45MjE4NzUsInkiOjEyNzcuMzY3MjcwNDU1OTY1MX1d" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M349.054,738L348.58,744.167C348.106,750.333,347.159,762.667,346.685,779.5C346.211,796.333,346.211,817.667,346.211,839C346.211,860.333,346.211,881.667,346.211,908.833C346.211,936,346.211,969,346.211,1002C346.211,1035,346.211,1068,346.211,1095.167C346.211,1122.333,346.211,1143.667,346.211,1165C346.211,1186.333,346.211,1207.667,339.176,1223.103C332.14,1238.539,318.069,1248.078,311.034,1252.847L303.998,1257.617" id="my-svg-L_MM_WR_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_MM_WR_0" data-points="W3sieCI6MzQ5LjA1NDAxOTMyNTY1NzksInkiOjczOH0seyJ4IjozNDYuMjEwOTM3NSwieSI6Nzc1fSx7IngiOjM0Ni4yMTA5Mzc1LCJ5Ijo4Mzl9LHsieCI6MzQ2LjIxMDkzNzUsInkiOjkwM30seyJ4IjozNDYuMjEwOTM3NSwieSI6MTAwMn0seyJ4IjozNDYuMjEwOTM3NSwieSI6MTEwMX0seyJ4IjozNDYuMjEwOTM3NSwieSI6MTE2NX0seyJ4IjozNDYuMjEwOTM3NSwieSI6MTIyOX0seyJ4IjozMDAuNjg3NSwieSI6MTI1OS44NjEzMDQyMDM5MDZ9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M166.068,1204L161.649,1208.167C157.23,1212.333,148.393,1220.667,153.956,1230.525C159.519,1240.383,179.483,1251.765,189.465,1257.457L199.447,1263.148" id="my-svg-L_PD_WR_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_PD_WR_0" data-points="W3sieCI6MTY2LjA2ODM1OTM3NSwieSI6MTIwNH0seyJ4IjoxMzkuNTU0Njg3NSwieSI6MTIyOX0seyJ4IjoyMDIuOTIxODc1LCJ5IjoxMjY1LjEyOTE3NTk0NjU0Nzh9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M721.914,1204L721.914,1208.167C721.914,1212.333,721.914,1220.667,683.94,1232.894C645.966,1245.122,570.017,1261.244,532.043,1269.305L494.069,1277.366" id="my-svg-L_RJ_SD_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_RJ_SD_0" data-points="W3sieCI6NzIxLjkxNDA2MjUsInkiOjEyMDR9LHsieCI6NzIxLjkxNDA2MjUsInkiOjEyMjl9LHsieCI6NDkwLjE1NjI1LCJ5IjoxMjc4LjE5Njk2MzAyMjQ2NjR9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M213.523,1204L214.174,1208.167C214.826,1212.333,216.128,1220.667,238.352,1231.635C260.577,1242.604,303.725,1256.207,325.299,1263.009L346.873,1269.811" id="my-svg-L_PD_SD_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_PD_SD_0" data-points="W3sieCI6MjEzLjUyMzQzNzUsInkiOjEyMDR9LHsieCI6MjE3LjQyOTY4NzUsInkiOjEyMjl9LHsieCI6MzUwLjY4NzUsInkiOjEyNzEuMDEzOTMyMTg2NDI5N31d" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M251.805,1332L251.805,1336.167C251.805,1340.333,251.805,1348.667,316.51,1361.951C381.216,1375.236,510.628,1393.472,575.333,1402.59L640.039,1411.708" id="my-svg-L_WR_WA_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_WR_WA_0" data-points="W3sieCI6MjUxLjgwNDY4NzUsInkiOjEzMzJ9LHsieCI6MjUxLjgwNDY4NzUsInkiOjEzNTd9LHsieCI6NjQ0LCJ5IjoxNDEyLjI2NjUyMjE3MjkxMDh9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M420.422,1332L420.422,1336.167C420.422,1340.333,420.422,1348.667,457.034,1361.039C493.647,1373.412,566.872,1389.823,603.484,1398.029L640.097,1406.235" id="my-svg-L_SD_WA_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_SD_WA_0" data-points="W3sieCI6NDIwLjQyMTg3NSwieSI6MTMzMn0seyJ4Ijo0MjAuNDIxODc1LCJ5IjoxMzU3fSx7IngiOjY0NCwieSI6MTQwNy4xMDk0OTA4NDg0MDM1fV0=" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M1119.3,738L1120.111,744.167C1120.922,750.333,1122.545,762.667,1123.357,779.5C1124.168,796.333,1124.168,817.667,1124.168,839C1124.168,860.333,1124.168,881.667,1124.168,908.833C1124.168,936,1124.168,969,1124.168,1002C1124.168,1035,1124.168,1068,1124.168,1095.167C1124.168,1122.333,1124.168,1143.667,1124.168,1165C1124.168,1186.333,1124.168,1207.667,1124.168,1229C1124.168,1250.333,1124.168,1271.667,1124.168,1293C1124.168,1314.333,1124.168,1335.667,1065.458,1355.318C1006.748,1374.97,889.327,1392.94,830.617,1401.925L771.907,1410.91" id="my-svg-L_DR_WA_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_DR_WA_0" data-points="W3sieCI6MTExOS4yOTk1NDc2OTczNjgzLCJ5Ijo3Mzh9LHsieCI6MTEyNC4xNjc5Njg3NSwieSI6Nzc1fSx7IngiOjExMjQuMTY3OTY4NzUsInkiOjgzOX0seyJ4IjoxMTI0LjE2Nzk2ODc1LCJ5Ijo5MDN9LHsieCI6MTEyNC4xNjc5Njg3NSwieSI6MTAwMn0seyJ4IjoxMTI0LjE2Nzk2ODc1LCJ5IjoxMTAxfSx7IngiOjExMjQuMTY3OTY4NzUsInkiOjExNjV9LHsieCI6MTEyNC4xNjc5Njg3NSwieSI6MTIyOX0seyJ4IjoxMTI0LjE2Nzk2ODc1LCJ5IjoxMjkzfSx7IngiOjExMjQuMTY3OTY4NzUsInkiOjEzNTd9LHsieCI6NzY3Ljk1MzEyNSwieSI6MTQxMS41MTUxMDg3NzM4MzA3fV0=" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M1242.785,1332L1242.785,1336.167C1242.785,1340.333,1242.785,1348.667,1164.308,1362.19C1085.832,1375.712,928.878,1394.425,850.402,1403.781L771.925,1413.137" id="my-svg-L_DS_WA_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_DS_WA_0" data-points="W3sieCI6MTI0Mi43ODUxNTYyNSwieSI6MTMzMn0seyJ4IjoxMjQyLjc4NTE1NjI1LCJ5IjoxMzU3fSx7IngiOjc2Ny45NTMxMjUsInkiOjE0MTMuNjEwOTYwMzE5NTk3fV0=" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M705.977,1460L705.977,1464.167C705.977,1468.333,705.977,1476.667,705.977,1484.333C705.977,1492,705.977,1499,705.977,1502.5L705.977,1506" id="my-svg-L_WA_WO_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_WA_WO_0" data-points="W3sieCI6NzA1Ljk3NjU2MjUsInkiOjE0NjB9LHsieCI6NzA1Ljk3NjU2MjUsInkiOjE0ODV9LHsieCI6NzA1Ljk3NjU2MjUsInkiOjE1MTB9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M790.445,1178.996L831.251,1187.33C872.057,1195.664,953.669,1212.332,933.629,1229.727C913.59,1247.121,791.898,1265.242,731.052,1274.303L670.206,1283.363" id="my-svg-L_RJ_WP_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_RJ_WP_0" data-points="W3sieCI6NzkwLjQ0NTMxMjUsInkiOjExNzguOTk2MzYwMTAwNzIwNX0seyJ4IjoxMDM1LjI4MTI1LCJ5IjoxMjI5fSx7IngiOjY2Ni4yNSwieSI6MTI4My45NTI1Mzg0OTA5MDJ9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M359.317,738L360.466,744.167C361.615,750.333,363.913,762.667,365.062,779.5C366.211,796.333,366.211,817.667,366.211,839C366.211,860.333,366.211,881.667,366.211,908.833C366.211,936,366.211,969,366.211,1002C366.211,1035,366.211,1068,366.211,1095.167C366.211,1122.333,366.211,1143.667,366.211,1165C366.211,1186.333,366.211,1207.667,395.321,1226.119C424.431,1244.572,482.65,1260.144,511.76,1267.93L540.87,1275.716" id="my-svg-L_MM_WP_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_MM_WP_0" data-points="W3sieCI6MzU5LjMxNzE3NzIyMDM5NDc0LCJ5Ijo3Mzh9LHsieCI6MzY2LjIxMDkzNzUsInkiOjc3NX0seyJ4IjozNjYuMjEwOTM3NSwieSI6ODM5fSx7IngiOjM2Ni4yMTA5Mzc1LCJ5Ijo5MDN9LHsieCI6MzY2LjIxMDkzNzUsInkiOjEwMDJ9LHsieCI6MzY2LjIxMDkzNzUsInkiOjExMDF9LHsieCI6MzY2LjIxMDkzNzUsInkiOjExNjV9LHsieCI6MzY2LjIxMDkzNzUsInkiOjEyMjl9LHsieCI6NTQ0LjczNDM3NSwieSI6MTI3Ni43NDkyNDkwNTMxNTR9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M721.914,878L721.914,882.167C721.914,886.333,721.914,894.667,721.914,902.333C721.914,910,721.914,917,721.914,920.5L721.914,924" id="my-svg-L_DP_fanout_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_DP_fanout_0" data-points="W3sieCI6NzIxLjkxNDA2MjUsInkiOjg3OH0seyJ4Ijo3MjEuOTE0MDYyNSwieSI6OTAzfSx7IngiOjcyMS45MTQwNjI1LCJ5Ijo5Mjh9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/><path d="M721.914,1076L721.914,1080.167C721.914,1084.333,721.914,1092.667,721.914,1100.333C721.914,1108,721.914,1115,721.914,1118.5L721.914,1122" id="my-svg-L_fanout_RJ_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_fanout_RJ_0" data-points="W3sieCI6NzIxLjkxNDA2MjUsInkiOjEwNzZ9LHsieCI6NzIxLjkxNDA2MjUsInkiOjExMDF9LHsieCI6NzIxLjkxNDA2MjUsInkiOjExMjZ9XQ==" data-look="classic" marker-end="url(#my-svg_flowchart-v2-pointEnd)"/></g><g class="edgeLabels"><g class="edgeLabel"><g class="label" data-id="L_CF_DI_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_DI_NV_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_DI_DR_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_DI_DS_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_NV_MI_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_NV_BU_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_MI_SE_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_MI_LD_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_SE_RC_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_LD_RC_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_BU_DP_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_RC_DP_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_BI_DP_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_MM_DP_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_DR_DP_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_RJ_WR_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_SC_WR_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_MM_WR_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_PD_WR_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_RJ_SD_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_PD_SD_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_WR_WA_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_SD_WA_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_DR_WA_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_DS_WA_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_WA_WO_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_RJ_WP_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_MM_WP_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_DP_fanout_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_fanout_RJ_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g></g><g class="nodes"><g class="root" transform="translate(410.6484375, 920)"><g class="clusters"><g class="cluster" id="my-svg-fanout" data-look="classic"><rect style="" x="8" y="8" width="606.53125" height="148"/><g class="cluster-label" transform="translate(311.265625, 8)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5;"><span class="nodeLabel"></span></div></foreignObject></g></g></g><g class="edgePaths"/><g class="edgeLabels"/><g class="nodes"><g class="node default worker" id="my-svg-flowchart-R0-34" data-look="classic" transform="translate(106.40625, 82)"><rect class="basic label-container" style="fill:#e8f0fe !important;stroke:#4285f4 !important" x="-60.90625" y="-39" width="121.8125" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-30.90625, -24)"><rect/><foreignObject width="61.8125" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>render:0<br /><small>[W]</small></p></span></div></foreignObject></g></g><g class="node default worker" id="my-svg-flowchart-R1-35" data-look="classic" transform="translate(303.21875, 82)"><rect class="basic label-container" style="fill:#e8f0fe !important;stroke:#4285f4 !important" x="-60.90625" y="-39" width="121.8125" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-30.90625, -24)"><rect/><foreignObject width="61.8125" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>render:1<br /><small>[W]</small></p></span></div></foreignObject></g></g><g class="node default worker" id="my-svg-flowchart-RN-36" data-look="classic" transform="translate(508.078125, 82)"><rect class="basic label-container" style="fill:#e8f0fe !important;stroke:#4285f4 !important" x="-68.953125" y="-39" width="137.90625" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-38.953125, -24)"><rect/><foreignObject width="77.90625" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>render:N-1<br /><small>[W]</small></p></span></div></foreignObject></g></g></g></g><g class="node default worker" id="my-svg-flowchart-BI-0" data-look="classic" transform="translate(178.58203125, 699)"><rect class="basic label-container" style="fill:#e8f0fe !important;stroke:#4285f4 !important" x="-61.7734375" y="-39" width="123.546875" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-31.7734375, -24)"><rect/><foreignObject width="63.546875" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>buildInfo<br /><small>[W]</small></p></span></div></foreignObject></g></g><g class="node default worker" id="my-svg-flowchart-SC-1" data-look="classic" transform="translate(51.6796875, 1165)"><rect class="basic label-container" style="fill:#e8f0fe !important;stroke:#4285f4 !important" x="-43.6796875" y="-39" width="87.359375" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-13.6796875, -24)"><rect/><foreignObject width="27.359375" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>scss<br /><small>[W]</small></p></span></div></foreignObject></g></g><g class="node default worker" id="my-svg-flowchart-MM-2" data-look="classic" transform="translate(352.05078125, 699)"><rect class="basic label-container" style="fill:#e8f0fe !important;stroke:#4285f4 !important" x="-61.6953125" y="-39" width="123.390625" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-31.6953125, -24)"><rect/><foreignObject width="63.390625" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>mermaid<br /><small>[W]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-PD-3" data-look="classic" transform="translate(207.4296875, 1165)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-62.0703125" y="-39" width="124.140625" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-32.0703125, -24)"><rect/><foreignObject width="64.140625" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>prepDest<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-CF-4" data-look="classic" transform="translate(1096.29296875, 47)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-51.8828125" y="-39" width="103.765625" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-21.8828125, -24)"><rect/><foreignObject width="43.765625" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>config<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-DI-5" data-look="classic" transform="translate(1096.29296875, 175)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-59.625" y="-39" width="119.25" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-29.625, -24)"><rect/><foreignObject width="59.25" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>discover<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-NV-7" data-look="classic" transform="translate(793.171875, 303)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-42.4921875" y="-39" width="84.984375" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-12.4921875, -24)"><rect/><foreignObject width="24.984375" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>nav<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-DR-9" data-look="classic" transform="translate(1114.16796875, 699)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-85.7734375" y="-39" width="171.546875" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-55.7734375, -24)"><rect/><foreignObject width="111.546875" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>deriveRedirects<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-DS-11" data-look="classic" transform="translate(1242.78515625, 1293)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-81.4609375" y="-39" width="162.921875" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-51.4609375, -24)"><rect/><foreignObject width="102.921875" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>deriveSitemap<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-MI-13" data-look="classic" transform="translate(700.31640625, 431)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-79.1171875" y="-39" width="158.234375" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-49.1171875, -24)"><rect/><foreignObject width="98.234375" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>markdownInit<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-BU-15" data-look="classic" transform="translate(918.41796875, 699)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-59.9765625" y="-39" width="119.953125" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-29.9765625, -24)"><rect/><foreignObject width="59.953125" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>buildInit<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-SE-17" data-look="classic" transform="translate(623.46875, 559)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-41.8984375" y="-39" width="83.796875" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-11.8984375, -24)"><rect/><foreignObject width="23.796875" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>seo<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-LD-19" data-look="classic" transform="translate(777.1640625, 559)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-61.796875" y="-39" width="123.59375" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-31.796875, -24)"><rect/><foreignObject width="63.59375" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>loadData<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-RC-21" data-look="classic" transform="translate(700.31640625, 699)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-75.734375" y="-51" width="151.46875" height="102"/><g class="label" style="color:#1a1a2e !important" transform="translate(-45.734375, -36)"><rect/><foreignObject width="91.46875" height="72"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>resolveBook-<br />Chapters<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-DP-25" data-look="classic" transform="translate(721.9140625, 839)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-60.140625" y="-39" width="120.28125" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-30.140625, -24)"><rect/><foreignObject width="60.28125" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>dispatch<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-RJ-40" data-look="classic" transform="translate(721.9140625, 1165)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-68.53125" y="-39" width="137.0625" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-38.53125, -24)"><rect/><foreignObject width="77.0625" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>renderJoin<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-WR-42" data-look="classic" transform="translate(251.8046875, 1293)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-48.8828125" y="-39" width="97.765625" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-18.8828125, -24)"><rect/><foreignObject width="37.765625" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>write<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-SD-50" data-look="classic" transform="translate(420.421875, 1293)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-69.734375" y="-39" width="139.46875" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-39.734375, -24)"><rect/><foreignObject width="79.46875" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>searchData<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-WA-54" data-look="classic" transform="translate(705.9765625, 1421)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-61.9765625" y="-39" width="123.953125" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-31.9765625, -24)"><rect/><foreignObject width="63.953125" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>writeAux<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-WO-62" data-look="classic" transform="translate(705.9765625, 1549)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-73.5625" y="-39" width="147.125" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-43.5625, -24)"><rect/><foreignObject width="87.125" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>writeOffline<br /><small>[M]</small></p></span></div></foreignObject></g></g><g class="node default main" id="my-svg-flowchart-WP-64" data-look="classic" transform="translate(605.4921875, 1293)"><rect class="basic label-container" style="fill:#fef7e0 !important;stroke:#f9ab00 !important" x="-60.7578125" y="-39" width="121.515625" height="78"/><g class="label" style="color:#1a1a2e !important" transform="translate(-30.7578125, -24)"><rect/><foreignObject width="61.515625" height="48"><div style="color: rgb(26, 26, 46) !important; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml"><span style="color:#1a1a2e !important" class="nodeLabel"><p>writePdf<br /><small>[M]</small></p></span></div></foreignObject></g></g></g></g></g><defs><filter id="my-svg-drop-shadow" height="130%" width="130%"><feDropShadow dx="4" dy="4" stdDeviation="0" flood-opacity="0.06" flood-color="#000000"/></filter></defs><defs><filter id="my-svg-drop-shadow-small" height="150%" width="150%"><feDropShadow dx="2" dy="2" stdDeviation="0" flood-opacity="0.06" flood-color="#000000"/></filter></defs></svg> \ No newline at end of file From efd08a3b155e3d72de363c12bd85909ed4558463 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sat, 30 May 2026 22:44:54 +0200 Subject: [PATCH 17/72] Make the build console output colorful. --- builder/scheduler.mjs | 26 ++++++++++++++++++++++---- builder/tbdocs.mjs | 24 ++++++++++++++---------- 2 files changed, 36 insertions(+), 14 deletions(-) diff --git a/builder/scheduler.mjs b/builder/scheduler.mjs index dd96ae10..19db5daf 100644 --- a/builder/scheduler.mjs +++ b/builder/scheduler.mjs @@ -1,6 +1,8 @@ // Task-graph scheduler for the tbdocs build pipeline. See PLAN-scheduler.md // for the full design, data-flow diagram, and task placement rationale. +import pc from "picocolors"; + export class SharedState { pages = []; // master copy; mutated in place by [M] tasks and render delta merges staticFiles = []; // master copy; mermaid.submit appends new SVG descriptors @@ -101,10 +103,26 @@ export class Scheduler { } summary() { - return [...this.timings.entries()] - .sort((a, b) => a[1].start - b[1].start) - .map(([id, { start, end }]) => `${id}=${end - start}ms`) - .join(" "); + const sorted = [...this.timings.entries()] + .sort((a, b) => a[1].start - b[1].start); + + const renderMap = new Map(); + const parts = []; + for (const [id, timing] of sorted) { + const m = id.match(/^render:(\d+)$/); + if (m) renderMap.set(Number(m[1]), timing); + else parts.push(`${id}=${timing.end - timing.start}ms`); + } + + let result = pc.dim(parts.join(" ")); + if (renderMap.size > 0) { + const renderEntries = [...renderMap.entries()].sort((a, b) => a[0] - b[0]); + const wallMs = Math.max(...renderEntries.map(([, t]) => t.end)) + - Math.min(...renderEntries.map(([, t]) => t.start)); + const inner = renderEntries.map(([i, t]) => `${i}=${t.end - t.start}ms`).join(", "); + result += `\n${pc.bold(pc.yellow("render:"))} ${pc.white(`${wallMs}ms,`)} ${pc.dim(inner)}`; + } + return result; } } diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index bbcc21d4..3598ac36 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -19,6 +19,7 @@ import path from "node:path"; import process from "node:process"; import { fileURLToPath } from "node:url"; import yaml from "js-yaml"; +import pc from "picocolors"; import { WorkerPool } from "./worker-pool.mjs"; import { Scheduler } from "./scheduler.mjs"; @@ -512,6 +513,7 @@ function chunkPages(pages, workers) { // ── Build entry point ───────────────────────────────────────────────────────── export async function runBuild(opts) { + const buildStart = Date.now(); const { src, dest } = opts; const srcRoot = path.resolve(process.cwd(), src); const destRoot = path.resolve(dest ?? path.join(srcRoot, "_site")); @@ -547,30 +549,32 @@ export async function runBuild(opts) { const offlineResult = results.get("writeOffline"); const pdfResult = results.get("writePdf"); - console.log(`Phase 1+2+3+4+5+6+7+8 done: ${pages.length} pages, ${staticFiles.length} static files`); - console.log(` wrote: ${writeStats.pages.written} pages (${writeStats.pages.skipped} skipped), ` + - `${writeStats.theme.copied} theme assets, ${writeStats.staticFiles.copied} static files ` + - `-> ${destRoot}`); + console.log(`Done in ${pc.bold(pc.green(`${Date.now() - buildStart}ms`))}: ${pages.length} pages, ${staticFiles.length} static files`); + console.log(` ${pc.bold("wrote:")} -> ${pc.cyan(destRoot)}`); + console.log(` ${writeStats.pages.written} pages (${writeStats.pages.skipped} skipped), ` + + `${writeStats.theme.copied} theme assets, ${writeStats.staticFiles.copied} static files`); if (auxResult?.redirectStats) { - console.log(` aux: ${auxResult.redirectStats.written} redirect stubs, ` + + console.log(` ${pc.bold("aux:")} ${auxResult.redirectStats.written} redirect stubs, ` + `${auxResult.sitemapStats.entries} sitemap entries, ` + `${auxResult.searchStats.entries} search-index entries`); } if (offlineResult) { - console.log(` offline: ${offlineResult.html} HTML, ${offlineResult.css} CSS, ` + + console.log(` ${pc.bold("offline:")} -> ${pc.cyan(`${destRoot}-offline`)}`); + console.log(` ${offlineResult.html} HTML, ${offlineResult.css} CSS, ` + `${offlineResult.redirects} redirect stubs, ` + `${offlineResult.statics + offlineResult.assets} assets, ` + `${offlineResult.excluded} excluded ` + - `(${offlineResult.unresolved} unresolved) -> ${destRoot}-offline`); + `(${offlineResult.unresolved} unresolved)`); if (opts.profileOffline && offlineResult.subT) { - console.log(` offline: ${offlineResult.subT.summary()}`); + console.log(` ${pc.bold("offline:")} ${offlineResult.subT.summary()}`); } } if (pdfResult) { const mb = (pdfResult.bookBytes / (1024 * 1024)).toFixed(1); const missingClause = pdfResult.missing > 0 ? ` (${pdfResult.missing} missing)` : ""; - console.log(` pdf: book.html (${mb} MB), ${pdfResult.css} CSS, ` + - `${pdfResult.images} images${missingClause} -> ${destRoot}-pdf`); + console.log(` ${pc.bold("pdf:")} -> ${pc.cyan(`${destRoot}-pdf`)}`); + console.log(` book.html (${mb} MB), ${pdfResult.css} CSS, ` + + `${pdfResult.images} images${missingClause}`); } console.log(scheduler.summary()); From 53605b8869cdf9e96589eab9b716190fb8fac08a Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sat, 30 May 2026 22:59:20 +0200 Subject: [PATCH 18/72] Output a Gantt chart of the build. --- .gitignore | 2 + builder/tbdocs.mjs | 105 ++++++++++++++++++++++++++++++++ docs/Documentation/BuildInfo.md | 15 +++++ 3 files changed, 122 insertions(+) create mode 100644 docs/Documentation/BuildInfo.md diff --git a/.gitignore b/.gitignore index 8ea362a8..686cdbcc 100644 --- a/.gitignore +++ b/.gitignore @@ -7,3 +7,5 @@ node_modules/ /before/ /after-*/ /findoverflow-baseline/ + +/build-gantt.mmd diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 3598ac36..b44d7b47 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -510,6 +510,104 @@ function chunkPages(pages, workers) { return chunks; } +// ── Gantt chart ─────────────────────────────────────────────────────────────── + +const GANTT_SECTION = { + config: "Seeds", buildInfo: "Seeds", scss: "Seeds", mermaid: "Seeds", prepDest: "Seeds", + discover: "Spine", nav: "Spine", markdownInit: "Spine", buildInit: "Spine", + seo: "Spine", loadData: "Spine", resolveBookChapters: "Spine", + deriveRedirects: "Spine", deriveSitemap: "Spine", + dispatch: "Render", renderJoin: "Render", + write: "Write", searchData: "Write", writeAux: "Write", writeOffline: "Write", writePdf: "Write", +}; +const GANTT_SECTION_ORDER = ["Seeds", "Spine", "Render", "Write"]; + +async function writeGantt(timings, outPath) { + if (timings.size === 0) return ""; + const t0 = Math.min(...[...timings.values()].map(t => t.start)); + + const grouped = new Map(GANTT_SECTION_ORDER.map(s => [s, []])); + for (const [id, { start, end }] of [...timings.entries()].sort((a, b) => a[1].start - b[1].start)) { + const section = /^render:\d+$/.test(id) ? "Render" : (GANTT_SECTION[id] ?? "Other"); + if (!grouped.has(section)) grouped.set(section, []); + grouped.get(section).push({ id, start: start - t0, end: end - t0 }); + } + + const lines = [ + "gantt", + " title Build task timeline", + " dateFormat x", + " axisFormat %S.%L", + "", + ]; + for (const [section, tasks] of grouped) { + if (tasks.length === 0) continue; + lines.push(` section ${section}`); + for (const { id, start, end } of tasks) { + const label = id.replace(":", " "); + const taskId = `t_${id.replace(/[^a-z0-9]/gi, "_")}`; + lines.push(` ${label} :done, ${taskId}, ${start}, ${Math.max(end, start + 1)}`); + } + lines.push(""); + } + + const content = lines.join("\n"); + await fs.writeFile(outPath, content, "utf8"); + return content; +} + +async function injectGanttChart(pages, destRoot, ganttContent) { + if (!ganttContent) return; + const page = pages.find(p => p.permalink === "/Documentation/Development/BuildInfo"); + if (!page) return; + + const chart = `<pre class="mermaid">\n${ganttContent}</pre>`; + // Match the rendered empty mmd code block — outer div + two closing divs. + const mmdBlockRe = /<div class="language-mmd highlighter-rouge">[\s\S]*?<\/div>\s*<\/div>/; + + // Online / serve: CDN ESM import. + const onlineScript = + `<script type="module">\n` + + `import mermaid from 'https://cdn.jsdelivr.net/npm/mermaid@11/dist/mermaid.esm.min.mjs';\n` + + `mermaid.initialize({ startOnLoad: true });\n` + + `</script>`; + + // Offline: self-contained IIFE bundle (the ESM entry point uses dynamic chunk + // imports that break on file:// URLs; the IIFE is a single self-contained file). + // Relative prefix climbs from the page's directory back to the site root. + const depth = page.destPath.split(/[/\\]/).length - 1; + const rel = "../".repeat(depth); + const offlineScript = + `<script src="${rel}assets/js/mermaid.min.js"></script>\n` + + `<script>mermaid.initialize({ startOnLoad: true });</script>`; + + const mermaidSrc = fileURLToPath( + new URL("../node_modules/mermaid/dist/mermaid.min.js", import.meta.url)); + + const targets = [ + { root: destRoot, script: onlineScript }, + { root: `${destRoot}-offline`, script: offlineScript, copyMermaid: true }, + ]; + + for (const { root, script, copyMermaid } of targets) { + const htmlPath = path.join(root, page.destPath); + let html; + try { + html = await fs.readFile(htmlPath, "utf8"); + } catch (e) { + if (e.code !== "ENOENT") throw e; + continue; + } + if (copyMermaid) { + const dest = path.join(root, "assets", "js", "mermaid.min.js"); + await fs.mkdir(path.dirname(dest), { recursive: true }); + await fs.copyFile(mermaidSrc, dest); + } + const patched = html.replace(mmdBlockRe, `${chart}\n${script}`); + if (patched !== html) await fs.writeFile(htmlPath, patched, "utf8"); + } +} + // ── Build entry point ───────────────────────────────────────────────────────── export async function runBuild(opts) { @@ -578,6 +676,13 @@ export async function runBuild(opts) { } console.log(scheduler.summary()); + const ganttPath = path.resolve(process.cwd(), "build-gantt.mmd"); + const ganttContent = await writeGantt(scheduler.timings, ganttPath); + + const injectStart = Date.now(); + await injectGanttChart(scheduler.state.pages, destRoot, ganttContent); + console.log(pc.dim(`gantt-inject=${Date.now() - injectStart}ms`)); + // Drift guard from PLAN-1.md §1. if (pages.length < 836) { console.error(`WARN: page count ${pages.length} below baseline 836`); diff --git a/docs/Documentation/BuildInfo.md b/docs/Documentation/BuildInfo.md new file mode 100644 index 00000000..4ad1e98f --- /dev/null +++ b/docs/Documentation/BuildInfo.md @@ -0,0 +1,15 @@ +--- +title: Build Info +parent: Documentation Development +nav_order: 100 +has_toc: false +permalink: /Documentation/Development/BuildInfo +--- + +# Build Info +{: .no_toc } + +Gantt chart of this build's task timeline. + +```mmd +``` From 022f4eee138a4e391324a11f4ca73c7ded1741e9 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sat, 30 May 2026 23:45:01 +0200 Subject: [PATCH 19/72] Move Shiki WASM init to a no-dep seed task (highlighterInit) --- builder/tbdocs.mjs | 39 ++++++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 13 deletions(-) diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index b44d7b47..0d29ac45 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -117,10 +117,10 @@ export function makeTimer() { // ── Task graph ──────────────────────────────────────────────────────────────── // -// Seeds (config, buildInfo, scss, mermaid, prepDest), the main-thread spine (discover → -// nav → markdownInit / buildInit → seo / loadData → resolveBookChapters + -// deriveRedirects / deriveSitemap), the render fan-out (dispatch → -// render:0..N → renderJoin), and write/post-write tasks +// Seeds (config, buildInfo, scss, mermaid, prepDest, highlighterInit), the main-thread +// spine (discover → nav → buildInit; discover + highlighterInit → markdownInit → +// seo / loadData → resolveBookChapters + deriveRedirects / deriveSitemap), the render +// fan-out (dispatch → render:0..N → renderJoin), and write/post-write tasks // (renderJoin + prepDest → searchData; write + searchData → writeAux → // writeOffline; renderJoin + mermaid → writePdf) are scheduler tasks. // runBuild() constructs the pool + scheduler, awaits start(), logs the @@ -195,6 +195,18 @@ const TASKS = { }, }, + // Shiki WASM init. Seed on main (the live highlighter object is not + // serializable across a worker boundary). Fires immediately so the WASM + // warms up while discover + nav are running. + highlighterInit: { + expected: [], + runOnMain: true, + async execute() { + return { highlighter: await initHighlighter() }; + }, + submit(out, emit) { emit("markdownInit", out); }, + }, + // ── Main-thread spine ───────────────────────────────────────────────────── discover: { @@ -210,6 +222,7 @@ const TASKS = { state.site.config = out.config; for (const p of out.pages) state.pageByDest.set(p.destPath, p); emit("nav", out); + emit("markdownInit", out); emit("deriveRedirects", out); emit("deriveSitemap", out); }, @@ -224,13 +237,12 @@ const TASKS = { return {}; }, submit(_, emit) { - emit("markdownInit", {}); - emit("buildInit", {}); + emit("buildInit", {}); }, }, // Pre-renders the sidebar/header/svg-sprite HTML used by templatePhase. - // Depends only on nav so it can start while markdownInit is in flight. + // Depends only on nav (needs state.site.navTree). buildInit: { expected: ["nav"], runOnMain: true, @@ -240,14 +252,14 @@ const TASKS = { submit(out, emit) { emit("dispatch", out); }, }, - // Shiki WASM init + link-table build + markdown-it instance creation. - // Not serializable (Shiki's highlighter is a live object), so it stays - // on main. Workers initialize their own independent highlighter instances. + // Link-table build + markdown-it assembly. The highlighter arrives + // pre-warmed from highlighterInit; pages + config come from discover. + // Synchronous: all async work is done upstream. Not serializable (live + // Shiki object), so it stays on main. Workers init their own instances. markdownInit: { - expected: ["nav"], + expected: ["discover", "highlighterInit"], runOnMain: true, - async execute(_, ctx, state) { - const highlighter = await initHighlighter(); + execute({ highlighterInit: { highlighter } }, ctx, state) { const linkTables = buildLinkTables(state.pages); const baseurl = String(state.site.config.baseurl || ""); const staticFileSet = new Set(state.staticFiles.map(s => s.srcRel)); @@ -514,6 +526,7 @@ function chunkPages(pages, workers) { const GANTT_SECTION = { config: "Seeds", buildInfo: "Seeds", scss: "Seeds", mermaid: "Seeds", prepDest: "Seeds", + highlighterInit: "Seeds", discover: "Spine", nav: "Spine", markdownInit: "Spine", buildInit: "Spine", seo: "Spine", loadData: "Spine", resolveBookChapters: "Spine", deriveRedirects: "Spine", deriveSitemap: "Spine", From 657dfe8ef5bef4d01e3b1790e37518c65a5b3606 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sat, 30 May 2026 23:51:39 +0200 Subject: [PATCH 20/72] Split scss into scssLight + scssDark workers to parallelize compilation --- builder/cpu-worker.mjs | 10 +++++++--- builder/scss.mjs | 40 +++++++++++++++++++++++---------------- builder/tbdocs.mjs | 43 +++++++++++++++++++++++++++++++----------- 3 files changed, 63 insertions(+), 30 deletions(-) diff --git a/builder/cpu-worker.mjs b/builder/cpu-worker.mjs index 46119f19..53960191 100644 --- a/builder/cpu-worker.mjs +++ b/builder/cpu-worker.mjs @@ -4,7 +4,7 @@ import { parentPort } from "node:worker_threads"; -import { compileScss } from "./scss.mjs"; +import { compileLightScss, compileDarkScss } from "./scss.mjs"; import { regenerateMermaid } from "./mermaid.mjs"; import { captureBuildInfo } from "./build-info.mjs"; @@ -24,8 +24,12 @@ import { deriveOfflinePage, deriveOfflinePageCached, const highlighterP = initHighlighter(); const handlers = { - async scss({ ctx }) { - return { scssResult: await compileScss(ctx.srcRoot) }; + async scssLight({ ctx }) { + return { scssLightResult: await compileLightScss(ctx.srcRoot) }; + }, + + async scssDark({ ctx }) { + return { scssDarkResult: await compileDarkScss(ctx.srcRoot) }; }, async mermaid({ ctx }) { diff --git a/builder/scss.mjs b/builder/scss.mjs index 1231c01a..84863866 100644 --- a/builder/scss.mjs +++ b/builder/scss.mjs @@ -13,7 +13,7 @@ // TWO sass.compile() calls are used: // 1. just-the-docs-combined.scss -- light theme // 2. just-the-docs-dark.scss -- dark theme (html.dark-mode { ... }) -// The results are concatenated into a single CSS asset. +// The results are concatenated into a single CSS asset by scssJoin in tbdocs.mjs. // // Two separate compilations are required because Dart Sass maintains one // module cache per compile() call, and a module URL can only be loaded once @@ -21,6 +21,9 @@ // modules.scss loaded with different variable values, which is only possible // in a fresh compilation with its own empty cache. // +// compileLightScss / compileDarkScss are called from separate cpu-worker.mjs +// handlers so both compilations run in parallel across two worker threads. +// // Failure modes: // SETUP -- sass not installed: throw. There is no pre-compiled fallback; // `npm install` is the fix. The error message points there. @@ -40,41 +43,46 @@ const VENDOR_JTD_SASS = path.join(__dirname, "vendor", "just-the-docs", "_sass") const SCSS_LIGHT_REL = path.join("assets", "css", "just-the-docs-combined.scss"); const SCSS_DARK_REL = path.join("assets", "css", "just-the-docs-dark.scss"); -export async function compileScss(srcRoot) { - let sass; +async function loadSass() { try { - sass = await import("sass"); + return await import("sass"); } catch (err) { throw new Error( "scss: sass not installed. Run `npm install` at the repo root to fetch it.", { cause: err }, ); } +} - const loadPaths = [ +function makeLoadPaths(srcRoot) { + return [ path.join(srcRoot, "_sass"), // our customizations first VENDOR_JTD_SASS, // gem fallback ]; +} - const compileOpts = { - style: "expanded", - sourceMap: false, - loadPaths, - }; - - let lightResult, darkResult; +export async function compileLightScss(srcRoot) { + const sass = await loadSass(); try { - lightResult = sass.compile(path.join(srcRoot, SCSS_LIGHT_REL), compileOpts); + const result = sass.compile(path.join(srcRoot, SCSS_LIGHT_REL), { + style: "expanded", sourceMap: false, loadPaths: makeLoadPaths(srcRoot), + }); + return { compiled: true, css: result.css }; } catch (err) { console.warn(`scss (light): compilation failed:\n ${err.message}`); return { compiled: false, failed: true }; } +} + +export async function compileDarkScss(srcRoot) { + const sass = await loadSass(); try { - darkResult = sass.compile(path.join(srcRoot, SCSS_DARK_REL), compileOpts); + const result = sass.compile(path.join(srcRoot, SCSS_DARK_REL), { + style: "expanded", sourceMap: false, loadPaths: makeLoadPaths(srcRoot), + }); + return { compiled: true, css: result.css }; } catch (err) { console.warn(`scss (dark): compilation failed:\n ${err.message}`); return { compiled: false, failed: true }; } - - return { compiled: true, css: lightResult.css + "\n" + darkResult.css }; } diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 0d29ac45..53b926ef 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -117,10 +117,11 @@ export function makeTimer() { // ── Task graph ──────────────────────────────────────────────────────────────── // -// Seeds (config, buildInfo, scss, mermaid, prepDest, highlighterInit), the main-thread -// spine (discover → nav → buildInit; discover + highlighterInit → markdownInit → -// seo / loadData → resolveBookChapters + deriveRedirects / deriveSitemap), the render -// fan-out (dispatch → render:0..N → renderJoin), and write/post-write tasks +// Seeds (config, buildInfo, scssLight + scssDark → scssJoin, mermaid, prepDest, +// highlighterInit), the main-thread spine (discover → nav → buildInit; +// discover + highlighterInit → markdownInit → seo / loadData → +// resolveBookChapters + deriveRedirects / deriveSitemap), the render fan-out +// (dispatch → render:0..N → renderJoin), and write/post-write tasks // (renderJoin + prepDest → searchData; write + searchData → writeAux → // writeOffline; renderJoin + mermaid → writePdf) are scheduler tasks. // runBuild() constructs the pool + scheduler, awaits start(), logs the @@ -155,10 +156,30 @@ const TASKS = { }, }, - // Sass compilation (~700 ms CPU). Worker so it overlaps with the main spine. - scss: { + // Sass compilation split across two workers so light + dark run in parallel. + // Each half is ~700 ms total serially; running concurrently saves ~200 ms. + scssLight: { expected: [], - // execute() runs in cpu-worker.mjs as the "scss" handler. + // execute() runs in cpu-worker.mjs as the "scssLight" handler. + submit(out, emit) { emit("scssJoin", out); }, + }, + + scssDark: { + expected: [], + // execute() runs in cpu-worker.mjs as the "scssDark" handler. + submit(out, emit) { emit("scssJoin", out); }, + }, + + // Joins the two parallel SCSS results and emits to write. + scssJoin: { + expected: ["scssLight", "scssDark"], + runOnMain: true, + execute({ scssLight: { scssLightResult }, scssDark: { scssDarkResult } }) { + if (scssLightResult.failed || scssDarkResult.failed) { + return { scssResult: { compiled: false, failed: true } }; + } + return { scssResult: { compiled: true, css: scssLightResult.css + "\n" + scssDarkResult.css } }; + }, submit(out, emit) { emit("write", out); }, }, @@ -422,9 +443,9 @@ const TASKS = { // mermaid (SVG descriptors appended to state.staticFiles by mermaid.submit), // and prepDest (_site/ cleaned and recreated). write: { - expected: ["renderJoin", "scss", "mermaid", "prepDest"], + expected: ["renderJoin", "scssJoin", "mermaid", "prepDest"], runOnMain: true, - async execute({ scss: { scssResult }, mermaid: { mermaidStats } }, ctx, state) { + async execute({ scssJoin: { scssResult }, mermaid: { mermaidStats } }, ctx, state) { void mermaidStats; // dependency signal only; append already happened in mermaid.submit const generatedAssets = []; if (state.site.highlighter?.themeCss) { @@ -525,7 +546,7 @@ function chunkPages(pages, workers) { // ── Gantt chart ─────────────────────────────────────────────────────────────── const GANTT_SECTION = { - config: "Seeds", buildInfo: "Seeds", scss: "Seeds", mermaid: "Seeds", prepDest: "Seeds", + config: "Seeds", buildInfo: "Seeds", scssLight: "Seeds", scssDark: "Seeds", scssJoin: "Seeds", mermaid: "Seeds", prepDest: "Seeds", highlighterInit: "Seeds", discover: "Spine", nav: "Spine", markdownInit: "Spine", buildInit: "Spine", seo: "Spine", loadData: "Spine", resolveBookChapters: "Spine", @@ -645,7 +666,7 @@ export async function runBuild(opts) { const site = scheduler.state.site; const { mermaidStats } = results.get("mermaid"); - const { scssResult } = results.get("scss"); + const { scssResult } = results.get("scssJoin"); if (mermaidStats.regenerated > 0 || mermaidStats.failed > 0) { const parts = [`regenerated ${mermaidStats.regenerated}`]; From 6e57a4056988737b778570371688204dc58a4484 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sat, 30 May 2026 23:56:00 +0200 Subject: [PATCH 21/72] Hide ...Join tasks from Gantt chart --- builder/tbdocs.mjs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 53b926ef..114d8377 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -546,12 +546,12 @@ function chunkPages(pages, workers) { // ── Gantt chart ─────────────────────────────────────────────────────────────── const GANTT_SECTION = { - config: "Seeds", buildInfo: "Seeds", scssLight: "Seeds", scssDark: "Seeds", scssJoin: "Seeds", mermaid: "Seeds", prepDest: "Seeds", + config: "Seeds", buildInfo: "Seeds", scssLight: "Seeds", scssDark: "Seeds", mermaid: "Seeds", prepDest: "Seeds", highlighterInit: "Seeds", discover: "Spine", nav: "Spine", markdownInit: "Spine", buildInit: "Spine", seo: "Spine", loadData: "Spine", resolveBookChapters: "Spine", deriveRedirects: "Spine", deriveSitemap: "Spine", - dispatch: "Render", renderJoin: "Render", + dispatch: "Render", write: "Write", searchData: "Write", writeAux: "Write", writeOffline: "Write", writePdf: "Write", }; const GANTT_SECTION_ORDER = ["Seeds", "Spine", "Render", "Write"]; @@ -562,6 +562,7 @@ async function writeGantt(timings, outPath) { const grouped = new Map(GANTT_SECTION_ORDER.map(s => [s, []])); for (const [id, { start, end }] of [...timings.entries()].sort((a, b) => a[1].start - b[1].start)) { + if (id.endsWith("Join")) continue; const section = /^render:\d+$/.test(id) ? "Render" : (GANTT_SECTION[id] ?? "Other"); if (!grouped.has(section)) grouped.set(section, []); grouped.get(section).push({ id, start: start - t0, end: end - t0 }); From 3e82bddcb2f8889c804e99ede73ac2f976c27239 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sun, 31 May 2026 00:02:34 +0200 Subject: [PATCH 22/72] Write serve Gantt to serve-gantt.mmd, separate from build-gantt.mmd --- .gitignore | 1 + builder/serve.mjs | 4 ++-- builder/tbdocs.mjs | 3 ++- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/.gitignore b/.gitignore index 686cdbcc..aedeb88c 100644 --- a/.gitignore +++ b/.gitignore @@ -9,3 +9,4 @@ node_modules/ /findoverflow-baseline/ /build-gantt.mmd +/serve-gantt.mmd diff --git a/builder/serve.mjs b/builder/serve.mjs index de09ed46..8cd701b7 100644 --- a/builder/serve.mjs +++ b/builder/serve.mjs @@ -167,7 +167,7 @@ export async function runServe(opts) { // Initial build try { - await runBuild({ ...opts, dest: destRoot, skipOffline: true, skipPdf: true }); + await runBuild({ ...opts, dest: destRoot, skipOffline: true, skipPdf: true, ganttFile: "serve-gantt.mmd" }); } catch (err) { console.error("serve: initial build failed:", err.message); process.exit(1); @@ -206,7 +206,7 @@ export async function runServe(opts) { if (running) { pending = true; return; } running = true; try { - await runBuild({ ...opts, dest: destRoot, skipOffline: true, skipPdf: true }); + await runBuild({ ...opts, dest: destRoot, skipOffline: true, skipPdf: true, ganttFile: "serve-gantt.mmd" }); notifyReload(); } catch (err) { console.error("rebuild failed:", err.message); diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 114d8377..e1e17b71 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -58,6 +58,7 @@ function parseArgs(argv) { profileOffline: false, serve: false, port: 4000, + ganttFile: "build-gantt.mmd", }; for (let i = 0; i < argv.length; i++) { const a = argv[i]; @@ -711,7 +712,7 @@ export async function runBuild(opts) { } console.log(scheduler.summary()); - const ganttPath = path.resolve(process.cwd(), "build-gantt.mmd"); + const ganttPath = path.resolve(process.cwd(), opts.ganttFile ?? "build-gantt.mmd"); const ganttContent = await writeGantt(scheduler.timings, ganttPath); const injectStart = Date.now(); From bea0cb6cca3ddb3f99952309102a934b4f6eaaa1 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sun, 31 May 2026 00:32:05 +0200 Subject: [PATCH 23/72] Annotate render Gantt bars with compute efficiency percentage --- builder/cpu-worker.mjs | 19 ++++++++++++------- builder/tbdocs.mjs | 21 ++++++++++++++------- 2 files changed, 26 insertions(+), 14 deletions(-) diff --git a/builder/cpu-worker.mjs b/builder/cpu-worker.mjs index 53960191..d2a92d70 100644 --- a/builder/cpu-worker.mjs +++ b/builder/cpu-worker.mjs @@ -41,6 +41,7 @@ const handlers = { }, async render({ inputs }) { + const workerStart = Date.now(); const { sharedSAB, chunk } = inputs; const { siteData, initData, linkTablesData, staticFilesArr, baseurl, buildInfo, sitePathsArr, @@ -94,15 +95,19 @@ const handlers = { } } + const workerEnd = Date.now(); // book-combined pages have renderedContent but no html (Phase 8 // handles them from renderedContent); send html: undefined for those. - return chunk.map(p => ({ - destPath: p.destPath, - renderedContent: p.renderedContent, - html: p.html, - offlineHtml: p.offlineHtml, - offlineMisses: p.offlineMisses, - })); + return { + workerStart, workerEnd, + pages: chunk.map(p => ({ + destPath: p.destPath, + renderedContent: p.renderedContent, + html: p.html, + offlineHtml: p.offlineHtml, + offlineMisses: p.offlineMisses, + })), + }; }, }; diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index e1e17b71..c5d7543f 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -417,8 +417,8 @@ const TASKS = { scheduler.register(id, { expected: [], handler: "render", - submit(renderOut, emit, state) { - for (const r of renderOut) { + submit(renderOut, emit, state, scheduler) { + for (const r of renderOut.pages) { const p = state.pageByDest.get(r.destPath); if (!p) continue; p.renderedContent = r.renderedContent; @@ -426,7 +426,9 @@ const TASKS = { if (r.offlineHtml !== undefined) p.offlineHtml = r.offlineHtml; if (r.offlineMisses !== undefined) p.offlineMisses = r.offlineMisses; } - emit("renderJoin", renderOut); + const t = scheduler.timings.get(id); + if (t) { t.workerStart = renderOut.workerStart; t.workerEnd = renderOut.workerEnd; } + emit("renderJoin", {}); }, }); scheduler.seed(id, { @@ -562,11 +564,13 @@ async function writeGantt(timings, outPath) { const t0 = Math.min(...[...timings.values()].map(t => t.start)); const grouped = new Map(GANTT_SECTION_ORDER.map(s => [s, []])); - for (const [id, { start, end }] of [...timings.entries()].sort((a, b) => a[1].start - b[1].start)) { + for (const [id, { start, end, workerStart, workerEnd }] of [...timings.entries()].sort((a, b) => a[1].start - b[1].start)) { if (id.endsWith("Join")) continue; const section = /^render:\d+$/.test(id) ? "Render" : (GANTT_SECTION[id] ?? "Other"); if (!grouped.has(section)) grouped.set(section, []); - grouped.get(section).push({ id, start: start - t0, end: end - t0 }); + const entry = { id, start: start - t0, end: end - t0 }; + if (workerStart != null) { entry.workerStart = workerStart - t0; entry.workerEnd = workerEnd - t0; } + grouped.get(section).push(entry); } const lines = [ @@ -579,8 +583,11 @@ async function writeGantt(timings, outPath) { for (const [section, tasks] of grouped) { if (tasks.length === 0) continue; lines.push(` section ${section}`); - for (const { id, start, end } of tasks) { - const label = id.replace(":", " "); + for (const { id, start, end, workerStart, workerEnd } of tasks) { + const pct = workerStart != null + ? ` (${Math.round((workerEnd - workerStart) / (end - start) * 100)}%)` + : ""; + const label = id.replace(":", " ") + pct; const taskId = `t_${id.replace(/[^a-z0-9]/gi, "_")}`; lines.push(` ${label} :done, ${taskId}, ${start}, ${Math.max(end, start + 1)}`); } From 295526af6aa35e6a438daec4ef89483c5952b7a0 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sun, 31 May 2026 00:35:09 +0200 Subject: [PATCH 24/72] Extend compute efficiency annotation to all worker tasks --- builder/cpu-worker.mjs | 16 ++++++++++++---- builder/scheduler.mjs | 5 ++++- builder/tbdocs.mjs | 4 +--- 3 files changed, 17 insertions(+), 8 deletions(-) diff --git a/builder/cpu-worker.mjs b/builder/cpu-worker.mjs index d2a92d70..ce17d0bb 100644 --- a/builder/cpu-worker.mjs +++ b/builder/cpu-worker.mjs @@ -25,19 +25,27 @@ const highlighterP = initHighlighter(); const handlers = { async scssLight({ ctx }) { - return { scssLightResult: await compileLightScss(ctx.srcRoot) }; + const workerStart = Date.now(); + const scssLightResult = await compileLightScss(ctx.srcRoot); + return { workerStart, workerEnd: Date.now(), scssLightResult }; }, async scssDark({ ctx }) { - return { scssDarkResult: await compileDarkScss(ctx.srcRoot) }; + const workerStart = Date.now(); + const scssDarkResult = await compileDarkScss(ctx.srcRoot); + return { workerStart, workerEnd: Date.now(), scssDarkResult }; }, async mermaid({ ctx }) { - return { mermaidStats: await regenerateMermaid(ctx.srcRoot) }; + const workerStart = Date.now(); + const mermaidStats = await regenerateMermaid(ctx.srcRoot); + return { workerStart, workerEnd: Date.now(), mermaidStats }; }, async buildInfo() { - return { buildInfo: await captureBuildInfo() }; + const workerStart = Date.now(); + const buildInfo = await captureBuildInfo(); + return { workerStart, workerEnd: Date.now(), buildInfo }; }, async render({ inputs }) { diff --git a/builder/scheduler.mjs b/builder/scheduler.mjs index 19db5daf..5fcccf43 100644 --- a/builder/scheduler.mjs +++ b/builder/scheduler.mjs @@ -83,7 +83,10 @@ export class Scheduler { } _onDone(task, output, start) { - this.timings.set(task.id, { start, end: Date.now() }); + const end = Date.now(); + const timing = { start, end }; + if (output?.workerStart != null) { timing.workerStart = output.workerStart; timing.workerEnd = output.workerEnd; } + this.timings.set(task.id, timing); this.results.set(task.id, output); this.inFlight--; // submit() must be synchronous; async work belongs in execute(). diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index c5d7543f..de22304e 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -417,7 +417,7 @@ const TASKS = { scheduler.register(id, { expected: [], handler: "render", - submit(renderOut, emit, state, scheduler) { + submit(renderOut, emit, state) { for (const r of renderOut.pages) { const p = state.pageByDest.get(r.destPath); if (!p) continue; @@ -426,8 +426,6 @@ const TASKS = { if (r.offlineHtml !== undefined) p.offlineHtml = r.offlineHtml; if (r.offlineMisses !== undefined) p.offlineMisses = r.offlineMisses; } - const t = scheduler.timings.get(id); - if (t) { t.workerStart = renderOut.workerStart; t.workerEnd = renderOut.workerEnd; } emit("renderJoin", {}); }, }); From 680deb7149ab022acf004c86680305f538f6df26 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sun, 31 May 2026 01:03:42 +0200 Subject: [PATCH 25/72] Show queue+compute split in worker task Gantt labels. Move Shiki WASM init to a no-dep seed task (highlighterInit) --- builder/cpu-worker.mjs | 1 - builder/render.mjs | 2 +- builder/tbdocs.mjs | 35 +++++++++++++++++++---------------- 3 files changed, 20 insertions(+), 18 deletions(-) diff --git a/builder/cpu-worker.mjs b/builder/cpu-worker.mjs index ce17d0bb..49aa0503 100644 --- a/builder/cpu-worker.mjs +++ b/builder/cpu-worker.mjs @@ -3,7 +3,6 @@ // See PLAN-scheduler.md §Worker for the full handler set. import { parentPort } from "node:worker_threads"; - import { compileLightScss, compileDarkScss } from "./scss.mjs"; import { regenerateMermaid } from "./mermaid.mjs"; import { captureBuildInfo } from "./build-info.mjs"; diff --git a/builder/render.mjs b/builder/render.mjs index f6c97e2e..a1bf4993 100644 --- a/builder/render.mjs +++ b/builder/render.mjs @@ -216,7 +216,7 @@ export function createMarkdownIt(ctx) { // kramdown smart_quotes; see PLAN-3 §5.9 for divergences. typographer: true, quotes: "“”‘’", - highlight: (code, lang) => ctx.highlighter.render(code, lang), + highlight: ctx.highlighter ? (code, lang) => ctx.highlighter.render(code, lang) : undefined, }); // Override the fence renderer so our highlight callback's wrapper HTML diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index de22304e..3602fd35 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -30,9 +30,10 @@ import { precomputeSeo } from "./seo.mjs"; import { resolveBookChapters } from "./book.mjs"; import { loadData } from "./data.mjs"; import { - createMarkdownIt, initHighlighter, + createMarkdownIt, buildLinkTables, serializeLinkTables, } from "./render.mjs"; +import { loadHighlightTheme } from "./highlight-theme.mjs"; import { buildInitFn } from "./template.mjs"; import { writePhase, prepareDestination } from "./write.mjs"; import { writeRedirects, deriveRedirectStubs } from "./redirects.mjs"; @@ -217,14 +218,16 @@ const TASKS = { }, }, - // Shiki WASM init. Seed on main (the live highlighter object is not - // serializable across a worker boundary). Fires immediately so the WASM - // warms up while discover + nav are running. + // Theme CSS load. Reads the vendored .theme files and generates the + // tb-highlight.css palette; does NOT init Shiki WASM (unneeded on main + // since no code blocks are rendered here). Workers init their own full + // highlighter instances independently. highlighterInit: { expected: [], runOnMain: true, async execute() { - return { highlighter: await initHighlighter() }; + const theme = await loadHighlightTheme(); + return { highlightCss: theme.css }; }, submit(out, emit) { emit("markdownInit", out); }, }, @@ -274,20 +277,20 @@ const TASKS = { submit(out, emit) { emit("dispatch", out); }, }, - // Link-table build + markdown-it assembly. The highlighter arrives - // pre-warmed from highlighterInit; pages + config come from discover. - // Synchronous: all async work is done upstream. Not serializable (live - // Shiki object), so it stays on main. Workers init their own instances. + // Link-table build + markdown-it assembly. Pages + config come from + // discover; highlighterInit supplies the theme CSS (no Shiki instance + // needed on main -- workers render all code blocks). Synchronous: all + // async work is done upstream. markdownInit: { expected: ["discover", "highlighterInit"], runOnMain: true, - execute({ highlighterInit: { highlighter } }, ctx, state) { + execute({ highlighterInit: { highlightCss } }, ctx, state) { const linkTables = buildLinkTables(state.pages); const baseurl = String(state.site.config.baseurl || ""); const staticFileSet = new Set(state.staticFiles.map(s => s.srcRel)); - state.site.highlighter = highlighter; + state.site.highlightCss = highlightCss; state.site.markdown = createMarkdownIt({ - highlighter, linkTables, baseurl, staticFiles: staticFileSet, + highlighter: null, linkTables, baseurl, staticFiles: staticFileSet, }); state.site.linkTablesSerialized = serializeLinkTables(linkTables); return {}; @@ -449,8 +452,8 @@ const TASKS = { async execute({ scssJoin: { scssResult }, mermaid: { mermaidStats } }, ctx, state) { void mermaidStats; // dependency signal only; append already happened in mermaid.submit const generatedAssets = []; - if (state.site.highlighter?.themeCss) { - generatedAssets.push({ rel: "assets/css/tb-highlight.css", content: state.site.highlighter.themeCss }); + if (state.site.highlightCss) { + generatedAssets.push({ rel: "assets/css/tb-highlight.css", content: state.site.highlightCss }); } if (scssResult.compiled) { generatedAssets.push({ rel: "assets/css/just-the-docs-combined.css", content: scssResult.css }); @@ -528,7 +531,7 @@ const TASKS = { if (ctx.opts.dryRun || skipPdf) return null; return writePdf(state.pages, state.staticFiles, state.site, ctx.destRoot, { tolerateMissingImages: ctx.opts.tolerateMissingImages, - highlightCss: state.site.highlighter?.themeCss, + highlightCss: state.site.highlightCss, }); }, submit() { /* terminal */ }, @@ -583,7 +586,7 @@ async function writeGantt(timings, outPath) { lines.push(` section ${section}`); for (const { id, start, end, workerStart, workerEnd } of tasks) { const pct = workerStart != null - ? ` (${Math.round((workerEnd - workerStart) / (end - start) * 100)}%)` + ? ` (${Math.round((workerStart - start) / (end - start) * 100)}%+${Math.round((workerEnd - workerStart) / (end - start) * 100)}%)` : ""; const label = id.replace(":", " ") + pct; const taskId = `t_${id.replace(/[^a-z0-9]/gi, "_")}`; From ebd104311fb4dd1db2cd996d1a08444ce5f2bb2c Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sun, 31 May 2026 01:15:06 +0200 Subject: [PATCH 26/72] Move loadData/resolveBookChapters earlier in task DAG --- builder/tbdocs.mjs | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 3602fd35..be57ce7f 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -120,9 +120,9 @@ export function makeTimer() { // ── Task graph ──────────────────────────────────────────────────────────────── // // Seeds (config, buildInfo, scssLight + scssDark → scssJoin, mermaid, prepDest, -// highlighterInit), the main-thread spine (discover → nav → buildInit; -// discover + highlighterInit → markdownInit → seo / loadData → -// resolveBookChapters + deriveRedirects / deriveSitemap), the render fan-out +// highlighterInit), the main-thread spine (config → discover → nav → buildInit; +// config → loadData; nav + loadData → resolveBookChapters; discover + highlighterInit → +// markdownInit → seo; deriveRedirects + deriveSitemap off discover), the render fan-out // (dispatch → render:0..N → renderJoin), and write/post-write tasks // (renderJoin + prepDest → searchData; write + searchData → writeAux → // writeOffline; renderJoin + mermaid → writePdf) are scheduler tasks. @@ -145,7 +145,7 @@ const TASKS = { if (ctx.opts.url != null) config.url = ctx.opts.url; return { config }; }, - submit(out, emit) { emit("discover", out); }, + submit(out, emit) { emit("discover", out); emit("loadData", out); }, }, // Git rev-parse / log shell-outs. Worker so they overlap with the main spine. @@ -263,6 +263,7 @@ const TASKS = { }, submit(_, emit) { emit("buildInit", {}); + emit("resolveBookChapters", {}); }, }, @@ -296,8 +297,7 @@ const TASKS = { return {}; }, submit(_, emit) { - emit("seo", {}); - emit("loadData", {}); + emit("seo", {}); }, }, @@ -311,11 +311,11 @@ const TASKS = { state.site.seoLogoUrl = seoLogoUrl; return {}; }, - submit(_, emit) { emit("resolveBookChapters", {}); }, + submit(_, emit) { emit("dispatch", {}); }, }, loadData: { - expected: ["markdownInit"], + expected: ["config"], runOnMain: true, async execute(_, ctx, state) { const data = await loadData(ctx.srcRoot); @@ -330,7 +330,7 @@ const TASKS = { // the same page objects must be read by writePdf later (after renderPhase // fills in renderedContent on those same objects). resolveBookChapters: { - expected: ["seo", "loadData"], + expected: ["nav", "loadData"], runOnMain: true, execute(_, ctx, state) { resolveBookChapters(state.site.bookData, state.pages); @@ -364,13 +364,14 @@ const TASKS = { // Slices state.pages into chunks and dynamically registers render:0..N // worker tasks plus a renderJoin barrier. Waits for buildInit (template - // chrome), resolveBookChapters (identity-critical page refs), and - // buildInfo (git metadata for the footer). + // chrome), resolveBookChapters (identity-critical page refs), seo (SEO + // fields on state.site), and buildInfo (git metadata for the footer). dispatch: { - expected: ["buildInit", "resolveBookChapters", "buildInfo", "mermaid", "deriveRedirects"], + expected: ["buildInit", "resolveBookChapters", "buildInfo", "mermaid", "deriveRedirects", "seo"], runOnMain: true, - execute({ buildInit: { initData }, buildInfo: { buildInfo }, mermaid: { mermaidStats }, deriveRedirects: { stubs } }, ctx, state) { + execute({ buildInit: { initData }, buildInfo: { buildInfo }, mermaid: { mermaidStats }, seo: _seoSignal, deriveRedirects: { stubs } }, ctx, state) { void mermaidStats; // dependency signal only -- static files already appended in mermaid.submit + void _seoSignal; // dependency signal only -- SEO fields already written to state.site const chunks = chunkPages(state.pages, ctx.workerCount); const excludePatterns = Array.isArray(state.site.config?.offline_exclude) ? state.site.config.offline_exclude.map(String) @@ -553,7 +554,7 @@ const GANTT_SECTION = { config: "Seeds", buildInfo: "Seeds", scssLight: "Seeds", scssDark: "Seeds", mermaid: "Seeds", prepDest: "Seeds", highlighterInit: "Seeds", discover: "Spine", nav: "Spine", markdownInit: "Spine", buildInit: "Spine", - seo: "Spine", loadData: "Spine", resolveBookChapters: "Spine", + seo: "Spine", loadData: "Seeds", resolveBookChapters: "Spine", deriveRedirects: "Spine", deriveSitemap: "Spine", dispatch: "Render", write: "Write", searchData: "Write", writeAux: "Write", writeOffline: "Write", writePdf: "Write", From 6c6bbbf0ffdc123b03fbf39855cd65d8a2f13c29 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sun, 31 May 2026 01:25:49 +0200 Subject: [PATCH 27/72] highlighterInit stores CSS directly; markdownInit depends only on discover --- builder/tbdocs.mjs | 36 +++++++++++++++++++----------------- 1 file changed, 19 insertions(+), 17 deletions(-) diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index be57ce7f..61ed9b5d 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -121,8 +121,8 @@ export function makeTimer() { // // Seeds (config, buildInfo, scssLight + scssDark → scssJoin, mermaid, prepDest, // highlighterInit), the main-thread spine (config → discover → nav → buildInit; -// config → loadData; nav + loadData → resolveBookChapters; discover + highlighterInit → -// markdownInit → seo; deriveRedirects + deriveSitemap off discover), the render fan-out +// config → loadData; nav + loadData → resolveBookChapters; discover → markdownInit → seo; +// deriveRedirects + deriveSitemap off discover), the render fan-out // (dispatch → render:0..N → renderJoin), and write/post-write tasks // (renderJoin + prepDest → searchData; write + searchData → writeAux → // writeOffline; renderJoin + mermaid → writePdf) are scheduler tasks. @@ -221,7 +221,8 @@ const TASKS = { // Theme CSS load. Reads the vendored .theme files and generates the // tb-highlight.css palette; does NOT init Shiki WASM (unneeded on main // since no code blocks are rendered here). Workers init their own full - // highlighter instances independently. + // highlighter instances independently. Stores highlightCss on state + // immediately (terminal -- no downstream task to emit). highlighterInit: { expected: [], runOnMain: true, @@ -229,7 +230,10 @@ const TASKS = { const theme = await loadHighlightTheme(); return { highlightCss: theme.css }; }, - submit(out, emit) { emit("markdownInit", out); }, + submit(out, emit, state) { + state.site.highlightCss = out.highlightCss; + emit("write", out); + }, }, // ── Main-thread spine ───────────────────────────────────────────────────── @@ -278,22 +282,19 @@ const TASKS = { submit(out, emit) { emit("dispatch", out); }, }, - // Link-table build + markdown-it assembly. Pages + config come from - // discover; highlighterInit supplies the theme CSS (no Shiki instance - // needed on main -- workers render all code blocks). Synchronous: all - // async work is done upstream. + // Link-table build + markdown-it assembly. Only needs discover (pages + + // config + staticFiles). Synchronous: all async work is done upstream. markdownInit: { - expected: ["discover", "highlighterInit"], + expected: ["discover"], runOnMain: true, - execute({ highlighterInit: { highlightCss } }, ctx, state) { + execute(_, ctx, state) { const linkTables = buildLinkTables(state.pages); const baseurl = String(state.site.config.baseurl || ""); const staticFileSet = new Set(state.staticFiles.map(s => s.srcRel)); - state.site.highlightCss = highlightCss; - state.site.markdown = createMarkdownIt({ + state.site.markdown = createMarkdownIt({ highlighter: null, linkTables, baseurl, staticFiles: staticFileSet, }); - state.site.linkTablesSerialized = serializeLinkTables(linkTables); + state.site.linkTablesSerialized = serializeLinkTables(linkTables); return {}; }, submit(_, emit) { @@ -446,12 +447,13 @@ const TASKS = { // Materialise pages + static files + generated CSS to _site/. // Waits for renderJoin (pages rendered + templated), scss (generated CSS), // mermaid (SVG descriptors appended to state.staticFiles by mermaid.submit), - // and prepDest (_site/ cleaned and recreated). + // prepDest (_site/ cleaned and recreated), and highlighterInit (highlight CSS). write: { - expected: ["renderJoin", "scssJoin", "mermaid", "prepDest"], + expected: ["renderJoin", "scssJoin", "mermaid", "prepDest", "highlighterInit"], runOnMain: true, - async execute({ scssJoin: { scssResult }, mermaid: { mermaidStats } }, ctx, state) { - void mermaidStats; // dependency signal only; append already happened in mermaid.submit + async execute({ scssJoin: { scssResult }, mermaid: { mermaidStats }, highlighterInit: _highlightSignal }, ctx, state) { + void mermaidStats; // dependency signal only; append already happened in mermaid.submit + void _highlightSignal; // dependency signal only; highlightCss already written to state.site const generatedAssets = []; if (state.site.highlightCss) { generatedAssets.push({ rel: "assets/css/tb-highlight.css", content: state.site.highlightCss }); From d50822e41d3aa535afe69cec43e92c2d2c39f687 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sun, 31 May 2026 01:28:37 +0200 Subject: [PATCH 28/72] Move resolveBookChapters dependency from dispatch to writePdf --- builder/tbdocs.mjs | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 61ed9b5d..a22cd431 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -337,7 +337,7 @@ const TASKS = { resolveBookChapters(state.site.bookData, state.pages); return {}; }, - submit(_, emit) { emit("dispatch", {}); }, + submit(_, emit) { emit("writePdf", {}); }, }, // Can run in parallel with nav/markdownInit -- only needs pages + config, @@ -365,10 +365,10 @@ const TASKS = { // Slices state.pages into chunks and dynamically registers render:0..N // worker tasks plus a renderJoin barrier. Waits for buildInit (template - // chrome), resolveBookChapters (identity-critical page refs), seo (SEO - // fields on state.site), and buildInfo (git metadata for the footer). + // chrome), seo (SEO fields on state.site), and buildInfo (git metadata + // for the footer). dispatch: { - expected: ["buildInit", "resolveBookChapters", "buildInfo", "mermaid", "deriveRedirects", "seo"], + expected: ["buildInit", "buildInfo", "mermaid", "deriveRedirects", "seo"], runOnMain: true, execute({ buildInit: { initData }, buildInfo: { buildInfo }, mermaid: { mermaidStats }, seo: _seoSignal, deriveRedirects: { stubs } }, ctx, state) { void mermaidStats; // dependency signal only -- static files already appended in mermaid.submit @@ -522,12 +522,13 @@ const TASKS = { submit() { /* terminal */ }, }, - // Produce _site-pdf/. Depends on renderJoin (pages have renderedContent) - // and mermaid (SVG descriptors in staticFiles). Sources CSS directly: + // Produce _site-pdf/. Depends on renderJoin (pages have renderedContent), + // resolveBookChapters (bookData._chapters refs into state.pages), and + // mermaid (SVG descriptors in staticFiles). Sources CSS directly: // tb-highlight.css from state.site.highlighter, print.css from staticFiles. // Runs in parallel with write → searchData → writeAux → writeOffline. writePdf: { - expected: ["renderJoin", "mermaid"], + expected: ["renderJoin", "mermaid", "resolveBookChapters"], runOnMain: true, async execute(_, ctx, state) { const skipPdf = ctx.opts.skipPdf ?? (state.site.config.also_build_pdf === false); From fcc0d8aad73f4cdd1512f18dc863563371f865ae Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sun, 31 May 2026 01:32:30 +0200 Subject: [PATCH 29/72] Defer deriveSitemap to after dispatch to reduce spine contention --- builder/tbdocs.mjs | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index a22cd431..e6eaf6b8 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -122,7 +122,7 @@ export function makeTimer() { // Seeds (config, buildInfo, scssLight + scssDark → scssJoin, mermaid, prepDest, // highlighterInit), the main-thread spine (config → discover → nav → buildInit; // config → loadData; nav + loadData → resolveBookChapters; discover → markdownInit → seo; -// deriveRedirects + deriveSitemap off discover), the render fan-out +// deriveRedirects off discover; deriveSitemap deferred to dispatch), the render fan-out // (dispatch → render:0..N → renderJoin), and write/post-write tasks // (renderJoin + prepDest → searchData; write + searchData → writeAux → // writeOffline; renderJoin + mermaid → writePdf) are scheduler tasks. @@ -253,7 +253,6 @@ const TASKS = { emit("nav", out); emit("markdownInit", out); emit("deriveRedirects", out); - emit("deriveSitemap", out); }, }, @@ -352,8 +351,10 @@ const TASKS = { submit(out, emit) { emit("writeAux", out); emit("dispatch", out); }, }, + // Deferred to after dispatch so it runs while the main thread is idle + // waiting for render workers, rather than contending during the spine. deriveSitemap: { - expected: ["discover"], + expected: ["dispatch"], runOnMain: true, execute(_, ctx, state) { return { urls: deriveSitemapUrls(state.pages, state.site) }; @@ -405,6 +406,7 @@ const TASKS = { }, submit(out, emit, _state, scheduler) { const N = out.chunks.length; + emit("deriveSitemap", {}); scheduler.register("renderJoin", { expected: Array.from({ length: N }, (_, i) => `render:${i}`), From af8e1000e682bd744de557ef20d4dfa9614e18a5 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sun, 31 May 2026 01:35:28 +0200 Subject: [PATCH 30/72] Defer resolveBookChapters to after deriveSitemap (idle window) --- builder/tbdocs.mjs | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index e6eaf6b8..a1b5616f 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -122,7 +122,7 @@ export function makeTimer() { // Seeds (config, buildInfo, scssLight + scssDark → scssJoin, mermaid, prepDest, // highlighterInit), the main-thread spine (config → discover → nav → buildInit; // config → loadData; nav + loadData → resolveBookChapters; discover → markdownInit → seo; -// deriveRedirects off discover; deriveSitemap deferred to dispatch), the render fan-out +// deriveRedirects off discover; deriveSitemap + resolveBookChapters deferred to dispatch), the render fan-out // (dispatch → render:0..N → renderJoin), and write/post-write tasks // (renderJoin + prepDest → searchData; write + searchData → writeAux → // writeOffline; renderJoin + mermaid → writePdf) are scheduler tasks. @@ -266,7 +266,6 @@ const TASKS = { }, submit(_, emit) { emit("buildInit", {}); - emit("resolveBookChapters", {}); }, }, @@ -323,14 +322,15 @@ const TASKS = { state.site.bookData = data.book ?? null; return {}; }, - submit(_, emit) { emit("resolveBookChapters", {}); }, + submit(_, emit) { }, }, // Mutates bookData._chapters with refs into state.pages. Identity-critical: // the same page objects must be read by writePdf later (after renderPhase - // fills in renderedContent on those same objects). + // fills in renderedContent on those same objects). Deferred to after + // deriveSitemap so it runs while the main thread is idle waiting for workers. resolveBookChapters: { - expected: ["nav", "loadData"], + expected: ["deriveSitemap"], runOnMain: true, execute(_, ctx, state) { resolveBookChapters(state.site.bookData, state.pages); @@ -359,7 +359,10 @@ const TASKS = { execute(_, ctx, state) { return { urls: deriveSitemapUrls(state.pages, state.site) }; }, - submit(out, emit) { emit("writeAux", out); }, + submit(out, emit) { + emit("writeAux", out); + emit("resolveBookChapters", {}); + }, }, // ── Render fan-out ───────────────────────────────────────────────────────── From b88d211b03af5eadf4453a63ad2eda259e1eb563 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sun, 31 May 2026 01:50:06 +0200 Subject: [PATCH 31/72] Split buildInit: sidebar into nav, config chrome runs after discover --- builder/tbdocs.mjs | 34 +++++++++++++++++----------------- builder/template.mjs | 28 ++++++++++++++++------------ 2 files changed, 33 insertions(+), 29 deletions(-) diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index a1b5616f..242156b6 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -34,7 +34,7 @@ import { buildLinkTables, serializeLinkTables, } from "./render.mjs"; import { loadHighlightTheme } from "./highlight-theme.mjs"; -import { buildInitFn } from "./template.mjs"; +import { buildInitConfig, renderSidebar } from "./template.mjs"; import { writePhase, prepareDestination } from "./write.mjs"; import { writeRedirects, deriveRedirectStubs } from "./redirects.mjs"; import { writeSitemap, deriveSitemapUrls } from "./sitemap.mjs"; @@ -120,8 +120,8 @@ export function makeTimer() { // ── Task graph ──────────────────────────────────────────────────────────────── // // Seeds (config, buildInfo, scssLight + scssDark → scssJoin, mermaid, prepDest, -// highlighterInit), the main-thread spine (config → discover → nav → buildInit; -// config → loadData; nav + loadData → resolveBookChapters; discover → markdownInit → seo; +// highlighterInit), the main-thread spine (config → discover → nav (sidebar) + buildInit (chrome); +// nav + buildInit → dispatch; config → loadData; discover → markdownInit → seo; // deriveRedirects off discover; deriveSitemap + resolveBookChapters deferred to dispatch), the render fan-out // (dispatch → render:0..N → renderJoin), and write/post-write tasks // (renderJoin + prepDest → searchData; write + searchData → writeAux → @@ -251,6 +251,7 @@ const TASKS = { state.site.config = out.config; for (const p of out.pages) state.pageByDest.set(p.destPath, p); emit("nav", out); + emit("buildInit", out); emit("markdownInit", out); emit("deriveRedirects", out); }, @@ -262,20 +263,20 @@ const TASKS = { execute(_, ctx, state) { const { navTree } = computeNav(state.pages, state.site.config); state.site.navTree = navTree; - return {}; - }, - submit(_, emit) { - emit("buildInit", {}); + return { sidebar: renderSidebar(state.site) }; }, + submit(out, emit) { emit("dispatch", out); }, }, - // Pre-renders the sidebar/header/svg-sprite HTML used by templatePhase. - // Depends only on nav (needs state.site.navTree). + // Pre-renders the config-only chrome (SVG sprites, header, search footer, + // mermaid script, favicon, GA). No nav-tree dependency -- runs after + // discover in parallel with nav. dispatch assembles the final initData + // by merging this with the sidebar from nav. buildInit: { - expected: ["nav"], + expected: ["discover"], runOnMain: true, execute(_, ctx, state) { - return { initData: buildInitFn(state.site) }; + return { initData: buildInitConfig(state.site) }; }, submit(out, emit) { emit("dispatch", out); }, }, @@ -368,13 +369,12 @@ const TASKS = { // ── Render fan-out ───────────────────────────────────────────────────────── // Slices state.pages into chunks and dynamically registers render:0..N - // worker tasks plus a renderJoin barrier. Waits for buildInit (template - // chrome), seo (SEO fields on state.site), and buildInfo (git metadata - // for the footer). + // worker tasks plus a renderJoin barrier. Assembles initData from the + // two parallel halves: nav (sidebar) + buildInit (config-only chrome). dispatch: { - expected: ["buildInit", "buildInfo", "mermaid", "deriveRedirects", "seo"], + expected: ["nav", "buildInit", "buildInfo", "mermaid", "deriveRedirects", "seo"], runOnMain: true, - execute({ buildInit: { initData }, buildInfo: { buildInfo }, mermaid: { mermaidStats }, seo: _seoSignal, deriveRedirects: { stubs } }, ctx, state) { + execute({ nav: { sidebar }, buildInit: { initData }, buildInfo: { buildInfo }, mermaid: { mermaidStats }, seo: _seoSignal, deriveRedirects: { stubs } }, ctx, state) { void mermaidStats; // dependency signal only -- static files already appended in mermaid.submit void _seoSignal; // dependency signal only -- SEO fields already written to state.site const chunks = chunkPages(state.pages, ctx.workerCount); @@ -395,7 +395,7 @@ const TASKS = { seoSiteTitle: state.site.seoSiteTitle, seoLogoUrl: state.site.seoLogoUrl, }, - initData, + initData: { ...initData, sidebar }, buildInfo, linkTablesData: state.site.linkTablesSerialized, staticFilesArr: state.staticFiles.map(f => f.srcRel), diff --git a/builder/template.mjs b/builder/template.mjs index 1c1f5a14..19c1bb09 100644 --- a/builder/template.mjs +++ b/builder/template.mjs @@ -34,24 +34,28 @@ export async function templatePhase(pages, site, initData) { })); } -// One-time per-build constants: pre-rendered SVG sprite, sidebar HTML, -// header static parts, aux-nav, search-footer, mermaid script, favicon -// link, GA snippet. Per §4 init order. -// Exported as buildInitFn for the scheduler's main-thread buildInit task. -export { buildInit as buildInitFn }; -function buildInit(site) { +// Config-only chrome fields -- no nav-tree dependency. Exported for the +// scheduler's buildInit task so it can run after discover, in parallel +// with nav. Does not include sidebar; that is rendered by nav and merged +// by dispatch. +export function buildInitConfig(site) { return { - svgSprites: buildSvgSprites(site.config), - sidebar: renderSidebar(site), - header: renderHeader(site), - searchFooter: renderSearchFooter(site), + svgSprites: buildSvgSprites(site.config), + header: renderHeader(site), + searchFooter: renderSearchFooter(site), mermaidScript: renderMermaidScript(site), - faviconLink: buildFaviconLink(site.config), - gaSnippet: buildGaSnippet(site.config), + faviconLink: buildFaviconLink(site.config), + gaSnippet: buildGaSnippet(site.config), searchEnabled: site.config.search_enabled !== false, }; } +// Full init -- used by the templatePhase fallback (no scheduler) and tools. +export { buildInit as buildInitFn, renderSidebar }; +function buildInit(site) { + return { ...buildInitConfig(site), sidebar: renderSidebar(site) }; +} + // ---------- §5.1 templatePage -------------------------------------------- function templatePage(page, site, init) { From 314e321e94d56b9969617fd2202239b06278723e Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sun, 31 May 2026 01:55:18 +0200 Subject: [PATCH 32/72] Chain config->highlighterInit->loadData to fill discover I/O window --- builder/tbdocs.mjs | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 242156b6..eea5d200 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -145,7 +145,7 @@ const TASKS = { if (ctx.opts.url != null) config.url = ctx.opts.url; return { config }; }, - submit(out, emit) { emit("discover", out); emit("loadData", out); }, + submit(out, emit) { emit("discover", out); emit("highlighterInit", out); }, }, // Git rev-parse / log shell-outs. Worker so they overlap with the main spine. @@ -221,10 +221,10 @@ const TASKS = { // Theme CSS load. Reads the vendored .theme files and generates the // tb-highlight.css palette; does NOT init Shiki WASM (unneeded on main // since no code blocks are rendered here). Workers init their own full - // highlighter instances independently. Stores highlightCss on state - // immediately (terminal -- no downstream task to emit). + // highlighter instances independently. Runs after config so it sits in + // the discover I/O window; chains to loadData for the same reason. highlighterInit: { - expected: [], + expected: ["config"], runOnMain: true, async execute() { const theme = await loadHighlightTheme(); @@ -232,7 +232,8 @@ const TASKS = { }, submit(out, emit, state) { state.site.highlightCss = out.highlightCss; - emit("write", out); + emit("write", out); + emit("loadData", out); }, }, @@ -315,7 +316,7 @@ const TASKS = { }, loadData: { - expected: ["config"], + expected: ["highlighterInit"], runOnMain: true, async execute(_, ctx, state) { const data = await loadData(ctx.srcRoot); @@ -560,9 +561,9 @@ function chunkPages(pages, workers) { const GANTT_SECTION = { config: "Seeds", buildInfo: "Seeds", scssLight: "Seeds", scssDark: "Seeds", mermaid: "Seeds", prepDest: "Seeds", - highlighterInit: "Seeds", + highlighterInit: "Seeds", loadData: "Seeds", discover: "Spine", nav: "Spine", markdownInit: "Spine", buildInit: "Spine", - seo: "Spine", loadData: "Seeds", resolveBookChapters: "Spine", + seo: "Spine", resolveBookChapters: "Spine", deriveRedirects: "Spine", deriveSitemap: "Spine", dispatch: "Render", write: "Write", searchData: "Write", writeAux: "Write", writeOffline: "Write", writePdf: "Write", From 38a57d99ae4cf8252847d4bb69b86dddd224c523 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sun, 31 May 2026 02:18:16 +0200 Subject: [PATCH 33/72] Centralise dest-dir wipe in prepareDestinations for all three output trees --- builder/offline.mjs | 19 ------------------- builder/pdf.mjs | 27 ++++----------------------- builder/tbdocs.mjs | 9 +++++---- builder/write.mjs | 18 ++++++++++-------- 4 files changed, 19 insertions(+), 54 deletions(-) diff --git a/builder/offline.mjs b/builder/offline.mjs index a1cbb365..3d1b87d2 100644 --- a/builder/offline.mjs +++ b/builder/offline.mjs @@ -137,9 +137,6 @@ export async function writeOffline(pages, staticFiles, site, destRoot, { auxStat const subT = profileOffline ? makeTimer() : null; - await setupOfflineDest(deps.offlineRoot); - subT?.lap("setup"); - const jtdSrc = path.join(destRoot, "assets/js/just-the-docs.js"); const jtdDest = path.join(deps.offlineRoot, "assets/js/just-the-docs.js"); const jtdPatches = await patchJustTheDocsJs(jtdSrc, jtdDest); @@ -213,22 +210,6 @@ export async function buildOfflineState(pages, staticFiles, site, destRoot, { st }; } -// §5.1 setupOfflineDest -- wipe-contents (not directory itself; see -// PLAN-7 §7.D1) then ensure the root exists. -async function setupOfflineDest(offlineRoot) { - if (!isUnderProject(offlineRoot)) { - throw new Error(`refusing to clean ${offlineRoot}: not under the project tree`); - } - if (existsSync(offlineRoot)) { - const entries = await fs.readdir(offlineRoot); - await Promise.all(entries.map(name => - fs.rm(path.join(offlineRoot, name), { recursive: true, force: true }), - )); - } else { - await fs.mkdir(offlineRoot, { recursive: true }); - } -} - // §5.2 writeOfflinePages -- per-page strip + rewrite + inject. // // PLAN-9 §5.3 (B7) nav-block cache: the just-the-docs sidebar in diff --git a/builder/pdf.mjs b/builder/pdf.mjs index 6178fa81..a8c80d80 100644 --- a/builder/pdf.mjs +++ b/builder/pdf.mjs @@ -13,9 +13,8 @@ // §A Top-level orchestration (writePdf entry point) // §B Image-path extraction (port of pdfify.rb's IMG_SRC_RE) // §C Static-file lookup -// §D Setup pass (wipe + recreate pdfRoot) -// §E Copy pass (book.html + CSS + images) -// §F Missing-image reporting (port of pdfify.rb's strict mode) +// §D Copy pass (book.html + CSS + images) +// §E Missing-image reporting (port of pdfify.rb's strict mode) import { promises as fs } from "node:fs"; @@ -24,7 +23,6 @@ import path from "node:path"; import { assembleBook } from "./book.mjs"; import { WRITE_LIMIT, - isUnderProject, mkdirRec, runLimited, safeWrite, @@ -52,8 +50,6 @@ export async function writePdf(pages, staticFiles, site, destRoot, { tolerateMis const staticByDestRel = new Map( staticFiles.map(s => [s.destRel.replaceAll("\\", "/"), s]), ); - await setupPdfDest(pdfRoot); - const counters = { bookBytes: 0, html: 0, css: 0, images: 0, missing: 0 }; const missingPaths = []; @@ -129,22 +125,7 @@ export function extractImagePaths(html) { } // --------------------------------------------------------------------------- -// §D Setup pass -// --------------------------------------------------------------------------- - -// PLAN-8 §5.4 + §7.D1: unlike Phase 7's wipe-contents-keep-directory -// pattern, Phase 8 wipes the entire <pdfRoot>/. Mirrors pdfify.rb's -// `rm_rf(dest)` + `mkdir_p(dest)`. -async function setupPdfDest(pdfRoot) { - if (!isUnderProject(pdfRoot)) { - throw new Error(`refusing to clean ${pdfRoot}: not under the project tree`); - } - await fs.rm(pdfRoot, { recursive: true, force: true }); - await fs.mkdir(pdfRoot, { recursive: true }); -} - -// --------------------------------------------------------------------------- -// §E Copy pass +// §D Copy pass // --------------------------------------------------------------------------- // PLAN-8 §5.5: write the assembled book.html. @@ -201,7 +182,7 @@ async function copyPdfImages(imagePaths, staticByDestRel, pdfRoot, counters, mis } // --------------------------------------------------------------------------- -// §F Missing-image reporting (port of pdfify.rb's strict mode) +// §E Missing-image reporting (port of pdfify.rb's strict mode) // --------------------------------------------------------------------------- // PLAN-8 §5.8: per-path error log, then throw if !tolerateMissingImages. diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index eea5d200..2a0d04ad 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -35,7 +35,7 @@ import { } from "./render.mjs"; import { loadHighlightTheme } from "./highlight-theme.mjs"; import { buildInitConfig, renderSidebar } from "./template.mjs"; -import { writePhase, prepareDestination } from "./write.mjs"; +import { writePhase, prepareDestinations } from "./write.mjs"; import { writeRedirects, deriveRedirectStubs } from "./redirects.mjs"; import { writeSitemap, deriveSitemapUrls } from "./sitemap.mjs"; import { writeSearchData } from "./search.mjs"; @@ -203,13 +203,14 @@ const TASKS = { }, }, - // Clean and recreate _site/. No dependencies -- overlaps with the entire - // main-thread spine and worker seeds. Joined by write and searchData. + // Clean and recreate _site/, _site-offline/, _site-pdf/. No dependencies -- + // overlaps with the entire main-thread spine and worker seeds. Joined by write and searchData. prepDest: { expected: [], runOnMain: true, async execute(_, ctx) { - await prepareDestination(ctx.destRoot, ctx.opts.dryRun); + const r = ctx.destRoot; + await prepareDestinations([r, r + "-offline", r + "-pdf"], ctx.opts.dryRun); return {}; }, submit(_, emit) { diff --git a/builder/write.mjs b/builder/write.mjs index 579bf689..fb3c7e0a 100644 --- a/builder/write.mjs +++ b/builder/write.mjs @@ -93,18 +93,20 @@ async function writeGeneratedAssets(assets, destRoot, limit, baseurl) { }); } -// ---------- §5.1 prepareDestination ------------------------------------- +// ---------- §5.1 prepareDestinations ------------------------------------ -export async function prepareDestination(destRoot, dryRun) { +export async function prepareDestinations(roots, dryRun) { if (dryRun) { - console.log(`[dry-run] would clean ${destRoot}`); + for (const root of roots) console.log(`[dry-run] would clean ${root}`); return; } - if (!isUnderProject(destRoot)) { - throw new Error(`refusing to clean ${destRoot}: not under the project tree`); - } - await fs.rm(destRoot, { recursive: true, force: true }); - await fs.mkdir(destRoot, { recursive: true }); + await Promise.all(roots.map(async (root) => { + if (!isUnderProject(root)) { + throw new Error(`refusing to clean ${root}: not under the project tree`); + } + await fs.rm(root, { recursive: true, force: true }); + await fs.mkdir(root, { recursive: true }); + })); } export function isUnderProject(destRoot) { From ffd1a85124200318e806b6d7e547b5ed24d766e7 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sun, 31 May 2026 02:22:58 +0200 Subject: [PATCH 34/72] Defer prepDest to after dispatch to avoid I/O contention with discover --- builder/tbdocs.mjs | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 2a0d04ad..1f846d24 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -119,11 +119,11 @@ export function makeTimer() { // ── Task graph ──────────────────────────────────────────────────────────────── // -// Seeds (config, buildInfo, scssLight + scssDark → scssJoin, mermaid, prepDest, +// Seeds (config, buildInfo, scssLight + scssDark → scssJoin, mermaid, // highlighterInit), the main-thread spine (config → discover → nav (sidebar) + buildInit (chrome); // nav + buildInit → dispatch; config → loadData; discover → markdownInit → seo; -// deriveRedirects off discover; deriveSitemap + resolveBookChapters deferred to dispatch), the render fan-out -// (dispatch → render:0..N → renderJoin), and write/post-write tasks +// deriveRedirects off discover; deriveSitemap + resolveBookChapters + prepDest deferred to dispatch), +// the render fan-out (dispatch → render:0..N → renderJoin), and write/post-write tasks // (renderJoin + prepDest → searchData; write + searchData → writeAux → // writeOffline; renderJoin + mermaid → writePdf) are scheduler tasks. // runBuild() constructs the pool + scheduler, awaits start(), logs the @@ -203,10 +203,11 @@ const TASKS = { }, }, - // Clean and recreate _site/, _site-offline/, _site-pdf/. No dependencies -- - // overlaps with the entire main-thread spine and worker seeds. Joined by write and searchData. + // Clean and recreate _site/, _site-offline/, _site-pdf/. Deferred to after + // dispatch so the wipe doesn't contend with discover's source-file reads. + // Joined by write and searchData. prepDest: { - expected: [], + expected: ["dispatch"], runOnMain: true, async execute(_, ctx) { const r = ctx.destRoot; @@ -412,6 +413,7 @@ const TASKS = { submit(out, emit, _state, scheduler) { const N = out.chunks.length; emit("deriveSitemap", {}); + emit("prepDest", {}); scheduler.register("renderJoin", { expected: Array.from({ length: N }, (_, i) => `render:${i}`), @@ -561,12 +563,12 @@ function chunkPages(pages, workers) { // ── Gantt chart ─────────────────────────────────────────────────────────────── const GANTT_SECTION = { - config: "Seeds", buildInfo: "Seeds", scssLight: "Seeds", scssDark: "Seeds", mermaid: "Seeds", prepDest: "Seeds", + config: "Seeds", buildInfo: "Seeds", scssLight: "Seeds", scssDark: "Seeds", mermaid: "Seeds", highlighterInit: "Seeds", loadData: "Seeds", discover: "Spine", nav: "Spine", markdownInit: "Spine", buildInit: "Spine", seo: "Spine", resolveBookChapters: "Spine", deriveRedirects: "Spine", deriveSitemap: "Spine", - dispatch: "Render", + dispatch: "Render", prepDest: "Render", write: "Write", searchData: "Write", writeAux: "Write", writeOffline: "Write", writePdf: "Write", }; const GANTT_SECTION_ORDER = ["Seeds", "Spine", "Render", "Write"]; From 98299cfae53ff355be68cad161ddce67de2c6ac1 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sun, 31 May 2026 15:44:59 +0200 Subject: [PATCH 35/72] Chain mermaid after buildInfo to reduce seed-phase CPU contention on CI --- builder/tbdocs.mjs | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 1f846d24..0d392e9c 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -119,7 +119,7 @@ export function makeTimer() { // ── Task graph ──────────────────────────────────────────────────────────────── // -// Seeds (config, buildInfo, scssLight + scssDark → scssJoin, mermaid, +// Seeds (config, buildInfo → mermaid, scssLight + scssDark → scssJoin, // highlighterInit), the main-thread spine (config → discover → nav (sidebar) + buildInit (chrome); // nav + buildInit → dispatch; config → loadData; discover → markdownInit → seo; // deriveRedirects off discover; deriveSitemap + resolveBookChapters + prepDest deferred to dispatch), @@ -149,12 +149,14 @@ const TASKS = { }, // Git rev-parse / log shell-outs. Worker so they overlap with the main spine. + // Chains into mermaid so the two don't compete with discover on 4-thread CI. buildInfo: { expected: [], // execute() runs in cpu-worker.mjs as the "buildInfo" handler. submit(out, emit, state) { state.site.buildInfo = out.buildInfo; emit("dispatch", out); + emit("mermaid", {}); }, }, @@ -185,9 +187,10 @@ const TASKS = { submit(out, emit) { emit("write", out); }, }, - // Stale mermaid SVG regeneration. Worker for the same reason. + // Stale mermaid SVG regeneration. Chained after buildInfo (not a seed) to + // avoid competing with discover on 4-thread CI. mermaid: { - expected: [], + expected: ["buildInfo"], // execute() runs in cpu-worker.mjs as the "mermaid" handler. submit(out, emit, state) { // Append any freshly-generated SVG descriptors that discover didn't see @@ -563,7 +566,7 @@ function chunkPages(pages, workers) { // ── Gantt chart ─────────────────────────────────────────────────────────────── const GANTT_SECTION = { - config: "Seeds", buildInfo: "Seeds", scssLight: "Seeds", scssDark: "Seeds", mermaid: "Seeds", + config: "Seeds", buildInfo: "Seeds", scssLight: "Seeds", scssDark: "Seeds", mermaid: "Spine", highlighterInit: "Seeds", loadData: "Seeds", discover: "Spine", nav: "Spine", markdownInit: "Spine", buildInit: "Spine", seo: "Spine", resolveBookChapters: "Spine", From 6222b261908afea9fe5eacc2a5ff7f52715365ca Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sun, 31 May 2026 16:28:54 +0200 Subject: [PATCH 36/72] Slice render chunks 10x finer for balanced worker utilisation --- builder/cpu-worker.mjs | 68 +++++++++++++++++++++++++---------------- builder/scheduler.mjs | 30 ++++++++++++------ builder/tbdocs.mjs | 29 ++++++++++++++++-- builder/worker-pool.mjs | 6 ++-- 4 files changed, 90 insertions(+), 43 deletions(-) diff --git a/builder/cpu-worker.mjs b/builder/cpu-worker.mjs index 49aa0503..f6610ba3 100644 --- a/builder/cpu-worker.mjs +++ b/builder/cpu-worker.mjs @@ -50,31 +50,16 @@ const handlers = { async render({ inputs }) { const workerStart = Date.now(); const { sharedSAB, chunk } = inputs; - const { siteData, initData, linkTablesData, staticFilesArr, - baseurl, buildInfo, sitePathsArr, - skipOffline } = unpackShared(sharedSAB); - - const highlighter = await highlighterP; - const linkTables = reconstructLinkTables(linkTablesData); - const staticFiles = new Set(staticFilesArr); - const markdown = createMarkdownIt({ highlighter, linkTables, baseurl, staticFiles }); - - const site = { ...siteData, markdown, buildInfo }; - await renderPhase(chunk, site); - await templatePhase(chunk, site, initData); - - // Offline rewrite pass (Phase III of PLAN-scheduler-offline.md). - // Runs the per-page URL rewrite inside the worker so it - // parallelises across CPUs. When skipOffline is true the entire - // pass is skipped — no Set construction, no rewriting. - if (!skipOffline) { - const sitePaths = new Set(sitePathsArr); - const caches = { rawResolution: new Map(), seg: new Map(), result: new Map() }; - const offlineState = { sitePaths, caches, baseurl: normalizeBaseurl(baseurl) }; - - // Nav-cache pre-pass: group chunk pages by dest dir, derive the - // first page per dir, cache nav block slices. Same logic as - // writeOfflinePages in offline.mjs. + const env = await getOrInitRenderEnv(sharedSAB); + + await renderPhase(chunk, env.site); + await templatePhase(chunk, env.site, env.initData); + + if (env.offlineBase) { + const offlineState = { ...env.offlineBase, + caches: { rawResolution: new Map(), seg: new Map(), result: new Map() }, + }; + const writable = chunk.filter(p => p.html !== undefined); const byDir = new Map(); for (const p of writable) { @@ -103,8 +88,6 @@ const handlers = { } const workerEnd = Date.now(); - // book-combined pages have renderedContent but no html (Phase 8 - // handles them from renderedContent); send html: undefined for those. return { workerStart, workerEnd, pages: chunk.map(p => ({ @@ -118,6 +101,37 @@ const handlers = { }, }; +// Cached per-worker render environment. The sharedSAB is identical for every +// chunk in a build, so we unpack + derive once and reuse across chunks. +let _renderSAB = null; +let _renderEnv = null; + +async function getOrInitRenderEnv(sharedSAB) { + if (_renderSAB === sharedSAB) return _renderEnv; + + const { siteData, initData, linkTablesData, staticFilesArr, + baseurl, buildInfo, sitePathsArr, + skipOffline } = unpackShared(sharedSAB); + + const highlighter = await highlighterP; + const linkTables = reconstructLinkTables(linkTablesData); + const staticFiles = new Set(staticFilesArr); + const markdown = createMarkdownIt({ highlighter, linkTables, baseurl, staticFiles }); + const site = { ...siteData, markdown, buildInfo }; + + let offlineBase = null; + if (!skipOffline) { + offlineBase = { + sitePaths: new Set(sitePathsArr), + baseurl: normalizeBaseurl(baseurl), + }; + } + + _renderSAB = sharedSAB; + _renderEnv = { site, initData, offlineBase }; + return _renderEnv; +} + parentPort.on("message", async (msg) => { const { name, ...payload } = msg; const handler = handlers[name]; diff --git a/builder/scheduler.mjs b/builder/scheduler.mjs index 5fcccf43..af256933 100644 --- a/builder/scheduler.mjs +++ b/builder/scheduler.mjs @@ -86,6 +86,9 @@ export class Scheduler { const end = Date.now(); const timing = { start, end }; if (output?.workerStart != null) { timing.workerStart = output.workerStart; timing.workerEnd = output.workerEnd; } + if (output?.lane != null) timing.lane = output.lane; + if (task.def.consolidate) timing.consolidate = true; + if (task.def.ganttSection) timing.ganttSection = task.def.ganttSection; this.timings.set(task.id, timing); this.results.set(task.id, output); this.inFlight--; @@ -109,21 +112,28 @@ export class Scheduler { const sorted = [...this.timings.entries()] .sort((a, b) => a[1].start - b[1].start); - const renderMap = new Map(); + const consolidated = new Map(); const parts = []; for (const [id, timing] of sorted) { - const m = id.match(/^render:(\d+)$/); - if (m) renderMap.set(Number(m[1]), timing); - else parts.push(`${id}=${timing.end - timing.start}ms`); + if (timing.consolidate && timing.lane != null) { + const section = timing.ganttSection ?? "worker"; + if (!consolidated.has(section)) consolidated.set(section, new Map()); + const byLane = consolidated.get(section); + const prev = byLane.get(timing.lane); + if (!prev) byLane.set(timing.lane, { start: timing.start, end: timing.end }); + else { prev.start = Math.min(prev.start, timing.start); prev.end = Math.max(prev.end, timing.end); } + } else { + parts.push(`${id}=${timing.end - timing.start}ms`); + } } let result = pc.dim(parts.join(" ")); - if (renderMap.size > 0) { - const renderEntries = [...renderMap.entries()].sort((a, b) => a[0] - b[0]); - const wallMs = Math.max(...renderEntries.map(([, t]) => t.end)) - - Math.min(...renderEntries.map(([, t]) => t.start)); - const inner = renderEntries.map(([i, t]) => `${i}=${t.end - t.start}ms`).join(", "); - result += `\n${pc.bold(pc.yellow("render:"))} ${pc.white(`${wallMs}ms,`)} ${pc.dim(inner)}`; + for (const [section, byLane] of consolidated) { + const lanes = [...byLane.entries()].sort((a, b) => a[0] - b[0]); + const wallMs = Math.max(...lanes.map(([, t]) => t.end)) + - Math.min(...lanes.map(([, t]) => t.start)); + const inner = lanes.map(([i, t]) => `w${i}=${t.end - t.start}ms`).join(", "); + result += `\n${pc.bold(pc.yellow(`${section.toLowerCase()}:`))} ${pc.white(`${wallMs}ms,`)} ${pc.dim(inner)}`; } return result; } diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 0d392e9c..90b7b10d 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -434,6 +434,8 @@ const TASKS = { scheduler.register(id, { expected: [], handler: "render", + consolidate: true, + ganttSection: "Render", submit(renderOut, emit, state) { for (const r of renderOut.pages) { const p = state.pageByDest.get(r.destPath); @@ -554,8 +556,10 @@ const TASKS = { }, }; +const SLICES_PER_WORKER = 10; + function chunkPages(pages, workers) { - const n = Math.min(workers, pages.length); + const n = Math.min(workers * SLICES_PER_WORKER, pages.length); if (n === 0) return []; const size = Math.ceil(pages.length / n); const chunks = []; @@ -581,15 +585,34 @@ async function writeGantt(timings, outPath) { const t0 = Math.min(...[...timings.values()].map(t => t.start)); const grouped = new Map(GANTT_SECTION_ORDER.map(s => [s, []])); - for (const [id, { start, end, workerStart, workerEnd }] of [...timings.entries()].sort((a, b) => a[1].start - b[1].start)) { + for (const [id, { start, end, workerStart, workerEnd, lane, consolidate, ganttSection }] of [...timings.entries()].sort((a, b) => a[1].start - b[1].start)) { if (id.endsWith("Join")) continue; - const section = /^render:\d+$/.test(id) ? "Render" : (GANTT_SECTION[id] ?? "Other"); + const section = ganttSection ?? GANTT_SECTION[id] ?? "Other"; if (!grouped.has(section)) grouped.set(section, []); const entry = { id, start: start - t0, end: end - t0 }; if (workerStart != null) { entry.workerStart = workerStart - t0; entry.workerEnd = workerEnd - t0; } + if (lane != null) entry.lane = lane; + if (consolidate) entry.consolidate = true; grouped.get(section).push(entry); } + // Condense tasks flagged consolidate into one bar per worker lane. + for (const [section, tasks] of grouped) { + const kept = []; + const byLane = new Map(); + for (const entry of tasks) { + if (!entry.consolidate || entry.lane == null) { kept.push(entry); continue; } + const prev = byLane.get(entry.lane); + if (!prev) byLane.set(entry.lane, { start: entry.start, end: entry.end }); + else { prev.start = Math.min(prev.start, entry.start); prev.end = Math.max(prev.end, entry.end); } + } + if (byLane.size === 0) continue; + const lanes = [...byLane.entries()] + .sort((a, b) => a[1].start - b[1].start) + .map(([lane, { start, end }]) => ({ id: `${section.toLowerCase()} w${lane}`, start, end })); + grouped.set(section, [...kept, ...lanes]); + } + const lines = [ "gantt", " title Build task timeline", diff --git a/builder/worker-pool.mjs b/builder/worker-pool.mjs index ebfec929..3d3a5589 100644 --- a/builder/worker-pool.mjs +++ b/builder/worker-pool.mjs @@ -10,10 +10,10 @@ export class WorkerPool { this._idle = []; // Worker[] this._busy = new Map(); // Worker → { resolve, reject } this._queue = []; // pending { message, transferList, resolve, reject } - this._workers = Array.from({ length: size }, () => this._spawn()); + this._workers = Array.from({ length: size }, (_, i) => this._spawn(i)); } - _spawn() { + _spawn(lane) { const w = new Worker(this._workerUrl); w.on("message", (msg) => { const entry = this._busy.get(w); @@ -21,7 +21,7 @@ export class WorkerPool { this._busy.delete(w); this._idle.push(w); if (msg.error) entry.reject(Object.assign(new Error(msg.error), { stack: msg.stack })); - else entry.resolve(msg.result); + else entry.resolve(Object.assign(msg.result, { lane })); this._drain(); }); w.on("error", (err) => { From ee89d55eafbeb9eb41c63b2b94ea8c77f137d0da Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sun, 31 May 2026 17:26:08 +0200 Subject: [PATCH 37/72] Defer Shiki init so critical-path seeds run without contention --- builder/cpu-worker.mjs | 34 +++++++++++++++++++++------------- builder/highlight.mjs | 3 +-- builder/scheduler.mjs | 1 + builder/worker-pool.mjs | 7 +++++++ 4 files changed, 30 insertions(+), 15 deletions(-) diff --git a/builder/cpu-worker.mjs b/builder/cpu-worker.mjs index f6610ba3..838901bc 100644 --- a/builder/cpu-worker.mjs +++ b/builder/cpu-worker.mjs @@ -7,20 +7,23 @@ import { compileLightScss, compileDarkScss } from "./scss.mjs"; import { regenerateMermaid } from "./mermaid.mjs"; import { captureBuildInfo } from "./build-info.mjs"; -import { initHighlighter } from "./highlight.mjs"; -import { createMarkdownIt, buildLinkTables, - renderPhase } from "./render.mjs"; -import { templatePhase } from "./template.mjs"; -import { unpackShared } from "./sab-broadcast.mjs"; - +import { createMarkdownIt, renderPhase } from "./render.mjs"; +import { templatePhase } from "./template.mjs"; +import { unpackShared } from "./sab-broadcast.mjs"; import { deriveOfflinePage, deriveOfflinePageCached, sliceNavBlock, normalizeBaseurl, - posixDirname } from "./offline-rewrite.mjs"; - -// Start WASM init immediately, do NOT await. Module evaluation finishes -// synchronously so the parentPort.on('message') dispatcher is installed -// before the pool sends any work. Only the `render` handler awaits. -const highlighterP = initHighlighter(); + posixDirname } from "./offline-rewrite.mjs"; + +// Shiki (highlight.mjs) is loaded lazily — its transitive import of the +// shiki package is the heaviest single module in the worker graph. A +// warmup signal from the pool triggers loading on idle workers so it +// overlaps with the main-thread discover phase; critical-path seeds +// (buildInfo, scss) finish before the import starts on their workers. +let _highlighterP = null; +function ensureHighlighterInit() { + if (!_highlighterP) _highlighterP = import("./highlight.mjs").then(m => m.initHighlighter()); + return _highlighterP; +} const handlers = { async scssLight({ ctx }) { @@ -113,7 +116,7 @@ async function getOrInitRenderEnv(sharedSAB) { baseurl, buildInfo, sitePathsArr, skipOffline } = unpackShared(sharedSAB); - const highlighter = await highlighterP; + const highlighter = await ensureHighlighterInit(); const linkTables = reconstructLinkTables(linkTablesData); const staticFiles = new Set(staticFilesArr); const markdown = createMarkdownIt({ highlighter, linkTables, baseurl, staticFiles }); @@ -133,6 +136,10 @@ async function getOrInitRenderEnv(sharedSAB) { } parentPort.on("message", async (msg) => { + // Warmup signal from the pool — start Shiki init without posting a + // response so the pool's busy-tracking is unaffected. + if (msg.warmup) { ensureHighlighterInit(); return; } + const { name, ...payload } = msg; const handler = handlers[name]; if (!handler) { @@ -145,6 +152,7 @@ parentPort.on("message", async (msg) => { } catch (err) { parentPort.postMessage({ error: err.message, stack: err.stack }); } + ensureHighlighterInit(); }); // linkTables values are page objects in the main pipeline, but diff --git a/builder/highlight.mjs b/builder/highlight.mjs index 688014f8..437b88ec 100644 --- a/builder/highlight.mjs +++ b/builder/highlight.mjs @@ -15,8 +15,6 @@ // </div> import { promises as fs } from "node:fs"; -import { createHighlighter } from "shiki"; - import { loadHighlightTheme } from "./highlight-theme.mjs"; // Fenced-info aliases that select the bundled tB grammar. @@ -67,6 +65,7 @@ export async function initHighlighter() { const grammarUrl = new URL("./twinbasic.tmLanguage.json", import.meta.url); const grammarText = await fs.readFile(grammarUrl, "utf8"); const tbGrammar = JSON.parse(grammarText); + const { createHighlighter } = await import("shiki"); shiki = await createHighlighter({ themes: [], langs: [tbGrammar, ...SHIKI_LANGS], diff --git a/builder/scheduler.mjs b/builder/scheduler.mjs index af256933..b05accf1 100644 --- a/builder/scheduler.mjs +++ b/builder/scheduler.mjs @@ -62,6 +62,7 @@ export class Scheduler { } } this._flush(); + this.pool.warmup(); return this._doneP; } diff --git a/builder/worker-pool.mjs b/builder/worker-pool.mjs index 3d3a5589..7676c6c4 100644 --- a/builder/worker-pool.mjs +++ b/builder/worker-pool.mjs @@ -48,6 +48,13 @@ export class WorkerPool { } } + // Send a no-response warmup signal to all currently idle workers. + // Workers handle { warmup: true } without posting back, so the + // pool's busy-tracking is unaffected. + warmup() { + for (const w of this._idle) w.postMessage({ warmup: true }); + } + destroy() { return Promise.all(this._workers.map(w => w.terminate())); } From e9ace9c425dc47abff984f36bfcc48eb0ac77396 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sun, 31 May 2026 18:37:04 +0200 Subject: [PATCH 38/72] Prefer Shiki-warm workers for render dispatch; seed mermaid when cores > 4 --- builder/cpu-worker.mjs | 15 ++++++++----- builder/scheduler.mjs | 3 ++- builder/tbdocs.mjs | 18 ++++++++++------ builder/worker-pool.mjs | 47 ++++++++++++++++++++++++++++++----------- 4 files changed, 59 insertions(+), 24 deletions(-) diff --git a/builder/cpu-worker.mjs b/builder/cpu-worker.mjs index 838901bc..9d0674c7 100644 --- a/builder/cpu-worker.mjs +++ b/builder/cpu-worker.mjs @@ -17,11 +17,16 @@ import { deriveOfflinePage, deriveOfflinePageCached, // Shiki (highlight.mjs) is loaded lazily — its transitive import of the // shiki package is the heaviest single module in the worker graph. A // warmup signal from the pool triggers loading on idle workers so it -// overlaps with the main-thread discover phase; critical-path seeds -// (buildInfo, scss) finish before the import starts on their workers. +// overlaps with the main-thread discover phase. buildInfo and mermaid +// skip the post-handler init so Shiki never contends with them. +// Once init completes, a { warmedUp: true } signal tells the pool this +// worker is warm so it can be preferred for render dispatch. let _highlighterP = null; function ensureHighlighterInit() { - if (!_highlighterP) _highlighterP = import("./highlight.mjs").then(m => m.initHighlighter()); + if (!_highlighterP) { + _highlighterP = import("./highlight.mjs").then(m => m.initHighlighter()); + _highlighterP.then(() => parentPort.postMessage({ warmedUp: true })); + } return _highlighterP; } @@ -140,7 +145,7 @@ parentPort.on("message", async (msg) => { // response so the pool's busy-tracking is unaffected. if (msg.warmup) { ensureHighlighterInit(); return; } - const { name, ...payload } = msg; + const { name, deferHighlighter, ...payload } = msg; const handler = handlers[name]; if (!handler) { parentPort.postMessage({ error: `unknown task: ${name}` }); @@ -152,7 +157,7 @@ parentPort.on("message", async (msg) => { } catch (err) { parentPort.postMessage({ error: err.message, stack: err.stack }); } - ensureHighlighterInit(); + if (!deferHighlighter) ensureHighlighterInit(); }); // linkTables values are page objects in the main pipeline, but diff --git a/builder/scheduler.mjs b/builder/scheduler.mjs index b05accf1..d7fd0732 100644 --- a/builder/scheduler.mjs +++ b/builder/scheduler.mjs @@ -75,7 +75,8 @@ export class Scheduler { this.inFlight++; const p = task.def.runOnMain ? Promise.resolve(task.def.execute(task.inputs, this._ctx, this.state)) - : this.pool.run({ inputs: task.inputs, ctx: this._ctx }, + : this.pool.run({ inputs: task.inputs, ctx: this._ctx, + deferHighlighter: task.def.deferHighlighter }, { name: task.def.handler ?? task.id }); p.then( (output) => this._onDone(task, output, start), diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 90b7b10d..028c5fbc 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -129,6 +129,9 @@ export function makeTimer() { // runBuild() constructs the pool + scheduler, awaits start(), logs the // summary, and returns. +const workerCount = os.availableParallelism(); +const mermaidIsSeed = workerCount > 4; + const TASKS = { // ── Seeds ───────────────────────────────────────────────────────────────── @@ -149,14 +152,16 @@ const TASKS = { }, // Git rev-parse / log shell-outs. Worker so they overlap with the main spine. - // Chains into mermaid so the two don't compete with discover on 4-thread CI. + // When workerCount <= 4, chains into mermaid so the two don't compete with + // discover on the CI runners. When workerCount > 4, mermaid is a seed. buildInfo: { expected: [], + deferHighlighter: true, // execute() runs in cpu-worker.mjs as the "buildInfo" handler. submit(out, emit, state) { state.site.buildInfo = out.buildInfo; emit("dispatch", out); - emit("mermaid", {}); + if (!mermaidIsSeed) emit("mermaid", {}); }, }, @@ -187,10 +192,12 @@ const TASKS = { submit(out, emit) { emit("write", out); }, }, - // Stale mermaid SVG regeneration. Chained after buildInfo (not a seed) to - // avoid competing with discover on 4-thread CI. + // Stale mermaid SVG regeneration. Seed when workerCount > 4 (enough cores + // to run without contention); chained after buildInfo otherwise so it + // doesn't compete with discover on 4-thread CI. mermaid: { - expected: ["buildInfo"], + expected: mermaidIsSeed ? [] : ["buildInfo"], + deferHighlighter: true, // execute() runs in cpu-worker.mjs as the "mermaid" handler. submit(out, emit, state) { // Append any freshly-generated SVG descriptors that discover didn't see @@ -699,7 +706,6 @@ export async function runBuild(opts) { const srcRoot = path.resolve(process.cwd(), src); const destRoot = path.resolve(dest ?? path.join(srcRoot, "_site")); - const workerCount = os.availableParallelism(); const pool = new WorkerPool(workerCount, CPU_WORKER_URL); const scheduler = new Scheduler({ pool, tasks: TASKS }); const ctx = { srcRoot, destRoot, opts, workerCount }; diff --git a/builder/worker-pool.mjs b/builder/worker-pool.mjs index 7676c6c4..3b5c7eca 100644 --- a/builder/worker-pool.mjs +++ b/builder/worker-pool.mjs @@ -1,25 +1,33 @@ // Worker pool over node:worker_threads. Spawns `size` workers eagerly at // construction, routes named tasks to whichever worker is idle, queues the // rest. No dynamic scaling, no recycling, no abort signals. +// +// Idle workers are split into two tiers: _idleWarm (Shiki-initialized) and +// _idleCold. _drain() pulls from warm first so render chunks land on +// workers that can start immediately; cold workers only get work when no +// warm ones are available. import { Worker } from "node:worker_threads"; export class WorkerPool { constructor(size, workerUrl) { this._workerUrl = workerUrl; - this._idle = []; // Worker[] - this._busy = new Map(); // Worker → { resolve, reject } - this._queue = []; // pending { message, transferList, resolve, reject } - this._workers = Array.from({ length: size }, (_, i) => this._spawn(i)); + this._idleWarm = []; // Worker[] — Shiki ready + this._idleCold = []; // Worker[] — not yet initialized + this._warm = new Set(); // all workers that have signalled warmedUp + this._busy = new Map(); // Worker → { resolve, reject } + this._queue = []; // pending { message, transferList, resolve, reject } + this._workers = Array.from({ length: size }, (_, i) => this._spawn(i)); } _spawn(lane) { const w = new Worker(this._workerUrl); w.on("message", (msg) => { + if (msg.warmedUp) { this._onWarmedUp(w); return; } const entry = this._busy.get(w); if (!entry) return; this._busy.delete(w); - this._idle.push(w); + this._pushIdle(w); if (msg.error) entry.reject(Object.assign(new Error(msg.error), { stack: msg.stack })); else entry.resolve(Object.assign(msg.result, { lane })); this._drain(); @@ -28,10 +36,25 @@ export class WorkerPool { const entry = this._busy.get(w); if (entry) { this._busy.delete(w); entry.reject(err); } }); - this._idle.push(w); + this._idleCold.push(w); return w; } + _pushIdle(w) { + if (this._warm.has(w)) this._idleWarm.push(w); + else this._idleCold.push(w); + } + + _onWarmedUp(w) { + this._warm.add(w); + const idx = this._idleCold.indexOf(w); + if (idx !== -1) { + this._idleCold.splice(idx, 1); + this._idleWarm.push(w); + this._drain(); + } + } + run(payload, { name, transferList } = {}) { return new Promise((resolve, reject) => { this._queue.push({ message: { name, ...payload }, transferList, resolve, reject }); @@ -40,19 +63,19 @@ export class WorkerPool { } _drain() { - while (this._queue.length && this._idle.length) { - const w = this._idle.shift(); + while (this._queue.length) { + const w = this._idleWarm.length ? this._idleWarm.shift() + : this._idleCold.length ? this._idleCold.shift() + : null; + if (!w) break; const { message, transferList, resolve, reject } = this._queue.shift(); this._busy.set(w, { resolve, reject }); w.postMessage(message, transferList); } } - // Send a no-response warmup signal to all currently idle workers. - // Workers handle { warmup: true } without posting back, so the - // pool's busy-tracking is unaffected. warmup() { - for (const w of this._idle) w.postMessage({ warmup: true }); + for (const w of this._idleCold) w.postMessage({ warmup: true }); } destroy() { From 23a6cc571b2d8b7ecbad1534463e9f5ec053cc77 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sun, 31 May 2026 18:45:53 +0200 Subject: [PATCH 39/72] Log changed files at rebuild time in serve mode --- builder/serve.mjs | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/builder/serve.mjs b/builder/serve.mjs index 8cd701b7..6e55fbc3 100644 --- a/builder/serve.mjs +++ b/builder/serve.mjs @@ -196,6 +196,7 @@ export async function runServe(opts) { let running = false; let pending = false; let debounceTimer = null; + const changedFiles = new Set(); function schedule() { clearTimeout(debounceTimer); @@ -205,6 +206,9 @@ export async function runServe(opts) { async function fire() { if (running) { pending = true; return; } running = true; + const files = [...changedFiles].sort(); + changedFiles.clear(); + console.log(`\nChanged: ${files.join(", ")}`); try { await runBuild({ ...opts, dest: destRoot, skipOffline: true, skipPdf: true, ganttFile: "serve-gantt.mmd" }); notifyReload(); @@ -224,6 +228,7 @@ export async function runServe(opts) { try { for await (const event of watcher) { if (!shouldRebuild(event.filename)) continue; + changedFiles.add(event.filename.replaceAll("\\", "/")); schedule(); } } catch (err) { From e6425aefefbc9427d2769fd186258b7330605ffe Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober <kuba@mareimbrium.org> Date: Sun, 31 May 2026 19:08:07 +0200 Subject: [PATCH 40/72] Draw the Gantt chart ourselves without Mermaid. This gives more control over formatting and placement of each bar. We'll need it to observe the workload in each worker. --- builder/gantt.mjs | 113 ++++++++++++++++++++++++++++++++ builder/tbdocs.mjs | 61 ++++------------- docs/Documentation/BuildInfo.md | 3 +- 3 files changed, 126 insertions(+), 51 deletions(-) create mode 100644 builder/gantt.mjs diff --git a/builder/gantt.mjs b/builder/gantt.mjs new file mode 100644 index 00000000..a31a8578 --- /dev/null +++ b/builder/gantt.mjs @@ -0,0 +1,113 @@ +// Inline SVG Gantt chart for the build timeline. +// Replaces the client-side Mermaid renderer — no JS runtime needed. + +const COLORS = { + Seeds: { light: "#86c7a3", dark: "#3d8b5e" }, + Spine: { light: "#6eb5d9", dark: "#3c7db0" }, + Render: { light: "#b09cd8", dark: "#8066a8" }, + Write: { light: "#e8a756", dark: "#c08030" }, + Other: { light: "#bbb", dark: "#666" }, +}; + +const SECTION_W = 60; +const SVG_W = 900; +const CHART_W = SVG_W - SECTION_W - 20; +const ROW_H = 20; +const BAR_H = 14; +const AXIS_H = 28; +const CHAR_W = 6.2; +const BAR_PAD = 4; + +export function renderGantt(grouped) { + const all = [...grouped.values()].flat(); + if (all.length === 0) return ""; + const maxT = Math.max(...all.map(t => t.end)); + if (maxT <= 0) return ""; + + let rows = 0; + for (const tasks of grouped.values()) rows += tasks.length; + const h = AXIS_H + rows * ROW_H + 5; + const xOf = t => SECTION_W + (t / maxT) * CHART_W; + + const tick = niceInterval(maxT); + const ticks = []; + for (let t = 0; t <= maxT + 0.5; t += tick) ticks.push(t); + + const o = []; + o.push(`<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 ${SVG_W} ${h}" style="width:100%;max-width:${SVG_W}px">`); + o.push(`<title>Build task timeline`); + + const css = [ + `.gantt{font-family:system-ui,-apple-system,sans-serif}`, + `.gl{fill:#333}.gs{fill:#333;font-weight:600}.ga{fill:#666}.gg{stroke:#e0e0e0}`, + ...Object.entries(COLORS).map(([s, c]) => `.gb-${s.toLowerCase()}{fill:${c.light}}`), + `html.dark-mode .gl{fill:#e6e1e8}`, + `html.dark-mode .gs{fill:#e6e1e8}`, + `html.dark-mode .ga{fill:#959396}`, + `html.dark-mode .gg{stroke:#44434d}`, + ...Object.entries(COLORS).map(([s, c]) => `html.dark-mode .gb-${s.toLowerCase()}{fill:${c.dark}}`), + ]; + o.push(``); + o.push(``); + + for (const t of ticks) { + const x = rd(xOf(t)); + o.push(``); + o.push(`${fmtMs(t)}`); + } + + let y = AXIS_H; + for (const [section, tasks] of grouped) { + if (tasks.length === 0) continue; + o.push(``); + const cls = `gb-${section.toLowerCase()}`; + for (let i = 0; i < tasks.length; i++) { + const t = tasks[i]; + const bx = rd(xOf(t.start)); + const bw = rd(Math.max(xOf(t.end) - xOf(t.start), 1)); + const by = rd(y + (ROW_H - BAR_H) / 2); + const ty = rd(y + ROW_H / 2 + 3.5); + if (i === 0) o.push(`${esc(section)}`); + o.push(``); + const lbl = taskLabel(t); + const textW = lbl.length * CHAR_W; + if (textW + BAR_PAD * 2 <= bw) { + o.push(`${esc(lbl)}`); + } else if (bx + bw + 4 + textW <= SVG_W) { + o.push(`${esc(lbl)}`); + } else { + o.push(`${esc(lbl)}`); + } + y += ROW_H; + } + } + + o.push(``); + return o.join("\n"); +} + +function niceInterval(max) { + for (const c of [100, 200, 250, 500, 1000, 2000, 2500, 5000]) + if (max / c <= 10) return c; + return Math.ceil(max / 10000) * 1000; +} + +function fmtMs(ms) { + return `${Math.floor(ms / 1000)}.${String(ms % 1000).padStart(3, "0")}`; +} + +function taskLabel(t) { + let s = t.id.replace(":", " "); + if (t.workerStart != null) { + const d = t.end - t.start; + if (d > 0) { + const a = Math.round((t.workerStart - t.start) / d * 100); + const b = Math.round((t.workerEnd - t.workerStart) / d * 100); + s += ` (${a}%+${b}%)`; + } + } + return s; +} + +function rd(n) { return Math.round(n * 10) / 10; } +function esc(s) { return s.replace(/&/g, "&").replace(//g, ">"); } diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 028c5fbc..620ed8de 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -23,6 +23,7 @@ import pc from "picocolors"; import { WorkerPool } from "./worker-pool.mjs"; import { Scheduler } from "./scheduler.mjs"; +import { renderGantt } from "./gantt.mjs"; import { discover } from "./discover.mjs"; import { computeNav } from "./nav.mjs"; @@ -588,7 +589,7 @@ const GANTT_SECTION = { const GANTT_SECTION_ORDER = ["Seeds", "Spine", "Render", "Write"]; async function writeGantt(timings, outPath) { - if (timings.size === 0) return ""; + if (timings.size === 0) return null; const t0 = Math.min(...[...timings.values()].map(t => t.start)); const grouped = new Map(GANTT_SECTION_ORDER.map(s => [s, []])); @@ -641,59 +642,21 @@ async function writeGantt(timings, outPath) { lines.push(""); } - const content = lines.join("\n"); - await fs.writeFile(outPath, content, "utf8"); - return content; + await fs.writeFile(outPath, lines.join("\n"), "utf8"); + return grouped; } -async function injectGanttChart(pages, destRoot, ganttContent) { - if (!ganttContent) return; +async function injectGanttChart(pages, destRoot, svgContent) { + if (!svgContent) return; const page = pages.find(p => p.permalink === "/Documentation/Development/BuildInfo"); if (!page) return; - const chart = `
\n${ganttContent}
`; - // Match the rendered empty mmd code block — outer div + two closing divs. - const mmdBlockRe = /
[\s\S]*?<\/div>\s*<\/div>/; - - // Online / serve: CDN ESM import. - const onlineScript = - ``; - - // Offline: self-contained IIFE bundle (the ESM entry point uses dynamic chunk - // imports that break on file:// URLs; the IIFE is a single self-contained file). - // Relative prefix climbs from the page's directory back to the site root. - const depth = page.destPath.split(/[/\\]/).length - 1; - const rel = "../".repeat(depth); - const offlineScript = - `\n` + - ``; - - const mermaidSrc = fileURLToPath( - new URL("../node_modules/mermaid/dist/mermaid.min.js", import.meta.url)); - - const targets = [ - { root: destRoot, script: onlineScript }, - { root: `${destRoot}-offline`, script: offlineScript, copyMermaid: true }, - ]; - - for (const { root, script, copyMermaid } of targets) { + for (const root of [destRoot, `${destRoot}-offline`]) { const htmlPath = path.join(root, page.destPath); let html; - try { - html = await fs.readFile(htmlPath, "utf8"); - } catch (e) { - if (e.code !== "ENOENT") throw e; - continue; - } - if (copyMermaid) { - const dest = path.join(root, "assets", "js", "mermaid.min.js"); - await fs.mkdir(path.dirname(dest), { recursive: true }); - await fs.copyFile(mermaidSrc, dest); - } - const patched = html.replace(mmdBlockRe, `${chart}\n${script}`); + try { html = await fs.readFile(htmlPath, "utf8"); } + catch (e) { if (e.code !== "ENOENT") throw e; continue; } + const patched = html.replace("", svgContent); if (patched !== html) await fs.writeFile(htmlPath, patched, "utf8"); } } @@ -766,10 +729,10 @@ export async function runBuild(opts) { console.log(scheduler.summary()); const ganttPath = path.resolve(process.cwd(), opts.ganttFile ?? "build-gantt.mmd"); - const ganttContent = await writeGantt(scheduler.timings, ganttPath); + const grouped = await writeGantt(scheduler.timings, ganttPath); const injectStart = Date.now(); - await injectGanttChart(scheduler.state.pages, destRoot, ganttContent); + await injectGanttChart(scheduler.state.pages, destRoot, grouped ? renderGantt(grouped) : ""); console.log(pc.dim(`gantt-inject=${Date.now() - injectStart}ms`)); // Drift guard from PLAN-1.md §1. diff --git a/docs/Documentation/BuildInfo.md b/docs/Documentation/BuildInfo.md index 4ad1e98f..f1c4d185 100644 --- a/docs/Documentation/BuildInfo.md +++ b/docs/Documentation/BuildInfo.md @@ -11,5 +11,4 @@ permalink: /Documentation/Development/BuildInfo Gantt chart of this build's task timeline. -```mmd -``` + From 2d19d30d5c24eee87d746d2e0fcba21c91171910 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Sun, 31 May 2026 19:46:45 +0200 Subject: [PATCH 41/72] Make task time accounting finer-grained. --- builder/cpu-worker.mjs | 9 +++- builder/gantt.mjs | 111 +++++++++++++++++++++++++++++++++------- builder/tbdocs.mjs | 26 ++++------ builder/worker-pool.mjs | 11 +++- 4 files changed, 117 insertions(+), 40 deletions(-) diff --git a/builder/cpu-worker.mjs b/builder/cpu-worker.mjs index 9d0674c7..d087de3c 100644 --- a/builder/cpu-worker.mjs +++ b/builder/cpu-worker.mjs @@ -2,7 +2,7 @@ // appropriate handler and posts back { result } or { error, stack }. // See PLAN-scheduler.md §Worker for the full handler set. -import { parentPort } from "node:worker_threads"; +import { parentPort, workerData } from "node:worker_threads"; import { compileLightScss, compileDarkScss } from "./scss.mjs"; import { regenerateMermaid } from "./mermaid.mjs"; import { captureBuildInfo } from "./build-info.mjs"; @@ -14,6 +14,10 @@ import { deriveOfflinePage, deriveOfflinePageCached, sliceNavBlock, normalizeBaseurl, posixDirname } from "./offline-rewrite.mjs"; +// Report cold-boot time: from worker spawn (passed via workerData) to +// the point where all static imports have resolved and top-level code runs. +if (workerData?.spawnTime) parentPort.postMessage({ coldBoot: { start: workerData.spawnTime, end: Date.now() } }); + // Shiki (highlight.mjs) is loaded lazily — its transitive import of the // shiki package is the heaviest single module in the worker graph. A // warmup signal from the pool triggers loading on idle workers so it @@ -24,8 +28,9 @@ import { deriveOfflinePage, deriveOfflinePageCached, let _highlighterP = null; function ensureHighlighterInit() { if (!_highlighterP) { + const warmStart = Date.now(); _highlighterP = import("./highlight.mjs").then(m => m.initHighlighter()); - _highlighterP.then(() => parentPort.postMessage({ warmedUp: true })); + _highlighterP.then(() => parentPort.postMessage({ warmedUp: true, warmBoot: { start: warmStart, end: Date.now() } })); } return _highlighterP; } diff --git a/builder/gantt.mjs b/builder/gantt.mjs index a31a8578..a8d2cd99 100644 --- a/builder/gantt.mjs +++ b/builder/gantt.mjs @@ -6,6 +6,7 @@ const COLORS = { Spine: { light: "#6eb5d9", dark: "#3c7db0" }, Render: { light: "#b09cd8", dark: "#8066a8" }, Write: { light: "#e8a756", dark: "#c08030" }, + Boot: { light: "#e57373", dark: "#c62828" }, Other: { light: "#bbb", dark: "#666" }, }; @@ -24,8 +25,32 @@ export function renderGantt(grouped) { const maxT = Math.max(...all.map(t => t.end)); if (maxT <= 0) return ""; - let rows = 0; - for (const tasks of grouped.values()) rows += tasks.length; + // Any task with a lane ran on a worker — pull it into the Workers + // section, tagged with its original section for bar colour. Leftover + // Render tasks (dispatch, prepDest) fold into Spine. + const seeds = [], spine = [], write = []; + const laneTasks = []; + for (const [section, tasks] of grouped) { + for (const t of tasks) { + if (t.lane != null) { t._color = section; laneTasks.push(t); } + else if (section === "Seeds") seeds.push(t); + else if (section === "Spine" || section === "Render") spine.push(t); + else if (section === "Write") write.push(t); + } + } + const mainSections = [["Seeds", seeds], ["Spine", spine], ["Write", write]]; + + const lanes = new Map(); + for (const t of laneTasks) { + if (!lanes.has(t.lane)) lanes.set(t.lane, []); + lanes.get(t.lane).push(t); + } + for (const tasks of lanes.values()) + tasks.sort((a, b) => a.workerStart - b.workerStart); + const sortedLanes = [...lanes.entries()].sort((a, b) => a[0] - b[0]); + + let rows = sortedLanes.length; + for (const [, tasks] of mainSections) rows += tasks.length; const h = AXIS_H + rows * ROW_H + 5; const xOf = t => SECTION_W + (t / maxT) * CHART_W; @@ -57,35 +82,79 @@ export function renderGantt(grouped) { } let y = AXIS_H; - for (const [section, tasks] of grouped) { + + // Seeds, Spine (with dispatch / prepDest folded in) + for (const [section, tasks] of mainSections.slice(0, 2)) { if (tasks.length === 0) continue; + y = renderMainSection(o, section, tasks, y, xOf); + } + + // Workers — one row per lane, individual task bars + if (sortedLanes.length > 0) { o.push(``); - const cls = `gb-${section.toLowerCase()}`; - for (let i = 0; i < tasks.length; i++) { - const t = tasks[i]; - const bx = rd(xOf(t.start)); - const bw = rd(Math.max(xOf(t.end) - xOf(t.start), 1)); - const by = rd(y + (ROW_H - BAR_H) / 2); + for (let li = 0; li < sortedLanes.length; li++) { + const [, tasks] = sortedLanes[li]; const ty = rd(y + ROW_H / 2 + 3.5); - if (i === 0) o.push(`${esc(section)}`); - o.push(``); - const lbl = taskLabel(t); - const textW = lbl.length * CHAR_W; - if (textW + BAR_PAD * 2 <= bw) { - o.push(`${esc(lbl)}`); - } else if (bx + bw + 4 + textW <= SVG_W) { - o.push(`${esc(lbl)}`); - } else { - o.push(`${esc(lbl)}`); + const by = rd(y + (ROW_H - BAR_H) / 2); + if (li === 0) o.push(`Workers`); + const bootBars = []; + for (const t of tasks) { + if (t._color === "Boot") { bootBars.push(t); continue; } + const bx = rd(xOf(t.workerStart)); + const bw = rd(Math.max(xOf(t.workerEnd) - xOf(t.workerStart), 1)); + o.push(``); + const lbl = workerLabel(t); + if (lbl.length * CHAR_W + BAR_PAD * 2 <= bw) + o.push(`${esc(lbl)}`); + } + const bootH = Math.round(BAR_H * 0.75); + for (const t of bootBars) { + const bx = rd(xOf(t.workerStart)); + const bw = rd(Math.max(xOf(t.workerEnd) - xOf(t.workerStart), 1)); + o.push(``); + const lbl = workerLabel(t); + if (lbl.length * CHAR_W + BAR_PAD * 2 <= bw) + o.push(`${esc(lbl)}`); } y += ROW_H; } } + // Write + for (const [section, tasks] of mainSections.slice(2)) { + if (tasks.length === 0) continue; + y = renderMainSection(o, section, tasks, y, xOf); + } + o.push(``); return o.join("\n"); } +function renderMainSection(o, section, tasks, y, xOf) { + o.push(``); + const cls = `gb-${section.toLowerCase()}`; + for (let i = 0; i < tasks.length; i++) { + const t = tasks[i]; + const bx = rd(xOf(t.start)); + const bw = rd(Math.max(xOf(t.end) - xOf(t.start), 1)); + const by = rd(y + (ROW_H - BAR_H) / 2); + const ty = rd(y + ROW_H / 2 + 3.5); + if (i === 0) o.push(`${esc(section)}`); + o.push(``); + const lbl = taskLabel(t); + const textW = lbl.length * CHAR_W; + if (textW + BAR_PAD * 2 <= bw) { + o.push(`${esc(lbl)}`); + } else if (bx + bw + 4 + textW <= SVG_W) { + o.push(`${esc(lbl)}`); + } else { + o.push(`${esc(lbl)}`); + } + y += ROW_H; + } + return y; +} + function niceInterval(max) { for (const c of [100, 200, 250, 500, 1000, 2000, 2500, 5000]) if (max / c <= 10) return c; @@ -109,5 +178,9 @@ function taskLabel(t) { return s; } +function workerLabel(t) { + return t.id.replace(/:.*/, "").replace(/ w\d+$/, ""); +} + function rd(n) { return Math.round(n * 10) / 10; } function esc(s) { return s.replace(/&/g, "&").replace(//g, ">"); } diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 620ed8de..2bf84247 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -604,23 +604,6 @@ async function writeGantt(timings, outPath) { grouped.get(section).push(entry); } - // Condense tasks flagged consolidate into one bar per worker lane. - for (const [section, tasks] of grouped) { - const kept = []; - const byLane = new Map(); - for (const entry of tasks) { - if (!entry.consolidate || entry.lane == null) { kept.push(entry); continue; } - const prev = byLane.get(entry.lane); - if (!prev) byLane.set(entry.lane, { start: entry.start, end: entry.end }); - else { prev.start = Math.min(prev.start, entry.start); prev.end = Math.max(prev.end, entry.end); } - } - if (byLane.size === 0) continue; - const lanes = [...byLane.entries()] - .sort((a, b) => a[1].start - b[1].start) - .map(([lane, { start, end }]) => ({ id: `${section.toLowerCase()} w${lane}`, start, end })); - grouped.set(section, [...kept, ...lanes]); - } - const lines = [ "gantt", " title Build task timeline", @@ -728,6 +711,15 @@ export async function runBuild(opts) { } console.log(scheduler.summary()); + for (const bt of pool.bootTimings) { + scheduler.timings.set(`${bt.type}:w${bt.lane}`, { + start: bt.start, end: bt.end, + workerStart: bt.start, workerEnd: bt.end, + lane: bt.lane, + ganttSection: "Boot", + }); + } + const ganttPath = path.resolve(process.cwd(), opts.ganttFile ?? "build-gantt.mmd"); const grouped = await writeGantt(scheduler.timings, ganttPath); diff --git a/builder/worker-pool.mjs b/builder/worker-pool.mjs index 3b5c7eca..bd7a1e81 100644 --- a/builder/worker-pool.mjs +++ b/builder/worker-pool.mjs @@ -17,13 +17,20 @@ export class WorkerPool { this._warm = new Set(); // all workers that have signalled warmedUp this._busy = new Map(); // Worker → { resolve, reject } this._queue = []; // pending { message, transferList, resolve, reject } + this.bootTimings = []; // { lane, type, start, end }[] this._workers = Array.from({ length: size }, (_, i) => this._spawn(i)); } _spawn(lane) { - const w = new Worker(this._workerUrl); + const spawnTime = Date.now(); + const w = new Worker(this._workerUrl, { workerData: { lane, spawnTime } }); w.on("message", (msg) => { - if (msg.warmedUp) { this._onWarmedUp(w); return; } + if (msg.coldBoot) { this.bootTimings.push({ lane, type: "cold", ...msg.coldBoot }); return; } + if (msg.warmedUp) { + if (msg.warmBoot) this.bootTimings.push({ lane, type: "warm", ...msg.warmBoot }); + this._onWarmedUp(w); + return; + } const entry = this._busy.get(w); if (!entry) return; this._busy.delete(w); From 5f6d35c6f9694d12ed6362bca47a0aac145a0791 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Sun, 31 May 2026 19:53:20 +0200 Subject: [PATCH 42/72] Fix a bug in serve mode that made it detect its own writes as file changes. --- builder/serve.mjs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/builder/serve.mjs b/builder/serve.mjs index 6e55fbc3..b11b4d59 100644 --- a/builder/serve.mjs +++ b/builder/serve.mjs @@ -134,7 +134,7 @@ function createStaticHandler(destRoot) { } // §D — Watcher filtering -const IGNORED_PREFIXES = ["_site", "_site-offline", "_site-pdf", "_serve", "_pdf", "node_modules", ".git"]; +const IGNORED_PREFIXES = ["_site", "_site-offline", "_site-pdf", "_serve", "_serve-offline", "_serve-pdf", "_pdf", "node_modules", ".git"]; const IGNORED_BASENAME_RE = /^\.|~$|\.tmp$|\.swp$|^4913$/; function shouldRebuild(filename) { From 55ec068252bb1ff0760e3ddc9188171b6ff61dbd Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Mon, 1 Jun 2026 13:56:22 +0200 Subject: [PATCH 43/72] Stop writing the Gantt .mmd file; the SVG is the only output. --- .gitignore | 3 --- builder/serve.mjs | 4 ++-- builder/tbdocs.mjs | 29 ++--------------------------- 3 files changed, 4 insertions(+), 32 deletions(-) diff --git a/.gitignore b/.gitignore index aedeb88c..8ea362a8 100644 --- a/.gitignore +++ b/.gitignore @@ -7,6 +7,3 @@ node_modules/ /before/ /after-*/ /findoverflow-baseline/ - -/build-gantt.mmd -/serve-gantt.mmd diff --git a/builder/serve.mjs b/builder/serve.mjs index b11b4d59..b9089f5c 100644 --- a/builder/serve.mjs +++ b/builder/serve.mjs @@ -167,7 +167,7 @@ export async function runServe(opts) { // Initial build try { - await runBuild({ ...opts, dest: destRoot, skipOffline: true, skipPdf: true, ganttFile: "serve-gantt.mmd" }); + await runBuild({ ...opts, dest: destRoot, skipOffline: true, skipPdf: true }); } catch (err) { console.error("serve: initial build failed:", err.message); process.exit(1); @@ -210,7 +210,7 @@ export async function runServe(opts) { changedFiles.clear(); console.log(`\nChanged: ${files.join(", ")}`); try { - await runBuild({ ...opts, dest: destRoot, skipOffline: true, skipPdf: true, ganttFile: "serve-gantt.mmd" }); + await runBuild({ ...opts, dest: destRoot, skipOffline: true, skipPdf: true }); notifyReload(); } catch (err) { console.error("rebuild failed:", err.message); diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 2bf84247..3b339cd7 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -60,7 +60,6 @@ function parseArgs(argv) { profileOffline: false, serve: false, port: 4000, - ganttFile: "build-gantt.mmd", }; for (let i = 0; i < argv.length; i++) { const a = argv[i]; @@ -588,7 +587,7 @@ const GANTT_SECTION = { }; const GANTT_SECTION_ORDER = ["Seeds", "Spine", "Render", "Write"]; -async function writeGantt(timings, outPath) { +function groupGanttTimings(timings) { if (timings.size === 0) return null; const t0 = Math.min(...[...timings.values()].map(t => t.start)); @@ -603,29 +602,6 @@ async function writeGantt(timings, outPath) { if (consolidate) entry.consolidate = true; grouped.get(section).push(entry); } - - const lines = [ - "gantt", - " title Build task timeline", - " dateFormat x", - " axisFormat %S.%L", - "", - ]; - for (const [section, tasks] of grouped) { - if (tasks.length === 0) continue; - lines.push(` section ${section}`); - for (const { id, start, end, workerStart, workerEnd } of tasks) { - const pct = workerStart != null - ? ` (${Math.round((workerStart - start) / (end - start) * 100)}%+${Math.round((workerEnd - workerStart) / (end - start) * 100)}%)` - : ""; - const label = id.replace(":", " ") + pct; - const taskId = `t_${id.replace(/[^a-z0-9]/gi, "_")}`; - lines.push(` ${label} :done, ${taskId}, ${start}, ${Math.max(end, start + 1)}`); - } - lines.push(""); - } - - await fs.writeFile(outPath, lines.join("\n"), "utf8"); return grouped; } @@ -720,8 +696,7 @@ export async function runBuild(opts) { }); } - const ganttPath = path.resolve(process.cwd(), opts.ganttFile ?? "build-gantt.mmd"); - const grouped = await writeGantt(scheduler.timings, ganttPath); + const grouped = groupGanttTimings(scheduler.timings); const injectStart = Date.now(); await injectGanttChart(scheduler.state.pages, destRoot, grouped ? renderGantt(grouped) : ""); From fc1b1f0638d5eb5bfac12d5e2ab6e896f63911f6 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Mon, 1 Jun 2026 16:07:50 +0200 Subject: [PATCH 44/72] Plan the SharedArrayBuffer (SAB)-based scheduler. --- builder/PLAN-sab-pull-scheduler.md | 979 +++++++++++++++++++++++++++++ 1 file changed, 979 insertions(+) create mode 100644 builder/PLAN-sab-pull-scheduler.md diff --git a/builder/PLAN-sab-pull-scheduler.md b/builder/PLAN-sab-pull-scheduler.md new file mode 100644 index 00000000..9262b3a1 --- /dev/null +++ b/builder/PLAN-sab-pull-scheduler.md @@ -0,0 +1,979 @@ +# SAB-based worker-pull scheduler + +Replaces the current push-based scheduler (main thread decides what's +ready, dispatches to workers via `pool.run()`) with a pull-based +model where workers resolve dependencies themselves via +SharedArrayBuffer atomics and claim the next task immediately after +completing one --- no main-thread round-trip for worker-to-worker +transitions. + +## Problem + +The scheduler runs on the main thread. When a `runOnMain` task's +`execute()` is running (e.g. `discover` at ~135 ms), the event loop is +blocked. Worker completion messages queue; no new tasks are dispatched +until the main-thread work finishes. On a 16-core machine, the idle +time across all threads sums to ~1 s --- significant against a <2 s +build. Worse on CI (4 threads) because fewer workers share the same +blocking windows. + +## Solution + +Put the mutable scheduling state (dependency counts, task status) in a +SharedArrayBuffer visible to all threads. Workers update dep counts and +claim ready tasks via `Atomics` operations. The main thread only +participates for `runOnMain` tasks and output merges into `SharedState`. + +Three new scheduling primitives handle the warmup case and +future worker-affinity needs: + +1. **`on_demand`** --- a seed task (no prerequisites) that is NOT + started at build start. It is triggered only when a dependent task + would otherwise be ready to run. Applies to both worker and + main-thread tasks. + +2. **`unique_per_worker`** --- instead of one global "done" flag, the + task has a done flag per worker lane. From worker W's perspective, + the task is done iff lane W's instance ran. Only applies to worker + tasks. + +3. **`pin_to_predecessor`** --- the task must run on the same worker + lane that ran a named predecessor. Applies to worker tasks. + +### warmInit under the new model + +`warmInit` is declared as an explicit task with both `unique_per_worker` +and `on_demand`. All `render` chunks list it as a per-worker dependency. +This replaces the current ad-hoc mechanism: the two-tier idle queue +(`_idleWarm` / `_idleCold`), the `warmup()` call in `scheduler.start()`, +the `deferHighlighter` flag on task defs, the `warmedUp` message +protocol, and the conditional `ensureHighlighterInit()` calls in +`cpu-worker.mjs`. + +## SAB memory layout + +A single SharedArrayBuffer allocated by the main thread before the +build starts. All arrays are Int32 for `Atomics` compatibility. + +``` +Constants: + MAX_TASKS = 256 // static tasks + max dynamic tasks (render chunks + renderJoin) + MAX_LANES = 64 // max worker threads + MAX_EDGES = 512 // total successor edges across all tasks + +Status values: + NOT_READY = 0 + READY = 1 + CLAIMED = 2 + DONE = 3 + +Flag bits: + F_ON_DEMAND = 1 + F_UNIQUE_PER_WORKER = 2 + F_RUN_ON_MAIN = 4 + F_PIN_TO_PRED = 8 + +Arrays (all Int32Array views into the SAB): + taskCount // [1] — current number of registered tasks (atomic) + depCount [MAX_TASKS] // remaining normal predecessor count per task + status [MAX_TASKS] // NOT_READY | READY | CLAIMED | DONE + flags [MAX_TASKS] // bitmask of F_* constants + succOffset [MAX_TASKS] // index into succList where this task's successors start + succCount [MAX_TASKS] // number of successors for this task + succList [MAX_EDGES] // flat array of successor task indices + affinityLane[MAX_TASKS] // -1 = any worker, 0..N-1 = pinned to this lane + pinnedTo [MAX_TASKS] // -1 = no pin, else = predecessor task index whose lane to inherit + completedOnLane[MAX_TASKS] // which lane completed this task (-1 = not done / ran on main) + perWorkerDone[MAX_TASKS * MAX_LANES] // 0 = not done, 1 = done (for unique_per_worker tasks) + edgeCount // [1] — current successor edge count (for dynamic append) + notify // [1] — generation counter for worker wakeup (see §Notify protocol) + firstReady // [1] — low-water mark: all tasks below this index are DONE (optimization) + buildDone // [1] — 0 = running, 1 = done, 2 = error (workers check and exit) + chunkOffset [MAX_RENDER_CHUNKS] // byte offset into chunkDataSAB for render chunk i + chunkLength [MAX_RENDER_CHUNKS] // byte length of render chunk i's JSON in chunkDataSAB +``` + + MAX_RENDER_CHUNKS = MAX_LANES * 10 // SLICES_PER_WORKER = 10 + +Total size: `(MAX_TASKS * 9 + MAX_EDGES + MAX_TASKS * MAX_LANES ++ MAX_RENDER_CHUNKS * 2 + 6) * 4` bytes. +With the defaults above: ~73 KB. Negligible. + +### Task ID mapping + +The SAB uses integer indices. A bidirectional mapping (name -> index, +index -> name) is built at startup for static tasks and extended by +`dispatch` for dynamic tasks. + +Static tasks get indices 0..N-1 in definition order. Dynamic task +slots (render chunks + renderJoin) are pre-reserved starting at index +`DYNAMIC_BASE`. The maximum number of render chunks is known: +`workerCount * SLICES_PER_WORKER` (currently 10 slices/worker). +`renderJoin` gets the slot after the last possible render chunk. + +### Graph metadata (init message to workers) + +The SAB encodes mutable scheduling state (dep counts, status, flags). +Immutable graph structure that workers also need is sent once at build +start via a `postMessage` init message. This includes: + +- **`taskMeta[]`** --- per-task-index metadata array: + - `handler`: string name of the worker handler function to call + (e.g. `"render"`, `"scssLight"`, `"warmInit"`). Workers use this + to look up the right function after claiming a task by index. + - `perWorkerDeps`: array of task indices that are + `unique_per_worker` dependencies. Empty for most tasks; render + chunks have `[warmInitIdx]`. Workers check these after claiming. + - `name`: string task name (for timing / error messages). + +- **`ctx`** --- the build context (`{ srcRoot, destRoot, opts, + workerCount }`). Small, immutable within a build. Workers cache it + and use it for seed handlers (`buildInfo`, `scssLight`, `scssDark`, + `mermaid`). + +- **`sabRef`** --- the scheduling SharedArrayBuffer itself. + +- **`idMapping`** --- task name to index and index to name maps + (for debug logging; not needed for the hot path). + +The init message is sent once per build. In serve mode, each rebuild +sends a fresh init message with a new SAB and ctx. Workers detect +the new init message, switch to the new SAB, reset their cached +`chunkDataSAB` / `renderEnv`, and re-enter the pull loop. + +Dynamic task registration (`dispatch` creating `render:i` tasks) +extends the metadata: `dispatch.submit()` broadcasts an update +message with the new task entries' metadata (handler names, +perWorkerDeps) for the dynamically-registered index range. Workers +merge this into their local `taskMeta` array. This arrives before +the render tasks become READY in the SAB (dispatch sets their status +to READY after broadcasting). + +## Task definition format + +Existing fields (`expected`, `handler`, `runOnMain`, `execute`, +`submit`) are retained. New fields: + +```js +warmInit: { + expected: [], + on_demand: true, // not started until a dependent needs it + unique_per_worker: true, // one instance per worker lane + handler: "warmInit", + // No submit --- unique_per_worker tasks don't participate in the + // normal dependency graph. +}, + +'render:0': { + expected: ["dispatch"], // normal dep (dispatch must complete) + perWorkerDeps: ["warmInit"], // checked at claim time, per-lane + handler: "render", + // ... +}, + +someTask: { + expected: ["priorTask"], + pin_to_predecessor: "priorTask", // must run on priorTask's worker lane + handler: "someHandler", + // ... +}, +``` + +### `submit()` split + +`submit()` currently does two things: (a) signal dependency completion +via `emit()`, and (b) mutate `SharedState`. Under the new model: + +- **(a) Dependency signaling** moves to the SAB. The completing thread + (worker or main) atomically decrements successor dep counts. This + is encoded in the SAB's successor adjacency list, not in `submit()`. + +- **(b) State mutation** stays in `submit()`, which runs on the main + thread as before. Worker tasks fire-and-forget their output via + `postMessage`; the main thread runs `submit()` to merge the output + into `SharedState` when it processes the message. + +The ordering constraint: a worker posts the output message BEFORE +updating successor dep counts in the SAB. Since worker-to-main +messages are FIFO, the merge message arrives before the main thread +would claim any downstream `runOnMain` task. The main thread drains +all pending messages before scanning the SAB for ready main-thread +tasks, ensuring merges complete first. + +For **worker-to-worker chains** (e.g. render:i -> renderJoin where +renderJoin is a trivial barrier), the successor dep count update +happens directly in the SAB with no main-thread involvement. If +`render:i.submit()` has state mutations (merging page deltas), the +merge message is fire-and-forget --- it doesn't gate the next worker +task because the downstream workers don't read `SharedState`. + +### How submit() is triggered + +Worker tasks: the main thread's message handler calls `submit()` +when it processes the output message. This is asynchronous relative +to the worker's progress (the worker has already moved on to its next +task via SAB). + +Main-thread tasks: `submit()` is called inline after `execute()` +completes, as today. + +## Worker pull loop + +Each worker runs a persistent loop after receiving the SAB and graph +metadata at startup: + +``` +function pullLoop(sab, views, myLane, handlers, graphMeta): + loop: + if Atomics.load(views.buildDone, 0) !== 0: return + + taskIdx = scanAndClaim(views, myLane) + if taskIdx === -1: + gen = Atomics.load(views.notify, 0) + // Double-check after reading gen — a task may have become READY + // between our scan and this load. + taskIdx = scanAndClaim(views, myLane) + if taskIdx === -1: + Atomics.wait(views.notify, 0, gen, 50) // sleep until gen changes, 50ms fallback + continue + + // Check per-worker deps (unique_per_worker) + unsatisfied = null + for each perWorkerDep D of graphMeta[taskIdx]: + if Atomics.load(views.perWorkerDone, D * MAX_LANES + myLane) === 0: + unsatisfied = D + break + + if unsatisfied !== null: + if flags[unsatisfied] & F_ON_DEMAND && !(flags[unsatisfied] & F_RUN_ON_MAIN): + // On-demand worker dep (e.g. warmInit): per-worker, no contention. + // Release the original task BEFORE executing the dep. + Atomics.store(views.status, taskIdx, READY) + Atomics.add(views.notify, 0, 1) + Atomics.notify(views.notify, 0, 1) // wake one worker for the released task + + execute handlers[unsatisfied] + Atomics.store(views.perWorkerDone, unsatisfied * MAX_LANES + myLane, 1) + continue // re-enter pull loop; may reclaim original task or get a different one + + if flags[unsatisfied] & F_ON_DEMAND && flags[unsatisfied] & F_RUN_ON_MAIN: + // On-demand main-thread dep: trigger it, release our task, wait. + Atomics.store(views.status, unsatisfied, READY) + postMessage({ triggerMainTask: unsatisfied }) + Atomics.store(views.status, taskIdx, READY) + Atomics.add(views.notify, 0, 1) + Atomics.notify(views.notify, 0, 1) // wake one worker for the released task + + // Check for other non-dependent work before sleeping + altTask = scanAndClaim(views, myLane) // may find unrelated work + if altTask !== -1: + // ... execute altTask (same path as below) ... + else: + // Wait on the dep's status slot rather than spinning claim-release cycles + Atomics.wait(views.status, unsatisfied, READY) + continue // re-enter pull loop + + // unique_per_worker without on_demand: should already be done + // (was started eagerly). Spin-wait. + Atomics.store(views.status, taskIdx, READY) + Atomics.add(views.notify, 0, 1) + Atomics.notify(views.notify, 0, 1) + waitForPerWorkerDone(views, unsatisfied, myLane) + continue + + // All deps satisfied --- execute + result = await handlers[graphMeta[taskIdx].handler](taskPayload) + postMessage({ done: taskIdx, output: result }) // fire-and-forget + onTaskDone(views, taskIdx, myLane, graphMeta) +``` + +### scanAndClaim + +``` +function scanAndClaim(views, myLane): + start = Atomics.load(views.firstReady, 0) + count = Atomics.load(views.taskCount, 0) + for i = start to count - 1: + if Atomics.load(views.status, i) !== READY: continue + if Atomics.load(views.flags, i) & F_RUN_ON_MAIN: continue + aff = Atomics.load(views.affinityLane, i) + if aff !== -1 && aff !== myLane: continue + if Atomics.compareExchange(views.status, i, READY, CLAIMED) === READY: + return i + return -1 +``` + +### onTaskDone (successor dep count update) + +``` +function onTaskDone(views, taskIdx, lane, graphMeta): + Atomics.store(views.status, taskIdx, DONE) + Atomics.store(views.completedOnLane, taskIdx, lane) + advanceFirstReady(views, taskIdx) + + readyCount = 0 + wakeMain = false + + off = Atomics.load(views.succOffset, taskIdx) + count = Atomics.load(views.succCount, taskIdx) + for i = off to off + count - 1: + succ = Atomics.load(views.succList, i) + + // Skip unique_per_worker successors (not tracked via depCount) + if Atomics.load(views.flags, succ) & F_UNIQUE_PER_WORKER: continue + + remaining = Atomics.sub(views.depCount, succ, 1) - 1 + if remaining === 0: + // Set affinity if successor is pinned + pin = Atomics.load(views.pinnedTo, succ) + if pin !== -1: + srcLane = Atomics.load(views.completedOnLane, pin) + Atomics.store(views.affinityLane, succ, srcLane) + + Atomics.store(views.status, succ, READY) + + if Atomics.load(views.flags, succ) & F_RUN_ON_MAIN: + wakeMain = true + else: + readyCount++ + + // Bump generation counter and wake the right number of workers + if readyCount > 0: + Atomics.add(views.notify, 0, 1) + Atomics.notify(views.notify, 0, readyCount) + if wakeMain: + postMessage({ mainTaskReady: true }) +``` + +### advanceFirstReady + +`firstReady` is a low-water mark: all task indices below it are DONE. +It only advances forward (monotonic). After any task transitions to +DONE, the completing thread tries to advance past consecutive DONE +tasks starting from the current value: + +``` +function advanceFirstReady(views, taskIdx): + count = Atomics.load(views.taskCount, 0) + cur = Atomics.load(views.firstReady, 0) + if taskIdx !== cur: return // not at the frontier; nothing to advance + next = cur + while next < count && Atomics.load(views.status, next) === DONE: + next++ + if next > cur: + Atomics.compareExchange(views.firstReady, 0, cur, next) + // CAS may fail if another thread advanced it further; that's fine. +``` + +This is a best-effort optimization. The CAS failure case is harmless +--- the competing thread advanced the pointer at least as far, so no +scan work is wasted. The scan is already microseconds, so this avoids +re-checking the early spine tasks (config, discover, nav, ...) once +they've completed and the build is in the render fan-out or write +phase. + +### Anti-thundering-herd for on-demand main-thread deps + +When a worker discovers an unsatisfied on-demand main-thread dep, it: + +1. Claims the dep (CAS on its status, or just sets it READY if it's + on_demand + not yet triggered --- first writer wins since + READY is idempotent) +2. Releases its original task back to READY +3. **Waits on the dep's status slot** (`Atomics.wait(status, depIdx, ...)`) + rather than re-entering the scan loop + +This prevents N workers from cycling through claim-release on render +chunks while the main thread runs the dep. When the dep completes +(main thread sets its status to DONE + notifies), all waiting workers +wake and re-scan productively. + +Workers first check if other non-dependent work is available before +waiting. The check is cheap (one scan of the status array) and avoids +unnecessary sleeping when there's useful work to do. + +## Main thread protocol + +The main thread is event-loop driven. It does NOT spin or use +`Atomics.wait` (which would block the event loop). Instead: + +### Input accumulation (results map) + +Main-thread tasks receive an `inputs` object keyed by predecessor +name: `{ predecessor1: output1, predecessor2: output2, ... }`. Under +the current push scheduler, `emit()` accumulates these in a +`pending.received` map. Under the SAB model, dependency *counting* +moves to the SAB, but the actual *data* still flows through the main +thread. + +The main thread maintains a `results` map: +`Map`. Every task's output is stored here --- +both worker tasks (when the `{ done, output }` message is processed) +and main-thread tasks (inline after execute). When a main-thread +task becomes READY and the main thread claims it, it assembles the +inputs: + +``` +function assembleInputs(taskIdx, taskDef, results, idMapping): + inputs = {} + for predName of taskDef.expected: + predIdx = idMapping.nameToIdx[predName] + inputs[predName] = results.get(predIdx) + return inputs +``` + +This is simpler than the current `pending` machinery --- no received +counting, no emit routing. The SAB dep count handles "when is it +ready"; the results map handles "what data does it get." + +Worker tasks do NOT read from the results map. They get their inputs +from the SAB (chunk data, shared payload) or from `ctx`. The results +map is main-thread-only. + +### Message handler + +``` +worker.on('message', msg => { + if (msg.done != null) { + // Store output for downstream main-thread tasks' input assembly + results.set(msg.done, msg.output) + // Run submit() to merge into SharedState + taskDef = tasks[msg.done] + taskDef.submit(msg.output, state) + } + if (msg.mainTaskReady != null || msg.triggerMainTask != null) { + scheduleMainScan() + } +}) +``` + +`scheduleMainScan()` uses `queueMicrotask()` (coalesced --- skip if +already scheduled) so all pending messages are processed (output +stored + merges complete) before the scan runs. + +### Main-thread task execution + +``` +function mainScan(): + start = Atomics.load(views.firstReady, 0) + count = Atomics.load(views.taskCount, 0) + for i = start to count - 1: + if Atomics.load(views.status, i) !== READY: continue + if !(Atomics.load(views.flags, i) & F_RUN_ON_MAIN): continue + if Atomics.compareExchange(views.status, i, READY, CLAIMED) !== READY: continue + + // Check on-demand deps (main-thread on_demand deps are global, not per-worker) + unsatisfied = checkOnDemandDeps(i) + if unsatisfied !== null: + // Run the on-demand dep inline (single-threaded, no concurrency concern) + output = await executeMainTask(unsatisfied) + results.set(unsatisfied, output) + Atomics.store(views.status, unsatisfied, DONE) + advanceFirstReady(views, unsatisfied) + Atomics.notify(views.status, unsatisfied) // wake waiting workers + + inputs = assembleInputs(i, taskDef, results, idMapping) + output = await taskDef.execute(inputs, ctx, state) + results.set(i, output) + Atomics.store(views.status, i, DONE) + taskDef.submit(output, state) // mutate SharedState + onTaskDone(views, i, -1, graphMeta) // -1 = main thread lane + + // Re-scan: the task we just completed may have made more main tasks ready + scheduleMainScan() + return +``` + +### Draining messages before scanning + +The main thread processes worker messages in the event loop's message +handler. `scheduleMainScan()` posts a microtask. Since microtasks run +after the current handler but before the next event, and multiple +worker messages in the same event-loop tick are processed sequentially, +all pending merges complete before the scan. + +If messages arrive while a `runOnMain` execute() is in progress, they +queue until execute() yields (await) or completes. This is the +irreducible cost of main-thread tasks --- same as today, but worker- +to-worker transitions no longer pay it. + +## Dynamic task registration (dispatch) + +`dispatch` runs on the main thread. After computing chunks: + +1. Write render:i task entries into pre-reserved SAB slots: + - `depCount[slot]` = 0 (render chunks are seeded directly) + - `flags[slot]` = 0 (worker task, not on_demand/unique_per_worker) + - `perWorkerDeps` metadata = `[warmInitIdx]` + - successor entries pointing to `renderJoinIdx` + +2. Write renderJoin entry: + - `depCount[renderJoinIdx]` = N (one per render chunk) + - `flags[renderJoinIdx]` = F_RUN_ON_MAIN + - successor entries to write, writePdf, searchData + +3. Append successor edges for render:i -> renderJoin to `succList`. + +4. Update `Atomics.store(views.taskCount, 0, newCount)`. + +5. Set each render:i's status to READY and notify workers. + +Workers that are waiting (Atomics.wait) wake up and see the new +render tasks. + +### Task inputs for render chunks + +The render chunks need their page data (the chunk array + the shared +SAB broadcast payload). The approach: extend the existing +`sab-broadcast.mjs` pattern. + +After `dispatch` creates chunks on the main thread: + +1. JSON-serialize each chunk, concatenate the byte arrays into one + buffer. +2. Pack into a single `chunkDataSAB` (SharedArrayBuffer). +3. Write offset/length per chunk into the scheduling SAB's + `chunkOffset` / `chunkLength` arrays. +4. Broadcast `{ chunkDataSAB }` to all workers in a single + postMessage (the SAB is a shared reference, not cloned). + +Workers store the `chunkDataSAB` reference when they receive the +message. When a worker claims `render:i`, it reads +`chunkOffset[i]` / `chunkLength[i]` from the scheduling SAB and +deserializes its slice from `chunkDataSAB`. Same cost as today's +structured clone per chunk, but without the main thread serializing +N copies and without blocking on per-worker postMessage delivery. + +The existing shared payload SAB (site data, initData, link tables) +is packed separately by `dispatch` as today and included in the same +broadcast message. + +Workers that are busy with non-render work (scss, buildInfo) when the +broadcast arrives queue the message and read it when they first claim +a render chunk. + +### Task inputs for non-render worker tasks + +`buildInfo`, `scssLight`, `scssDark`, and `mermaid` need `ctx` +(mainly `ctx.srcRoot`). The `ctx` object is small and immutable +within a build. It is sent once at build start alongside the +scheduling SAB as part of the init message. Workers cache it. + +In serve mode, a fresh `ctx` is sent with each rebuild's new SAB. + +## Build start sequence + +Step-by-step from `runBuild()` entry to the first task executing: + +1. **Allocate the scheduling SAB.** `allocSchedulerSAB(TASKS, + workerCount)` reads the task definitions, assigns integer indices, + computes the successor adjacency list, and writes all static + fields (depCount, flags, succOffset/succCount/succList, pinnedTo, + affinityLane initialized to -1, completedOnLane initialized to -1, + perWorkerDone zeroed, firstReady = 0, notify = 0, buildDone = 0). + Returns `{ sab, views, idMapping, taskMeta }`. + + Seed tasks (expected.length === 0 AND NOT on_demand) have their + status set to READY. All others are NOT_READY. On-demand seeds + stay NOT_READY until triggered. + +2. **Construct or reuse the worker pool.** In `runBuild()`, the pool + is created fresh (and destroyed after the build). In serve mode, + the pool persists and is reused. + +3. **Post init message to all workers.** Each worker receives: + ``` + { init: true, sab, taskMeta, ctx, idMapping } + ``` + Workers store these, create Int32Array views over the SAB, and + enter the pull loop. Workers that were sleeping from a previous + build (serve mode) receive this as a regular message, detect the + `init` flag, switch to the new SAB, reset cached state + (`chunkDataSAB`, `renderEnv`), and re-enter the pull loop. + +4. **Main thread enters its scan loop.** `scheduleMainScan()` is + called once to kick off the first scan. Seed `runOnMain` tasks + (e.g. `config`) are already READY in the SAB, so the first + `mainScan()` claims and executes them. + +5. **Workers wake and scan.** Workers see seed worker tasks + (`buildInfo`, `scssLight`, `scssDark`, optionally `mermaid`) as + READY in the SAB and claim them. + +6. **Build proceeds.** Workers and main thread independently claim + and execute tasks via the SAB. Worker outputs flow to the main + thread as fire-and-forget messages. The main thread merges, + accumulates results, and claims main-thread tasks as they become + ready. + +7. **Dispatch phase.** When `dispatch` (runOnMain) executes, it: + - Computes render chunks and builds the chunkDataSAB + sharedSAB. + - Writes dynamic task entries (render:i, renderJoin) into the + pre-reserved SAB slots. + - Broadcasts `{ renderData: true, chunkDataSAB, sharedSAB, + taskMeta: [...new entries...] }` to all workers. + - Sets render:i status to READY and bumps the notify generation + counter: `Atomics.add(views.notify, 0, 1)` + + `Atomics.notify(views.notify, 0, Infinity)`. + - Workers wake, merge the new taskMeta entries, store the + chunkDataSAB, and claim render chunks. + +8. **Completion.** The main thread detects all tasks DONE (or + untriggered on_demand). Sets `buildDone = 1` in the SAB, notifies + all workers. Workers exit the pull loop. `runBuild()` resolves. + +## What gets removed + +| Current code | Replacement | +|---|---| +| `WorkerPool._idleWarm` / `_idleCold` / `_warm` | Workers are equal; warmth is emergent | +| `WorkerPool._onWarmedUp()` | No warm/cold distinction | +| `WorkerPool._drain()` + `_queue` | Workers pull from SAB; no push queue | +| `WorkerPool.warmup()` | `warmInit` task (on_demand + unique_per_worker) | +| `WorkerPool.run()` for worker tasks | Workers self-schedule; main-thread tasks still use a thin dispatch | +| `Scheduler._flush()` / `_run()` push logic | SAB atomics | +| `Scheduler.pending` / `ready` / `emit()` | SAB depCount + status | +| `deferHighlighter` flag on task defs | Gone; warmInit is explicit | +| `ensureHighlighterInit()` in cpu-worker.mjs | `warmInit` handler | +| `warmedUp` / `warmBoot` message protocol | Gone | +| `warmup: true` message handling | Gone | + +`WorkerPool` reduces to a lifecycle manager: spawn workers at +construction, send them the SAB + metadata, terminate on destroy. +The message forwarding (output merges, main-task signals) remains. + +## Serve mode + +The pool persists across rebuilds. Per rebuild: + +1. Main thread allocates a new scheduling SAB (new dep counts, fresh + status array). +2. Main thread posts the new SAB to all workers as a "new build" + message. +3. Workers switch to the new SAB and enter the pull loop. +4. Build completes (main thread detects: all tasks DONE, no pending + work). +5. Workers go idle (`Atomics.wait` on the status array --- nothing + is READY). + +On the next rebuild, step 2 wakes all workers (they're waiting) and +they switch to the fresh SAB. + +The `chunkDataSAB` from the previous build is garbage-collected once +no worker holds a reference. + +## Completion detection + +The build is complete when: + +- All tasks have status DONE (or are unreachable --- e.g. on_demand + tasks that were never triggered). +- No worker is executing (all are in Atomics.wait or have exited + the pull loop). + +The main thread tracks this by counting: every time a task transitions +to DONE, increment a `doneCount` atomic. When `doneCount` equals the +number of non-on_demand tasks plus the number of triggered on_demand +tasks, the build is done. + +Simpler approach: the main thread's `mainScan()` checks after each +task completion whether any tasks remain (status != DONE, excluding +untriggered on_demand). When none remain, resolve the build promise. + +Workers are notified of build completion via a `buildDone` atomic in +the SAB. Workers check this in their pull loop and exit cleanly. + +## Error handling + +### Worker task failure + +Worker catches the error in its handler, posts +`{ error: taskIdx, message, stack }` to the main thread, and sets +the task's SAB status to a new `FAILED` state (value 4). The main +thread's error handler rejects the build promise (same as today's +`_onError`). All other workers see `FAILED` when scanning and skip +the task. + +Workers do NOT abort on a sibling's failure --- they continue +processing ready tasks until the main thread signals build abort +via the `buildDone` atomic (set to an error sentinel). Workers +check `buildDone` in their pull loop and exit. + +### Main-thread task failure + +Same as today: the main thread catches the error in `execute()`, +rejects the build promise, and signals workers to stop via +`buildDone`. + +### Worker crash + +Same as today: the crashed worker is not respawned; the pool +degrades. The worker's in-progress task stays CLAIMED forever +(no successor dep counts are decremented). The main thread +detects a stalled build via a timeout or the worker's `exit` event, +and aborts. + +## Timing / instrumentation + +Workers post timing data alongside their output: +`{ done: taskIdx, output, timing: { start, end } }`. The main +thread collects these into the existing `timings` map. The summary +and Gantt chart code are unchanged. + +For `unique_per_worker` tasks (`warmInit`), each worker posts its own +timing. The summary consolidates these per-lane, same as render chunks. + +## Migration phases + +### Current state (phases 0--4 are done) + +Phases 0--4 from [PLAN-scheduler.md](PLAN-scheduler.md) are fully +implemented. The codebase already has: + +- `builder/scheduler.mjs` --- push-based `Scheduler` class with + `SharedState`, `pending`/`ready` maps, `_flush()`/`_run()` dispatch. +- `builder/worker-pool.mjs` --- `WorkerPool` with two-tier idle queue + (`_idleWarm` / `_idleCold`), `warmup()`, `run()` push dispatch. +- `builder/cpu-worker.mjs` --- worker harness with `parentPort` + message loop, `ensureHighlighterInit()`, `getOrInitRenderEnv()`, + named handlers (`scssLight`, `scssDark`, `mermaid`, `buildInfo`, + `render`). +- `builder/sab-broadcast.mjs` --- `packShared()` / `unpackShared()` + for the render fan-out's shared payload SAB. +- `builder/tbdocs.mjs` --- full task DAG (`TASKS` object) with all + static and dynamic task definitions, `dispatch.submit()` dynamic + registration of `render:i` + `renderJoin`, Gantt chart instrumentation. +- `builder/serve.mjs` --- dev server reusing the pool across rebuilds. + +The build runs end-to-end through the push scheduler with worker +fan-out. `build.bat && check.bat` is clean at baseline. + +The phases below (5--8) replace the push scheduler internals with +the SAB-based pull model while preserving the task definitions, +handler functions, and external behavior. + +### Phase 5: SAB scheduler skeleton + +**Files:** new `builder/sab-scheduler.mjs`, modifications to +`scheduler.mjs`, `worker-pool.mjs`, `cpu-worker.mjs`. + +1. Define SAB constants (`MAX_TASKS`, `MAX_LANES`, `MAX_EDGES`, + `MAX_RENDER_CHUNKS`, status values, flag bits), byte-offset + calculations, and a `createViews(sab)` helper that returns an + object of named Int32Array views over the SAB. + +2. Add `allocSchedulerSAB(taskDefs, workerCount)`: + - Assigns integer indices to each static task (definition order). + - Pre-reserves `DYNAMIC_BASE` through + `DYNAMIC_BASE + MAX_RENDER_CHUNKS` for render chunks, plus one + slot for `renderJoin`. + - Builds the successor adjacency list from `taskDef.expected` + (inverting predecessor lists to successor lists). + - Writes depCount, flags (from `runOnMain`, `on_demand`, + `unique_per_worker`, `pin_to_predecessor`), succOffset/succCount/ + succList, pinnedTo, affinityLane (-1), completedOnLane (-1). + - Sets seed tasks' status to READY (except on_demand seeds). + - Returns `{ sab, views, idMapping, taskMeta }`. + +3. `idMapping` contains: + - `nameToIdx`: `Map` (task name -> SAB index). + - `idxToName`: `string[]` (SAB index -> task name). + - `DYNAMIC_BASE`, `RENDER_JOIN_IDX`: constants for dispatch. + +4. `taskMeta` is a plain array indexed by task index: + - `taskMeta[i].handler`: handler function name (string). + - `taskMeta[i].perWorkerDeps`: array of task indices (for + unique_per_worker deps). Empty for most tasks. + - `taskMeta[i].name`: task name (for debug/timing). + +5. Add the `warmInit` task definition to `TASKS` in `tbdocs.mjs`: + ```js + warmInit: { + expected: [], + on_demand: true, + unique_per_worker: true, + handler: "warmInit", + submit() {}, + }, + ``` + No runtime behavior change yet --- the push scheduler ignores the + new flags and the warmInit handler is not wired up. + +6. No runtime behavior change. The existing push scheduler still + runs. This phase adds data structures only. + +**Verification:** assert at build time that the SAB encodes the +expected dep counts and successor edges for the static task graph. +`build.bat && check.bat` clean; output unchanged. + +### Phase 6: Worker pull loop + +**This is the critical phase.** Worker-to-worker transitions move +to the SAB. Main-thread tasks still run via the existing push +scheduler, bridged into the SAB. + +1. **Init message handling.** `WorkerPool` sends `{ init: true, sab, + taskMeta, ctx, idMapping }` to each worker after construction (or + after each rebuild in serve mode). Workers store these and create + SAB views. + +2. **Handler table.** `cpu-worker.mjs` keeps its existing named + handlers (`scssLight`, `scssDark`, `mermaid`, `buildInfo`, + `render`) and adds `warmInit`: + ```js + async warmInit() { + const start = Date.now(); + const highlighter = await (await import("./highlight.mjs")).initHighlighter(); + return { warmInit: true, timing: { start, end: Date.now() } }; + } + ``` + The pull loop looks up the handler by name: + `handlers[taskMeta[taskIdx].handler]`. + +3. **Pull loop.** Replace the `parentPort.on('message')` dispatch + with the persistent pull loop (§Worker pull loop pseudocode). + The message handler is retained only for: + - `{ init }` --- switch to new SAB + metadata. + - `{ renderData, chunkDataSAB, sharedSAB, taskMeta }` --- store + chunk data and merge new taskMeta entries from dispatch. + +4. **Output posting.** After executing a task, the worker posts: + ``` + { done: taskIdx, output: result, timing: { start, end } } + ``` + Then calls `onTaskDone()` to update the SAB. The `postMessage` + happens BEFORE the SAB update (ordering constraint from §submit() + split). + +5. **Bridge: main thread updates SAB after its tasks.** The existing + push scheduler's `_onDone()` is extended: after running `submit()` + and `emit()` as today, it also calls `onTaskDone(views, taskIdx, + -1, graphMeta)` to decrement successor dep counts in the SAB and + set newly-ready tasks to READY. This lets workers see downstream + tasks become ready immediately after a main-thread task completes, + without waiting for the push scheduler's `_flush()`. + + The push scheduler's `_flush()` / `_run()` still handles + main-thread tasks. Worker tasks are no longer dispatched through + `pool.run()` --- they're pulled from the SAB. + +6. **warmInit replaces ensureHighlighterInit().** The `warmInit` + handler does the same work (dynamic import of highlight.mjs + + initHighlighter). The on-demand + unique_per_worker flags ensure + it runs once per lane, only when needed. The `deferHighlighter` + flag and `ensureHighlighterInit()` calls are removed. + +**Verification:** `build.bat && check.bat` clean. Timing summary +shows render chunks starting without main-thread gaps. `warmInit` +appears in per-lane timing (consolidated like render chunks). + +### Phase 7: Main-thread SAB integration + +Replace the push scheduler's main-thread dispatch with SAB-based +claiming. The `Scheduler` class is rewritten. + +1. **`results` map.** The scheduler maintains + `results: Map`. Populated in two places: + - Worker output messages: `results.set(msg.done, msg.output)`. + - Main-thread task completion: `results.set(idx, output)` inline. + +2. **`assembleInputs()`.** Before executing a `runOnMain` task, the + scheduler reads the task definition's `expected` array, maps each + predecessor name to its index via `idMapping`, looks up the output + in `results`, and builds the `inputs` object: + ```js + function assembleInputs(taskIdx, taskDef, results, idMapping) { + const inputs = {}; + for (const predName of taskDef.expected) { + const predIdx = idMapping.nameToIdx.get(predName); + inputs[predName] = results.get(predIdx); + } + return inputs; + } + ``` + This replaces the current `pending.received` accumulation and + `emit()` routing. + +3. **`mainScan()`.** Replaces `_flush()` / `_run()`. Scans the SAB + for READY + F_RUN_ON_MAIN tasks, claims via CAS, assembles inputs, + executes, runs `submit()`, calls `onTaskDone()` to update successor + dep counts. See §Main-thread task execution pseudocode. + +4. **Message handler.** Replaces the pool's completion callback. + Processes `{ done, output }` (store + submit), `{ mainTaskReady }` + and `{ triggerMainTask }` (schedule scan), `{ error }` (abort). + Uses `queueMicrotask` coalescing so all pending messages drain + before scanning. + +5. **Completion detection.** After each `onTaskDone()` call from the + main thread, check: scan the SAB for any task that is not DONE + and not an untriggered on_demand task. If none remain, set + `buildDone = 1`, notify all workers, resolve the build promise. + +6. **Remove push machinery.** Delete `Scheduler.pending`, `ready`, + `emit()`, `_flush()`, `_run()`, `seed()`, `register()`. + `dispatch.submit()` now writes directly to the SAB and broadcasts + to workers (see §Build start sequence step 7) instead of calling + `scheduler.register()` / `scheduler.seed()`. + +**Verification:** `build.bat && check.bat` clean. Full build runs +through the SAB scheduler with no push-based code paths. The timing +summary and Gantt chart are identical to Phase 6 (same tasks, same +concurrency, different dispatch mechanism). + +### Phase 8: Cleanup + +1. Remove `WorkerPool._idleWarm`, `_idleCold`, `_warm`, + `_onWarmedUp`, `_drain`, `_queue`, `warmup()`. +2. Remove `deferHighlighter` from task defs and cpu-worker. +3. Remove `warmedUp` / `warmBoot` message protocol. +4. Remove `warmup: true` handling in cpu-worker. +5. `WorkerPool` becomes a thin lifecycle manager: spawn, forward + messages, terminate. +6. Update `serve.mjs` per-rebuild SAB reallocation. +7. Update Gantt chart to include `warmInit` per-lane entries. + +**Verification:** `build.bat && check.bat` clean. Serve mode works +(rebuild on file change, workers reuse across rebuilds). No +warmup-related code remains. + +## Notify protocol + +Workers sleep on a single generation-counter slot (`views.notify`) +rather than per-task status slots. The protocol: + +``` +// Worker (after scanAndClaim returns -1): +gen = Atomics.load(views.notify, 0) +taskIdx = scanAndClaim(views, myLane) // double-check: race window +if taskIdx === -1: + Atomics.wait(views.notify, 0, gen, 50) // sleep until gen changes, 50ms fallback + +// Any thread making N worker tasks READY (in onTaskDone): +Atomics.add(views.notify, 0, 1) // bump generation +Atomics.notify(views.notify, 0, readyCount) // wake exactly readyCount workers + +// dispatch seeding N render chunks: +Atomics.add(views.notify, 0, 1) +Atomics.notify(views.notify, 0, Infinity) // wake all workers +``` + +The double-check in the worker's sleep path prevents a race: a task +could become READY between the failed scan and the `Atomics.load` of +the generation counter. Without the double-check, the worker would +sleep with stale `gen` and miss the notification. + +The 50 ms timeout is a safety net for edge cases where a notification +is lost (e.g. the bump and notify happen between the worker's +`Atomics.load` and `Atomics.wait`, and no other notification follows). +50 ms is long enough to avoid busy-spinning but short enough to not +stall a build. + +**Exception: on-demand main-thread deps.** When a worker is waiting +for a specific on-demand main-thread task to complete, it waits on +that task's **status slot** (`Atomics.wait(views.status, depIdx, +READY)`) --- not the generation counter. This is targeted: the worker +knows exactly what it's waiting for, and wakes as soon as the main +thread sets the task to DONE and notifies the slot. Before waiting, +the worker checks if other non-dependent work is available (one scan); +if so, it does that work instead of sleeping. From 309c03d489c5ed65988b82701d59ea202264d977 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Mon, 1 Jun 2026 16:07:54 +0200 Subject: [PATCH 45/72] Add SAB scheduler skeleton (Phase 5). --- builder/sab-scheduler.mjs | 281 ++++++++++++++++++++++++++++++++++++++ builder/scheduler.mjs | 2 +- builder/tbdocs.mjs | 18 ++- 3 files changed, 298 insertions(+), 3 deletions(-) create mode 100644 builder/sab-scheduler.mjs diff --git a/builder/sab-scheduler.mjs b/builder/sab-scheduler.mjs new file mode 100644 index 00000000..ae8e1a88 --- /dev/null +++ b/builder/sab-scheduler.mjs @@ -0,0 +1,281 @@ +// SAB-based scheduling data structures. Phase 5: layout, allocation, and +// verification only; the push scheduler still runs the build. +// See PLAN-sab-pull-scheduler.md for the full design. + +// ── Constants ──────────────────────────────────────────────────────────────── + +export const MAX_TASKS = 256; +export const MAX_LANES = 64; +export const MAX_EDGES = 512; +export const SLICES_PER_WORKER = 10; +export const MAX_RENDER_CHUNKS = MAX_LANES * SLICES_PER_WORKER; + +// Status values (Int32) +export const NOT_READY = 0; +export const READY = 1; +export const CLAIMED = 2; +export const DONE = 3; + +// Flag bits +export const F_ON_DEMAND = 1; +export const F_UNIQUE_PER_WORKER = 2; +export const F_RUN_ON_MAIN = 4; +export const F_PIN_TO_PRED = 8; + +// ── SAB layout ─────────────────────────────────────────────────────────────── +// +// All arrays are Int32. Offsets are in Int32 elements (multiply by 4 for bytes). + +const L = (() => { + let o = 0; + const a = n => { const off = o; o += n; return off; }; + return { + taskCount: a(1), + depCount: a(MAX_TASKS), + status: a(MAX_TASKS), + flags: a(MAX_TASKS), + succOffset: a(MAX_TASKS), + succCount: a(MAX_TASKS), + succList: a(MAX_EDGES), + affinityLane: a(MAX_TASKS), + pinnedTo: a(MAX_TASKS), + completedOnLane: a(MAX_TASKS), + perWorkerDone: a(MAX_TASKS * MAX_LANES), + edgeCount: a(1), + notify: a(1), + firstReady: a(1), + buildDone: a(1), + chunkOffset: a(MAX_RENDER_CHUNKS), + chunkLength: a(MAX_RENDER_CHUNKS), + TOTAL: o, + }; +})(); + +export const SAB_BYTE_LENGTH = L.TOTAL * 4; + +// ── View creation ──────────────────────────────────────────────────────────── + +export function createViews(sab) { + const v = (off, len) => new Int32Array(sab, off * 4, len); + return { + taskCount: v(L.taskCount, 1), + depCount: v(L.depCount, MAX_TASKS), + status: v(L.status, MAX_TASKS), + flags: v(L.flags, MAX_TASKS), + succOffset: v(L.succOffset, MAX_TASKS), + succCount: v(L.succCount, MAX_TASKS), + succList: v(L.succList, MAX_EDGES), + affinityLane: v(L.affinityLane, MAX_TASKS), + pinnedTo: v(L.pinnedTo, MAX_TASKS), + completedOnLane: v(L.completedOnLane, MAX_TASKS), + perWorkerDone: v(L.perWorkerDone, MAX_TASKS * MAX_LANES), + edgeCount: v(L.edgeCount, 1), + notify: v(L.notify, 1), + firstReady: v(L.firstReady, 1), + buildDone: v(L.buildDone, 1), + chunkOffset: v(L.chunkOffset, MAX_RENDER_CHUNKS), + chunkLength: v(L.chunkLength, MAX_RENDER_CHUNKS), + }; +} + +// ── Allocation ─────────────────────────────────────────────────────────────── + +export function allocSchedulerSAB(taskDefs, workerCount) { + // 1. Assign indices to static tasks in definition order + const nameToIdx = new Map(); + const idxToName = []; + + for (const name of Object.keys(taskDefs)) { + nameToIdx.set(name, idxToName.length); + idxToName.push(name); + } + + const DYNAMIC_BASE = idxToName.length; + const maxChunks = workerCount * SLICES_PER_WORKER; + const RENDER_JOIN_IDX = DYNAMIC_BASE + maxChunks; + + // Pre-reserve render chunk and renderJoin slots + for (let i = 0; i < maxChunks; i++) { + nameToIdx.set(`render:${i}`, DYNAMIC_BASE + i); + idxToName.push(`render:${i}`); + } + nameToIdx.set("renderJoin", RENDER_JOIN_IDX); + idxToName.push("renderJoin"); + + const totalTasks = RENDER_JOIN_IDX + 1; + if (totalTasks > MAX_TASKS) + throw new Error(`${totalTasks} tasks exceeds MAX_TASKS (${MAX_TASKS})`); + + // 2. Build successor adjacency list by inverting expected[] predecessors + const successors = Array.from({ length: totalTasks }, () => []); + + for (const [name, def] of Object.entries(taskDefs)) { + const taskIdx = nameToIdx.get(name); + for (const pred of def.expected) { + const predIdx = nameToIdx.get(pred); + if (predIdx == null) + throw new Error(`"${name}" expects unknown predecessor "${pred}"`); + successors[predIdx].push(taskIdx); + } + } + + // 3. Allocate SAB and create views + const sab = new SharedArrayBuffer(SAB_BYTE_LENGTH); + const views = createViews(sab); + + Atomics.store(views.taskCount, 0, totalTasks); + + // 4. Populate per-task arrays + let edgePos = 0; + + for (let i = 0; i < totalTasks; i++) { + const name = idxToName[i]; + const def = taskDefs[name]; // undefined for dynamic slots + + // depCount: static tasks use expected.length; dynamic slots start at 0 + if (def) views.depCount[i] = def.expected.length; + + // flags + let f = 0; + if (def?.on_demand) f |= F_ON_DEMAND; + if (def?.unique_per_worker) f |= F_UNIQUE_PER_WORKER; + if (def?.runOnMain) f |= F_RUN_ON_MAIN; + if (def?.pin_to_predecessor) f |= F_PIN_TO_PRED; + views.flags[i] = f; + + // successor edges + const succs = successors[i]; + views.succOffset[i] = edgePos; + views.succCount[i] = succs.length; + for (const s of succs) { + if (edgePos >= MAX_EDGES) + throw new Error(`Edge count exceeds MAX_EDGES (${MAX_EDGES})`); + views.succList[edgePos++] = s; + } + + // defaults + views.affinityLane[i] = -1; + views.completedOnLane[i] = -1; + + // pinnedTo + if (def?.pin_to_predecessor) { + const pinIdx = nameToIdx.get(def.pin_to_predecessor); + if (pinIdx == null) + throw new Error(`"${name}" pinned to unknown "${def.pin_to_predecessor}"`); + views.pinnedTo[i] = pinIdx; + } else { + views.pinnedTo[i] = -1; + } + + // status: non-on_demand seeds get READY + if (def && def.expected.length === 0 && !def.on_demand) { + views.status[i] = READY; + } + } + + Atomics.store(views.edgeCount, 0, edgePos); + + // 5. Build taskMeta + const warmInitIdx = nameToIdx.get("warmInit"); + const taskMeta = new Array(totalTasks).fill(null); + + for (const [name, def] of Object.entries(taskDefs)) { + taskMeta[nameToIdx.get(name)] = { + handler: def.handler ?? name, + perWorkerDeps: [], + name, + }; + } + + for (let i = 0; i < maxChunks; i++) { + taskMeta[DYNAMIC_BASE + i] = { + handler: "render", + perWorkerDeps: warmInitIdx != null ? [warmInitIdx] : [], + name: `render:${i}`, + }; + } + + taskMeta[RENDER_JOIN_IDX] = { + handler: "renderJoin", + perWorkerDeps: [], + name: "renderJoin", + }; + + const idMapping = { + nameToIdx, + idxToName, + DYNAMIC_BASE, + RENDER_JOIN_IDX, + maxRenderChunks: maxChunks, + }; + + return { sab, views, idMapping, taskMeta }; +} + +// ── Verification ───────────────────────────────────────────────────────────── + +export function verifySchedulerSAB(taskDefs, views, idMapping) { + const { nameToIdx, idxToName } = idMapping; + const errors = []; + + // Verify dep counts for static tasks + for (const [name, def] of Object.entries(taskDefs)) { + const idx = nameToIdx.get(name); + const actual = views.depCount[idx]; + if (actual !== def.expected.length) + errors.push(`depCount "${name}": got ${actual}, want ${def.expected.length}`); + } + + // Verify successor edges (rebuild expected set and compare) + const expectedSucc = new Map(); + for (let i = 0; i < idxToName.length; i++) expectedSucc.set(i, new Set()); + + for (const [name, def] of Object.entries(taskDefs)) { + const taskIdx = nameToIdx.get(name); + for (const pred of def.expected) { + expectedSucc.get(nameToIdx.get(pred)).add(taskIdx); + } + } + + for (const [predIdx, want] of expectedSucc) { + const off = views.succOffset[predIdx]; + const cnt = views.succCount[predIdx]; + const got = new Set(); + for (let i = off; i < off + cnt; i++) got.add(views.succList[i]); + + for (const s of want) { + if (!got.has(s)) + errors.push(`missing edge: ${idxToName[predIdx]} -> ${idxToName[s]}`); + } + for (const s of got) { + if (!want.has(s)) + errors.push(`extra edge: ${idxToName[predIdx]} -> ${idxToName[s]}`); + } + } + + // Verify flags + for (const [name, def] of Object.entries(taskDefs)) { + const idx = nameToIdx.get(name); + let want = 0; + if (def.on_demand) want |= F_ON_DEMAND; + if (def.unique_per_worker) want |= F_UNIQUE_PER_WORKER; + if (def.runOnMain) want |= F_RUN_ON_MAIN; + if (def.pin_to_predecessor) want |= F_PIN_TO_PRED; + if (views.flags[idx] !== want) + errors.push(`flags "${name}": got ${views.flags[idx]}, want ${want}`); + } + + // Verify seed status + for (const [name, def] of Object.entries(taskDefs)) { + const idx = nameToIdx.get(name); + const isSeed = def.expected.length === 0 && !def.on_demand; + const status = views.status[idx]; + if (isSeed && status !== READY) + errors.push(`seed "${name}" should be READY, got ${status}`); + if (!isSeed && status !== NOT_READY) + errors.push(`non-seed "${name}" should be NOT_READY, got ${status}`); + } + + if (errors.length > 0) + throw new Error("SAB verification failed:\n " + errors.join("\n ")); +} diff --git a/builder/scheduler.mjs b/builder/scheduler.mjs index d7fd0732..cbb0845a 100644 --- a/builder/scheduler.mjs +++ b/builder/scheduler.mjs @@ -58,7 +58,7 @@ export class Scheduler { for (const [id, def] of this.tasks) { if (def.expected.length === 0) { this.pending.delete(id); // seeds have no inputs; remove from pending before dispatching - this.ready.push({ id, def, inputs: {} }); + if (!def.on_demand) this.ready.push({ id, def, inputs: {} }); } } this._flush(); diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 3b339cd7..e1557d0b 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -44,6 +44,7 @@ import { writeOffline, enumerateVendoredThemeAssets } from "./offline.mjs"; import { buildSitePathsSync } from "./offline-rewrite.mjs"; import { writePdf } from "./pdf.mjs"; import { packShared } from "./sab-broadcast.mjs"; +import { allocSchedulerSAB, verifySchedulerSAB, SLICES_PER_WORKER } from "./sab-scheduler.mjs"; const CPU_WORKER_URL = new URL("./cpu-worker.mjs", import.meta.url); @@ -249,6 +250,16 @@ const TASKS = { }, }, + // On-demand per-worker Shiki initializer. Not dispatched by the push + // scheduler (on_demand flag); becomes active in the SAB pull model. + warmInit: { + expected: [], + on_demand: true, + unique_per_worker: true, + handler: "warmInit", + submit() {}, + }, + // ── Main-thread spine ───────────────────────────────────────────────────── discover: { @@ -563,8 +574,6 @@ const TASKS = { }, }; -const SLICES_PER_WORKER = 10; - function chunkPages(pages, workers) { const n = Math.min(workers * SLICES_PER_WORKER, pages.length); if (n === 0) return []; @@ -632,6 +641,11 @@ export async function runBuild(opts) { const scheduler = new Scheduler({ pool, tasks: TASKS }); const ctx = { srcRoot, destRoot, opts, workerCount }; + // Phase 5 verification: allocate a SAB from the task graph and assert + // that dep counts, successor edges, flags, and seed status are correct. + const { views: sabViews, idMapping: sabMapping } = allocSchedulerSAB(TASKS, workerCount); + verifySchedulerSAB(TASKS, sabViews, sabMapping); + let results; try { results = await scheduler.start(ctx); From 0c7a9f7569180c595bab9f4e3029747bfd9edfde Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Mon, 1 Jun 2026 16:34:33 +0200 Subject: [PATCH 46/72] Implement SAB worker pull loop (Phase 6). --- builder/cpu-worker.mjs | 222 +++++++++++++++++++++++++++++--------- builder/sab-scheduler.mjs | 109 ++++++++++++++++++- builder/scheduler.mjs | 175 ++++++++++++++++++++++++------ builder/tbdocs.mjs | 47 ++++---- builder/worker-pool.mjs | 57 +++++++--- 5 files changed, 494 insertions(+), 116 deletions(-) diff --git a/builder/cpu-worker.mjs b/builder/cpu-worker.mjs index d087de3c..2dfe70f4 100644 --- a/builder/cpu-worker.mjs +++ b/builder/cpu-worker.mjs @@ -1,6 +1,8 @@ -// Worker harness for the tbdocs build pipeline. Routes named tasks to the -// appropriate handler and posts back { result } or { error, stack }. -// See PLAN-scheduler.md §Worker for the full handler set. +// Worker harness for the tbdocs build pipeline. Phase 6: persistent pull +// loop over the scheduling SAB. Workers claim tasks via Atomics, execute +// the named handler, post the output, and update successor dep counts --- +// no main-thread round-trip for worker-to-worker transitions. +// See PLAN-sab-pull-scheduler.md §Worker pull loop. import { parentPort, workerData } from "node:worker_threads"; import { compileLightScss, compileDarkScss } from "./scss.mjs"; @@ -14,41 +16,48 @@ import { deriveOfflinePage, deriveOfflinePageCached, sliceNavBlock, normalizeBaseurl, posixDirname } from "./offline-rewrite.mjs"; -// Report cold-boot time: from worker spawn (passed via workerData) to -// the point where all static imports have resolved and top-level code runs. +import { + createViews, scanAndClaim, onTaskDone, + READY, F_ON_DEMAND, F_RUN_ON_MAIN, + MAX_LANES, +} from "./sab-scheduler.mjs"; + if (workerData?.spawnTime) parentPort.postMessage({ coldBoot: { start: workerData.spawnTime, end: Date.now() } }); -// Shiki (highlight.mjs) is loaded lazily — its transitive import of the -// shiki package is the heaviest single module in the worker graph. A -// warmup signal from the pool triggers loading on idle workers so it -// overlaps with the main-thread discover phase. buildInfo and mermaid -// skip the post-handler init so Shiki never contends with them. -// Once init completes, a { warmedUp: true } signal tells the pool this -// worker is warm so it can be preferred for render dispatch. -let _highlighterP = null; -function ensureHighlighterInit() { - if (!_highlighterP) { - const warmStart = Date.now(); - _highlighterP = import("./highlight.mjs").then(m => m.initHighlighter()); - _highlighterP.then(() => parentPort.postMessage({ warmedUp: true, warmBoot: { start: warmStart, end: Date.now() } })); - } - return _highlighterP; -} +const myLane = workerData?.lane ?? 0; + +// ── Mutable state set by init / renderData messages ───────────────────────── + +let views = null; // Int32Array views into the scheduling SAB +let taskMeta = null; // per-index { handler, perWorkerDeps, name } +let ctx = null; // { srcRoot, destRoot, opts, workerCount } +let idMapping = null; // { nameToIdx, idxToName, DYNAMIC_BASE, … } + +let _chunkDataSAB = null; // SharedArrayBuffer with packed render chunks +let _sharedSAB = null; // SharedArrayBuffer with packed shared payload + +// ── Handler table ─────────────────────────────────────────────────────────── const handlers = { - async scssLight({ ctx }) { + async warmInit() { + const { initHighlighter } = await import("./highlight.mjs"); + await initHighlighter(); + return {}; + }, + + async scssLight() { const workerStart = Date.now(); const scssLightResult = await compileLightScss(ctx.srcRoot); return { workerStart, workerEnd: Date.now(), scssLightResult }; }, - async scssDark({ ctx }) { + async scssDark() { const workerStart = Date.now(); const scssDarkResult = await compileDarkScss(ctx.srcRoot); return { workerStart, workerEnd: Date.now(), scssDarkResult }; }, - async mermaid({ ctx }) { + async mermaid() { const workerStart = Date.now(); const mermaidStats = await regenerateMermaid(ctx.srcRoot); return { workerStart, workerEnd: Date.now(), mermaidStats }; @@ -60,10 +69,24 @@ const handlers = { return { workerStart, workerEnd: Date.now(), buildInfo }; }, - async render({ inputs }) { + async render(taskIdx) { const workerStart = Date.now(); - const { sharedSAB, chunk } = inputs; - const env = await getOrInitRenderEnv(sharedSAB); + + // The renderData broadcast may not yet have been processed (it was + // posted before the SAB set these tasks READY, but message delivery + // is async). Yield until the data arrives. + while (!_chunkDataSAB || !_sharedSAB) { + await new Promise(resolve => setImmediate(resolve)); + } + + const chunkIndex = taskIdx - idMapping.DYNAMIC_BASE; + const offset = Atomics.load(views.chunkOffset, chunkIndex); + const length = Atomics.load(views.chunkLength, chunkIndex); + const chunk = JSON.parse( + new TextDecoder().decode(new Uint8Array(_chunkDataSAB, offset, length)), + ); + + const env = await getOrInitRenderEnv(_sharedSAB); await renderPhase(chunk, env.site); await templatePhase(chunk, env.site, env.initData); @@ -114,8 +137,8 @@ const handlers = { }, }; -// Cached per-worker render environment. The sharedSAB is identical for every -// chunk in a build, so we unpack + derive once and reuse across chunks. +// ── Cached per-worker render environment ──────────────────────────────────── + let _renderSAB = null; let _renderEnv = null; @@ -126,7 +149,8 @@ async function getOrInitRenderEnv(sharedSAB) { baseurl, buildInfo, sitePathsArr, skipOffline } = unpackShared(sharedSAB); - const highlighter = await ensureHighlighterInit(); + const { initHighlighter } = await import("./highlight.mjs"); + const highlighter = await initHighlighter(); const linkTables = reconstructLinkTables(linkTablesData); const staticFiles = new Set(staticFilesArr); const markdown = createMarkdownIt({ highlighter, linkTables, baseurl, staticFiles }); @@ -145,30 +169,132 @@ async function getOrInitRenderEnv(sharedSAB) { return _renderEnv; } -parentPort.on("message", async (msg) => { - // Warmup signal from the pool — start Shiki init without posting a - // response so the pool's busy-tracking is unaffected. - if (msg.warmup) { ensureHighlighterInit(); return; } +// ── Message handler (init + renderData only) ──────────────────────────────── - const { name, deferHighlighter, ...payload } = msg; - const handler = handlers[name]; - if (!handler) { - parentPort.postMessage({ error: `unknown task: ${name}` }); +parentPort.on("message", (msg) => { + if (msg.init) { + views = createViews(msg.sab); + taskMeta = msg.taskMeta; + ctx = msg.ctx; + idMapping = msg.idMapping; + _chunkDataSAB = null; + _sharedSAB = null; + _renderSAB = null; + _renderEnv = null; + pullLoop(); return; } - try { - const result = await handler(payload); - parentPort.postMessage({ result }); - } catch (err) { - parentPort.postMessage({ error: err.message, stack: err.stack }); + if (msg.renderData) { + _chunkDataSAB = msg.chunkDataSAB; + _sharedSAB = msg.sharedSAB; + return; } - if (!deferHighlighter) ensureHighlighterInit(); }); -// linkTables values are page objects in the main pipeline, but -// resolveLink() in the relative-links plugin only reads .permalink. -// The serialized form ships [key, permalink] pairs; we reconstruct -// minimal { permalink } stubs in the worker. +// ── Pull loop ─────────────────────────────────────────────────────────────── + +async function pullLoop() { + while (true) { + if (Atomics.load(views.buildDone, 0) !== 0) return; + + let taskIdx = scanAndClaim(views, myLane); + + if (taskIdx === -1) { + const gen = Atomics.load(views.notify, 0); + // Double-check after reading gen (race: a task may have become + // READY between the failed scan and this load). + taskIdx = scanAndClaim(views, myLane); + if (taskIdx === -1) { + Atomics.wait(views.notify, 0, gen, 50); + continue; + } + } + + // ── Per-worker deps (unique_per_worker) ── + const meta = taskMeta[taskIdx]; + let unsatisfied = null; + if (meta?.perWorkerDeps) { + for (const depIdx of meta.perWorkerDeps) { + if (Atomics.load(views.perWorkerDone, depIdx * MAX_LANES + myLane) === 0) { + unsatisfied = depIdx; + break; + } + } + } + + if (unsatisfied !== null) { + const depFlags = Atomics.load(views.flags, unsatisfied); + + if ((depFlags & F_ON_DEMAND) && !(depFlags & F_RUN_ON_MAIN)) { + // On-demand worker dep (warmInit). Release the claimed task so + // other workers can pick it up, then execute the dep inline. + Atomics.store(views.status, taskIdx, READY); + Atomics.add(views.notify, 0, 1); + Atomics.notify(views.notify, 0, 1); + + const depMeta = taskMeta[unsatisfied]; + const depStart = Date.now(); + try { + await handlers[depMeta.handler](); + } catch (err) { + parentPort.postMessage({ taskFailed: unsatisfied, message: err.message, stack: err.stack }); + return; + } + Atomics.store(views.perWorkerDone, unsatisfied * MAX_LANES + myLane, 1); + + parentPort.postMessage({ + warmInit: true, + timing: { start: depStart, end: Date.now() }, + lane: myLane, + }); + continue; + } + + // Other unsatisfied dep types: release and re-scan. + Atomics.store(views.status, taskIdx, READY); + Atomics.add(views.notify, 0, 1); + Atomics.notify(views.notify, 0, 1); + continue; + } + + // ── Execute task ── + const handler = handlers[meta.handler]; + if (!handler) { + parentPort.postMessage({ taskFailed: taskIdx, message: `unknown handler: ${meta.handler}`, stack: "" }); + return; + } + + const start = Date.now(); + let result; + try { + result = await handler(taskIdx); + } catch (err) { + parentPort.postMessage({ taskFailed: taskIdx, message: err.message, stack: err.stack }); + Atomics.store(views.status, taskIdx, 4); // FAILED + return; + } + + // Post output BEFORE the SAB update (ordering constraint: the merge + // message must arrive on the main thread before any downstream + // main-thread task could be claimed). + parentPort.postMessage({ + done: taskIdx, + output: result, + timing: { start, end: Date.now() }, + lane: myLane, + }); + + const { readyCount, wakeMain } = onTaskDone(views, taskIdx, myLane); + if (readyCount > 0) { + Atomics.add(views.notify, 0, 1); + Atomics.notify(views.notify, 0, readyCount); + } + if (wakeMain) { + parentPort.postMessage({ mainTaskReady: true }); + } + } +} + function reconstructLinkTables({ byPath, byUrl, byRedirect }) { const make = (pairs) => new Map(pairs.map(([k, pl]) => [k, { permalink: pl }])); return { byPath: make(byPath), byUrl: make(byUrl), byRedirect: make(byRedirect) }; diff --git a/builder/sab-scheduler.mjs b/builder/sab-scheduler.mjs index ae8e1a88..2c5ac273 100644 --- a/builder/sab-scheduler.mjs +++ b/builder/sab-scheduler.mjs @@ -1,6 +1,6 @@ -// SAB-based scheduling data structures. Phase 5: layout, allocation, and -// verification only; the push scheduler still runs the build. -// See PLAN-sab-pull-scheduler.md for the full design. +// SAB-based scheduling data structures and worker-pull primitives. +// Phase 6: workers pull tasks via atomics; main-thread tasks still use the +// push scheduler, bridged into the SAB. See PLAN-sab-pull-scheduler.md. // ── Constants ──────────────────────────────────────────────────────────────── @@ -15,6 +15,7 @@ export const NOT_READY = 0; export const READY = 1; export const CLAIMED = 2; export const DONE = 3; +export const FAILED = 4; // Flag bits export const F_ON_DEMAND = 1; @@ -212,6 +213,108 @@ export function allocSchedulerSAB(taskDefs, workerCount) { return { sab, views, idMapping, taskMeta }; } +// ── Worker pull-loop primitives ────────────────────────────────────────────── +// +// Used by cpu-worker.mjs (worker thread) and by the main-thread bridge in +// scheduler.mjs. All operate directly on the SAB via Atomics so they are +// thread-safe without locks. + +export function scanAndClaim(views, myLane) { + const start = Atomics.load(views.firstReady, 0); + const count = Atomics.load(views.taskCount, 0); + for (let i = start; i < count; i++) { + if (Atomics.load(views.status, i) !== READY) continue; + if (Atomics.load(views.flags, i) & F_RUN_ON_MAIN) continue; + const aff = Atomics.load(views.affinityLane, i); + if (aff !== -1 && aff !== myLane) continue; + if (Atomics.compareExchange(views.status, i, READY, CLAIMED) === READY) + return i; + } + return -1; +} + +export function onTaskDone(views, taskIdx, lane) { + Atomics.store(views.status, taskIdx, DONE); + Atomics.store(views.completedOnLane, taskIdx, lane); + advanceFirstReady(views, taskIdx); + + let readyCount = 0; + let wakeMain = false; + + const off = Atomics.load(views.succOffset, taskIdx); + const cnt = Atomics.load(views.succCount, taskIdx); + for (let i = off; i < off + cnt; i++) { + const succ = Atomics.load(views.succList, i); + if (Atomics.load(views.flags, succ) & F_UNIQUE_PER_WORKER) continue; + + const remaining = Atomics.sub(views.depCount, succ, 1) - 1; + if (remaining === 0) { + const pin = Atomics.load(views.pinnedTo, succ); + if (pin !== -1) { + const srcLane = Atomics.load(views.completedOnLane, pin); + Atomics.store(views.affinityLane, succ, srcLane); + } + Atomics.store(views.status, succ, READY); + + if (Atomics.load(views.flags, succ) & F_RUN_ON_MAIN) wakeMain = true; + else readyCount++; + } + } + + return { readyCount, wakeMain }; +} + +export function advanceFirstReady(views, taskIdx) { + const count = Atomics.load(views.taskCount, 0); + const cur = Atomics.load(views.firstReady, 0); + if (taskIdx !== cur) return; + let next = cur; + while (next < count && Atomics.load(views.status, next) === DONE) next++; + if (next > cur) Atomics.compareExchange(views.firstReady, 0, cur, next); +} + +// ── Dynamic task registration (called by dispatch on the main thread) ─────── + +const encoder = new TextEncoder(); + +export function registerDynamicRender(views, idMapping, numChunks) { + let edgePos = Atomics.load(views.edgeCount, 0); + for (let i = 0; i < numChunks; i++) { + const idx = idMapping.DYNAMIC_BASE + i; + views.succOffset[idx] = edgePos; + views.succCount[idx] = 1; + if (edgePos >= MAX_EDGES) + throw new Error(`Edge count exceeds MAX_EDGES (${MAX_EDGES})`); + views.succList[edgePos++] = idMapping.RENDER_JOIN_IDX; + } + Atomics.store(views.edgeCount, 0, edgePos); + Atomics.store(views.depCount, idMapping.RENDER_JOIN_IDX, numChunks); + views.flags[idMapping.RENDER_JOIN_IDX] = F_RUN_ON_MAIN; +} + +export function packChunkData(chunks, views) { + const buffers = chunks.map(c => encoder.encode(JSON.stringify(c))); + const totalBytes = buffers.reduce((sum, b) => sum + b.byteLength, 0); + const sab = new SharedArrayBuffer(totalBytes); + const full = new Uint8Array(sab); + let offset = 0; + for (let i = 0; i < buffers.length; i++) { + full.set(buffers[i], offset); + Atomics.store(views.chunkOffset, i, offset); + Atomics.store(views.chunkLength, i, buffers[i].byteLength); + offset += buffers[i].byteLength; + } + return sab; +} + +export function activateRenderTasks(views, idMapping, numChunks) { + for (let i = 0; i < numChunks; i++) { + Atomics.store(views.status, idMapping.DYNAMIC_BASE + i, READY); + } + Atomics.add(views.notify, 0, 1); + Atomics.notify(views.notify, 0, Infinity); +} + // ── Verification ───────────────────────────────────────────────────────────── export function verifySchedulerSAB(taskDefs, views, idMapping) { diff --git a/builder/scheduler.mjs b/builder/scheduler.mjs index cbb0845a..cdffe4b2 100644 --- a/builder/scheduler.mjs +++ b/builder/scheduler.mjs @@ -1,40 +1,77 @@ -// Task-graph scheduler for the tbdocs build pipeline. See PLAN-scheduler.md -// for the full design, data-flow diagram, and task placement rationale. +// Task-graph scheduler for the tbdocs build pipeline. Phase 6: main-thread +// tasks still use the push-based pending/ready/flush mechanism; worker tasks +// are pulled from the scheduling SAB. A bridge in _onDone() updates the SAB +// after each main-thread task so downstream worker tasks become READY +// immediately. See PLAN-sab-pull-scheduler.md §Phase 6. import pc from "picocolors"; +import { + onTaskDone as sabOnTaskDone, + registerDynamicRender, packChunkData, activateRenderTasks, +} from "./sab-scheduler.mjs"; export class SharedState { - pages = []; // master copy; mutated in place by [M] tasks and render delta merges - staticFiles = []; // master copy; mermaid.submit appends new SVG descriptors - site = {}; // config, navTree, seoSiteTitle, seoLogoUrl, bookData, data, markdown, … - pageByDest = new Map(); // destPath → page; built once in discover.submit + pages = []; + staticFiles = []; + site = {}; + pageByDest = new Map(); } export class Scheduler { - constructor({ pool, tasks }) { - this.pool = pool; - this.tasks = new Map(Object.entries(tasks)); - this.pending = new Map(); - this.ready = []; - this.results = new Map(); - this.timings = new Map(); - this.state = new SharedState(); - this.inFlight = 0; + constructor({ pool, tasks, views, idMapping }) { + this.pool = pool; + this.tasks = new Map(Object.entries(tasks)); + this.pending = new Map(); // only main-thread tasks + this.ready = []; + this.results = new Map(); + this.timings = new Map(); + this.state = new SharedState(); + this.inFlight = 0; // main-thread tasks currently executing + this._views = views; + this._idMapping = idMapping; + + // Count static worker tasks (SAB-tracked, not on_demand). + this._workerRemaining = 0; + for (const [, def] of this.tasks) { + if (!def.runOnMain && !def.on_demand) this._workerRemaining++; + } + [this._doneP, this._doneResolve, this._doneReject] = deferred(); - for (const [id, def] of this.tasks) this._initPending(id, def); + + // Only main-thread tasks participate in the push scheduler's + // pending/ready/flush mechanism. + for (const [id, def] of this.tasks) { + if (def.runOnMain) this._initPending(id, def); + } } _initPending(id, def) { this.pending.set(id, { expected: def.expected.length, received: new Map() }); } + // Register a new main-thread task (used by dispatch.submit for renderJoin). register(id, def) { this.tasks.set(id, def); - this._initPending(id, def); + if (def.runOnMain) this._initPending(id, def); + } + + // Register a worker task definition (for submit() lookup) without adding + // it to the push scheduler's pending map. Increments _workerRemaining. + registerWorkerTask(id, def) { + this.tasks.set(id, def); + this._workerRemaining++; } - // Seed a freshly-registered task directly (used by dispatch.submit to feed - // each render:i its chunk without going through emit()). + // Write render SAB entries, broadcast chunk data, and activate tasks. + dispatchRender(chunks, sharedSAB) { + const N = chunks.length; + registerDynamicRender(this._views, this._idMapping, N); + const chunkDataSAB = packChunkData(chunks, this._views); + this.pool.broadcastRenderData(chunkDataSAB, sharedSAB); + activateRenderTasks(this._views, this._idMapping, N); + } + + // Seed a freshly-registered main-thread task directly. seed(id, inputs) { this.pending.delete(id); this.ready.push({ id, def: this.tasks.get(id), inputs }); @@ -43,7 +80,11 @@ export class Scheduler { emit(targetId, data, sourceId) { const entry = this.pending.get(targetId); - if (!entry) throw new Error(`unknown or already-dispatched task: ${targetId}`); + if (!entry) { + // Worker task or already-dispatched — SAB handles readiness. + if (this.tasks.has(targetId)) return; + throw new Error(`unknown or already-dispatched task: ${targetId}`); + } entry.received.set(sourceId, data); if (entry.received.size === entry.expected) { this.pending.delete(targetId); @@ -56,13 +97,14 @@ export class Scheduler { async start(ctx) { this._ctx = ctx; for (const [id, def] of this.tasks) { - if (def.expected.length === 0) { - this.pending.delete(id); // seeds have no inputs; remove from pending before dispatching - if (!def.on_demand) this.ready.push({ id, def, inputs: {} }); + if (def.expected.length === 0 && !def.on_demand) { + this.pending.delete(id); + // Only seed main-thread tasks; worker seeds are already READY in + // the SAB (set by allocSchedulerSAB). + if (def.runOnMain) this.ready.push({ id, def, inputs: {} }); } } this._flush(); - this.pool.warmup(); return this._doneP; } @@ -73,11 +115,9 @@ export class Scheduler { _run(task) { const start = Date.now(); this.inFlight++; - const p = task.def.runOnMain - ? Promise.resolve(task.def.execute(task.inputs, this._ctx, this.state)) - : this.pool.run({ inputs: task.inputs, ctx: this._ctx, - deferHighlighter: task.def.deferHighlighter }, - { name: task.def.handler ?? task.id }); + // Phase 6: only main-thread tasks reach _run(). Worker tasks are + // SAB-pulled and never enter the ready queue. + const p = Promise.resolve(task.def.execute(task.inputs, this._ctx, this.state)); p.then( (output) => this._onDone(task, output, start), (err) => this._onError(task, err), @@ -94,19 +134,92 @@ export class Scheduler { this.timings.set(task.id, timing); this.results.set(task.id, output); this.inFlight--; - // submit() must be synchronous; async work belongs in execute(). task.def.submit( output, (tgt, data) => this.emit(tgt, data, task.id), this.state, this, ); - if (this.inFlight === 0 && this.ready.length === 0 && this.pending.size === 0) { + + // Bridge: update the SAB so downstream worker tasks become READY + // without waiting for the push scheduler's _flush(). + const taskIdx = this._idMapping.nameToIdx.get(task.id); + if (taskIdx != null) { + const { readyCount } = sabOnTaskDone(this._views, taskIdx, -1); + if (readyCount > 0) { + Atomics.add(this._views.notify, 0, 1); + Atomics.notify(this._views.notify, 0, readyCount); + } + } + + this._checkDone(); + } + + // Called by the pool's message handler when a worker posts { done }. + _onWorkerDone({ done: taskIdx, output, timing, lane }) { + const name = this._idMapping.idxToName[taskIdx]; + const def = this.tasks.get(name); + if (!def) return; + + const t = { start: timing.start, end: timing.end }; + if (output?.workerStart != null) { t.workerStart = output.workerStart; t.workerEnd = output.workerEnd; } + if (lane != null) t.lane = lane; + if (def.consolidate) t.consolidate = true; + if (def.ganttSection) t.ganttSection = def.ganttSection; + this.timings.set(name, t); + this.results.set(name, output); + + def.submit( + output, + (tgt, data) => this.emit(tgt, data, name), + this.state, + this, + ); + + this._workerRemaining--; + this._checkDone(); + } + + _onWorkerError({ taskFailed: taskIdx, message, stack }) { + const name = this._idMapping.idxToName[taskIdx] ?? `task#${taskIdx}`; + const err = Object.assign(new Error(message), { stack }); + + // Signal workers to stop. + Atomics.store(this._views.buildDone, 0, 2); + Atomics.add(this._views.notify, 0, 1); + Atomics.notify(this._views.notify, 0, Infinity); + + this._doneReject(new Error(`task ${name} failed`, { cause: err })); + } + + _onWarmInitTiming({ timing, lane }) { + this.timings.set(`warmInit:w${lane}`, { + start: timing.start, end: timing.end, + workerStart: timing.start, workerEnd: timing.end, + lane, + consolidate: true, + ganttSection: "Boot", + }); + } + + _checkDone() { + if (this.inFlight === 0 && this._workerRemaining === 0 + && this.ready.length === 0 && this.pending.size === 0) { + // Signal workers to exit. + Atomics.store(this._views.buildDone, 0, 1); + Atomics.add(this._views.notify, 0, 1); + Atomics.notify(this._views.notify, 0, Infinity); + this._doneResolve(this.results); } } _onError(task, err) { + // Signal workers to stop. + Atomics.store(this._views.buildDone, 0, 2); + Atomics.add(this._views.notify, 0, 1); + Atomics.notify(this._views.notify, 0, Infinity); + this._doneReject(new Error(`task ${task.id} failed`, { cause: err })); } diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index e1557d0b..fd1dffb4 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -1,5 +1,4 @@ -// tbdocs orchestrator. Phases 1+2+3+4+5+6+7+8: DISCOVER + COMPUTE + -// RENDER + TEMPLATE + WRITE ONLINE + AUXILIARIES + WRITE OFFLINE + WRITE PDF. +// tbdocs orchestrator. Phases 1-4 pipeline + Phase 5+6 SAB scheduler. // // Usage: node builder/tbdocs.mjs [--src ] [--dest ] // [--baseurl ] [--url ] [--dry-run] @@ -157,8 +156,7 @@ const TASKS = { // discover on the CI runners. When workerCount > 4, mermaid is a seed. buildInfo: { expected: [], - deferHighlighter: true, - // execute() runs in cpu-worker.mjs as the "buildInfo" handler. + handler: "buildInfo", submit(out, emit, state) { state.site.buildInfo = out.buildInfo; emit("dispatch", out); @@ -170,13 +168,13 @@ const TASKS = { // Each half is ~700 ms total serially; running concurrently saves ~200 ms. scssLight: { expected: [], - // execute() runs in cpu-worker.mjs as the "scssLight" handler. + handler: "scssLight", submit(out, emit) { emit("scssJoin", out); }, }, scssDark: { expected: [], - // execute() runs in cpu-worker.mjs as the "scssDark" handler. + handler: "scssDark", submit(out, emit) { emit("scssJoin", out); }, }, @@ -198,8 +196,7 @@ const TASKS = { // doesn't compete with discover on 4-thread CI. mermaid: { expected: mermaidIsSeed ? [] : ["buildInfo"], - deferHighlighter: true, - // execute() runs in cpu-worker.mjs as the "mermaid" handler. + handler: "mermaid", submit(out, emit, state) { // Append any freshly-generated SVG descriptors that discover didn't see // (because mermaid and discover run concurrently). Dedup by srcRel so @@ -436,6 +433,7 @@ const TASKS = { emit("deriveSitemap", {}); emit("prepDest", {}); + // renderJoin is a main-thread barrier — tracked by the push scheduler. scheduler.register("renderJoin", { expected: Array.from({ length: N }, (_, i) => `render:${i}`), runOnMain: true, @@ -447,9 +445,12 @@ const TASKS = { }, }); + // render:i tasks are worker-pulled from the SAB. Register their + // submit() with the scheduler (so merge + emit fires when the main + // thread processes the output message) but don't seed them through + // the push scheduler. for (let i = 0; i < N; i++) { - const id = `render:${i}`; - scheduler.register(id, { + scheduler.registerWorkerTask(`render:${i}`, { expected: [], handler: "render", consolidate: true, @@ -466,11 +467,10 @@ const TASKS = { emit("renderJoin", {}); }, }); - scheduler.seed(id, { - sharedSAB: out.sharedSAB, - chunk: out.chunks[i], - }); } + + // Write SAB entries, broadcast chunk data, set render tasks READY. + scheduler.dispatchRender(out.chunks, out.sharedSAB); }, }, @@ -637,14 +637,21 @@ export async function runBuild(opts) { const srcRoot = path.resolve(process.cwd(), src); const destRoot = path.resolve(dest ?? path.join(srcRoot, "_site")); - const pool = new WorkerPool(workerCount, CPU_WORKER_URL); - const scheduler = new Scheduler({ pool, tasks: TASKS }); const ctx = { srcRoot, destRoot, opts, workerCount }; - // Phase 5 verification: allocate a SAB from the task graph and assert - // that dep counts, successor edges, flags, and seed status are correct. - const { views: sabViews, idMapping: sabMapping } = allocSchedulerSAB(TASKS, workerCount); - verifySchedulerSAB(TASKS, sabViews, sabMapping); + // Allocate the scheduling SAB before the pool so workers receive it at + // init and start pulling tasks immediately. + const { sab, views, idMapping, taskMeta } = allocSchedulerSAB(TASKS, workerCount); + verifySchedulerSAB(TASKS, views, idMapping); + + const pool = new WorkerPool(workerCount, CPU_WORKER_URL); + const scheduler = new Scheduler({ pool, tasks: TASKS, views, idMapping }); + + pool.onWorkerDone = (msg) => scheduler._onWorkerDone(msg); + pool.onWorkerError = (msg) => scheduler._onWorkerError(msg); + pool.onWarmInitTiming = (msg) => scheduler._onWarmInitTiming(msg); + + pool.sendInit(sab, taskMeta, ctx, idMapping); let results; try { diff --git a/builder/worker-pool.mjs b/builder/worker-pool.mjs index bd7a1e81..74f4a65e 100644 --- a/builder/worker-pool.mjs +++ b/builder/worker-pool.mjs @@ -1,31 +1,44 @@ -// Worker pool over node:worker_threads. Spawns `size` workers eagerly at -// construction, routes named tasks to whichever worker is idle, queues the -// rest. No dynamic scaling, no recycling, no abort signals. +// Worker pool over node:worker_threads. Phase 6: workers pull tasks from a +// scheduling SAB; the pool is a lifecycle manager that spawns workers, sends +// them the SAB + metadata, forwards output/error messages to the scheduler, +// and terminates workers on destroy. // -// Idle workers are split into two tiers: _idleWarm (Shiki-initialized) and -// _idleCold. _drain() pulls from warm first so render chunks land on -// workers that can start immediately; cold workers only get work when no -// warm ones are available. +// Legacy push-model fields (_idleWarm, _idleCold, _warm, _busy, _queue, +// run, warmup, _drain, _pushIdle, _onWarmedUp) are retained as dead code; +// Phase 8 removes them. import { Worker } from "node:worker_threads"; export class WorkerPool { constructor(size, workerUrl) { this._workerUrl = workerUrl; - this._idleWarm = []; // Worker[] — Shiki ready - this._idleCold = []; // Worker[] — not yet initialized - this._warm = new Set(); // all workers that have signalled warmedUp - this._busy = new Map(); // Worker → { resolve, reject } - this._queue = []; // pending { message, transferList, resolve, reject } - this.bootTimings = []; // { lane, type, start, end }[] - this._workers = Array.from({ length: size }, (_, i) => this._spawn(i)); + this._idleWarm = []; // (Phase 8 removes) + this._idleCold = []; // (Phase 8 removes) + this._warm = new Set(); // (Phase 8 removes) + this._busy = new Map(); // (Phase 8 removes) + this._queue = []; // (Phase 8 removes) + this.bootTimings = []; + + // Callbacks wired by the caller after construction. + this.onWorkerDone = null; // ({ done, output, timing, lane }) => void + this.onWorkerError = null; // ({ taskFailed, message, stack }) => void + this.onWarmInitTiming = null; // ({ warmInit, timing, lane }) => void + + this._workers = Array.from({ length: size }, (_, i) => this._spawn(i)); } _spawn(lane) { const spawnTime = Date.now(); const w = new Worker(this._workerUrl, { workerData: { lane, spawnTime } }); w.on("message", (msg) => { + // ── Phase 6 SAB-based message routing ── if (msg.coldBoot) { this.bootTimings.push({ lane, type: "cold", ...msg.coldBoot }); return; } + if (msg.warmInit) { this.onWarmInitTiming?.(msg); return; } + if (msg.done != null) { this.onWorkerDone?.(msg); return; } + if (msg.taskFailed != null) { this.onWorkerError?.(msg); return; } + if (msg.mainTaskReady) { /* Phase 7 wires this up */ return; } + + // ── Legacy push-model routing (Phase 8 removes) ── if (msg.warmedUp) { if (msg.warmBoot) this.bootTimings.push({ lane, type: "warm", ...msg.warmBoot }); this._onWarmedUp(w); @@ -47,6 +60,22 @@ export class WorkerPool { return w; } + // ── Phase 6: SAB init + broadcast ────────────────────────────────────────── + + sendInit(sab, taskMeta, ctx, idMapping) { + for (const w of this._workers) { + w.postMessage({ init: true, sab, taskMeta, ctx, idMapping }); + } + } + + broadcastRenderData(chunkDataSAB, sharedSAB) { + for (const w of this._workers) { + w.postMessage({ renderData: true, chunkDataSAB, sharedSAB }); + } + } + + // ── Legacy push-model methods (Phase 8 removes) ─────────────────────────── + _pushIdle(w) { if (this._warm.has(w)) this._idleWarm.push(w); else this._idleCold.push(w); From 70ea89c9315a6519eab4e89f2f8f0b9d811a4333 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Mon, 1 Jun 2026 16:55:26 +0200 Subject: [PATCH 47/72] Integrate main-thread tasks into the SAB scheduler (Phase 7). --- builder/cpu-worker.mjs | 2 +- builder/sab-scheduler.mjs | 4 +- builder/scheduler.mjs | 282 ++++++++++++++++++++------------------ builder/tbdocs.mjs | 100 +++++--------- builder/worker-pool.mjs | 9 +- 5 files changed, 190 insertions(+), 207 deletions(-) diff --git a/builder/cpu-worker.mjs b/builder/cpu-worker.mjs index 2dfe70f4..f6c4965d 100644 --- a/builder/cpu-worker.mjs +++ b/builder/cpu-worker.mjs @@ -1,4 +1,4 @@ -// Worker harness for the tbdocs build pipeline. Phase 6: persistent pull +// Worker harness for the tbdocs build pipeline. Phase 6+7: persistent pull // loop over the scheduling SAB. Workers claim tasks via Atomics, execute // the named handler, post the output, and update successor dep counts --- // no main-thread round-trip for worker-to-worker transitions. diff --git a/builder/sab-scheduler.mjs b/builder/sab-scheduler.mjs index 2c5ac273..feeb91da 100644 --- a/builder/sab-scheduler.mjs +++ b/builder/sab-scheduler.mjs @@ -1,6 +1,6 @@ // SAB-based scheduling data structures and worker-pull primitives. -// Phase 6: workers pull tasks via atomics; main-thread tasks still use the -// push scheduler, bridged into the SAB. See PLAN-sab-pull-scheduler.md. +// Phase 7: workers pull tasks via atomics; main-thread tasks scan the SAB +// for READY work and claim via CAS. See PLAN-sab-pull-scheduler.md. // ── Constants ──────────────────────────────────────────────────────────────── diff --git a/builder/scheduler.mjs b/builder/scheduler.mjs index cdffe4b2..ea41f218 100644 --- a/builder/scheduler.mjs +++ b/builder/scheduler.mjs @@ -1,13 +1,14 @@ -// Task-graph scheduler for the tbdocs build pipeline. Phase 6: main-thread -// tasks still use the push-based pending/ready/flush mechanism; worker tasks -// are pulled from the scheduling SAB. A bridge in _onDone() updates the SAB -// after each main-thread task so downstream worker tasks become READY -// immediately. See PLAN-sab-pull-scheduler.md §Phase 6. +// Task-graph scheduler for the tbdocs build pipeline. Phase 7: main-thread +// tasks scan the scheduling SAB for READY work and claim via CAS; worker +// tasks are pulled from the SAB by workers (Phase 6). The push-based +// pending/ready/emit/flush mechanism is removed entirely. +// See PLAN-sab-pull-scheduler.md §Phase 7. import pc from "picocolors"; import { onTaskDone as sabOnTaskDone, registerDynamicRender, packChunkData, activateRenderTasks, + READY, CLAIMED, DONE, F_RUN_ON_MAIN, } from "./sab-scheduler.mjs"; export class SharedState { @@ -21,45 +22,24 @@ export class Scheduler { constructor({ pool, tasks, views, idMapping }) { this.pool = pool; this.tasks = new Map(Object.entries(tasks)); - this.pending = new Map(); // only main-thread tasks - this.ready = []; - this.results = new Map(); + this.results = new Map(); // task name → output this.timings = new Map(); this.state = new SharedState(); - this.inFlight = 0; // main-thread tasks currently executing this._views = views; this._idMapping = idMapping; - // Count static worker tasks (SAB-tracked, not on_demand). - this._workerRemaining = 0; + // Count non-on_demand static tasks for completion detection. + // Dynamic tasks (render:i, renderJoin) are added via addDynamicTasks(). + this._remaining = 0; for (const [, def] of this.tasks) { - if (!def.runOnMain && !def.on_demand) this._workerRemaining++; + if (!def.on_demand) this._remaining++; } - [this._doneP, this._doneResolve, this._doneReject] = deferred(); - - // Only main-thread tasks participate in the push scheduler's - // pending/ready/flush mechanism. - for (const [id, def] of this.tasks) { - if (def.runOnMain) this._initPending(id, def); - } - } + this._scanning = false; + this._mainScanScheduled = false; + this._finished = false; - _initPending(id, def) { - this.pending.set(id, { expected: def.expected.length, received: new Map() }); - } - - // Register a new main-thread task (used by dispatch.submit for renderJoin). - register(id, def) { - this.tasks.set(id, def); - if (def.runOnMain) this._initPending(id, def); - } - - // Register a worker task definition (for submit() lookup) without adding - // it to the push scheduler's pending map. Increments _workerRemaining. - registerWorkerTask(id, def) { - this.tasks.set(id, def); - this._workerRemaining++; + [this._doneP, this._doneResolve, this._doneReject] = deferred(); } // Write render SAB entries, broadcast chunk data, and activate tasks. @@ -71,125 +51,154 @@ export class Scheduler { activateRenderTasks(this._views, this._idMapping, N); } - // Seed a freshly-registered main-thread task directly. - seed(id, inputs) { - this.pending.delete(id); - this.ready.push({ id, def: this.tasks.get(id), inputs }); - this._flush(); - } - - emit(targetId, data, sourceId) { - const entry = this.pending.get(targetId); - if (!entry) { - // Worker task or already-dispatched — SAB handles readiness. - if (this.tasks.has(targetId)) return; - throw new Error(`unknown or already-dispatched task: ${targetId}`); - } - entry.received.set(sourceId, data); - if (entry.received.size === entry.expected) { - this.pending.delete(targetId); - const def = this.tasks.get(targetId); - this.ready.push({ id: targetId, def, inputs: Object.fromEntries(entry.received) }); - this._flush(); - } + // Increment remaining-task count for dynamically-registered tasks. + addDynamicTasks(count) { + this._remaining += count; } async start(ctx) { this._ctx = ctx; - for (const [id, def] of this.tasks) { - if (def.expected.length === 0 && !def.on_demand) { - this.pending.delete(id); - // Only seed main-thread tasks; worker seeds are already READY in - // the SAB (set by allocSchedulerSAB). - if (def.runOnMain) this.ready.push({ id, def, inputs: {} }); - } - } - this._flush(); + this._scheduleMainScan(); return this._doneP; } - _flush() { - while (this.ready.length > 0) this._run(this.ready.shift()); + // ── Main-thread SAB scan (replaces _flush / _run) ───────────────────────── + + _scheduleMainScan() { + if (this._mainScanScheduled) return; + this._mainScanScheduled = true; + // setImmediate lets pending worker messages drain before scanning. + setImmediate(() => { + this._mainScanScheduled = false; + this._mainScan(); + }); } - _run(task) { - const start = Date.now(); - this.inFlight++; - // Phase 6: only main-thread tasks reach _run(). Worker tasks are - // SAB-pulled and never enter the ready queue. - const p = Promise.resolve(task.def.execute(task.inputs, this._ctx, this.state)); - p.then( - (output) => this._onDone(task, output, start), - (err) => this._onError(task, err), - ); + async _mainScan() { + if (this._scanning || this._finished) return; + this._scanning = true; + try { + while (this._remaining > 0 && !this._finished) { + const claimed = this._claimMainTask(); + if (!claimed) break; + await this._executeMainTask(claimed); + } + } finally { + this._scanning = false; + } } - _onDone(task, output, start) { - const end = Date.now(); - const timing = { start, end }; + // Scan the SAB for a READY main-thread task whose predecessor outputs are + // all available in the results map. Returns { taskIdx, name, def, inputs } + // or null if nothing is claimable. + _claimMainTask() { + const views = this._views; + const start = Atomics.load(views.firstReady, 0); + const count = Atomics.load(views.taskCount, 0); + for (let i = start; i < count; i++) { + if (Atomics.load(views.status, i) !== READY) continue; + if (!(Atomics.load(views.flags, i) & F_RUN_ON_MAIN)) continue; + if (Atomics.compareExchange(views.status, i, READY, CLAIMED) !== READY) continue; + + const name = this._idMapping.idxToName[i]; + const def = this.tasks.get(name); + if (!def) { + Atomics.store(views.status, i, DONE); + continue; + } + + const inputs = this._assembleInputs(def); + if (inputs === null) { + // Predecessor output not yet received (message in flight); release. + Atomics.store(views.status, i, READY); + continue; + } + return { taskIdx: i, name, def, inputs }; + } + return null; + } + + async _executeMainTask({ taskIdx, name, def, inputs }) { + const views = this._views; + const t0 = Date.now(); + let output; + try { + output = await def.execute(inputs, this._ctx, this.state); + } catch (err) { + this._abort(name, err); + return; + } + if (this._finished) return; + const t1 = Date.now(); + + // Timing. + const timing = { start: t0, end: t1 }; if (output?.workerStart != null) { timing.workerStart = output.workerStart; timing.workerEnd = output.workerEnd; } if (output?.lane != null) timing.lane = output.lane; - if (task.def.consolidate) timing.consolidate = true; - if (task.def.ganttSection) timing.ganttSection = task.def.ganttSection; - this.timings.set(task.id, timing); - this.results.set(task.id, output); - this.inFlight--; - task.def.submit( - output, - (tgt, data) => this.emit(tgt, data, task.id), - this.state, - this, - ); - - // Bridge: update the SAB so downstream worker tasks become READY - // without waiting for the push scheduler's _flush(). - const taskIdx = this._idMapping.nameToIdx.get(task.id); - if (taskIdx != null) { - const { readyCount } = sabOnTaskDone(this._views, taskIdx, -1); - if (readyCount > 0) { - Atomics.add(this._views.notify, 0, 1); - Atomics.notify(this._views.notify, 0, readyCount); - } + if (def.consolidate) timing.consolidate = true; + if (def.ganttSection) timing.ganttSection = def.ganttSection; + this.timings.set(name, timing); + + // Store result. + this.results.set(name, output); + + // State mutation. + def.submit(output, this.state, this); + + // Update SAB: mark DONE, decrement successor dep counts. + const { readyCount } = sabOnTaskDone(views, taskIdx, -1); + if (readyCount > 0) { + Atomics.add(views.notify, 0, 1); + Atomics.notify(views.notify, 0, readyCount); } - this._checkDone(); + this._remaining--; + if (this._remaining === 0) this._finish(); } - // Called by the pool's message handler when a worker posts { done }. + _assembleInputs(def) { + const inputs = {}; + for (const predName of def.expected) { + if (!this.results.has(predName)) return null; + inputs[predName] = this.results.get(predName); + } + return inputs; + } + + // ── Worker output handling ──────────────────────────────────────────────── + _onWorkerDone({ done: taskIdx, output, timing, lane }) { const name = this._idMapping.idxToName[taskIdx]; const def = this.tasks.get(name); - if (!def) return; + // Timing. const t = { start: timing.start, end: timing.end }; if (output?.workerStart != null) { t.workerStart = output.workerStart; t.workerEnd = output.workerEnd; } if (lane != null) t.lane = lane; - if (def.consolidate) t.consolidate = true; - if (def.ganttSection) t.ganttSection = def.ganttSection; + if (def?.consolidate) t.consolidate = true; + if (def?.ganttSection) t.ganttSection = def.ganttSection; this.timings.set(name, t); + + // Store result. this.results.set(name, output); - def.submit( - output, - (tgt, data) => this.emit(tgt, data, name), - this.state, - this, - ); + // State mutation. + if (def) def.submit(output, this.state, this); + + this._remaining--; + if (this._remaining === 0) { + this._finish(); + return; + } - this._workerRemaining--; - this._checkDone(); + // A newly-stored result may satisfy a previously-blocked main task. + this._scheduleMainScan(); } _onWorkerError({ taskFailed: taskIdx, message, stack }) { const name = this._idMapping.idxToName[taskIdx] ?? `task#${taskIdx}`; const err = Object.assign(new Error(message), { stack }); - - // Signal workers to stop. - Atomics.store(this._views.buildDone, 0, 2); - Atomics.add(this._views.notify, 0, 1); - Atomics.notify(this._views.notify, 0, Infinity); - - this._doneReject(new Error(`task ${name} failed`, { cause: err })); + this._abort(name, err); } _onWarmInitTiming({ timing, lane }) { @@ -202,27 +211,32 @@ export class Scheduler { }); } - _checkDone() { - if (this.inFlight === 0 && this._workerRemaining === 0 - && this.ready.length === 0 && this.pending.size === 0) { - // Signal workers to exit. - Atomics.store(this._views.buildDone, 0, 1); - Atomics.add(this._views.notify, 0, 1); - Atomics.notify(this._views.notify, 0, Infinity); + _onMainTaskReady() { + this._scheduleMainScan(); + } + + // ── Completion / abort ──────────────────────────────────────────────────── - this._doneResolve(this.results); - } + _finish() { + if (this._finished) return; + this._finished = true; + Atomics.store(this._views.buildDone, 0, 1); + Atomics.add(this._views.notify, 0, 1); + Atomics.notify(this._views.notify, 0, Infinity); + this._doneResolve(this.results); } - _onError(task, err) { - // Signal workers to stop. + _abort(taskName, err) { + if (this._finished) return; + this._finished = true; Atomics.store(this._views.buildDone, 0, 2); Atomics.add(this._views.notify, 0, 1); Atomics.notify(this._views.notify, 0, Infinity); - - this._doneReject(new Error(`task ${task.id} failed`, { cause: err })); + this._doneReject(new Error(`task ${taskName} failed`, { cause: err })); } + // ── Summary ─────────────────────────────────────────────────────────────── + summary() { const sorted = [...this.timings.entries()] .sort((a, b) => a[1].start - b[1].start); diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index fd1dffb4..07b6c2f4 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -1,4 +1,4 @@ -// tbdocs orchestrator. Phases 1-4 pipeline + Phase 5+6 SAB scheduler. +// tbdocs orchestrator. Phases 1-4 pipeline + Phase 5-7 SAB scheduler. // // Usage: node builder/tbdocs.mjs [--src ] [--dest ] // [--baseurl ] [--url ] [--dry-run] @@ -148,7 +148,7 @@ const TASKS = { if (ctx.opts.url != null) config.url = ctx.opts.url; return { config }; }, - submit(out, emit) { emit("discover", out); emit("highlighterInit", out); }, + submit() {}, }, // Git rev-parse / log shell-outs. Worker so they overlap with the main spine. @@ -157,10 +157,8 @@ const TASKS = { buildInfo: { expected: [], handler: "buildInfo", - submit(out, emit, state) { + submit(out, state) { state.site.buildInfo = out.buildInfo; - emit("dispatch", out); - if (!mermaidIsSeed) emit("mermaid", {}); }, }, @@ -169,16 +167,16 @@ const TASKS = { scssLight: { expected: [], handler: "scssLight", - submit(out, emit) { emit("scssJoin", out); }, + submit() {}, }, scssDark: { expected: [], handler: "scssDark", - submit(out, emit) { emit("scssJoin", out); }, + submit() {}, }, - // Joins the two parallel SCSS results and emits to write. + // Joins the two parallel SCSS results. scssJoin: { expected: ["scssLight", "scssDark"], runOnMain: true, @@ -188,7 +186,7 @@ const TASKS = { } return { scssResult: { compiled: true, css: scssLightResult.css + "\n" + scssDarkResult.css } }; }, - submit(out, emit) { emit("write", out); }, + submit() {}, }, // Stale mermaid SVG regeneration. Seed when workerCount > 4 (enough cores @@ -197,17 +195,11 @@ const TASKS = { mermaid: { expected: mermaidIsSeed ? [] : ["buildInfo"], handler: "mermaid", - submit(out, emit, state) { - // Append any freshly-generated SVG descriptors that discover didn't see - // (because mermaid and discover run concurrently). Dedup by srcRel so - // SVGs already on disk at discover time aren't double-counted. + submit(out, state) { const known = new Set(state.staticFiles.map((f) => f.srcRel)); for (const f of out.mermaidStats.svgFiles ?? []) { if (!known.has(f.srcRel)) state.staticFiles.push(f); } - emit("write", out); - emit("writePdf", out); - emit("dispatch", out); }, }, @@ -222,10 +214,7 @@ const TASKS = { await prepareDestinations([r, r + "-offline", r + "-pdf"], ctx.opts.dryRun); return {}; }, - submit(_, emit) { - emit("write", {}); - emit("searchData", {}); - }, + submit() {}, }, // Theme CSS load. Reads the vendored .theme files and generates the @@ -240,15 +229,13 @@ const TASKS = { const theme = await loadHighlightTheme(); return { highlightCss: theme.css }; }, - submit(out, emit, state) { + submit(out, state) { state.site.highlightCss = out.highlightCss; - emit("write", out); - emit("loadData", out); }, }, - // On-demand per-worker Shiki initializer. Not dispatched by the push - // scheduler (on_demand flag); becomes active in the SAB pull model. + // On-demand per-worker Shiki initializer. Workers execute it the first + // time they claim a render chunk (per-worker dep in the SAB). warmInit: { expected: [], on_demand: true, @@ -266,15 +253,11 @@ const TASKS = { const { pages, staticFiles } = await discover(ctx.srcRoot, config.exclude ?? []); return { pages, staticFiles, config }; }, - submit(out, emit, state) { + submit(out, state) { state.pages = out.pages; state.staticFiles = out.staticFiles; state.site.config = out.config; for (const p of out.pages) state.pageByDest.set(p.destPath, p); - emit("nav", out); - emit("buildInit", out); - emit("markdownInit", out); - emit("deriveRedirects", out); }, }, @@ -286,7 +269,7 @@ const TASKS = { state.site.navTree = navTree; return { sidebar: renderSidebar(state.site) }; }, - submit(out, emit) { emit("dispatch", out); }, + submit() {}, }, // Pre-renders the config-only chrome (SVG sprites, header, search footer, @@ -299,7 +282,7 @@ const TASKS = { execute(_, ctx, state) { return { initData: buildInitConfig(state.site) }; }, - submit(out, emit) { emit("dispatch", out); }, + submit() {}, }, // Link-table build + markdown-it assembly. Only needs discover (pages + @@ -317,9 +300,7 @@ const TASKS = { state.site.linkTablesSerialized = serializeLinkTables(linkTables); return {}; }, - submit(_, emit) { - emit("seo", {}); - }, + submit() {}, }, seo: { @@ -332,7 +313,7 @@ const TASKS = { state.site.seoLogoUrl = seoLogoUrl; return {}; }, - submit(_, emit) { emit("dispatch", {}); }, + submit() {}, }, loadData: { @@ -344,7 +325,7 @@ const TASKS = { state.site.bookData = data.book ?? null; return {}; }, - submit(_, emit) { }, + submit() {}, }, // Mutates bookData._chapters with refs into state.pages. Identity-critical: @@ -358,7 +339,7 @@ const TASKS = { resolveBookChapters(state.site.bookData, state.pages); return {}; }, - submit(_, emit) { emit("writePdf", {}); }, + submit() {}, }, // Can run in parallel with nav/markdownInit -- only needs pages + config, @@ -370,7 +351,7 @@ const TASKS = { execute(_, ctx, state) { return { stubs: deriveRedirectStubs(state.pages, state.site) }; }, - submit(out, emit) { emit("writeAux", out); emit("dispatch", out); }, + submit() {}, }, // Deferred to after dispatch so it runs while the main thread is idle @@ -381,10 +362,7 @@ const TASKS = { execute(_, ctx, state) { return { urls: deriveSitemapUrls(state.pages, state.site) }; }, - submit(out, emit) { - emit("writeAux", out); - emit("resolveBookChapters", {}); - }, + submit() {}, }, // ── Render fan-out ───────────────────────────────────────────────────────── @@ -428,34 +406,26 @@ const TASKS = { const sharedSAB = packShared(shared); return { chunks, sharedSAB }; }, - submit(out, emit, _state, scheduler) { + submit(out, _state, scheduler) { const N = out.chunks.length; - emit("deriveSitemap", {}); - emit("prepDest", {}); - // renderJoin is a main-thread barrier — tracked by the push scheduler. - scheduler.register("renderJoin", { + // Register task definitions for dynamic tasks. The SAB already has + // pre-reserved slots and dep-count edges for render:i → renderJoin; + // registerDynamicRender (called by dispatchRender) populates them. + scheduler.tasks.set("renderJoin", { expected: Array.from({ length: N }, (_, i) => `render:${i}`), runOnMain: true, execute() { return {}; }, - submit(_, emit) { - emit("write", {}); - emit("writePdf", {}); - emit("searchData", {}); - }, + submit() {}, }); - // render:i tasks are worker-pulled from the SAB. Register their - // submit() with the scheduler (so merge + emit fires when the main - // thread processes the output message) but don't seed them through - // the push scheduler. for (let i = 0; i < N; i++) { - scheduler.registerWorkerTask(`render:${i}`, { + scheduler.tasks.set(`render:${i}`, { expected: [], handler: "render", consolidate: true, ganttSection: "Render", - submit(renderOut, emit, state) { + submit(renderOut, state) { for (const r of renderOut.pages) { const p = state.pageByDest.get(r.destPath); if (!p) continue; @@ -464,12 +434,11 @@ const TASKS = { if (r.offlineHtml !== undefined) p.offlineHtml = r.offlineHtml; if (r.offlineMisses !== undefined) p.offlineMisses = r.offlineMisses; } - emit("renderJoin", {}); }, }); } - // Write SAB entries, broadcast chunk data, set render tasks READY. + scheduler.addDynamicTasks(N + 1); scheduler.dispatchRender(out.chunks, out.sharedSAB); }, }, @@ -500,7 +469,7 @@ const TASKS = { baseurl: String(state.site.config.baseurl || ""), }); }, - submit(out, emit) { emit("writeAux", out); }, + submit() {}, }, // Write search-data.json. Depends on renderJoin (pages have @@ -513,7 +482,7 @@ const TASKS = { if (ctx.opts.dryRun) return { entries: 0, json: "" }; return writeSearchData(state.pages, state.site, ctx.destRoot); }, - submit(out, emit) { emit("writeAux", out); }, + submit() {}, }, // Write redirect stubs + sitemap/robots. Waits for write (pages on disk), @@ -530,9 +499,7 @@ const TASKS = { ]); return { redirectStats, sitemapStats, searchStats }; }, - submit(out, emit) { - emit("writeOffline", out); - }, + submit() {}, }, // Produce _site-offline/. Runs in parallel with writePdf on the main thread; @@ -650,6 +617,7 @@ export async function runBuild(opts) { pool.onWorkerDone = (msg) => scheduler._onWorkerDone(msg); pool.onWorkerError = (msg) => scheduler._onWorkerError(msg); pool.onWarmInitTiming = (msg) => scheduler._onWarmInitTiming(msg); + pool.onMainTaskReady = () => scheduler._onMainTaskReady(); pool.sendInit(sab, taskMeta, ctx, idMapping); diff --git a/builder/worker-pool.mjs b/builder/worker-pool.mjs index 74f4a65e..5ca5717d 100644 --- a/builder/worker-pool.mjs +++ b/builder/worker-pool.mjs @@ -1,7 +1,7 @@ -// Worker pool over node:worker_threads. Phase 6: workers pull tasks from a +// Worker pool over node:worker_threads. Phase 7: workers pull tasks from a // scheduling SAB; the pool is a lifecycle manager that spawns workers, sends -// them the SAB + metadata, forwards output/error messages to the scheduler, -// and terminates workers on destroy. +// them the SAB + metadata, forwards output/error/mainTaskReady messages to +// the scheduler, and terminates workers on destroy. // // Legacy push-model fields (_idleWarm, _idleCold, _warm, _busy, _queue, // run, warmup, _drain, _pushIdle, _onWarmedUp) are retained as dead code; @@ -23,6 +23,7 @@ export class WorkerPool { this.onWorkerDone = null; // ({ done, output, timing, lane }) => void this.onWorkerError = null; // ({ taskFailed, message, stack }) => void this.onWarmInitTiming = null; // ({ warmInit, timing, lane }) => void + this.onMainTaskReady = null; // () => void this._workers = Array.from({ length: size }, (_, i) => this._spawn(i)); } @@ -36,7 +37,7 @@ export class WorkerPool { if (msg.warmInit) { this.onWarmInitTiming?.(msg); return; } if (msg.done != null) { this.onWorkerDone?.(msg); return; } if (msg.taskFailed != null) { this.onWorkerError?.(msg); return; } - if (msg.mainTaskReady) { /* Phase 7 wires this up */ return; } + if (msg.mainTaskReady || msg.triggerMainTask != null) { this.onMainTaskReady?.(); return; } // ── Legacy push-model routing (Phase 8 removes) ── if (msg.warmedUp) { From 090493feaf5794397337ef52e0c32fb7a35a1546 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Mon, 1 Jun 2026 17:01:05 +0200 Subject: [PATCH 48/72] Remove push-model dead code from WorkerPool (Phase 8). --- builder/PLAN-sab-pull-scheduler.md | 88 ++++++++++++++++++++++++++++++ builder/worker-pool.mjs | 78 ++------------------------ 2 files changed, 93 insertions(+), 73 deletions(-) diff --git a/builder/PLAN-sab-pull-scheduler.md b/builder/PLAN-sab-pull-scheduler.md index 9262b3a1..51329641 100644 --- a/builder/PLAN-sab-pull-scheduler.md +++ b/builder/PLAN-sab-pull-scheduler.md @@ -40,6 +40,15 @@ future worker-affinity needs: 3. **`pin_to_predecessor`** --- the task must run on the same worker lane that ran a named predecessor. Applies to worker tasks. +4. **`run_when_idle`** --- when a worker has no claimable tasks and + would otherwise sleep, it speculatively executes this task. This + is distinct from `on_demand` (triggered by a dependent) --- it is + triggered by worker idleness. The primary use case is overlapping + `warmInit` with the main-thread spine: workers finish their seed + tasks (scss, buildInfo) well before render chunks appear, and + would otherwise sit idle. With `run_when_idle`, they warm up + during that dead time. Applies to worker tasks. + ### warmInit under the new model `warmInit` is declared as an explicit task with both `unique_per_worker` @@ -72,6 +81,7 @@ Flag bits: F_UNIQUE_PER_WORKER = 2 F_RUN_ON_MAIN = 4 F_PIN_TO_PRED = 8 + F_RUN_WHEN_IDLE = 16 Arrays (all Int32Array views into the SAB): taskCount // [1] — current number of registered tasks (atomic) @@ -159,6 +169,7 @@ warmInit: { expected: [], on_demand: true, // not started until a dependent needs it unique_per_worker: true, // one instance per worker lane + run_when_idle: true, // speculatively run when worker has no other work handler: "warmInit", // No submit --- unique_per_worker tasks don't participate in the // normal dependency graph. @@ -937,6 +948,83 @@ concurrency, different dispatch mechanism). (rebuild on file change, workers reuse across rebuilds). No warmup-related code remains. +### Phase 9: Speculative idle execution (`run_when_idle`) + +After Phase 8, `warmInit` is on-demand: it runs only when a worker +claims a render chunk and discovers the per-worker dep is unsatisfied. +This is correct but leaves performance on the table --- workers that +finish seed tasks (scss, buildInfo, mermaid) sit idle during the +main-thread spine (~200 ms) when they could be warming up. + +This phase adds `F_RUN_WHEN_IDLE` and wires `warmInit` to use it. + +1. **Flag bit.** Add `F_RUN_WHEN_IDLE = 16` to the SAB flag + constants and `run_when_idle` to the task definition schema. + `allocSchedulerSAB` sets the bit when the task def has + `run_when_idle: true`. + +2. **Pull loop change.** In the worker pull loop, after + `scanAndClaim` returns -1 (no claimable work) and before the + sleep path, insert a speculative-execution check: + + ``` + if taskIdx === -1: + // Speculative: run idle-eligible tasks before sleeping + idleTask = findIdleTask(views, myLane) + if idleTask !== -1: + execute handlers[idleTask] + Atomics.store(views.perWorkerDone, idleTask * MAX_LANES + myLane, 1) + postMessage({ done: idleTask, output, timing: { start, end } }) + continue // re-enter pull loop (real work may have appeared) + + // Nothing to do — sleep + gen = Atomics.load(views.notify, 0) + taskIdx = scanAndClaim(views, myLane) // double-check + if taskIdx === -1: + Atomics.wait(views.notify, 0, gen, 50) + continue + ``` + +3. **`findIdleTask`.** Scans task indices for tasks with + `F_RUN_WHEN_IDLE` set: + + ``` + function findIdleTask(views, myLane): + count = Atomics.load(views.taskCount, 0) + for i = 0 to count - 1: + if !(Atomics.load(views.flags, i) & F_RUN_WHEN_IDLE): continue + if Atomics.load(views.flags, i) & F_UNIQUE_PER_WORKER: + if Atomics.load(views.perWorkerDone, i * MAX_LANES + myLane) === 0: + return i // per-worker: no contention, no CAS needed + else: + if Atomics.load(views.status, i) !== DONE: + if Atomics.compareExchange(views.status, i, NOT_READY, CLAIMED) === NOT_READY: + return i + return -1 + ``` + + In practice, only `warmInit` has this flag, and it's + `unique_per_worker`, so the scan hits one task and checks one + per-worker-done flag. After a worker has run its `warmInit`, the + check short-circuits on every subsequent idle pass. + +4. **`warmInit` task def.** Add `run_when_idle: true` alongside the + existing `on_demand: true` and `unique_per_worker: true`. + +5. **No change to the render claim path.** Render chunks still list + `warmInit` in `perWorkerDeps`. If a render chunk becomes ready + before the idle-speculative path ran (e.g. the worker was busy + with scss the whole time), the existing on-demand claim-release + protocol handles it. The two paths are complementary, not + alternatives. + +**Verification:** `build.bat && check.bat` clean. Timing summary +shows `warmInit` per-lane timings overlapping with the main-thread +spine (starting around t=100--200 ms, while discover/nav/seo are +running), rather than clustering at render-chunk claim time +(t=400+ ms). This is the same overlap the old `pool.warmup()` +achieved, now expressed declaratively. + ## Notify protocol Workers sleep on a single generation-counter slot (`views.notify`) diff --git a/builder/worker-pool.mjs b/builder/worker-pool.mjs index 5ca5717d..19f1700e 100644 --- a/builder/worker-pool.mjs +++ b/builder/worker-pool.mjs @@ -1,22 +1,13 @@ -// Worker pool over node:worker_threads. Phase 7: workers pull tasks from a -// scheduling SAB; the pool is a lifecycle manager that spawns workers, sends -// them the SAB + metadata, forwards output/error/mainTaskReady messages to -// the scheduler, and terminates workers on destroy. -// -// Legacy push-model fields (_idleWarm, _idleCold, _warm, _busy, _queue, -// run, warmup, _drain, _pushIdle, _onWarmedUp) are retained as dead code; -// Phase 8 removes them. +// Worker pool over node:worker_threads. Lifecycle manager: spawns workers, +// sends them the scheduling SAB + metadata, forwards output/error/mainTaskReady +// messages to the scheduler, and terminates workers on destroy. Workers pull +// tasks from the SAB; the pool has no dispatch or queue logic. import { Worker } from "node:worker_threads"; export class WorkerPool { constructor(size, workerUrl) { this._workerUrl = workerUrl; - this._idleWarm = []; // (Phase 8 removes) - this._idleCold = []; // (Phase 8 removes) - this._warm = new Set(); // (Phase 8 removes) - this._busy = new Map(); // (Phase 8 removes) - this._queue = []; // (Phase 8 removes) this.bootTimings = []; // Callbacks wired by the caller after construction. @@ -32,37 +23,18 @@ export class WorkerPool { const spawnTime = Date.now(); const w = new Worker(this._workerUrl, { workerData: { lane, spawnTime } }); w.on("message", (msg) => { - // ── Phase 6 SAB-based message routing ── if (msg.coldBoot) { this.bootTimings.push({ lane, type: "cold", ...msg.coldBoot }); return; } if (msg.warmInit) { this.onWarmInitTiming?.(msg); return; } if (msg.done != null) { this.onWorkerDone?.(msg); return; } if (msg.taskFailed != null) { this.onWorkerError?.(msg); return; } if (msg.mainTaskReady || msg.triggerMainTask != null) { this.onMainTaskReady?.(); return; } - - // ── Legacy push-model routing (Phase 8 removes) ── - if (msg.warmedUp) { - if (msg.warmBoot) this.bootTimings.push({ lane, type: "warm", ...msg.warmBoot }); - this._onWarmedUp(w); - return; - } - const entry = this._busy.get(w); - if (!entry) return; - this._busy.delete(w); - this._pushIdle(w); - if (msg.error) entry.reject(Object.assign(new Error(msg.error), { stack: msg.stack })); - else entry.resolve(Object.assign(msg.result, { lane })); - this._drain(); }); w.on("error", (err) => { - const entry = this._busy.get(w); - if (entry) { this._busy.delete(w); entry.reject(err); } + this.onWorkerError?.({ taskFailed: -1, message: err.message, stack: err.stack }); }); - this._idleCold.push(w); return w; } - // ── Phase 6: SAB init + broadcast ────────────────────────────────────────── - sendInit(sab, taskMeta, ctx, idMapping) { for (const w of this._workers) { w.postMessage({ init: true, sab, taskMeta, ctx, idMapping }); @@ -75,46 +47,6 @@ export class WorkerPool { } } - // ── Legacy push-model methods (Phase 8 removes) ─────────────────────────── - - _pushIdle(w) { - if (this._warm.has(w)) this._idleWarm.push(w); - else this._idleCold.push(w); - } - - _onWarmedUp(w) { - this._warm.add(w); - const idx = this._idleCold.indexOf(w); - if (idx !== -1) { - this._idleCold.splice(idx, 1); - this._idleWarm.push(w); - this._drain(); - } - } - - run(payload, { name, transferList } = {}) { - return new Promise((resolve, reject) => { - this._queue.push({ message: { name, ...payload }, transferList, resolve, reject }); - this._drain(); - }); - } - - _drain() { - while (this._queue.length) { - const w = this._idleWarm.length ? this._idleWarm.shift() - : this._idleCold.length ? this._idleCold.shift() - : null; - if (!w) break; - const { message, transferList, resolve, reject } = this._queue.shift(); - this._busy.set(w, { resolve, reject }); - w.postMessage(message, transferList); - } - } - - warmup() { - for (const w of this._idleCold) w.postMessage({ warmup: true }); - } - destroy() { return Promise.all(this._workers.map(w => w.terminate())); } From 7f8f4503333e145accc043f884f2cf6153d8475f Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Mon, 1 Jun 2026 17:06:42 +0200 Subject: [PATCH 49/72] Add speculative idle execution for warmInit (Phase 9). --- builder/cpu-worker.mjs | 39 +++++++++++++++++++++++++++++++++++++++ builder/sab-scheduler.mjs | 3 +++ builder/tbdocs.mjs | 1 + 3 files changed, 43 insertions(+) diff --git a/builder/cpu-worker.mjs b/builder/cpu-worker.mjs index f6c4965d..aa526bd7 100644 --- a/builder/cpu-worker.mjs +++ b/builder/cpu-worker.mjs @@ -19,6 +19,7 @@ import { deriveOfflinePage, deriveOfflinePageCached, import { createViews, scanAndClaim, onTaskDone, READY, F_ON_DEMAND, F_RUN_ON_MAIN, + F_RUN_WHEN_IDLE, F_UNIQUE_PER_WORKER, MAX_LANES, } from "./sab-scheduler.mjs"; @@ -191,6 +192,24 @@ parentPort.on("message", (msg) => { } }); +// ── Idle-task scan (speculative warmup) ───────────────────────────────────── + +function findIdleTask(views, lane) { + const count = Atomics.load(views.taskCount, 0); + for (let i = 0; i < count; i++) { + if (!(Atomics.load(views.flags, i) & F_RUN_WHEN_IDLE)) continue; + if (Atomics.load(views.flags, i) & F_UNIQUE_PER_WORKER) { + if (Atomics.load(views.perWorkerDone, i * MAX_LANES + lane) === 0) + return i; + } else { + if (Atomics.load(views.status, i) !== READY) continue; + if (Atomics.compareExchange(views.status, i, READY, CLAIMED) === READY) + return i; + } + } + return -1; +} + // ── Pull loop ─────────────────────────────────────────────────────────────── async function pullLoop() { @@ -200,6 +219,26 @@ async function pullLoop() { let taskIdx = scanAndClaim(views, myLane); if (taskIdx === -1) { + // Speculative: run idle-eligible tasks before sleeping + const idleTask = findIdleTask(views, myLane); + if (idleTask !== -1) { + const idleMeta = taskMeta[idleTask]; + const idleStart = Date.now(); + try { + await handlers[idleMeta.handler](); + } catch (err) { + parentPort.postMessage({ taskFailed: idleTask, message: err.message, stack: err.stack }); + return; + } + Atomics.store(views.perWorkerDone, idleTask * MAX_LANES + myLane, 1); + parentPort.postMessage({ + warmInit: true, + timing: { start: idleStart, end: Date.now() }, + lane: myLane, + }); + continue; + } + const gen = Atomics.load(views.notify, 0); // Double-check after reading gen (race: a task may have become // READY between the failed scan and this load). diff --git a/builder/sab-scheduler.mjs b/builder/sab-scheduler.mjs index feeb91da..ddf4b641 100644 --- a/builder/sab-scheduler.mjs +++ b/builder/sab-scheduler.mjs @@ -22,6 +22,7 @@ export const F_ON_DEMAND = 1; export const F_UNIQUE_PER_WORKER = 2; export const F_RUN_ON_MAIN = 4; export const F_PIN_TO_PRED = 8; +export const F_RUN_WHEN_IDLE = 16; // ── SAB layout ─────────────────────────────────────────────────────────────── // @@ -142,6 +143,7 @@ export function allocSchedulerSAB(taskDefs, workerCount) { if (def?.unique_per_worker) f |= F_UNIQUE_PER_WORKER; if (def?.runOnMain) f |= F_RUN_ON_MAIN; if (def?.pin_to_predecessor) f |= F_PIN_TO_PRED; + if (def?.run_when_idle) f |= F_RUN_WHEN_IDLE; views.flags[i] = f; // successor edges @@ -364,6 +366,7 @@ export function verifySchedulerSAB(taskDefs, views, idMapping) { if (def.unique_per_worker) want |= F_UNIQUE_PER_WORKER; if (def.runOnMain) want |= F_RUN_ON_MAIN; if (def.pin_to_predecessor) want |= F_PIN_TO_PRED; + if (def.run_when_idle) want |= F_RUN_WHEN_IDLE; if (views.flags[idx] !== want) errors.push(`flags "${name}": got ${views.flags[idx]}, want ${want}`); } diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 07b6c2f4..0f6d8cf1 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -240,6 +240,7 @@ const TASKS = { expected: [], on_demand: true, unique_per_worker: true, + run_when_idle: true, handler: "warmInit", submit() {}, }, From 3d3cf1aece8983bce77680fcb30a803db3fd7907 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Mon, 1 Jun 2026 17:38:05 +0200 Subject: [PATCH 50/72] Plan phases 10 and 11 of the SAB scheduler. --- builder/PLAN-sab-pull-scheduler.md | 388 ++++++++++++++++++++++++++++- 1 file changed, 387 insertions(+), 1 deletion(-) diff --git a/builder/PLAN-sab-pull-scheduler.md b/builder/PLAN-sab-pull-scheduler.md index 51329641..c930a3e9 100644 --- a/builder/PLAN-sab-pull-scheduler.md +++ b/builder/PLAN-sab-pull-scheduler.md @@ -133,7 +133,13 @@ start via a `postMessage` init message. This includes: to look up the right function after claiming a task by index. - `perWorkerDeps`: array of task indices that are `unique_per_worker` dependencies. Empty for most tasks; render - chunks have `[warmInitIdx]`. Workers check these after claiming. + chunks have `[renderEnvInitIdx]`. Workers check these after + claiming. + - `expectedIdxs`: array of predecessor task indices used as + preconditions for `unique_per_worker` tasks (Phase 10). Workers + verify each predecessor is DONE before executing the per-worker + instance. Empty for tasks without predecessors (e.g. `warmInit`); + `renderEnvInit` has `[dispatchIdx]`. - `name`: string task name (for timing / error messages). - **`ctx`** --- the build context (`{ srcRoot, destRoot, opts, @@ -795,6 +801,10 @@ handler functions, and external behavior. - `taskMeta[i].handler`: handler function name (string). - `taskMeta[i].perWorkerDeps`: array of task indices (for unique_per_worker deps). Empty for most tasks. + - `taskMeta[i].expectedIdxs`: array of predecessor task indices + (for precondition checking on unique_per_worker tasks with + `expected` predecessors, Phase 10). Empty for most tasks. + Populated by mapping `def.expected` names through `nameToIdx`. - `taskMeta[i].name`: task name (for debug/timing). 5. Add the `warmInit` task definition to `TASKS` in `tbdocs.mjs`: @@ -1025,6 +1035,382 @@ running), rather than clustering at render-chunk claim time (t=400+ ms). This is the same overlap the old `pool.warmup()` achieved, now expressed declaratively. +### Phase 10: Explicit render env init (`renderEnvInit`) + +After Phase 9, the first render chunk on each worker pays a hidden +startup cost inside `getOrInitRenderEnv`: unpack ~300 KB shared +payload (JSON.parse), reconstruct three link-table Maps (~857 entries +each), instantiate markdown-it with plugins, build two Sets +(`staticFilesArr`, `sitePathsArr`). This cost is invisible in +timing --- it's buried inside the first render chunk's wall-clock --- +and it front-loads onto one chunk per worker, making that chunk +appear ~10--15 ms slower than the rest. + +This phase extracts the init into an explicit per-worker task, +making the cost visible, moving it off the render hot path, and +eliminating the `while (!_chunkDataSAB)` polling loop from the +render handler. + +#### Design extension: `unique_per_worker` tasks with predecessors + +Phases 5--9 treat `unique_per_worker` tasks as seeds (no `expected` +predecessors). `renderEnvInit` needs `dispatch` to be DONE before it +can run (the sharedSAB doesn't exist until then). This requires +allowing `expected` on `unique_per_worker` tasks. + +The semantics: `expected` predecessors on a `unique_per_worker` task +are **preconditions**, checked as read-only SAB status reads before +the per-worker instance executes. They are NOT tracked via depCount +(the task still doesn't participate in normal dependency counting). +The worker simply verifies each predecessor is DONE: + +``` +// About to run on-demand unique_per_worker dep D: +for each pred of D.expected: + predIdx = nameToIdx[pred] + if Atomics.load(views.status, predIdx) !== DONE: + // Precondition not met; release original task and re-scan. + Atomics.store(views.status, taskIdx, READY) + Atomics.add(views.notify, 0, 1) + Atomics.notify(views.notify, 0, 1) + continue outer loop +``` + +The precondition check runs after the per-worker dep check (nested +deps are checked first). If `renderEnvInit` depends on `warmInit` +via `perWorkerDeps`, the flow for a render chunk is: + +1. Worker claims render:i +2. Checks perWorkerDeps: `renderEnvInit[W]` not done +3. About to run renderEnvInit on-demand; checks its perWorkerDeps: + `warmInit[W]` not done +4. Runs warmInit[W] (on-demand, releases render:i first) +5. Re-enters pull loop, claims render:i again +6. Checks perWorkerDeps: `renderEnvInit[W]` not done +7. About to run renderEnvInit; checks its perWorkerDeps: + `warmInit[W]` done +8. Checks renderEnvInit's preconditions: `dispatch` DONE? Yes +9. Runs renderEnvInit[W] (releases render:i first) +10. Re-enters pull loop, claims render:i again +11. Checks perWorkerDeps: `renderEnvInit[W]` done +12. Executes render:i --- env already initialized, no polling + +If Phase 9's `run_when_idle` already ran warmInit during the spine, +steps 3--5 are skipped (warmInit[W] is already done). + +#### Task definitions + +```js +warmInit: { + expected: [], + on_demand: true, + unique_per_worker: true, + run_when_idle: true, + handler: "warmInit", + submit() {}, +}, + +renderEnvInit: { + expected: ["dispatch"], // precondition: sharedSAB exists + perWorkerDeps: ["warmInit"], // needs Shiki loaded + on_demand: true, + unique_per_worker: true, + handler: "renderEnvInit", + submit() {}, +}, +``` + +Render chunks change from `perWorkerDeps: ["warmInit"]` to +`perWorkerDeps: ["renderEnvInit"]`. This chains the dependency: +render → renderEnvInit → warmInit. + +#### Handler + +The `renderEnvInit` handler does what `getOrInitRenderEnv` does +today, minus the highlighter init (already done by warmInit): + +```js +async renderEnvInit() { + // Wait for renderData message (sharedSAB delivered via postMessage + // after dispatch; may not be processed yet if we were in + // Atomics.wait when it arrived). + while (!_sharedSAB) { + await new Promise(resolve => setImmediate(resolve)); + } + + const { siteData, initData, linkTablesData, staticFilesArr, + baseurl, buildInfo, sitePathsArr, + skipOffline } = unpackShared(_sharedSAB); + + const { initHighlighter } = await import("./highlight.mjs"); + const highlighter = await initHighlighter(); // cached; instant after warmInit + const linkTables = reconstructLinkTables(linkTablesData); + const staticFiles = new Set(staticFilesArr); + const markdown = createMarkdownIt({ highlighter, linkTables, baseurl, staticFiles }); + const site = { ...siteData, markdown, buildInfo }; + + let offlineBase = null; + if (!skipOffline) { + offlineBase = { + sitePaths: new Set(sitePathsArr), + baseurl: normalizeBaseurl(baseurl), + }; + } + + _renderEnv = { site, initData, offlineBase }; + return {}; +} +``` + +The render handler simplifies --- no lazy init, no polling: + +```js +async render(taskIdx) { + const workerStart = Date.now(); + + const chunkIndex = taskIdx - idMapping.DYNAMIC_BASE; + const offset = Atomics.load(views.chunkOffset, chunkIndex); + const length = Atomics.load(views.chunkLength, chunkIndex); + const chunk = JSON.parse( + new TextDecoder().decode(new Uint8Array(_chunkDataSAB, offset, length)), + ); + + // _renderEnv guaranteed initialized by renderEnvInit (perWorkerDep). + const env = _renderEnv; + + await renderPhase(chunk, env.site); + await templatePhase(chunk, env.site, env.initData); + // ... offline rewriting ... +} +``` + +`getOrInitRenderEnv` is deleted. + +#### Changes to `findIdleTask` (Phase 9) + +`findIdleTask` must respect preconditions for `run_when_idle` tasks +that have `expected` predecessors. `renderEnvInit` is NOT +`run_when_idle` (no benefit --- there's no idle window between +dispatch and render chunks), so this doesn't apply to it. But if a +future `run_when_idle` task has predecessors, `findIdleTask` should +check them: + +``` +function findIdleTask(views, myLane, taskMeta): + count = Atomics.load(views.taskCount, 0) + for i = 0 to count - 1: + if !(Atomics.load(views.flags, i) & F_RUN_WHEN_IDLE): continue + if Atomics.load(views.flags, i) & F_UNIQUE_PER_WORKER: + if Atomics.load(views.perWorkerDone, i * MAX_LANES + myLane) !== 0: + continue // already done on this lane + + // Check preconditions and perWorkerDeps before running + if !preconditionsMet(views, i, taskMeta): continue + if !perWorkerDepsMet(views, i, myLane, taskMeta): continue + + return i + // ... non-unique_per_worker case unchanged ... + return -1 +``` + +For `warmInit` (no predecessors, no perWorkerDeps), these checks +are no-ops. The guard exists for forward-compatibility. + +#### Implementation steps + +1. Extend the on-demand dep execution path in the pull loop: + before executing an on-demand `unique_per_worker` dep, check + its `expected` predecessors against SAB status. If any + predecessor is not DONE, release the original task and + re-scan. + +2. Extend the on-demand dep execution path to recursively check + `perWorkerDeps` on the dep itself. If `renderEnvInit` has + `perWorkerDeps: ["warmInit"]`, and warmInit is not done on + this lane, handle warmInit first (same on-demand protocol). + +3. Add the `renderEnvInit` task definition to `TASKS`. + +4. Add the `renderEnvInit` handler to `cpu-worker.mjs`. + +5. Move the `while (!_sharedSAB)` polling from the render handler + to `renderEnvInit`. + +6. Simplify the render handler: read `_renderEnv` directly. + +7. Delete `getOrInitRenderEnv` and the `_renderSAB` cache-check + variable. + +8. Update render chunk `perWorkerDeps` from `["warmInit"]` to + `["renderEnvInit"]` (in `allocSchedulerSAB`'s taskMeta + construction and in any task def that references it). + +9. Update `findIdleTask` to check preconditions and perWorkerDeps + for `run_when_idle` tasks (forward-compatibility). + +**Verification:** `build.bat && check.bat` clean. Timing summary +shows `renderEnvInit:w0`, `renderEnvInit:w1`, ... per-lane entries +(consolidated in the Boot section, ~10--15 ms each) appearing after +dispatch. First render chunk per worker no longer shows an inflated +wall-clock relative to subsequent chunks. Total render wall-clock +is unchanged (the init cost moved, not eliminated). + +### Phase 11: Amortize chunk packing into discover I/O gaps + +`packChunkData` runs inside `dispatch.submit()`, after the dispatch +timing window closes. It JSON-serializes every chunk array (~858 +pages across ~16 chunks), creating a ~40 ms gap between the dispatch +bar and the first render bar in the Gantt. The serialization is +pure synchronous CPU work on the main thread. + +`discover` spends ~200 ms in async disk I/O (`fs.readFile` on every +source file via `Promise.all`). During each I/O wait the main +thread's event loop is idle. This phase pre-serializes page objects +inside discover's `Promise.all` callbacks, overlapping the ~40 ms of +CPU work with the ~200 ms of libuv reads. At dispatch time, +`packChunkData` concatenates pre-serialized strings instead of +re-traversing the page data. + +Independent of Phase 10 --- neither changes code the other touches. + +#### Precondition + +Page objects are NOT mutated between `discover.submit()` and +`dispatch.submit()`. The spine tasks (`nav`, `seo`, `markdownInit`, +`deriveRedirects`, `buildInit`) all read `state.pages` but write +their results to `state.site.*`, not back to individual page +objects. JSON serialized during discover is therefore identical to +what `JSON.stringify` would produce at pack time. + +A debug assertion (step 6) guards this invariant. + +#### Side-table, not page property + +Pre-serialized strings are stored in a `Map` returned +alongside `{ pages, staticFiles }` from `discover()` --- NOT as a +`_json` property on the page object. A property would be included +by `JSON.stringify(page)`, embedding a JSON-escaped copy of the +whole page inside itself and roughly doubling the payload. + +#### Data flow + +1. **`discover()` returns the cache.** After `buildPage()` creates + a page object, `JSON.stringify(page)` produces the pre-serialized + string. Both happen in the same `Promise.all` callback, right + after `await fs.readFile()`: + + ``` + await Promise.all(allFiles.map(async (srcRel) => { + ... + const raw = await fs.readFile(srcPath, "utf8") + const page = buildPage(srcRoot, srcRel, parsed) + jsonCache.set(page, JSON.stringify(page)) + pages.push(page) + })) + return { pages, staticFiles, jsonCache } + ``` + + Each `JSON.stringify` takes ~47 μs (40 ms / 858 pages). The + libuv thread pool continues servicing reads while the main thread + does this work. + + Object identity is preserved through `pages.sort()` and + `chunkPages()` --- both reorder or slice the same objects --- so + `jsonCache.get(page)` resolves correctly at pack time. + +2. **`discover.submit()` stores the cache on `state`.** + + ``` + submit(out, state) { + state.pages = out.pages + state.staticFiles = out.staticFiles + state.site.config = out.config + state.jsonCache = out.jsonCache // new + for (const p of out.pages) state.pageByDest.set(p.destPath, p) + } + ``` + +3. **`Scheduler.dispatchRender()` passes the cache to + `packChunkData`.** + + ``` + dispatchRender(chunks, sharedSAB) { + ... + const chunkDataSAB = packChunkData(chunks, this._views, + this.state.jsonCache) + ... + } + ``` + +4. **`packChunkData` concatenates instead of serializing.** + + ``` + export function packChunkData(chunks, views, jsonCache) { + const buffers = jsonCache + ? chunks.map(chunk => + encoder.encode("[" + chunk.map(p => jsonCache.get(p)).join(",") + "]")) + : chunks.map(c => encoder.encode(JSON.stringify(c))) + ... // remainder unchanged: allocate SAB, copy buffers, write offsets + } + ``` + + The fallback path (no cache) is retained so any caller that omits + the argument gets the original behavior. + +#### Why `Promise.all` still works + +The current `discover` fires all reads at once via +`Promise.all(allFiles.map(async ...))`. Adding `JSON.stringify` +inside each callback does not reduce I/O concurrency --- all reads +are already dispatched to libuv before any callback runs. As I/O +completions arrive, the event loop runs each callback synchronously +(parse frontmatter, build page, serialize). Each ~47 μs serialize +is invisible between I/O completions; libuv workers keep reading +disk in the background. + +#### Memory + +The cache holds ~858 JSON strings averaging ~4 KB each --- ~3.4 MB +total. Negligible. The cache is dropped when the build state is +GC'd (after each build, or after each rebuild in serve mode). + +#### Implementation steps + +**Files:** `builder/discover.mjs`, `builder/tbdocs.mjs` (the +`discover` task definition and `SharedState`), `builder/scheduler.mjs` +(`Scheduler` class), `builder/sab-scheduler.mjs` (`packChunkData`). + +1. Add a `jsonCache` field to `SharedState` in + `builder/scheduler.mjs` (initialized to `null`). + +2. In `discover()` (`builder/discover.mjs`), create a `new Map()`, + populate it inside the `Promise.all` callback after + `buildPage()`, and include it in the return value. + +3. In the `discover` task definition in `builder/tbdocs.mjs`, update + `execute()` to destructure `jsonCache` from `discover()`'s + return value and pass it through in the output object. Update + `submit()` to store `state.jsonCache = out.jsonCache`. + +4. Add a `jsonCache` parameter to `packChunkData` in + `builder/sab-scheduler.mjs`. When present, use string + concatenation (`"[" + ... + "]"`); otherwise fall back to + `JSON.stringify`. + +5. In `Scheduler.dispatchRender()` (`builder/scheduler.mjs`), pass + `this.state.jsonCache` to `packChunkData`. + +6. Add a debug assertion (gated on `process.env.TBDOCS_DEBUG`) that + compares each concatenated chunk JSON with + `JSON.stringify(chunk)` to catch any unexpected page mutation + between discover and dispatch. + +**Verification:** `build.bat && check.bat` clean. The dispatch-to- +first-render gap in the Gantt shrinks from ~40 ms to <5 ms. +Discover's wall-clock time does not increase meaningfully (~1--3 ms). +Rendered output is byte-identical to the pre-change build. Run once +with `TBDOCS_DEBUG=1` to exercise the assertion. + ## Notify protocol Workers sleep on a single generation-counter slot (`views.notify`) From 3598bed85a71fca4872c7a3c67a4aa1fd512c922 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Mon, 1 Jun 2026 17:48:25 +0200 Subject: [PATCH 51/72] Extract renderEnvInit as explicit per-worker task (Phase 10). --- builder/PLAN-sab-pull-scheduler.md | 3 +- builder/cpu-worker.mjs | 162 ++++++++++++++++++++--------- builder/sab-scheduler.mjs | 34 +++++- builder/scheduler.mjs | 4 +- builder/tbdocs.mjs | 16 ++- builder/worker-pool.mjs | 4 +- 6 files changed, 164 insertions(+), 59 deletions(-) diff --git a/builder/PLAN-sab-pull-scheduler.md b/builder/PLAN-sab-pull-scheduler.md index c930a3e9..05578663 100644 --- a/builder/PLAN-sab-pull-scheduler.md +++ b/builder/PLAN-sab-pull-scheduler.md @@ -1066,8 +1066,7 @@ The worker simply verifies each predecessor is DONE: ``` // About to run on-demand unique_per_worker dep D: -for each pred of D.expected: - predIdx = nameToIdx[pred] +for each predIdx of taskMeta[D].expectedIdxs: if Atomics.load(views.status, predIdx) !== DONE: // Precondition not met; release original task and re-scan. Atomics.store(views.status, taskIdx, READY) diff --git a/builder/cpu-worker.mjs b/builder/cpu-worker.mjs index aa526bd7..74c23898 100644 --- a/builder/cpu-worker.mjs +++ b/builder/cpu-worker.mjs @@ -18,7 +18,7 @@ import { deriveOfflinePage, deriveOfflinePageCached, import { createViews, scanAndClaim, onTaskDone, - READY, F_ON_DEMAND, F_RUN_ON_MAIN, + READY, DONE, F_ON_DEMAND, F_RUN_ON_MAIN, F_RUN_WHEN_IDLE, F_UNIQUE_PER_WORKER, MAX_LANES, } from "./sab-scheduler.mjs"; @@ -46,6 +46,34 @@ const handlers = { return {}; }, + async renderEnvInit() { + while (!_sharedSAB) { + await new Promise(resolve => setImmediate(resolve)); + } + + const { siteData, initData, linkTablesData, staticFilesArr, + baseurl, buildInfo, sitePathsArr, + skipOffline } = unpackShared(_sharedSAB); + + const { initHighlighter } = await import("./highlight.mjs"); + const highlighter = await initHighlighter(); + const linkTables = reconstructLinkTables(linkTablesData); + const staticFiles = new Set(staticFilesArr); + const markdown = createMarkdownIt({ highlighter, linkTables, baseurl, staticFiles }); + const site = { ...siteData, markdown, buildInfo }; + + let offlineBase = null; + if (!skipOffline) { + offlineBase = { + sitePaths: new Set(sitePathsArr), + baseurl: normalizeBaseurl(baseurl), + }; + } + + _renderEnv = { site, initData, offlineBase }; + return {}; + }, + async scssLight() { const workerStart = Date.now(); const scssLightResult = await compileLightScss(ctx.srcRoot); @@ -73,13 +101,6 @@ const handlers = { async render(taskIdx) { const workerStart = Date.now(); - // The renderData broadcast may not yet have been processed (it was - // posted before the SAB set these tasks READY, but message delivery - // is async). Yield until the data arrives. - while (!_chunkDataSAB || !_sharedSAB) { - await new Promise(resolve => setImmediate(resolve)); - } - const chunkIndex = taskIdx - idMapping.DYNAMIC_BASE; const offset = Atomics.load(views.chunkOffset, chunkIndex); const length = Atomics.load(views.chunkLength, chunkIndex); @@ -87,7 +108,7 @@ const handlers = { new TextDecoder().decode(new Uint8Array(_chunkDataSAB, offset, length)), ); - const env = await getOrInitRenderEnv(_sharedSAB); + const env = _renderEnv; await renderPhase(chunk, env.site); await templatePhase(chunk, env.site, env.initData); @@ -138,38 +159,8 @@ const handlers = { }, }; -// ── Cached per-worker render environment ──────────────────────────────────── - -let _renderSAB = null; let _renderEnv = null; -async function getOrInitRenderEnv(sharedSAB) { - if (_renderSAB === sharedSAB) return _renderEnv; - - const { siteData, initData, linkTablesData, staticFilesArr, - baseurl, buildInfo, sitePathsArr, - skipOffline } = unpackShared(sharedSAB); - - const { initHighlighter } = await import("./highlight.mjs"); - const highlighter = await initHighlighter(); - const linkTables = reconstructLinkTables(linkTablesData); - const staticFiles = new Set(staticFilesArr); - const markdown = createMarkdownIt({ highlighter, linkTables, baseurl, staticFiles }); - const site = { ...siteData, markdown, buildInfo }; - - let offlineBase = null; - if (!skipOffline) { - offlineBase = { - sitePaths: new Set(sitePathsArr), - baseurl: normalizeBaseurl(baseurl), - }; - } - - _renderSAB = sharedSAB; - _renderEnv = { site, initData, offlineBase }; - return _renderEnv; -} - // ── Message handler (init + renderData only) ──────────────────────────────── parentPort.on("message", (msg) => { @@ -180,7 +171,6 @@ parentPort.on("message", (msg) => { idMapping = msg.idMapping; _chunkDataSAB = null; _sharedSAB = null; - _renderSAB = null; _renderEnv = null; pullLoop(); return; @@ -199,8 +189,24 @@ function findIdleTask(views, lane) { for (let i = 0; i < count; i++) { if (!(Atomics.load(views.flags, i) & F_RUN_WHEN_IDLE)) continue; if (Atomics.load(views.flags, i) & F_UNIQUE_PER_WORKER) { - if (Atomics.load(views.perWorkerDone, i * MAX_LANES + lane) === 0) - return i; + if (Atomics.load(views.perWorkerDone, i * MAX_LANES + lane) !== 0) + continue; + const meta = taskMeta[i]; + if (meta?.expectedIdxs) { + let skip = false; + for (const predIdx of meta.expectedIdxs) { + if (Atomics.load(views.status, predIdx) !== DONE) { skip = true; break; } + } + if (skip) continue; + } + if (meta?.perWorkerDeps) { + let skip = false; + for (const depIdx of meta.perWorkerDeps) { + if (Atomics.load(views.perWorkerDone, depIdx * MAX_LANES + lane) === 0) { skip = true; break; } + } + if (skip) continue; + } + return i; } else { if (Atomics.load(views.status, i) !== READY) continue; if (Atomics.compareExchange(views.status, i, READY, CLAIMED) === READY) @@ -232,7 +238,8 @@ async function pullLoop() { } Atomics.store(views.perWorkerDone, idleTask * MAX_LANES + myLane, 1); parentPort.postMessage({ - warmInit: true, + perWorkerTiming: true, + taskName: idleMeta.name, timing: { start: idleStart, end: Date.now() }, lane: myLane, }); @@ -265,14 +272,72 @@ async function pullLoop() { const depFlags = Atomics.load(views.flags, unsatisfied); if ((depFlags & F_ON_DEMAND) && !(depFlags & F_RUN_ON_MAIN)) { - // On-demand worker dep (warmInit). Release the claimed task so - // other workers can pick it up, then execute the dep inline. + const depMeta = taskMeta[unsatisfied]; + + // Check the dep's own perWorkerDeps (e.g. renderEnvInit → warmInit). + let nestedUnsatisfied = null; + if (depMeta?.perWorkerDeps) { + for (const nestedIdx of depMeta.perWorkerDeps) { + if (Atomics.load(views.perWorkerDone, nestedIdx * MAX_LANES + myLane) === 0) { + nestedUnsatisfied = nestedIdx; + break; + } + } + } + + if (nestedUnsatisfied !== null) { + const nestedFlags = Atomics.load(views.flags, nestedUnsatisfied); + if ((nestedFlags & F_ON_DEMAND) && !(nestedFlags & F_RUN_ON_MAIN)) { + Atomics.store(views.status, taskIdx, READY); + Atomics.add(views.notify, 0, 1); + Atomics.notify(views.notify, 0, 1); + + const nestedMeta = taskMeta[nestedUnsatisfied]; + const nestedStart = Date.now(); + try { + await handlers[nestedMeta.handler](); + } catch (err) { + parentPort.postMessage({ taskFailed: nestedUnsatisfied, message: err.message, stack: err.stack }); + return; + } + Atomics.store(views.perWorkerDone, nestedUnsatisfied * MAX_LANES + myLane, 1); + parentPort.postMessage({ + perWorkerTiming: true, + taskName: nestedMeta.name, + timing: { start: nestedStart, end: Date.now() }, + lane: myLane, + }); + continue; + } + Atomics.store(views.status, taskIdx, READY); + Atomics.add(views.notify, 0, 1); + Atomics.notify(views.notify, 0, 1); + continue; + } + + // Check preconditions (expected predecessors on the dep). + let precondFailed = false; + if (depMeta?.expectedIdxs) { + for (const predIdx of depMeta.expectedIdxs) { + if (Atomics.load(views.status, predIdx) !== DONE) { + precondFailed = true; + break; + } + } + } + if (precondFailed) { + Atomics.store(views.status, taskIdx, READY); + Atomics.add(views.notify, 0, 1); + Atomics.notify(views.notify, 0, 1); + continue; + } + + // All dep's deps satisfied. Release original task, execute the dep. Atomics.store(views.status, taskIdx, READY); Atomics.add(views.notify, 0, 1); Atomics.notify(views.notify, 0, 1); - const depMeta = taskMeta[unsatisfied]; - const depStart = Date.now(); + const depStart = Date.now(); try { await handlers[depMeta.handler](); } catch (err) { @@ -282,7 +347,8 @@ async function pullLoop() { Atomics.store(views.perWorkerDone, unsatisfied * MAX_LANES + myLane, 1); parentPort.postMessage({ - warmInit: true, + perWorkerTiming: true, + taskName: depMeta.name, timing: { start: depStart, end: Date.now() }, lane: myLane, }); diff --git a/builder/sab-scheduler.mjs b/builder/sab-scheduler.mjs index ddf4b641..e88cf506 100644 --- a/builder/sab-scheduler.mjs +++ b/builder/sab-scheduler.mjs @@ -179,13 +179,37 @@ export function allocSchedulerSAB(taskDefs, workerCount) { Atomics.store(views.edgeCount, 0, edgePos); // 5. Build taskMeta - const warmInitIdx = nameToIdx.get("warmInit"); - const taskMeta = new Array(totalTasks).fill(null); + const renderEnvInitIdx = nameToIdx.get("renderEnvInit"); + const taskMeta = new Array(totalTasks).fill(null); for (const [name, def] of Object.entries(taskDefs)) { + // Map perWorkerDeps from definition (if any) through nameToIdx. + let perWorkerDeps = []; + if (def.perWorkerDeps) { + perWorkerDeps = def.perWorkerDeps.map(depName => { + const depIdx = nameToIdx.get(depName); + if (depIdx == null) + throw new Error(`"${name}" has unknown perWorkerDep "${depName}"`); + return depIdx; + }); + } + + // Map expected predecessors to indices for precondition checking + // on unique_per_worker tasks (Phase 10). + let expectedIdxs = []; + if (def.unique_per_worker && def.expected.length > 0) { + expectedIdxs = def.expected.map(predName => { + const predIdx = nameToIdx.get(predName); + if (predIdx == null) + throw new Error(`"${name}" has unknown expected predecessor "${predName}"`); + return predIdx; + }); + } + taskMeta[nameToIdx.get(name)] = { handler: def.handler ?? name, - perWorkerDeps: [], + perWorkerDeps, + expectedIdxs, name, }; } @@ -193,7 +217,8 @@ export function allocSchedulerSAB(taskDefs, workerCount) { for (let i = 0; i < maxChunks; i++) { taskMeta[DYNAMIC_BASE + i] = { handler: "render", - perWorkerDeps: warmInitIdx != null ? [warmInitIdx] : [], + perWorkerDeps: renderEnvInitIdx != null ? [renderEnvInitIdx] : [], + expectedIdxs: [], name: `render:${i}`, }; } @@ -201,6 +226,7 @@ export function allocSchedulerSAB(taskDefs, workerCount) { taskMeta[RENDER_JOIN_IDX] = { handler: "renderJoin", perWorkerDeps: [], + expectedIdxs: [], name: "renderJoin", }; diff --git a/builder/scheduler.mjs b/builder/scheduler.mjs index ea41f218..b2c24805 100644 --- a/builder/scheduler.mjs +++ b/builder/scheduler.mjs @@ -201,8 +201,8 @@ export class Scheduler { this._abort(name, err); } - _onWarmInitTiming({ timing, lane }) { - this.timings.set(`warmInit:w${lane}`, { + _onPerWorkerTiming({ taskName, timing, lane }) { + this.timings.set(`${taskName}:w${lane}`, { start: timing.start, end: timing.end, workerStart: timing.start, workerEnd: timing.end, lane, diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 0f6d8cf1..b7139ebe 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -245,6 +245,20 @@ const TASKS = { submit() {}, }, + // On-demand per-worker render environment init: unpacks the shared + // payload, reconstructs link-table Maps, instantiates markdown-it. + // Depends on dispatch (sharedSAB must exist) and warmInit (Shiki + // must be loaded). Moves the hidden first-chunk init cost off the + // render hot path. + renderEnvInit: { + expected: ["dispatch"], + perWorkerDeps: ["warmInit"], + on_demand: true, + unique_per_worker: true, + handler: "renderEnvInit", + submit() {}, + }, + // ── Main-thread spine ───────────────────────────────────────────────────── discover: { @@ -617,7 +631,7 @@ export async function runBuild(opts) { pool.onWorkerDone = (msg) => scheduler._onWorkerDone(msg); pool.onWorkerError = (msg) => scheduler._onWorkerError(msg); - pool.onWarmInitTiming = (msg) => scheduler._onWarmInitTiming(msg); + pool.onPerWorkerTiming = (msg) => scheduler._onPerWorkerTiming(msg); pool.onMainTaskReady = () => scheduler._onMainTaskReady(); pool.sendInit(sab, taskMeta, ctx, idMapping); diff --git a/builder/worker-pool.mjs b/builder/worker-pool.mjs index 19f1700e..cd1e60f4 100644 --- a/builder/worker-pool.mjs +++ b/builder/worker-pool.mjs @@ -13,7 +13,7 @@ export class WorkerPool { // Callbacks wired by the caller after construction. this.onWorkerDone = null; // ({ done, output, timing, lane }) => void this.onWorkerError = null; // ({ taskFailed, message, stack }) => void - this.onWarmInitTiming = null; // ({ warmInit, timing, lane }) => void + this.onPerWorkerTiming = null; // ({ perWorkerTiming, taskName, timing, lane }) => void this.onMainTaskReady = null; // () => void this._workers = Array.from({ length: size }, (_, i) => this._spawn(i)); @@ -24,7 +24,7 @@ export class WorkerPool { const w = new Worker(this._workerUrl, { workerData: { lane, spawnTime } }); w.on("message", (msg) => { if (msg.coldBoot) { this.bootTimings.push({ lane, type: "cold", ...msg.coldBoot }); return; } - if (msg.warmInit) { this.onWarmInitTiming?.(msg); return; } + if (msg.perWorkerTiming) { this.onPerWorkerTiming?.(msg); return; } if (msg.done != null) { this.onWorkerDone?.(msg); return; } if (msg.taskFailed != null) { this.onWorkerError?.(msg); return; } if (msg.mainTaskReady || msg.triggerMainTask != null) { this.onMainTaskReady?.(); return; } From e8559c2ce5e4b5493bd5b52f45cbe4abd6396845 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Mon, 1 Jun 2026 18:00:13 +0200 Subject: [PATCH 52/72] Update the plan - phase 11 is a no-go. --- builder/PLAN-sab-pull-scheduler.md | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/builder/PLAN-sab-pull-scheduler.md b/builder/PLAN-sab-pull-scheduler.md index 05578663..d7e5d7ac 100644 --- a/builder/PLAN-sab-pull-scheduler.md +++ b/builder/PLAN-sab-pull-scheduler.md @@ -1254,7 +1254,32 @@ dispatch. First render chunk per worker no longer shows an inflated wall-clock relative to subsequent chunks. Total render wall-clock is unchanged (the init cost moved, not eliminated). -### Phase 11: Amortize chunk packing into discover I/O gaps +### Phase 11: Amortize chunk packing into discover I/O gaps — DECLINED + +**Decision:** precondition is false; pages are mutated between +discover and dispatch. + +The plan assumed page objects are NOT mutated between +`discover.submit()` and `dispatch.submit()`, so JSON serialized +during discover would be identical to what `JSON.stringify` would +produce at pack time. In practice, `nav` mutates every page in +place (adding `navPath`, `breadcrumbs`, `children`, `navLevels`) +and `seo` adds `seoTitle`, `seoFullTitle`, `seoCanonical`, +`seoIsHome`. Both run between discover and dispatch. A debug +assertion (`TBDOCS_DEBUG=1`) confirmed the mismatch immediately +on the first chunk. + +The ~40 ms `packChunkData` cost is real but cannot be amortized +into discover's I/O gaps without either (a) serializing only the +discover-time fields and reconstructing the full page on the worker +(fragile, couples the cache to every future spine mutation), or +(b) re-serializing after the last mutation (which puts the work +back on the critical path and defeats the purpose). Neither is +worth the complexity for a 40 ms saving. + +--- + +*Original plan text retained below for reference.* `packChunkData` runs inside `dispatch.submit()`, after the dispatch timing window closes. It JSON-serializes every chunk array (~858 From 63edd0c7700d5adbc3e630b59b277683488540fe Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Mon, 1 Jun 2026 18:11:44 +0200 Subject: [PATCH 53/72] Update the Gantt chart formatting. --- builder/gantt.mjs | 41 +++++++++++++++++++++++++---------------- 1 file changed, 25 insertions(+), 16 deletions(-) diff --git a/builder/gantt.mjs b/builder/gantt.mjs index a8d2cd99..b5fc29d5 100644 --- a/builder/gantt.mjs +++ b/builder/gantt.mjs @@ -7,10 +7,18 @@ const COLORS = { Render: { light: "#b09cd8", dark: "#8066a8" }, Write: { light: "#e8a756", dark: "#c08030" }, Boot: { light: "#e57373", dark: "#c62828" }, + Cold: { light: "#6eb5d9", dark: "#3c7db0" }, + Env: { light: "#e8a756", dark: "#c08030" }, Other: { light: "#bbb", dark: "#666" }, }; -const SECTION_W = 60; +const SECTION_W = 24; + +const BOOT_STYLE = { + cold: { label: "cold", cls: "cold" }, + warmInit: { label: "warm", cls: "boot" }, + renderEnvInit: { label: "env", cls: "env" }, +}; const SVG_W = 900; const CHART_W = SVG_W - SECTION_W - 20; const ROW_H = 20; @@ -92,26 +100,24 @@ export function renderGantt(grouped) { // Workers — one row per lane, individual task bars if (sortedLanes.length > 0) { o.push(``); + const lx = Math.round(SECTION_W / 2); + const ly = rd(y + sortedLanes.length * ROW_H / 2); + o.push(`Workers`); for (let li = 0; li < sortedLanes.length; li++) { const [, tasks] = sortedLanes[li]; const ty = rd(y + ROW_H / 2 + 3.5); const by = rd(y + (ROW_H - BAR_H) / 2); - if (li === 0) o.push(`Workers`); - const bootBars = []; for (const t of tasks) { - if (t._color === "Boot") { bootBars.push(t); continue; } - const bx = rd(xOf(t.workerStart)); - const bw = rd(Math.max(xOf(t.workerEnd) - xOf(t.workerStart), 1)); - o.push(``); - const lbl = workerLabel(t); - if (lbl.length * CHAR_W + BAR_PAD * 2 <= bw) - o.push(`${esc(lbl)}`); - } - const bootH = Math.round(BAR_H * 0.75); - for (const t of bootBars) { const bx = rd(xOf(t.workerStart)); const bw = rd(Math.max(xOf(t.workerEnd) - xOf(t.workerStart), 1)); - o.push(``); + let cls; + if (t._color === "Boot") { + const base = t.id.replace(/:.*/, ""); + cls = `gb-${BOOT_STYLE[base]?.cls ?? "boot"}`; + } else { + cls = `gb-${(t._color || "render").toLowerCase()}`; + } + o.push(``); const lbl = workerLabel(t); if (lbl.length * CHAR_W + BAR_PAD * 2 <= bw) o.push(`${esc(lbl)}`); @@ -133,13 +139,15 @@ export function renderGantt(grouped) { function renderMainSection(o, section, tasks, y, xOf) { o.push(``); const cls = `gb-${section.toLowerCase()}`; + const lx = Math.round(SECTION_W / 2); + const ly = rd(y + tasks.length * ROW_H / 2); + o.push(`${esc(section)}`); for (let i = 0; i < tasks.length; i++) { const t = tasks[i]; const bx = rd(xOf(t.start)); const bw = rd(Math.max(xOf(t.end) - xOf(t.start), 1)); const by = rd(y + (ROW_H - BAR_H) / 2); const ty = rd(y + ROW_H / 2 + 3.5); - if (i === 0) o.push(`${esc(section)}`); o.push(``); const lbl = taskLabel(t); const textW = lbl.length * CHAR_W; @@ -179,7 +187,8 @@ function taskLabel(t) { } function workerLabel(t) { - return t.id.replace(/:.*/, "").replace(/ w\d+$/, ""); + const base = t.id.replace(/:.*/, "").replace(/ w\d+$/, ""); + return BOOT_STYLE[base]?.label ?? base; } function rd(n) { return Math.round(n * 10) / 10; } From de524e0bf4487605ad0444a6feffa50b8e091f28 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Mon, 1 Jun 2026 18:17:50 +0200 Subject: [PATCH 54/72] Pre-create page output dirs during render window (prepPageDirs). --- builder/tbdocs.mjs | 27 +++++++++++++++++++++------ builder/write.mjs | 17 ++++++++++++----- 2 files changed, 33 insertions(+), 11 deletions(-) diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index b7139ebe..515313d7 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -35,7 +35,7 @@ import { } from "./render.mjs"; import { loadHighlightTheme } from "./highlight-theme.mjs"; import { buildInitConfig, renderSidebar } from "./template.mjs"; -import { writePhase, prepareDestinations } from "./write.mjs"; +import { writePhase, prepareDestinations, preparePageDirs } from "./write.mjs"; import { writeRedirects, deriveRedirectStubs } from "./redirects.mjs"; import { writeSitemap, deriveSitemapUrls } from "./sitemap.mjs"; import { writeSearchData } from "./search.mjs"; @@ -124,8 +124,9 @@ export function makeTimer() { // nav + buildInit → dispatch; config → loadData; discover → markdownInit → seo; // deriveRedirects off discover; deriveSitemap + resolveBookChapters + prepDest deferred to dispatch), // the render fan-out (dispatch → render:0..N → renderJoin), and write/post-write tasks -// (renderJoin + prepDest → searchData; write + searchData → writeAux → -// writeOffline; renderJoin + mermaid → writePdf) are scheduler tasks. +// (prepDest → prepPageDirs; renderJoin + prepPageDirs → write + searchData; +// write + searchData → writeAux → writeOffline; renderJoin + mermaid → writePdf) +// are scheduler tasks. // runBuild() constructs the pool + scheduler, awaits start(), logs the // summary, and returns. @@ -217,6 +218,19 @@ const TASKS = { submit() {}, }, + // Pre-create all page output directories while render workers are busy. + // Lets writePages skip mkdir entirely — pure writeFile. + prepPageDirs: { + expected: ["prepDest"], + runOnMain: true, + async execute(_, ctx, state) { + if (ctx.opts.dryRun) return {}; + await preparePageDirs(state.pages, state.staticFiles, ctx.destRoot); + return {}; + }, + submit() {}, + }, + // Theme CSS load. Reads the vendored .theme files and generates the // tb-highlight.css palette; does NOT init Shiki WASM (unneeded on main // since no code blocks are rendered here). Workers init their own full @@ -463,9 +477,10 @@ const TASKS = { // Materialise pages + static files + generated CSS to _site/. // Waits for renderJoin (pages rendered + templated), scss (generated CSS), // mermaid (SVG descriptors appended to state.staticFiles by mermaid.submit), - // prepDest (_site/ cleaned and recreated), and highlighterInit (highlight CSS). + // prepPageDirs (page output directories pre-created), and highlighterInit + // (highlight CSS). write: { - expected: ["renderJoin", "scssJoin", "mermaid", "prepDest", "highlighterInit"], + expected: ["renderJoin", "scssJoin", "mermaid", "prepPageDirs", "highlighterInit"], runOnMain: true, async execute({ scssJoin: { scssResult }, mermaid: { mermaidStats }, highlighterInit: _highlightSignal }, ctx, state) { void mermaidStats; // dependency signal only; append already happened in mermaid.submit @@ -573,7 +588,7 @@ const GANTT_SECTION = { discover: "Spine", nav: "Spine", markdownInit: "Spine", buildInit: "Spine", seo: "Spine", resolveBookChapters: "Spine", deriveRedirects: "Spine", deriveSitemap: "Spine", - dispatch: "Render", prepDest: "Render", + dispatch: "Render", prepDest: "Render", prepPageDirs: "Render", write: "Write", searchData: "Write", writeAux: "Write", writeOffline: "Write", writePdf: "Write", }; const GANTT_SECTION_ORDER = ["Seeds", "Spine", "Render", "Write"]; diff --git a/builder/write.mjs b/builder/write.mjs index fb3c7e0a..8ddb0a98 100644 --- a/builder/write.mjs +++ b/builder/write.mjs @@ -43,8 +43,6 @@ export async function writePhase(pages, staticFiles, { destRoot, dryRun = false, mkdirCache.clear(); mkdirInflight.clear(); - assertNoDestinationCollisions(pages, staticFiles); - if (dryRun) { const pagesToWrite = pages.filter(p => p.html !== undefined).length; const skipped = pages.length - pagesToWrite; @@ -109,6 +107,17 @@ export async function prepareDestinations(roots, dryRun) { })); } +// Pre-create all page output directories so writePages can skip mkdir +// entirely. Runs as a separate task concurrently with render workers. +export async function preparePageDirs(pages, staticFiles, destRoot) { + assertNoDestinationCollisions(pages, staticFiles); + const dirs = new Set(); + for (const page of pages) { + if (page.destPath) dirs.add(path.dirname(path.join(destRoot, page.destPath))); + } + await Promise.all([...dirs].map(d => fs.mkdir(d, { recursive: true }))); +} + export function isUnderProject(destRoot) { const rel = path.relative(PROJECT_ROOT, destRoot); return rel !== "" && !rel.startsWith("..") && !path.isAbsolute(rel); @@ -121,12 +130,10 @@ async function writePages(pages, destRoot, limit) { let skipped = 0; await runLimited(pages, limit, async (page) => { if (page.html === undefined) { - // book.html (layout: book-combined) -- Phase 8 owns it. skipped++; return; } const dest = path.join(destRoot, page.destPath); - await mkdirRec(path.dirname(dest)); await safeWrite(dest, () => fs.writeFile(dest, page.html, "utf8")); written++; }); @@ -188,7 +195,7 @@ async function copyStaticFiles(staticFiles, destRoot, limit, baseurl) { // ---------- §6.4 assertNoDestinationCollisions -------------------------- -function assertNoDestinationCollisions(pages, staticFiles) { +export function assertNoDestinationCollisions(pages, staticFiles) { const pageDests = new Set( pages.filter(p => p.html !== undefined).map(p => p.destPath), ); From 6c879226fe92cc699ae5751141c17d3d631e5585 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Mon, 1 Jun 2026 19:20:40 +0200 Subject: [PATCH 55/72] Per-worker page flush, replacing renderJoin barrier (Phase 12). --- builder/PLAN-sab-pull-scheduler.md | 275 +++++++++++++++++++++++++++++ builder/cpu-worker.mjs | 77 ++++++-- builder/offline.mjs | 7 +- builder/sab-scheduler.mjs | 32 +--- builder/scheduler.mjs | 33 +++- builder/tbdocs.mjs | 102 ++++++----- builder/write.mjs | 11 +- 7 files changed, 442 insertions(+), 95 deletions(-) diff --git a/builder/PLAN-sab-pull-scheduler.md b/builder/PLAN-sab-pull-scheduler.md index d7e5d7ac..be667747 100644 --- a/builder/PLAN-sab-pull-scheduler.md +++ b/builder/PLAN-sab-pull-scheduler.md @@ -1435,6 +1435,281 @@ Discover's wall-clock time does not increase meaningfully (~1--3 ms). Rendered output is byte-identical to the pre-change build. Run once with `TBDOCS_DEBUG=1` to exercise the assertion. +### Phase 12: Per-worker page flush + +**Suggested model:** Opus. + +**Motivation.** Today the entire `writePages` pass --- ~1,080 files to +disk --- waits behind `renderJoin`, then runs on the main thread. The +per-page I/O is embarrassingly parallel and the data is already in +worker memory after `render`. This phase moves page writes into +workers, overlapping I/O with the render tail and eliminating the +`html` / `offlineHtml` fields from the render delta (the two largest +structured-clone payloads per chunk). + +**Design.** Three changes: + +1. **Page stash.** Each worker keeps a module-scope array + (`_pageStash = []`) initialized empty at startup. The `render` + handler appends `{ destPath, html, offlineHtml }` for each rendered + page to the stash instead of returning those fields in the delta. + The render delta shrinks to `{ destPath, renderedContent, + offlineMisses }`. + +2. **`flushPages` task.** A new `unique_per_worker` + `on_demand` task. + When activated (by `prepPageDirs` completing on main), it becomes + eligible in the idle-task scan. The handler writes every stashed + page to `_site/` and `_site-offline/`, clears the stash, and returns + write stats. A `flushJoin` barrier on main collects all per-worker + flush completions. + +3. **Priority-ordered idle scan.** The current `F_RUN_WHEN_IDLE` + boolean becomes a numeric priority (`idle_priority`). + `findIdleTask` picks the eligible task with the lowest + (= highest-priority) value. Assignment: `warmInit` = 0 (run first + --- Shiki must load before rendering), `flushPages` = 1 (run after + render drains). In practice they never compete (`warmInit` finishes + long before `flushPages` becomes eligible), but the priority makes + the ordering explicit and defensive. + + Implementation: store `idle_priority` in `taskMeta` (the JS-side + per-task metadata already sent to workers at init), not in the SAB + layout. `findIdleTask` keeps the flag-bit scan + (`F_RUN_WHEN_IDLE`) to identify idle-eligible tasks, then among + eligible candidates picks the one with the lowest + `taskMeta[i].idlePriority`. With only 2--3 idle tasks the extra + comparison is negligible. + +**Edge case: worker with zero render chunks.** Under high worker +counts (or small page sets), one or more workers may never claim a +render task. Their stash stays at the initialized-empty `[]`. +`flushPages` on such a worker writes zero pages and returns +`{ written: 0, offlineWritten: 0 }`. This is safe --- `flushJoin` +counts it as done. Guard: `_pageStash` must be preset to `[]` both +at module scope and in the `msg.init` handler (serve-mode pool reuse +across rebuilds). + +**Graph changes.** `renderJoin` is removed entirely. All downstream +tasks that depended on it switch to `flushJoin`: + +``` +render:i [W] (stashes pages locally; delta carries renderedContent + offlineMisses only) + render:i.submit() merges renderedContent into state.pages on main + +prepPageDirs [M] → activates flushPages ON_DEMAND slots + +flushPages [W, unique_per_worker, on_demand, idle_priority: 1] + writes stashed html → _site/ + writes stashed offlineHtml → _site-offline/ + → flushJoin [M] + +flushJoin + scssJoin + mermaid + highlighterInit → writeAssets [M] + (copyTheme + copyStaticFiles + writeGeneratedAssets --- no page writes) + +flushJoin + prepDest → searchData [M] + +flushJoin + searchData + deriveRedirects + deriveSitemap → writeAux [M] + +writeAux + writeAssets → writeOffline [M] + (offline theme / static / aux only --- page HTML already on disk from flush) + +flushJoin + mermaid → writePdf [M] + (reads renderedContent from state.pages --- not html) +``` + +**Why `flushJoin` subsumes `renderJoin`.** Messages from a single +worker to main are FIFO. Each worker posts all `render:i` completion +messages (triggering `render:i.submit()` delta merges on main) before +posting its `flushPages` completion. By the time main processes the +last worker's flush-done --- which is when `flushJoin` fires --- every +`render:i.submit()` has already executed. So `flushJoin` implies all +render deltas are merged, and `searchData` / `writePdf` can safely +read `renderedContent` from `state.pages`. + +**Implementation details.** + +- **Stash initialization.** `let _pageStash = []` at module scope in + `cpu-worker.mjs`. Reset to `[]` in the `msg.init` handler (for + serve-mode pool reuse across rebuilds). + +- **Render handler change.** After `renderPhase` + `templatePhase` + + offline derivation, the handler pushes `{ destPath, html, + offlineHtml }` onto `_pageStash` for each writable page + (`html !== undefined`). The return value drops `html` and + `offlineHtml`: + ```js + return { + workerStart, workerEnd, + pages: chunk.map(p => ({ + destPath: p.destPath, + renderedContent: p.renderedContent, + offlineMisses: p.offlineMisses, + })), + }; + ``` + +- **`flushPages` handler.** Reads `ctx.destRoot` (already available on + the worker via the init message). Writes each stashed page to + `path.join(destRoot, p.destPath)` and, when `offlineHtml` is + defined, to `path.join(destRoot + '-offline', p.destPath)`. Skips + actual writes when `ctx.opts.dryRun` is true. Returns + `{ written, offlineWritten, offlineMisses }` (`offlineMisses` is + the sum of per-page `offlineMisses` counts from the stash). + + **Stats delivery.** The `perWorkerTiming` message format gains an + optional `output` field. The `flushPages` completion path in + `cpu-worker.mjs` sets it to the handler's return value so the stats + reach main alongside the timing. Existing per-worker tasks + (`warmInit`, `renderEnvInit`) omit the field; the main-thread + handler ignores it when absent. + +- **`flushPages` task definition.** + ```js + flushPages: { + expected: ["prepPageDirs"], + on_demand: true, + unique_per_worker: true, + run_when_idle: true, + idle_priority: 1, + handler: "flushPages", + submit() {}, + }, + ``` +- **`flushJoin` task definition and activation.** `flushJoin` is + `on_demand` + `runOnMain`. It does not participate in the normal + SAB successor system (because `flushPages` is `unique_per_worker`, + which uses `perWorkerDone` + `perWorkerTiming` --- not the regular + task-completion path that decrements successor dep counts). + + Instead, **counter-based activation in `_onPerWorkerTiming`:** + the scheduler keeps a `_flushCount` counter (initialized to 0) and + a `_flushStats` accumulator. When `_onPerWorkerTiming` receives a + message with `taskName === "flushPages"`, it increments the counter + and folds the message's `output` into the accumulator (summing + `written`, `offlineWritten`, `offlineMisses`). When the counter + reaches `workerCount`, it: + + 1. Stores the aggregated stats on `this.results` under + `"flushPages"` so downstream tasks can read them. + 2. Calls `addDynamicTasks(1)` (so `_remaining` includes + `flushJoin`). + 3. Sets `flushJoin`'s SAB status to `READY`. + 4. Calls `_scheduleMainScan()`. + + `flushJoin` then runs as a no-op barrier; its `submit()` is empty + (downstream tasks declare `"flushJoin"` in their `expected` arrays + and the scheduler's `_assembleInputs` resolves it from + `this.results`). + + ```js + flushJoin: { + expected: [], + on_demand: true, + runOnMain: true, + execute() { return {}; }, + submit() {}, + }, + ``` + + Reset `_flushCount` and `_flushStats` in the constructor (and on + each rebuild in serve mode). + +- **`render:i.submit()` change.** Drops the `html` and `offlineHtml` + assignments from the delta merge. Only merges `renderedContent` and + `offlineMisses`. + +- **`prepPageDirs` extension.** Currently creates directories under + `destRoot` only. Extended to also create the corresponding + directories under `destRoot + '-offline'` so `flushPages` workers + can write without per-file mkdir. + +- **`write` → `writeAssets`.** The current `write` task is renamed. + Its `expected` changes from + `["renderJoin", "scssJoin", "mermaid", "prepPageDirs", + "highlighterInit"]` to + `["flushJoin", "scssJoin", "mermaid", "prepPageDirs", + "highlighterInit"]`. It no longer calls `writePages` --- only + `copyTheme`, `copyStaticFiles`, and `writeGeneratedAssets`. + +- **`writeOffline` change.** The `writeOfflinePages` call is removed + from `writeOffline`'s `Promise.all` orchestration. With + `offlineHtml` no longer merged back into `state.pages` (it stays on + the worker), the `precomputed` branch at `offline.mjs:235` would + filter to zero pages --- the offline page HTML is already on disk + from the flush. `writeOffline` keeps: JS patches + (`just-the-docs.js`), `search-data.js` wrapper, redirect-stub + rewrites, theme-asset copy, static-file copy. + + The `deps.counters.html` and `deps.counters.unresolved` tallies + that `writeOfflinePages` used to maintain move to `flushPages` + return stats, aggregated on main via `flushJoin`. + + `writeOffline` gains a direct dependency on `writeAssets` (in + addition to `writeAux`): it reads `_site/assets/js/just-the-docs.js` + to produce the patched offline copy, and walks `_site/assets/` to + mirror theme files with CSS URL rewrites. Today this is covered + transitively through `writeAux → write`; with `write` split into + `flushPages` + `writeAssets`, the edge must be explicit. + +- **`searchData`, `writePdf` dependency change.** Both switch from + `renderJoin` to `flushJoin` in their `expected` arrays. No other + changes --- `searchData` reads `renderedContent` from + `state.pages`; `writePdf` reads `renderedContent` via + `bookData._chapters` refs. Neither reads `html`. + +- **`GANTT_SECTION` map** (`tbdocs.mjs`). Remove `write`, add + `writeAssets: "Write"`, `flushJoin: "Write"`. Per-worker + `flushPages` timings are recorded by `_onPerWorkerTiming` under + `"flushPages:wN"` with `ganttSection: "Write"` (update the + hard-coded `"Boot"` section in `_onPerWorkerTiming` to read from + taskMeta, or special-case `flushPages`). + +- **Summary output** (`tbdocs.mjs`). The build summary currently + reports write stats from the `write` task result. With the split, + page-write stats come from the aggregated `flushPages` result + (stored under `"flushPages"` in `this.results` by the counter + activation), and asset-write stats come from `writeAssets`. + +**Files touched:** + +| File | Changes | +|---|---| +| `cpu-worker.mjs` | `_pageStash` module var + reset in `msg.init`; render handler: push to stash, drop `html`/`offlineHtml` from return; new `flushPages` handler; `perWorkerTiming` message gains `output` field for flush stats | +| `tbdocs.mjs` | New `flushPages` + `flushJoin` task defs; rename `write` → `writeAssets` and strip `writePages` call; update `expected` arrays (`searchData`, `writePdf`, `writeOffline`); `GANTT_SECTION` map; summary output | +| `scheduler.mjs` | `_flushCount` / `_flushStats` counter; `_onPerWorkerTiming` extension for flush activation + stats aggregation; Gantt section for per-worker flush timings | +| `sab-scheduler.mjs` | `idlePriority` in `taskMeta` wire-up (minor --- already passed to workers, just needs to be populated from the task def); `flushJoin` SAB slot allocation | +| `write.mjs` | `preparePageDirs` extended to create dirs under `destRoot + '-offline'` | +| `offline.mjs` | Remove `writeOfflinePages` call from `writeOffline`'s `Promise.all`; adjust counter reporting | + +**Expected savings.** + +Three sources: +1. **Wall-clock overlap.** Pages start hitting disk as soon as a worker + exhausts its render tasks, instead of waiting for `renderJoin` + + main-thread `writePages`. The overlap between the render tail and + the first flush is the direct win. +2. **Reduced structured-clone cost.** `html` (~5--15 KB per page) and + `offlineHtml` (similar size) no longer cross the worker boundary. + On ~1,080 pages across ~16 chunks, this drops the total return-path + clone volume substantially. +3. **Decoupled `writeAssets`.** Theme and static-file copies no longer + wait for rendered pages. They start as soon as `flushJoin` + their + seed deps are ready. + +Conservative estimate: 50--100 ms wall-clock on a 16-core machine. +The main value is architectural --- the write pipeline is no longer a +main-thread bottleneck gated on the render barrier. + +**Verification.** `build.bat && check.bat` clean. The timing summary +should show: +- Per-worker `flushPages` timings appearing after the last `render:i` + per worker, with earlier workers' flushes overlapping later workers' + render tails. +- `writeAssets` replacing `write` in the Write section, with a shorter + duration (no `writePages`). +- `writeOffline` duration dropping (no per-page HTML writing). +- `renderJoin` absent from the summary. + ## Notify protocol Workers sleep on a single generation-counter slot (`views.notify`) diff --git a/builder/cpu-worker.mjs b/builder/cpu-worker.mjs index 74c23898..9656bbdd 100644 --- a/builder/cpu-worker.mjs +++ b/builder/cpu-worker.mjs @@ -4,6 +4,8 @@ // no main-thread round-trip for worker-to-worker transitions. // See PLAN-sab-pull-scheduler.md §Worker pull loop. +import { promises as fsP } from "node:fs"; +import path from "node:path"; import { parentPort, workerData } from "node:worker_threads"; import { compileLightScss, compileDarkScss } from "./scss.mjs"; import { regenerateMermaid } from "./mermaid.mjs"; @@ -74,6 +76,30 @@ const handlers = { return {}; }, + async flushPages() { + let written = 0, offlineWritten = 0, offlineMisses = 0; + if (!ctx.opts.dryRun) { + const items = _pageStash; + let next = 0; + const limit = 64; + const workers = Array.from({ length: Math.min(limit, items.length || 1) }, async () => { + while (next < items.length) { + const p = items[next++]; + await fsP.writeFile(path.join(ctx.destRoot, p.destPath), p.html, "utf8"); + written++; + if (p.offlineHtml !== undefined) { + await fsP.writeFile(path.join(ctx.destRoot + "-offline", p.destPath), p.offlineHtml, "utf8"); + offlineWritten++; + } + offlineMisses += p.offlineMisses ?? 0; + } + }); + await Promise.all(workers); + } + _pageStash = []; + return { written, offlineWritten, offlineMisses }; + }, + async scssLight() { const workerStart = Date.now(); const scssLightResult = await compileLightScss(ctx.srcRoot); @@ -145,21 +171,33 @@ const handlers = { } } + // Stash writable pages for flushPages (avoids structured-clone of + // html + offlineHtml across the worker boundary). + for (const p of chunk) { + if (p.html !== undefined) { + _pageStash.push({ + destPath: p.destPath, + html: p.html, + offlineHtml: p.offlineHtml, + offlineMisses: p.offlineMisses, + }); + } + } + const workerEnd = Date.now(); return { workerStart, workerEnd, pages: chunk.map(p => ({ destPath: p.destPath, renderedContent: p.renderedContent, - html: p.html, - offlineHtml: p.offlineHtml, offlineMisses: p.offlineMisses, })), }; }, }; -let _renderEnv = null; +let _renderEnv = null; +let _pageStash = []; // ── Message handler (init + renderData only) ──────────────────────────────── @@ -172,6 +210,7 @@ parentPort.on("message", (msg) => { _chunkDataSAB = null; _sharedSAB = null; _renderEnv = null; + _pageStash = []; pullLoop(); return; } @@ -186,12 +225,16 @@ parentPort.on("message", (msg) => { function findIdleTask(views, lane) { const count = Atomics.load(views.taskCount, 0); + let bestIdx = -1; + let bestPri = Infinity; for (let i = 0; i < count; i++) { if (!(Atomics.load(views.flags, i) & F_RUN_WHEN_IDLE)) continue; + const meta = taskMeta[i]; + const pri = meta?.idlePriority ?? 0; + if (pri >= bestPri) continue; if (Atomics.load(views.flags, i) & F_UNIQUE_PER_WORKER) { if (Atomics.load(views.perWorkerDone, i * MAX_LANES + lane) !== 0) continue; - const meta = taskMeta[i]; if (meta?.expectedIdxs) { let skip = false; for (const predIdx of meta.expectedIdxs) { @@ -206,14 +249,20 @@ function findIdleTask(views, lane) { } if (skip) continue; } - return i; + bestIdx = i; + bestPri = pri; } else { if (Atomics.load(views.status, i) !== READY) continue; - if (Atomics.compareExchange(views.status, i, READY, CLAIMED) === READY) - return i; + bestIdx = i; + bestPri = pri; } } - return -1; + // For non-unique_per_worker tasks, CAS-claim at the end. + if (bestIdx !== -1 && !(Atomics.load(views.flags, bestIdx) & F_UNIQUE_PER_WORKER)) { + if (Atomics.compareExchange(views.status, bestIdx, READY, CLAIMED) !== READY) + return -1; + } + return bestIdx; } // ── Pull loop ─────────────────────────────────────────────────────────────── @@ -230,8 +279,9 @@ async function pullLoop() { if (idleTask !== -1) { const idleMeta = taskMeta[idleTask]; const idleStart = Date.now(); + let idleResult; try { - await handlers[idleMeta.handler](); + idleResult = await handlers[idleMeta.handler](); } catch (err) { parentPort.postMessage({ taskFailed: idleTask, message: err.message, stack: err.stack }); return; @@ -242,6 +292,7 @@ async function pullLoop() { taskName: idleMeta.name, timing: { start: idleStart, end: Date.now() }, lane: myLane, + output: idleResult, }); continue; } @@ -294,8 +345,9 @@ async function pullLoop() { const nestedMeta = taskMeta[nestedUnsatisfied]; const nestedStart = Date.now(); + let nestedResult; try { - await handlers[nestedMeta.handler](); + nestedResult = await handlers[nestedMeta.handler](); } catch (err) { parentPort.postMessage({ taskFailed: nestedUnsatisfied, message: err.message, stack: err.stack }); return; @@ -306,6 +358,7 @@ async function pullLoop() { taskName: nestedMeta.name, timing: { start: nestedStart, end: Date.now() }, lane: myLane, + output: nestedResult, }); continue; } @@ -338,8 +391,9 @@ async function pullLoop() { Atomics.notify(views.notify, 0, 1); const depStart = Date.now(); + let depResult; try { - await handlers[depMeta.handler](); + depResult = await handlers[depMeta.handler](); } catch (err) { parentPort.postMessage({ taskFailed: unsatisfied, message: err.message, stack: err.stack }); return; @@ -351,6 +405,7 @@ async function pullLoop() { taskName: depMeta.name, timing: { start: depStart, end: Date.now() }, lane: myLane, + output: depResult, }); continue; } diff --git a/builder/offline.mjs b/builder/offline.mjs index 3d1b87d2..5a635f15 100644 --- a/builder/offline.mjs +++ b/builder/offline.mjs @@ -158,24 +158,23 @@ export async function writeOffline(pages, staticFiles, site, destRoot, { auxStat // synchronous prefix (e.g. nav-block cache pre-pass) immediately, // before any await happens. Folding that work into the same lap as // the parallel await keeps the timing report honest. + // Offline page HTML is already on disk from per-worker flushPages. + // Only redirect stubs, static files, and theme assets are written here. if (subT) { const t0Pages = Date.now(); - let dPages = 0, dRedirects = 0, dStatics = 0, dThemes = 0; + let dRedirects = 0, dStatics = 0, dThemes = 0; const branches = [ - writeOfflinePages(pages, deps, { precomputed }).then(() => { dPages = Date.now() - t0Pages; }), writeOfflineRedirects(auxStats?.redirects?.stubs ?? [], deps).then(() => { dRedirects = Date.now() - t0Pages; }), copyOfflineStatics(staticFiles, deps).then(() => { dStatics = Date.now() - t0Pages; }), copyOfflineThemeAssets(deps).then(() => { dThemes = Date.now() - t0Pages; }), ]; await Promise.all(branches); subT.lap("parallel"); - console.log(` offline.pages (concurrent): ${dPages} ms`); console.log(` offline.redirects (concurrent): ${dRedirects} ms`); console.log(` offline.statics (concurrent): ${dStatics} ms`); console.log(` offline.themeAssets (concurrent): ${dThemes} ms`); } else { await Promise.all([ - writeOfflinePages(pages, deps, { precomputed }), writeOfflineRedirects(auxStats?.redirects?.stubs ?? [], deps), copyOfflineStatics(staticFiles, deps), copyOfflineThemeAssets(deps), diff --git a/builder/sab-scheduler.mjs b/builder/sab-scheduler.mjs index e88cf506..53e80193 100644 --- a/builder/sab-scheduler.mjs +++ b/builder/sab-scheduler.mjs @@ -94,17 +94,15 @@ export function allocSchedulerSAB(taskDefs, workerCount) { const DYNAMIC_BASE = idxToName.length; const maxChunks = workerCount * SLICES_PER_WORKER; - const RENDER_JOIN_IDX = DYNAMIC_BASE + maxChunks; - // Pre-reserve render chunk and renderJoin slots + // Pre-reserve render chunk slots (no renderJoin — flushJoin is a + // static task activated by counter, not by SAB dep counts). for (let i = 0; i < maxChunks; i++) { nameToIdx.set(`render:${i}`, DYNAMIC_BASE + i); idxToName.push(`render:${i}`); } - nameToIdx.set("renderJoin", RENDER_JOIN_IDX); - idxToName.push("renderJoin"); - const totalTasks = RENDER_JOIN_IDX + 1; + const totalTasks = DYNAMIC_BASE + maxChunks; if (totalTasks > MAX_TASKS) throw new Error(`${totalTasks} tasks exceeds MAX_TASKS (${MAX_TASKS})`); @@ -211,6 +209,7 @@ export function allocSchedulerSAB(taskDefs, workerCount) { perWorkerDeps, expectedIdxs, name, + idlePriority: def.idle_priority ?? 0, }; } @@ -223,18 +222,10 @@ export function allocSchedulerSAB(taskDefs, workerCount) { }; } - taskMeta[RENDER_JOIN_IDX] = { - handler: "renderJoin", - perWorkerDeps: [], - expectedIdxs: [], - name: "renderJoin", - }; - const idMapping = { nameToIdx, idxToName, DYNAMIC_BASE, - RENDER_JOIN_IDX, maxRenderChunks: maxChunks, }; @@ -305,21 +296,6 @@ export function advanceFirstReady(views, taskIdx) { const encoder = new TextEncoder(); -export function registerDynamicRender(views, idMapping, numChunks) { - let edgePos = Atomics.load(views.edgeCount, 0); - for (let i = 0; i < numChunks; i++) { - const idx = idMapping.DYNAMIC_BASE + i; - views.succOffset[idx] = edgePos; - views.succCount[idx] = 1; - if (edgePos >= MAX_EDGES) - throw new Error(`Edge count exceeds MAX_EDGES (${MAX_EDGES})`); - views.succList[edgePos++] = idMapping.RENDER_JOIN_IDX; - } - Atomics.store(views.edgeCount, 0, edgePos); - Atomics.store(views.depCount, idMapping.RENDER_JOIN_IDX, numChunks); - views.flags[idMapping.RENDER_JOIN_IDX] = F_RUN_ON_MAIN; -} - export function packChunkData(chunks, views) { const buffers = chunks.map(c => encoder.encode(JSON.stringify(c))); const totalBytes = buffers.reduce((sum, b) => sum + b.byteLength, 0); diff --git a/builder/scheduler.mjs b/builder/scheduler.mjs index b2c24805..91a34e4e 100644 --- a/builder/scheduler.mjs +++ b/builder/scheduler.mjs @@ -7,7 +7,7 @@ import pc from "picocolors"; import { onTaskDone as sabOnTaskDone, - registerDynamicRender, packChunkData, activateRenderTasks, + packChunkData, activateRenderTasks, READY, CLAIMED, DONE, F_RUN_ON_MAIN, } from "./sab-scheduler.mjs"; @@ -19,7 +19,7 @@ export class SharedState { } export class Scheduler { - constructor({ pool, tasks, views, idMapping }) { + constructor({ pool, tasks, views, idMapping, ganttSections }) { this.pool = pool; this.tasks = new Map(Object.entries(tasks)); this.results = new Map(); // task name → output @@ -27,14 +27,19 @@ export class Scheduler { this.state = new SharedState(); this._views = views; this._idMapping = idMapping; + this._ganttSections = ganttSections ?? {}; // Count non-on_demand static tasks for completion detection. - // Dynamic tasks (render:i, renderJoin) are added via addDynamicTasks(). + // Dynamic tasks (render:i) are added via addDynamicTasks(). this._remaining = 0; for (const [, def] of this.tasks) { if (!def.on_demand) this._remaining++; } + // flushPages counter: activated by per-worker timing messages. + this._flushCount = 0; + this._flushStats = { written: 0, offlineWritten: 0, offlineMisses: 0 }; + this._scanning = false; this._mainScanScheduled = false; this._finished = false; @@ -45,7 +50,6 @@ export class Scheduler { // Write render SAB entries, broadcast chunk data, and activate tasks. dispatchRender(chunks, sharedSAB) { const N = chunks.length; - registerDynamicRender(this._views, this._idMapping, N); const chunkDataSAB = packChunkData(chunks, this._views); this.pool.broadcastRenderData(chunkDataSAB, sharedSAB); activateRenderTasks(this._views, this._idMapping, N); @@ -201,14 +205,31 @@ export class Scheduler { this._abort(name, err); } - _onPerWorkerTiming({ taskName, timing, lane }) { + _onPerWorkerTiming({ taskName, timing, lane, output }) { this.timings.set(`${taskName}:w${lane}`, { start: timing.start, end: timing.end, workerStart: timing.start, workerEnd: timing.end, lane, consolidate: true, - ganttSection: "Boot", + ganttSection: this._ganttSections[taskName] ?? "Boot", }); + + if (taskName === "flushPages" && output) { + this._flushCount++; + this._flushStats.written += output.written ?? 0; + this._flushStats.offlineWritten += output.offlineWritten ?? 0; + this._flushStats.offlineMisses += output.offlineMisses ?? 0; + + if (this._flushCount === this._ctx.workerCount) { + this.results.set("flushPages", { ...this._flushStats }); + this.addDynamicTasks(1); + const joinIdx = this._idMapping.nameToIdx.get("flushJoin"); + if (joinIdx != null) { + Atomics.store(this._views.status, joinIdx, READY); + this._scheduleMainScan(); + } + } + } } _onMainTaskReady() { diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 515313d7..cfcc5a68 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -123,9 +123,11 @@ export function makeTimer() { // highlighterInit), the main-thread spine (config → discover → nav (sidebar) + buildInit (chrome); // nav + buildInit → dispatch; config → loadData; discover → markdownInit → seo; // deriveRedirects off discover; deriveSitemap + resolveBookChapters + prepDest deferred to dispatch), -// the render fan-out (dispatch → render:0..N → renderJoin), and write/post-write tasks -// (prepDest → prepPageDirs; renderJoin + prepPageDirs → write + searchData; -// write + searchData → writeAux → writeOffline; renderJoin + mermaid → writePdf) +// the render fan-out (dispatch → render:0..N, each worker stashes html locally), +// the per-worker flush (prepPageDirs → flushPages [per worker] → flushJoin [counter barrier]), +// and write/post-write tasks +// (flushJoin + prepPageDirs → writeAssets + searchData; +// writeAssets + searchData → writeAux → writeOffline; flushJoin + mermaid → writePdf) // are scheduler tasks. // runBuild() constructs the pool + scheduler, awaits start(), logs the // summary, and returns. @@ -225,7 +227,9 @@ const TASKS = { runOnMain: true, async execute(_, ctx, state) { if (ctx.opts.dryRun) return {}; - await preparePageDirs(state.pages, state.staticFiles, ctx.destRoot); + const skipOffline = ctx.opts.skipOffline ?? (state.site.config.also_build_offline === false); + const offlineRoot = skipOffline ? null : ctx.destRoot + "-offline"; + await preparePageDirs(state.pages, state.staticFiles, ctx.destRoot, offlineRoot); return {}; }, submit() {}, @@ -273,6 +277,30 @@ const TASKS = { submit() {}, }, + // On-demand per-worker page flush: writes stashed html + offlineHtml + // to disk, overlapping I/O with the render tail. Activated by + // prepPageDirs (directories must exist). Counter-based flushJoin + // barrier fires when all workers complete. + flushPages: { + expected: ["prepPageDirs"], + on_demand: true, + unique_per_worker: true, + run_when_idle: true, + idle_priority: 1, + handler: "flushPages", + submit() {}, + }, + + // Barrier: all workers have flushed their stashed pages to disk. + // Activated by counter in _onPerWorkerTiming, not by SAB dep counts. + flushJoin: { + expected: [], + on_demand: true, + runOnMain: true, + execute() { return {}; }, + submit() {}, + }, + // ── Main-thread spine ───────────────────────────────────────────────────── discover: { @@ -438,16 +466,6 @@ const TASKS = { submit(out, _state, scheduler) { const N = out.chunks.length; - // Register task definitions for dynamic tasks. The SAB already has - // pre-reserved slots and dep-count edges for render:i → renderJoin; - // registerDynamicRender (called by dispatchRender) populates them. - scheduler.tasks.set("renderJoin", { - expected: Array.from({ length: N }, (_, i) => `render:${i}`), - runOnMain: true, - execute() { return {}; }, - submit() {}, - }); - for (let i = 0; i < N; i++) { scheduler.tasks.set(`render:${i}`, { expected: [], @@ -459,28 +477,24 @@ const TASKS = { const p = state.pageByDest.get(r.destPath); if (!p) continue; p.renderedContent = r.renderedContent; - if (r.html !== undefined) p.html = r.html; - if (r.offlineHtml !== undefined) p.offlineHtml = r.offlineHtml; if (r.offlineMisses !== undefined) p.offlineMisses = r.offlineMisses; } }, }); } - scheduler.addDynamicTasks(N + 1); + scheduler.addDynamicTasks(N); scheduler.dispatchRender(out.chunks, out.sharedSAB); }, }, // ── Write and post-write tasks ───────────────────────────────────────────── - // Materialise pages + static files + generated CSS to _site/. - // Waits for renderJoin (pages rendered + templated), scss (generated CSS), - // mermaid (SVG descriptors appended to state.staticFiles by mermaid.submit), - // prepPageDirs (page output directories pre-created), and highlighterInit - // (highlight CSS). - write: { - expected: ["renderJoin", "scssJoin", "mermaid", "prepPageDirs", "highlighterInit"], + // Materialise static files + generated CSS to _site/. Page HTML is + // written by per-worker flushPages; this task handles theme, static + // files, and generated CSS only. + writeAssets: { + expected: ["flushJoin", "scssJoin", "mermaid", "prepPageDirs", "highlighterInit"], runOnMain: true, async execute({ scssJoin: { scssResult }, mermaid: { mermaidStats }, highlighterInit: _highlightSignal }, ctx, state) { void mermaidStats; // dependency signal only; append already happened in mermaid.submit @@ -497,16 +511,17 @@ const TASKS = { dryRun: ctx.opts.dryRun, generatedAssets, baseurl: String(state.site.config.baseurl || ""), + skipPages: true, }); }, submit() {}, }, - // Write search-data.json. Depends on renderJoin (pages have + // Write search-data.json. Depends on flushJoin (pages have // renderedContent) and prepDest (_site/ exists). Result passes // through to writeAux so its search.json field reaches writeOffline. searchData: { - expected: ["renderJoin", "prepDest"], + expected: ["flushJoin", "prepDest"], runOnMain: true, async execute(_, ctx, state) { if (ctx.opts.dryRun) return { entries: 0, json: "" }; @@ -515,11 +530,11 @@ const TASKS = { submit() {}, }, - // Write redirect stubs + sitemap/robots. Waits for write (pages on disk), - // searchData, deriveRedirects, and deriveSitemap. + // Write redirect stubs + sitemap/robots. Waits for writeAssets (theme on + // disk), searchData, deriveRedirects, and deriveSitemap. // Passes searchStats through to writeOffline (for search-data.js). writeAux: { - expected: ["write", "searchData", "deriveRedirects", "deriveSitemap"], + expected: ["writeAssets", "searchData", "deriveRedirects", "deriveSitemap"], runOnMain: true, async execute({ searchData: searchStats, deriveRedirects: { stubs }, deriveSitemap: { urls } }, ctx, state) { if (ctx.opts.dryRun) return { redirectStats: null, sitemapStats: null, searchStats }; @@ -532,10 +547,11 @@ const TASKS = { submit() {}, }, - // Produce _site-offline/. Runs in parallel with writePdf on the main thread; - // the gain is interleaved async I/O windows. + // Produce _site-offline/. Depends on writeAux (redirects + sitemap on + // disk) and writeAssets (theme assets on disk for the CSS-rewrite + + // JTD-patch passes). Offline page HTML is already on disk from flushPages. writeOffline: { - expected: ["writeAux"], + expected: ["writeAux", "writeAssets"], runOnMain: true, async execute({ writeAux: { redirectStats, sitemapStats, searchStats } }, ctx, state) { const skipOffline = ctx.opts.skipOffline ?? (state.site.config.also_build_offline === false); @@ -551,13 +567,13 @@ const TASKS = { submit() { /* terminal */ }, }, - // Produce _site-pdf/. Depends on renderJoin (pages have renderedContent), + // Produce _site-pdf/. Depends on flushJoin (pages have renderedContent), // resolveBookChapters (bookData._chapters refs into state.pages), and // mermaid (SVG descriptors in staticFiles). Sources CSS directly: // tb-highlight.css from state.site.highlighter, print.css from staticFiles. - // Runs in parallel with write → searchData → writeAux → writeOffline. + // Runs in parallel with writeAssets → searchData → writeAux → writeOffline. writePdf: { - expected: ["renderJoin", "mermaid", "resolveBookChapters"], + expected: ["flushJoin", "mermaid", "resolveBookChapters"], runOnMain: true, async execute(_, ctx, state) { const skipPdf = ctx.opts.skipPdf ?? (state.site.config.also_build_pdf === false); @@ -589,7 +605,8 @@ const GANTT_SECTION = { seo: "Spine", resolveBookChapters: "Spine", deriveRedirects: "Spine", deriveSitemap: "Spine", dispatch: "Render", prepDest: "Render", prepPageDirs: "Render", - write: "Write", searchData: "Write", writeAux: "Write", writeOffline: "Write", writePdf: "Write", + flushPages: "Write", flushJoin: "Write", + writeAssets: "Write", searchData: "Write", writeAux: "Write", writeOffline: "Write", writePdf: "Write", }; const GANTT_SECTION_ORDER = ["Seeds", "Spine", "Render", "Write"]; @@ -642,7 +659,7 @@ export async function runBuild(opts) { verifySchedulerSAB(TASKS, views, idMapping); const pool = new WorkerPool(workerCount, CPU_WORKER_URL); - const scheduler = new Scheduler({ pool, tasks: TASKS, views, idMapping }); + const scheduler = new Scheduler({ pool, tasks: TASKS, views, idMapping, ganttSections: GANTT_SECTION }); pool.onWorkerDone = (msg) => scheduler._onWorkerDone(msg); pool.onWorkerError = (msg) => scheduler._onWorkerError(msg); @@ -672,15 +689,16 @@ export async function runBuild(opts) { if (mermaidStats.failed > 0) process.exitCode = 1; if (scssResult.failed) process.exitCode = 1; - const writeStats = results.get("write"); + const flushStats = results.get("flushPages"); + const assetStats = results.get("writeAssets"); const auxResult = results.get("writeAux"); const offlineResult = results.get("writeOffline"); const pdfResult = results.get("writePdf"); console.log(`Done in ${pc.bold(pc.green(`${Date.now() - buildStart}ms`))}: ${pages.length} pages, ${staticFiles.length} static files`); console.log(` ${pc.bold("wrote:")} -> ${pc.cyan(destRoot)}`); - console.log(` ${writeStats.pages.written} pages (${writeStats.pages.skipped} skipped), ` + - `${writeStats.theme.copied} theme assets, ${writeStats.staticFiles.copied} static files`); + console.log(` ${flushStats.written} pages, ` + + `${assetStats.theme.copied} theme assets, ${assetStats.staticFiles.copied} static files`); if (auxResult?.redirectStats) { console.log(` ${pc.bold("aux:")} ${auxResult.redirectStats.written} redirect stubs, ` + `${auxResult.sitemapStats.entries} sitemap entries, ` + @@ -688,11 +706,11 @@ export async function runBuild(opts) { } if (offlineResult) { console.log(` ${pc.bold("offline:")} -> ${pc.cyan(`${destRoot}-offline`)}`); - console.log(` ${offlineResult.html} HTML, ${offlineResult.css} CSS, ` + + console.log(` ${flushStats.offlineWritten} HTML, ${offlineResult.css} CSS, ` + `${offlineResult.redirects} redirect stubs, ` + `${offlineResult.statics + offlineResult.assets} assets, ` + `${offlineResult.excluded} excluded ` + - `(${offlineResult.unresolved} unresolved)`); + `(${flushStats.offlineMisses} unresolved)`); if (opts.profileOffline && offlineResult.subT) { console.log(` ${pc.bold("offline:")} ${offlineResult.subT.summary()}`); } diff --git a/builder/write.mjs b/builder/write.mjs index 8ddb0a98..3c33d0d5 100644 --- a/builder/write.mjs +++ b/builder/write.mjs @@ -35,7 +35,7 @@ export const WRITE_LIMIT = LIMIT; const mkdirCache = new Set(); const mkdirInflight = new Map(); -export async function writePhase(pages, staticFiles, { destRoot, dryRun = false, generatedAssets = [], baseurl = "" } = {}) { +export async function writePhase(pages, staticFiles, { destRoot, dryRun = false, generatedAssets = [], baseurl = "", skipPages = false } = {}) { if (!destRoot) { throw new Error("writePhase requires a destRoot"); } @@ -61,7 +61,7 @@ export async function writePhase(pages, staticFiles, { destRoot, dryRun = false, // generated asset ever land at the same rel as a vendored file, the // generated content wins. No such collision exists today. const [pagesStats, themeStats, staticStats] = await Promise.all([ - writePages(pages, destRoot, LIMIT), + skipPages ? { written: 0, skipped: 0 } : writePages(pages, destRoot, LIMIT), copyTheme(BUILDER_ASSETS, destRoot, LIMIT, baseurl), copyStaticFiles(staticFiles, destRoot, LIMIT, baseurl), ]); @@ -109,11 +109,14 @@ export async function prepareDestinations(roots, dryRun) { // Pre-create all page output directories so writePages can skip mkdir // entirely. Runs as a separate task concurrently with render workers. -export async function preparePageDirs(pages, staticFiles, destRoot) { +export async function preparePageDirs(pages, staticFiles, destRoot, offlineRoot) { assertNoDestinationCollisions(pages, staticFiles); const dirs = new Set(); for (const page of pages) { - if (page.destPath) dirs.add(path.dirname(path.join(destRoot, page.destPath))); + if (page.destPath) { + dirs.add(path.dirname(path.join(destRoot, page.destPath))); + if (offlineRoot) dirs.add(path.dirname(path.join(offlineRoot, page.destPath))); + } } await Promise.all([...dirs].map(d => fs.mkdir(d, { recursive: true }))); } From 30d13d68c60a68bca51cd577ed48c2245cd6a651 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Mon, 1 Jun 2026 19:28:38 +0200 Subject: [PATCH 56/72] Gantt formatting changes. --- builder/cpu-worker.mjs | 4 ++-- builder/gantt.mjs | 2 +- builder/offline.mjs | 2 +- builder/scheduler.mjs | 6 +++--- builder/tbdocs.mjs | 14 +++++++------- 5 files changed, 14 insertions(+), 14 deletions(-) diff --git a/builder/cpu-worker.mjs b/builder/cpu-worker.mjs index 9656bbdd..75edcc0f 100644 --- a/builder/cpu-worker.mjs +++ b/builder/cpu-worker.mjs @@ -76,7 +76,7 @@ const handlers = { return {}; }, - async flushPages() { + async flush() { let written = 0, offlineWritten = 0, offlineMisses = 0; if (!ctx.opts.dryRun) { const items = _pageStash; @@ -171,7 +171,7 @@ const handlers = { } } - // Stash writable pages for flushPages (avoids structured-clone of + // Stash writable pages for flush (avoids structured-clone of // html + offlineHtml across the worker boundary). for (const p of chunk) { if (p.html !== undefined) { diff --git a/builder/gantt.mjs b/builder/gantt.mjs index b5fc29d5..95edb592 100644 --- a/builder/gantt.mjs +++ b/builder/gantt.mjs @@ -7,7 +7,7 @@ const COLORS = { Render: { light: "#b09cd8", dark: "#8066a8" }, Write: { light: "#e8a756", dark: "#c08030" }, Boot: { light: "#e57373", dark: "#c62828" }, - Cold: { light: "#6eb5d9", dark: "#3c7db0" }, + Cold: { light: "#5b7fb5", dark: "#2c4a7c" }, Env: { light: "#e8a756", dark: "#c08030" }, Other: { light: "#bbb", dark: "#666" }, }; diff --git a/builder/offline.mjs b/builder/offline.mjs index 5a635f15..4c9221e2 100644 --- a/builder/offline.mjs +++ b/builder/offline.mjs @@ -158,7 +158,7 @@ export async function writeOffline(pages, staticFiles, site, destRoot, { auxStat // synchronous prefix (e.g. nav-block cache pre-pass) immediately, // before any await happens. Folding that work into the same lap as // the parallel await keeps the timing report honest. - // Offline page HTML is already on disk from per-worker flushPages. + // Offline page HTML is already on disk from per-worker flush. // Only redirect stubs, static files, and theme assets are written here. if (subT) { const t0Pages = Date.now(); diff --git a/builder/scheduler.mjs b/builder/scheduler.mjs index 91a34e4e..44997d16 100644 --- a/builder/scheduler.mjs +++ b/builder/scheduler.mjs @@ -36,7 +36,7 @@ export class Scheduler { if (!def.on_demand) this._remaining++; } - // flushPages counter: activated by per-worker timing messages. + // flush counter: activated by per-worker timing messages. this._flushCount = 0; this._flushStats = { written: 0, offlineWritten: 0, offlineMisses: 0 }; @@ -214,14 +214,14 @@ export class Scheduler { ganttSection: this._ganttSections[taskName] ?? "Boot", }); - if (taskName === "flushPages" && output) { + if (taskName === "flush" && output) { this._flushCount++; this._flushStats.written += output.written ?? 0; this._flushStats.offlineWritten += output.offlineWritten ?? 0; this._flushStats.offlineMisses += output.offlineMisses ?? 0; if (this._flushCount === this._ctx.workerCount) { - this.results.set("flushPages", { ...this._flushStats }); + this.results.set("flush", { ...this._flushStats }); this.addDynamicTasks(1); const joinIdx = this._idMapping.nameToIdx.get("flushJoin"); if (joinIdx != null) { diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index cfcc5a68..31dd13dd 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -124,7 +124,7 @@ export function makeTimer() { // nav + buildInit → dispatch; config → loadData; discover → markdownInit → seo; // deriveRedirects off discover; deriveSitemap + resolveBookChapters + prepDest deferred to dispatch), // the render fan-out (dispatch → render:0..N, each worker stashes html locally), -// the per-worker flush (prepPageDirs → flushPages [per worker] → flushJoin [counter barrier]), +// the per-worker flush (prepPageDirs → flush [per worker] → flushJoin [counter barrier]), // and write/post-write tasks // (flushJoin + prepPageDirs → writeAssets + searchData; // writeAssets + searchData → writeAux → writeOffline; flushJoin + mermaid → writePdf) @@ -281,13 +281,13 @@ const TASKS = { // to disk, overlapping I/O with the render tail. Activated by // prepPageDirs (directories must exist). Counter-based flushJoin // barrier fires when all workers complete. - flushPages: { + flush: { expected: ["prepPageDirs"], on_demand: true, unique_per_worker: true, run_when_idle: true, idle_priority: 1, - handler: "flushPages", + handler: "flush", submit() {}, }, @@ -491,7 +491,7 @@ const TASKS = { // ── Write and post-write tasks ───────────────────────────────────────────── // Materialise static files + generated CSS to _site/. Page HTML is - // written by per-worker flushPages; this task handles theme, static + // written by per-worker flush; this task handles theme, static // files, and generated CSS only. writeAssets: { expected: ["flushJoin", "scssJoin", "mermaid", "prepPageDirs", "highlighterInit"], @@ -549,7 +549,7 @@ const TASKS = { // Produce _site-offline/. Depends on writeAux (redirects + sitemap on // disk) and writeAssets (theme assets on disk for the CSS-rewrite + - // JTD-patch passes). Offline page HTML is already on disk from flushPages. + // JTD-patch passes). Offline page HTML is already on disk from flush. writeOffline: { expected: ["writeAux", "writeAssets"], runOnMain: true, @@ -605,7 +605,7 @@ const GANTT_SECTION = { seo: "Spine", resolveBookChapters: "Spine", deriveRedirects: "Spine", deriveSitemap: "Spine", dispatch: "Render", prepDest: "Render", prepPageDirs: "Render", - flushPages: "Write", flushJoin: "Write", + flush: "Write", flushJoin: "Write", writeAssets: "Write", searchData: "Write", writeAux: "Write", writeOffline: "Write", writePdf: "Write", }; const GANTT_SECTION_ORDER = ["Seeds", "Spine", "Render", "Write"]; @@ -689,7 +689,7 @@ export async function runBuild(opts) { if (mermaidStats.failed > 0) process.exitCode = 1; if (scssResult.failed) process.exitCode = 1; - const flushStats = results.get("flushPages"); + const flushStats = results.get("flush"); const assetStats = results.get("writeAssets"); const auxResult = results.get("writeAux"); const offlineResult = results.get("writeOffline"); From 64a6c190ffd0af9ca3f363b7c6ea527e37752e78 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Mon, 1 Jun 2026 19:34:30 +0200 Subject: [PATCH 57/72] Move the flushJoin dependency to writeAux. Having the dependency in writeAssets unnecessarily delayed the write. --- builder/tbdocs.mjs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 31dd13dd..ad0be439 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -494,7 +494,7 @@ const TASKS = { // written by per-worker flush; this task handles theme, static // files, and generated CSS only. writeAssets: { - expected: ["flushJoin", "scssJoin", "mermaid", "prepPageDirs", "highlighterInit"], + expected: ["scssJoin", "mermaid", "prepPageDirs", "highlighterInit"], runOnMain: true, async execute({ scssJoin: { scssResult }, mermaid: { mermaidStats }, highlighterInit: _highlightSignal }, ctx, state) { void mermaidStats; // dependency signal only; append already happened in mermaid.submit @@ -534,7 +534,7 @@ const TASKS = { // disk), searchData, deriveRedirects, and deriveSitemap. // Passes searchStats through to writeOffline (for search-data.js). writeAux: { - expected: ["writeAssets", "searchData", "deriveRedirects", "deriveSitemap"], + expected: ["writeAssets", "searchData", "flushJoin", "deriveRedirects", "deriveSitemap"], runOnMain: true, async execute({ searchData: searchStats, deriveRedirects: { stubs }, deriveSitemap: { urls } }, ctx, state) { if (ctx.opts.dryRun) return { redirectStats: null, sitemapStats: null, searchStats }; From 05b3aa8e3309dda512eddb08973baf600a322ec3 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Mon, 1 Jun 2026 19:47:28 +0200 Subject: [PATCH 58/72] Plan phase 13 to capture uniform task timing. --- builder/PLAN-sab-pull-scheduler.md | 230 +++++++++++++++++++++++++++++ 1 file changed, 230 insertions(+) diff --git a/builder/PLAN-sab-pull-scheduler.md b/builder/PLAN-sab-pull-scheduler.md index be667747..c4c36310 100644 --- a/builder/PLAN-sab-pull-scheduler.md +++ b/builder/PLAN-sab-pull-scheduler.md @@ -1710,6 +1710,236 @@ should show: - `writeOffline` duration dropping (no per-page HTML writing). - `renderJoin` absent from the summary. +### Phase 13: Uniform task timing (t0 / t1 / t3) + +**Suggested model:** Opus. + +**Motivation.** The Gantt chart shows a gap between `dispatch` ending +and the first `renderEnvInit` starting. Investigation reveals the +gap is real but uncharted: `dispatch.submit()` runs *after* the +execute timing window closes (t1), and it does substantial work --- +`packChunkData`, `broadcastRenderData`, `activateRenderTasks` --- that +is invisible in the timeline. The same blind spot exists for every +main-thread task: `submit()` is never timed. + +A secondary problem: every worker handler (`scssLight`, `scssDark`, +`mermaid`, `buildInfo`, `render`) redundantly captures its own +`workerStart` / `workerEnd` timestamps, near-identical to the pull +loop's `start` / `end` that already wrap the same call. The timing +should live in one place --- the runner (pull loop on the worker side, +`_executeMainTask` / `_onWorkerDone` on main) --- with handlers +unaware of timing. + +**Design.** Two boundary timestamps per task, with a third on +main-thread tasks only: + +| Timestamp | Main-thread tasks | Worker tasks | +|-----------|-------------------|--------------| +| t0 | before `execute()` | before `handler()` | +| t1 | after `execute()` | after `handler()` | +| *(t2)* | *(reserved, unused)* | *(reserved, unused)* | +| t3 | after `submit()` | *(not captured --- see below)* | + +t2 is reserved for a future split (e.g. timing `results.set()` +separately) but not captured now. + +**Why t3 is main-thread only.** For main-thread tasks, `submit()` +runs between t1 and `sabOnTaskDone` --- it gates successor activation, +so its cost is on the critical path. For worker tasks, the worker +itself calls `onTaskDone` (SAB update) *before* posting the result +message; `submit()` runs later on the main thread when +`_onWorkerDone` processes the message, off the critical path. +Worker-side `postMessage` cost (structured-clone serialization) +cannot be included in the message it is measuring; the gap between +t1 and the main thread's receipt time serves as a proxy if needed. + +The `workerStart` / `workerEnd` fields on handler return values are +removed. Handlers no longer capture timing; the runner does it +uniformly. + +#### Changes to `scheduler.mjs` + +1. **`_executeMainTask`.** Capture t3 after `def.submit()`: + + ```js + const t0 = Date.now(); + output = await def.execute(inputs, this._ctx, this.state); + const t1 = Date.now(); + + // ... results.set, submit ... + def.submit(output, this.state, this); + const t3 = Date.now(); + + const timing = { start: t0, end: t1, t3 }; + ``` + + The existing `start` / `end` semantics are preserved (t0 / t1) for + backwards compatibility with the summary and Gantt. `t3` is a new + optional field. + +2. **`_onWorkerDone`.** Drop the `output.workerStart` / + `output.workerEnd` extraction. Populate `workerStart` / + `workerEnd` from the timing message (t0 / t1): + + ```js + const t = { start: timing.start, end: timing.end }; + if (lane != null) { + t.workerStart = timing.start; + t.workerEnd = timing.end; + t.lane = lane; + } + ``` + + No t3 --- worker `submit()` is off the critical path. + +3. **`_onPerWorkerTiming`.** Per-worker tasks (`warmInit`, + `renderEnvInit`, `flushPages`) arrive via this path. The runner + sends `{ start, end }` (t0 / t1). No change to the timing + fields stored --- `workerStart` / `workerEnd` are already + populated from `timing.start` / `timing.end`: + + ```js + _onPerWorkerTiming({ taskName, timing, lane, output }) { + this.timings.set(`${taskName}:w${lane}`, { + start: timing.start, end: timing.end, + workerStart: timing.start, workerEnd: timing.end, + lane, + consolidate: true, + ganttSection: this._ganttSections[taskName] ?? "Boot", + }); + ``` + + This is unchanged from the current code. + +4. **Drop `workerStart` / `workerEnd` extraction from output.** The + lines in `_executeMainTask` and `_onWorkerDone` that read + `output.workerStart` / `output.workerEnd` are deleted. The runner + provides these timestamps; handlers no longer carry them. + +#### Changes to `cpu-worker.mjs` + +1. **Pull loop --- regular task path** (~line 427). Capture t0 / t1 + as explicit variables. The timing object sent to main carries + `{ start, end }` = t0 / t1: + + ```js + const t0 = Date.now(); + result = await handler(taskIdx); + const t1 = Date.now(); + + parentPort.postMessage({ + done: taskIdx, + output: result, + timing: { start: t0, end: t1 }, + lane: myLane, + }); + ``` + + This is the same data the current code sends (it evaluates + `Date.now()` inside the postMessage args); the only change is + naming the variable `t1` before the call instead of inlining it. + +2. **Pull loop --- per-worker dep and idle paths.** Three distinct + `perWorkerTiming` send sites need the same t0 / t1 treatment: + + a. **Idle-task completion** (~line 281). Currently: + `timing: { start: idleStart, end: Date.now() }`. + Change to capture t1 before the postMessage: + ```js + const t0 = Date.now(); + idleResult = await handlers[idleMeta.handler](); + const t1 = Date.now(); + Atomics.store(views.perWorkerDone, idleTask * MAX_LANES + myLane, 1); + parentPort.postMessage({ + perWorkerTiming: true, + taskName: idleMeta.name, + timing: { start: t0, end: t1 }, + lane: myLane, + output: idleResult, + }); + ``` + + b. **Nested per-worker dep completion** (~line 347). Currently: + `timing: { start: nestedStart, end: Date.now() }`. + Same pattern --- capture t1, send `{ start: t0, end: t1 }`. + + c. **Direct per-worker dep completion** (~line 393). Currently: + `timing: { start: depStart, end: Date.now() }`. + Same pattern. + +3. **Remove `workerStart` / `workerEnd` from handlers.** Delete the + `workerStart = Date.now()` / `workerEnd: Date.now()` boilerplate + from: `scssLight`, `scssDark`, `mermaid`, `buildInfo`, `render`. + Each handler returns only its domain data (e.g. `{ scssLightResult }`, + `{ buildInfo }`, `{ pages: [...] }`). + +#### Changes to `gantt.mjs` + +1. **Worker lane bars.** Currently uses `t.workerStart` / + `t.workerEnd`. After Phase 13, these are populated from the + runner's t0 / t1 in `_onWorkerDone` and `_onPerWorkerTiming` + (see scheduler changes above), so the Gantt renderer needs no + change for basic rendering. + +2. **Submit / dispatch overlay.** When `t3` is present on a timing + entry, render a second narrower or lighter-shaded rect from + `end` to `t3` on main-thread task bars. This makes the + `dispatch.submit()` cost visible in the Gantt --- the gap that + motivated this phase. Only main-thread tasks carry `t3`, so + worker lane bars are unaffected. + +#### Changes to `groupGanttTimings` (`tbdocs.mjs`) + +Pass through the `t3` field when present: + +```js +const entry = { id, start: start - t0, end: end - t0 }; +if (t3 != null) entry.t3 = t3 - t0; +``` + +The destructuring on line 601 gains `t3`. + +#### Implementation steps + +1. Remove `workerStart` / `workerEnd` from the five worker handlers + (`scssLight`, `scssDark`, `mermaid`, `buildInfo`, `render`). + +2. Update the pull loop's regular-task path (~line 427) to capture + t0 / t1 as named variables. Send `{ start: t0, end: t1 }` in + the timing object. + +3. Update all three per-worker timing send sites in the pull loop: + idle-task completion (~line 281), nested per-worker dep completion + (~line 347), direct per-worker dep completion (~line 393). Each + gets the same t0 / t1 pattern. + +4. In `_executeMainTask`, capture t3 after `submit()`. Store it on + the timing entry. + +5. In `_onWorkerDone`, stop reading `output.workerStart` / + `output.workerEnd`. Populate `workerStart` / `workerEnd` from + the timing message's `start` / `end`. + +6. In `groupGanttTimings`, pass through t3. + +7. In `gantt.mjs`, render the `end`--`t3` overlay rect on + main-thread task bars when `t3` is present. + +**Files touched:** + +| File | Changes | +|---|---| +| `cpu-worker.mjs` | Remove `workerStart` / `workerEnd` from 5 handlers; name t0 / t1 in pull loop regular-task path; same pattern in all 3 per-worker timing send sites | +| `scheduler.mjs` | `_executeMainTask`: capture t3 after submit; `_onWorkerDone`: drop `output.workerStart` extraction, populate from timing message; `_onPerWorkerTiming`: no change | +| `tbdocs.mjs` | `groupGanttTimings`: pass through t3 | +| `gantt.mjs` | Render end--t3 overlay rect on main-thread task bars | + +**Verification.** `build.bat && check.bat` clean. The timing summary +is unchanged (it reads `start` / `end`, which remain t0 / t1). The +Gantt chart shows a visible submit-phase tail on main-thread task +bars --- most notably on `dispatch`, where the end--t3 overlay accounts +for the previously invisible gap before `renderEnvInit`. + ## Notify protocol Workers sleep on a single generation-counter slot (`views.notify`) From be4ffd657c477642032251022485a2c010340dd0 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Mon, 1 Jun 2026 20:08:15 +0200 Subject: [PATCH 59/72] scss task writes combined CSS directly to both site trees. --- builder/offline.mjs | 2 ++ builder/tbdocs.mjs | 65 +++++++++++++++++++++++++++++++++------------ 2 files changed, 50 insertions(+), 17 deletions(-) diff --git a/builder/offline.mjs b/builder/offline.mjs index 4c9221e2..07dddbe3 100644 --- a/builder/offline.mjs +++ b/builder/offline.mjs @@ -313,6 +313,7 @@ async function copyOfflineThemeAssets(deps) { await runLimited(themeEntries, LIMIT, async (e) => { if (e.isJtdJs) return; + if (e.isCombinedCss) return; const relAsset = "assets/" + e.relUnderAssets; if (offlineExcluded(relAsset, deps.excludePatterns)) return; const dest = path.join(offlineRoot, "assets", e.relUnderAssets); @@ -600,6 +601,7 @@ async function collectThemeFiles(themeRoot) { relUnderAssets: childRel, srcAbs: path.join(themeRoot, childRel), isCss: childRel.endsWith(".css"), + isCombinedCss: childRel === "css/just-the-docs-combined.css", isJtdJs: childRel === "js/just-the-docs.js", }); } diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index ad0be439..24d16d69 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -40,7 +40,8 @@ import { writeRedirects, deriveRedirectStubs } from "./redirects.mjs"; import { writeSitemap, deriveSitemapUrls } from "./sitemap.mjs"; import { writeSearchData } from "./search.mjs"; import { writeOffline, enumerateVendoredThemeAssets } from "./offline.mjs"; -import { buildSitePathsSync } from "./offline-rewrite.mjs"; +import { buildSitePathsSync, deriveOfflineCss, + normalizeBaseurl } from "./offline-rewrite.mjs"; import { writePdf } from "./pdf.mjs"; import { packShared } from "./sab-broadcast.mjs"; import { allocSchedulerSAB, verifySchedulerSAB, SLICES_PER_WORKER } from "./sab-scheduler.mjs"; @@ -119,7 +120,7 @@ export function makeTimer() { // ── Task graph ──────────────────────────────────────────────────────────────── // -// Seeds (config, buildInfo → mermaid, scssLight + scssDark → scssJoin, +// Seeds (config, buildInfo → mermaid, scssLight + scssDark → scss, // highlighterInit), the main-thread spine (config → discover → nav (sidebar) + buildInit (chrome); // nav + buildInit → dispatch; config → loadData; discover → markdownInit → seo; // deriveRedirects off discover; deriveSitemap + resolveBookChapters + prepDest deferred to dispatch), @@ -179,15 +180,48 @@ const TASKS = { submit() {}, }, - // Joins the two parallel SCSS results. - scssJoin: { - expected: ["scssLight", "scssDark"], + // Joins the two parallel SCSS results and writes the combined CSS to + // _site/ and _site-offline/. Depends on prepDest so the output dirs + // exist (prepDest cleans them first). + scss: { + expected: ["scssLight", "scssDark", "prepDest"], runOnMain: true, - execute({ scssLight: { scssLightResult }, scssDark: { scssDarkResult } }) { + async execute({ scssLight: { scssLightResult }, scssDark: { scssDarkResult } }, ctx, state) { if (scssLightResult.failed || scssDarkResult.failed) { return { scssResult: { compiled: false, failed: true } }; } - return { scssResult: { compiled: true, css: scssLightResult.css + "\n" + scssDarkResult.css } }; + const combined = scssLightResult.css + "\n" + scssDarkResult.css; + if (ctx.opts.dryRun) { + return { scssResult: { compiled: true, css: combined } }; + } + + const rel = "assets/css/just-the-docs-combined.css"; + const baseurl = String(state.site.config.baseurl || ""); + const online = baseurl + ? combined.replace( + /url\((["']?)\/(?!\/)([^)"']*)\1\)/g, + (_, q, rest) => `url(${q}${baseurl}/${rest}${q})`, + ) + : combined; + const dest = path.join(ctx.destRoot, rel); + await fs.mkdir(path.dirname(dest), { recursive: true }); + await fs.writeFile(dest, online, "utf8"); + + const skipOffline = ctx.opts.skipOffline ?? (state.site.config.also_build_offline === false); + let offlineMisses = 0; + if (!skipOffline) { + const offlineState = { + sitePaths: state.sitePaths, + caches: { rawResolution: new Map(), seg: new Map(), result: new Map() }, + baseurl: normalizeBaseurl(baseurl), + }; + const { css: offlineCss, misses } = deriveOfflineCss(online, rel, offlineState); + offlineMisses = misses; + const offDest = path.join(ctx.destRoot + "-offline", rel); + await fs.mkdir(path.dirname(offDest), { recursive: true }); + await fs.writeFile(offDest, offlineCss, "utf8"); + } + return { scssResult: { compiled: true, css: combined }, offlineMisses }; }, submit() {}, }, @@ -490,22 +524,19 @@ const TASKS = { // ── Write and post-write tasks ───────────────────────────────────────────── - // Materialise static files + generated CSS to _site/. Page HTML is - // written by per-worker flush; this task handles theme, static - // files, and generated CSS only. + // Materialise theme JS, static files, and highlight CSS to _site/. + // Page HTML is written by per-worker flush; combined SCSS is written + // by the scss task. writeAssets: { - expected: ["scssJoin", "mermaid", "prepPageDirs", "highlighterInit"], + expected: ["mermaid", "prepPageDirs", "highlighterInit"], runOnMain: true, - async execute({ scssJoin: { scssResult }, mermaid: { mermaidStats }, highlighterInit: _highlightSignal }, ctx, state) { + async execute({ mermaid: { mermaidStats }, highlighterInit: _highlightSignal }, ctx, state) { void mermaidStats; // dependency signal only; append already happened in mermaid.submit void _highlightSignal; // dependency signal only; highlightCss already written to state.site const generatedAssets = []; if (state.site.highlightCss) { generatedAssets.push({ rel: "assets/css/tb-highlight.css", content: state.site.highlightCss }); } - if (scssResult.compiled) { - generatedAssets.push({ rel: "assets/css/just-the-docs-combined.css", content: scssResult.css }); - } return writePhase(state.pages, state.staticFiles, { destRoot: ctx.destRoot, dryRun: ctx.opts.dryRun, @@ -599,7 +630,7 @@ function chunkPages(pages, workers) { // ── Gantt chart ─────────────────────────────────────────────────────────────── const GANTT_SECTION = { - config: "Seeds", buildInfo: "Seeds", scssLight: "Seeds", scssDark: "Seeds", mermaid: "Spine", + config: "Seeds", buildInfo: "Seeds", scssLight: "Seeds", scssDark: "Seeds", scss: "Write", mermaid: "Spine", highlighterInit: "Seeds", loadData: "Seeds", discover: "Spine", nav: "Spine", markdownInit: "Spine", buildInit: "Spine", seo: "Spine", resolveBookChapters: "Spine", @@ -679,7 +710,7 @@ export async function runBuild(opts) { const site = scheduler.state.site; const { mermaidStats } = results.get("mermaid"); - const { scssResult } = results.get("scssJoin"); + const { scssResult } = results.get("scss"); if (mermaidStats.regenerated > 0 || mermaidStats.failed > 0) { const parts = [`regenerated ${mermaidStats.regenerated}`]; From a370c5f770ee1330c3746bd263cdadaeb129c007 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Mon, 1 Jun 2026 20:18:30 +0200 Subject: [PATCH 60/72] Reintroduce renderJoin barrier; searchData depends on it, not flushJoin. --- builder/scheduler.mjs | 16 ++++++++++++++++ builder/tbdocs.mjs | 23 ++++++++++++++++++----- 2 files changed, 34 insertions(+), 5 deletions(-) diff --git a/builder/scheduler.mjs b/builder/scheduler.mjs index 44997d16..a47e8bb9 100644 --- a/builder/scheduler.mjs +++ b/builder/scheduler.mjs @@ -36,6 +36,10 @@ export class Scheduler { if (!def.on_demand) this._remaining++; } + // render counter: activated when all render:i tasks complete. + this._renderCount = 0; + this._renderExpected = 0; + // flush counter: activated by per-worker timing messages. this._flushCount = 0; this._flushStats = { written: 0, offlineWritten: 0, offlineMisses: 0 }; @@ -189,6 +193,18 @@ export class Scheduler { // State mutation. if (def) def.submit(output, this.state, this); + // Render counter: activate renderJoin when all render:i complete. + if (name?.startsWith("render:") && this._renderExpected > 0) { + this._renderCount++; + if (this._renderCount === this._renderExpected) { + this.addDynamicTasks(1); + const joinIdx = this._idMapping.nameToIdx.get("renderJoin"); + if (joinIdx != null) { + Atomics.store(this._views.status, joinIdx, READY); + } + } + } + this._remaining--; if (this._remaining === 0) { this._finish(); diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 24d16d69..024b1ac0 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -325,6 +325,17 @@ const TASKS = { submit() {}, }, + // Barrier: all render:i deltas merged into state.pages (renderedContent + // available). Activated by counter in _onWorkerDone. Tasks that only + // need renderedContent (not page HTML on disk) depend on this. + renderJoin: { + expected: [], + on_demand: true, + runOnMain: true, + execute() { return {}; }, + submit() {}, + }, + // Barrier: all workers have flushed their stashed pages to disk. // Activated by counter in _onPerWorkerTiming, not by SAB dep counts. flushJoin: { @@ -499,6 +510,7 @@ const TASKS = { }, submit(out, _state, scheduler) { const N = out.chunks.length; + scheduler._renderExpected = N; for (let i = 0; i < N; i++) { scheduler.tasks.set(`render:${i}`, { @@ -548,11 +560,12 @@ const TASKS = { submit() {}, }, - // Write search-data.json. Depends on flushJoin (pages have - // renderedContent) and prepDest (_site/ exists). Result passes - // through to writeAux so its search.json field reaches writeOffline. + // Write search-data.json. Depends on renderJoin (pages have + // renderedContent in memory) and prepDest (_site/ exists). Result + // passes through to writeAux so its search.json field reaches + // writeOffline. searchData: { - expected: ["flushJoin", "prepDest"], + expected: ["renderJoin", "prepDest"], runOnMain: true, async execute(_, ctx, state) { if (ctx.opts.dryRun) return { entries: 0, json: "" }; @@ -636,7 +649,7 @@ const GANTT_SECTION = { seo: "Spine", resolveBookChapters: "Spine", deriveRedirects: "Spine", deriveSitemap: "Spine", dispatch: "Render", prepDest: "Render", prepPageDirs: "Render", - flush: "Write", flushJoin: "Write", + renderJoin: "Render", flush: "Write", flushJoin: "Write", writeAssets: "Write", searchData: "Write", writeAux: "Write", writeOffline: "Write", writePdf: "Write", }; const GANTT_SECTION_ORDER = ["Seeds", "Spine", "Render", "Write"]; From 46363db68cc612b6a2ddae6d3947f18584dc80b5 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Mon, 1 Jun 2026 21:21:34 +0200 Subject: [PATCH 61/72] Plan a deferred phase 14 for moving writePdf off the main thread. --- builder/PLAN-sab-pull-scheduler.md | 59 ++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/builder/PLAN-sab-pull-scheduler.md b/builder/PLAN-sab-pull-scheduler.md index c4c36310..c35d720f 100644 --- a/builder/PLAN-sab-pull-scheduler.md +++ b/builder/PLAN-sab-pull-scheduler.md @@ -1980,3 +1980,62 @@ knows exactly what it's waiting for, and wakes as soon as the main thread sets the task to DONE and notifies the slot. Before waiting, the worker checks if other non-dependent work is available (one scan); if so, it does that work instead of sleeping. + +### Phase 14: Move `writePdf` to a worker --- DEFERRED + +**Motivation.** `writePdf` depends on `flushJoin` + `mermaid` + +`resolveBookChapters` --- no data dependency on the offline pipeline +(`searchData` → `writeAux` → `writeOffline`). But both `writePdf` and +the offline pipeline are `runOnMain`, so they serialize on the main +thread. On a machine where `flushJoin` lands at ~1.2 s, the ~150 ms +`writePdf` cost is 12 % of the build. + +**Investigation.** Splitting `writePdf` into two main-thread tasks +(`assemblePdf` + `writePdfFiles`) to measure the compute-vs-I/O +breakdown showed: + +``` +assemblePdf=160ms writePdfFiles=30ms +``` + +The compute half (`assembleBook`: chapter walking, body transforms, +href rewriting, html-compress) is ~84 % of the cost. The file-write +half (one `book.html` + 2 CSS files + ~100 images) is only ~30 ms. + +**Consequence.** Moving just the file writes to a worker saves ~30 ms +--- not enough to justify the SAB broadcast plumbing. The real win +requires moving `assembleBook` itself off main. Two blockers prevent +that: + +1. **Live page-object references.** `resolveBookChapters` stores page + objects in `bookData._chapters[]`, `_foreword`, `_landing`. These + are identity-linked to `state.pages` entries where `renderedContent` + was merged after render. Structured clone to a worker breaks the + identity link. + +2. **`site.markdown` dependency.** `assembleBook` → + `renderPartDivider` calls `site.markdown.render()` for part + subtitles and intros. The markdown-it instance is not serializable. + +**Paths forward (not yet committed to):** + +- **Index-based chapter references.** `resolveBookChapters` stores + permalink strings instead of page objects; `assembleBook` builds a + `Map` at the start and resolves refs through it. + Removes blocker 1. + +- **Pre-render book text.** Pre-render subtitles/intros during + `resolveBookChapters` (which runs after `markdownInit`), storing the + HTML on `bookData` entries. `renderPartDivider` reads the + pre-rendered strings instead of calling `site.markdown`. Removes + blocker 2. + +- **Full worker migration.** With both blockers removed, the entire + `writePdf` (compute + I/O) can run on a worker via SAB broadcast of + a page projection (~10 MB: all pages' `permalink`, `navPath`, + `renderedContent`, `frontmatter` subset). Packing cost ~30--50 ms; + net main-thread savings ~100--120 ms. + +Deferred: the refactoring cost is significant for a ~120 ms saving on +a 4 s build. Revisit if the build wall-clock shrinks enough that the +PDF task becomes a larger fraction. From 9b26a533b265f771e8e633b3708d85453983e8d1 Mon Sep 17 00:00:00 2001 From: Kuba Sunderland-Ober Date: Mon, 1 Jun 2026 23:07:46 +0200 Subject: [PATCH 62/72] Add more timing information to the Gannt and make it zoomable/saveable/copyable. --- builder/cpu-worker.mjs | 39 ++++++++++++++++++--------------------- builder/gantt.mjs | 7 +++++++ builder/scheduler.mjs | 22 ++++++++++++---------- builder/tbdocs.mjs | 33 +++++++++++++++++++++++++++++++-- 4 files changed, 68 insertions(+), 33 deletions(-) diff --git a/builder/cpu-worker.mjs b/builder/cpu-worker.mjs index 75edcc0f..3162ff9d 100644 --- a/builder/cpu-worker.mjs +++ b/builder/cpu-worker.mjs @@ -101,31 +101,26 @@ const handlers = { }, async scssLight() { - const workerStart = Date.now(); const scssLightResult = await compileLightScss(ctx.srcRoot); - return { workerStart, workerEnd: Date.now(), scssLightResult }; + return { scssLightResult }; }, async scssDark() { - const workerStart = Date.now(); const scssDarkResult = await compileDarkScss(ctx.srcRoot); - return { workerStart, workerEnd: Date.now(), scssDarkResult }; + return { scssDarkResult }; }, async mermaid() { - const workerStart = Date.now(); const mermaidStats = await regenerateMermaid(ctx.srcRoot); - return { workerStart, workerEnd: Date.now(), mermaidStats }; + return { mermaidStats }; }, async buildInfo() { - const workerStart = Date.now(); const buildInfo = await captureBuildInfo(); - return { workerStart, workerEnd: Date.now(), buildInfo }; + return { buildInfo }; }, async render(taskIdx) { - const workerStart = Date.now(); const chunkIndex = taskIdx - idMapping.DYNAMIC_BASE; const offset = Atomics.load(views.chunkOffset, chunkIndex); @@ -184,9 +179,7 @@ const handlers = { } } - const workerEnd = Date.now(); return { - workerStart, workerEnd, pages: chunk.map(p => ({ destPath: p.destPath, renderedContent: p.renderedContent, @@ -277,8 +270,8 @@ async function pullLoop() { // Speculative: run idle-eligible tasks before sleeping const idleTask = findIdleTask(views, myLane); if (idleTask !== -1) { - const idleMeta = taskMeta[idleTask]; - const idleStart = Date.now(); + const idleMeta = taskMeta[idleTask]; + const t0 = Date.now(); let idleResult; try { idleResult = await handlers[idleMeta.handler](); @@ -286,11 +279,12 @@ async function pullLoop() { parentPort.postMessage({ taskFailed: idleTask, message: err.message, stack: err.stack }); return; } + const t1 = Date.now(); Atomics.store(views.perWorkerDone, idleTask * MAX_LANES + myLane, 1); parentPort.postMessage({ perWorkerTiming: true, taskName: idleMeta.name, - timing: { start: idleStart, end: Date.now() }, + timing: { start: t0, end: t1 }, lane: myLane, output: idleResult, }); @@ -343,8 +337,8 @@ async function pullLoop() { Atomics.add(views.notify, 0, 1); Atomics.notify(views.notify, 0, 1); - const nestedMeta = taskMeta[nestedUnsatisfied]; - const nestedStart = Date.now(); + const nestedMeta = taskMeta[nestedUnsatisfied]; + const t0 = Date.now(); let nestedResult; try { nestedResult = await handlers[nestedMeta.handler](); @@ -352,11 +346,12 @@ async function pullLoop() { parentPort.postMessage({ taskFailed: nestedUnsatisfied, message: err.message, stack: err.stack }); return; } + const t1 = Date.now(); Atomics.store(views.perWorkerDone, nestedUnsatisfied * MAX_LANES + myLane, 1); parentPort.postMessage({ perWorkerTiming: true, taskName: nestedMeta.name, - timing: { start: nestedStart, end: Date.now() }, + timing: { start: t0, end: t1 }, lane: myLane, output: nestedResult, }); @@ -390,7 +385,7 @@ async function pullLoop() { Atomics.add(views.notify, 0, 1); Atomics.notify(views.notify, 0, 1); - const depStart = Date.now(); + const t0 = Date.now(); let depResult; try { depResult = await handlers[depMeta.handler](); @@ -398,12 +393,13 @@ async function pullLoop() { parentPort.postMessage({ taskFailed: unsatisfied, message: err.message, stack: err.stack }); return; } + const t1 = Date.now(); Atomics.store(views.perWorkerDone, unsatisfied * MAX_LANES + myLane, 1); parentPort.postMessage({ perWorkerTiming: true, taskName: depMeta.name, - timing: { start: depStart, end: Date.now() }, + timing: { start: t0, end: t1 }, lane: myLane, output: depResult, }); @@ -424,7 +420,7 @@ async function pullLoop() { return; } - const start = Date.now(); + const t0 = Date.now(); let result; try { result = await handler(taskIdx); @@ -433,6 +429,7 @@ async function pullLoop() { Atomics.store(views.status, taskIdx, 4); // FAILED return; } + const t1 = Date.now(); // Post output BEFORE the SAB update (ordering constraint: the merge // message must arrive on the main thread before any downstream @@ -440,7 +437,7 @@ async function pullLoop() { parentPort.postMessage({ done: taskIdx, output: result, - timing: { start, end: Date.now() }, + timing: { start: t0, end: t1 }, lane: myLane, }); diff --git a/builder/gantt.mjs b/builder/gantt.mjs index 95edb592..04c2ddd8 100644 --- a/builder/gantt.mjs +++ b/builder/gantt.mjs @@ -149,6 +149,13 @@ function renderMainSection(o, section, tasks, y, xOf) { const by = rd(y + (ROW_H - BAR_H) / 2); const ty = rd(y + ROW_H / 2 + 3.5); o.push(``); + if (t.t3 != null) { + const t3x = rd(xOf(t.t3)); + const t3w = rd(Math.max(t3x - (bx + bw), 1)); + const t3h = Math.round(BAR_H / 2); + const t3by = rd(by + BAR_H - t3h); + o.push(``); + } const lbl = taskLabel(t); const textW = lbl.length * CHAR_W; if (textW + BAR_PAD * 2 <= bw) { diff --git a/builder/scheduler.mjs b/builder/scheduler.mjs index a47e8bb9..3db693f3 100644 --- a/builder/scheduler.mjs +++ b/builder/scheduler.mjs @@ -139,19 +139,18 @@ export class Scheduler { if (this._finished) return; const t1 = Date.now(); - // Timing. - const timing = { start: t0, end: t1 }; - if (output?.workerStart != null) { timing.workerStart = output.workerStart; timing.workerEnd = output.workerEnd; } - if (output?.lane != null) timing.lane = output.lane; - if (def.consolidate) timing.consolidate = true; - if (def.ganttSection) timing.ganttSection = def.ganttSection; - this.timings.set(name, timing); - // Store result. this.results.set(name, output); // State mutation. def.submit(output, this.state, this); + const t3 = Date.now(); + + // Timing. + const timing = { start: t0, end: t1, t3 }; + if (def.consolidate) timing.consolidate = true; + if (def.ganttSection) timing.ganttSection = def.ganttSection; + this.timings.set(name, timing); // Update SAB: mark DONE, decrement successor dep counts. const { readyCount } = sabOnTaskDone(views, taskIdx, -1); @@ -181,8 +180,11 @@ export class Scheduler { // Timing. const t = { start: timing.start, end: timing.end }; - if (output?.workerStart != null) { t.workerStart = output.workerStart; t.workerEnd = output.workerEnd; } - if (lane != null) t.lane = lane; + if (lane != null) { + t.workerStart = timing.start; + t.workerEnd = timing.end; + t.lane = lane; + } if (def?.consolidate) t.consolidate = true; if (def?.ganttSection) t.ganttSection = def.ganttSection; this.timings.set(name, t); diff --git a/builder/tbdocs.mjs b/builder/tbdocs.mjs index 024b1ac0..2fe17f89 100644 --- a/builder/tbdocs.mjs +++ b/builder/tbdocs.mjs @@ -659,11 +659,12 @@ function groupGanttTimings(timings) { const t0 = Math.min(...[...timings.values()].map(t => t.start)); const grouped = new Map(GANTT_SECTION_ORDER.map(s => [s, []])); - for (const [id, { start, end, workerStart, workerEnd, lane, consolidate, ganttSection }] of [...timings.entries()].sort((a, b) => a[1].start - b[1].start)) { + for (const [id, { start, end, t3, workerStart, workerEnd, lane, consolidate, ganttSection }] of [...timings.entries()].sort((a, b) => a[1].start - b[1].start)) { if (id.endsWith("Join")) continue; const section = ganttSection ?? GANTT_SECTION[id] ?? "Other"; if (!grouped.has(section)) grouped.set(section, []); const entry = { id, start: start - t0, end: end - t0 }; + if (t3 != null) entry.t3 = t3 - t0; if (workerStart != null) { entry.workerStart = workerStart - t0; entry.workerEnd = workerEnd - t0; } if (lane != null) entry.lane = lane; if (consolidate) entry.consolidate = true; @@ -682,7 +683,35 @@ async function injectGanttChart(pages, destRoot, svgContent) { let html; try { html = await fs.readFile(htmlPath, "utf8"); } catch (e) { if (e.code !== "ENOENT") throw e; continue; } - const patched = html.replace("", svgContent); + const dataUrl = `data:image/svg+xml;charset=utf-8,${encodeURIComponent(svgContent)}`; + const ls = `font-size:0.85em;float:right;margin-left:1em`; + const downloadLink = `Download SVG`; + const copyLink = `Copy SVG`; + const header = `
${downloadLink}${copyLink}
`; + const zoomScript = [ + ``, ``, a ``, ``, and a visible `` fallback for no-script/no-meta-refresh environments. - -**Reads:** `page.frontmatter.redirect_from`, `page.permalink`, `site.config`. -**Writes:** redirect stub HTML files under `/`. - -**All exports** - -| Symbol | Signature | Description | -|---|---|---| -| `writeRedirects` | `(pages, site, destRoot, stubs?) → Promise<{ written }>` | Main entry point. Accepts optional pre-computed `stubs` from `deriveRedirectStubs`. | -| `deriveRedirectStubs` | `(pages, site) → Array<{ from, to, destPath }>` | Pure derivation of the stub list without writing to disk. Exported so `offline.mjs` can read the list without re-running the derivation. | - ---- - -### `sitemap.mjs` +Calls `writePhase(state.pages, state.staticFiles, { destRoot, dryRun, generatedAssets, baseurl, skipPages: true })` from `write.mjs`. Copies vendored theme JS, copies project static files, writes generated CSS (`tb-highlight.css` from `state.site.highlightCss`). **Does not** write page HTML --- the per-chunk `flush:i` tasks already did that. The CSS baseurl rewrite (`url("/path")` → `url("/path")`) applies to both copy paths and to generated assets. -**Entry point** +### `searchData` (main) ```js -writeSitemap(pages: Page[], site: object, destRoot: string, urls?: string[]): Promise<{ entries: number }> +searchData.expected = ["renderJoin", "prepDest"] ``` -When `urls` is provided (pre-computed by the `deriveSitemap` scheduler task), the URL derivation step is skipped. Filters pages by jekyll-sitemap rules (drops `sitemap: false` and `/404.html`), sorts absolute URLs alphabetically for byte-identical re-runs, and emits `sitemap.xml`. Also writes `robots.txt` with a `Sitemap:` reference. +Calls `writeSearchDataFromChunks(state.searchChunks, destRoot)` from `search.mjs`. Flattens the per-chunk entry arrays, renumbers the global `i` index sequentially, writes `assets/js/search-data.json`. Returns `{ entries, json }`. The heavy work (heading split, content sanitisation, URL encoding) ran on the workers; this task only concatenates. -**Reads:** `page.permalink`, `page.frontmatter.sitemap`, `site.config`. -**Writes:** `/sitemap.xml`, `/robots.txt`. +### `writeAux` (main) -**All exports** +```js +writeAux.expected = ["writeAssets", "searchData", "flushJoin", "deriveRedirects", "deriveSitemap"] +``` -| Symbol | Signature | Description | -|---|---|---| -| `writeSitemap` | `(pages, site, destRoot, urls?) → Promise<{ entries }>` | Main entry point. Accepts optional pre-computed `urls` from `deriveSitemapUrls`. | -| `deriveSitemapUrls` | `(pages, site) → string[]` | Returns the sorted list of absolute URLs that would appear in the sitemap, without writing to disk. | -| `extractSitemapUrls` | `(xml: string) → string[]` | Parses an existing `sitemap.xml` string and extracts its `` values. Useful for diffing two builds. | -| `renderRobotsTxt` | `(config: object) → string` | Produces the `robots.txt` content string. | +In parallel: ---- +- `writeRedirects(state.pages, state.site, destRoot, stubs)` from `redirects.mjs` --- one HTML stub per `redirect_from:` entry, each with `` + `` + `` + a noscript `` fallback. +- `writeSitemap(state.pages, state.site, destRoot, urls)` from `sitemap.mjs` --- `sitemap.xml` + `robots.txt` with a `Sitemap:` reference. -### `search.mjs` +Returns `{ redirectStats, sitemapStats, searchStats }` (the search stats pass through from the `searchData` input). -**Entry point** +### `writeOffline` (main, terminal) ```js -writeSearchData(pages: Page[], site: object, destRoot: string): Promise<{ entries: number }> +writeOffline.expected = ["writeAux", "writeAssets"] ``` -Splits each titled, non-`search_exclude` page by headings, emits one search-index entry per heading-bounded section, and writes a Lunr-compatible JSON index. +Calls `writeOffline(state.pages, state.staticFiles, state.site, destRoot, { auxStats, precomputed: true, sitePaths, profileOffline })` from `offline.mjs`. With `precomputed: true`, the per-page HTML rewrite is skipped --- it was already done inside `render:i` and written by `flush:i`. This task handles the cross-cutting work: CSS `url()` rewriting, the `just-the-docs.js` AST patch (`deriveOfflineJtdJs`), the `search-data.js` wrapper (`deriveOfflineSearchDataJs`), theme assets, redirect stubs. Reads `sitePaths` from `state.sitePaths` (computed by `dispatch`). -**Reads:** `page.renderedContent`, `page.frontmatter.title`, `page.frontmatter.search_exclude`, `page.permalink`, `page.seoTitle`, `site.config`. -**Writes:** `/assets/js/search-data.json`. +### `writePdf` (main, terminal) -**All exports** +```js +writePdf.expected = ["flushJoin", "mermaid", "resolveBookChapters"] +``` -| Symbol | Signature | Description | -|---|---|---| -| `writeSearchData` | `(pages, site, destRoot) → Promise<{ entries }>` | Main entry point. | -| `deriveSearchEntries` | `(pages, site) → object[]` | Returns the search-index entry array without writing to disk. | +Calls `writePdf(state.pages, state.staticFiles, state.site, destRoot, { tolerateMissingImages, highlightCss })` from `pdf.mjs`. Internally calls `assembleBook(site, pages)` from `book.mjs` for the `book.html` HTML string, writes `tb-highlight.css` from the highlight string passed in, copies `print.css` via the `staticFiles` inventory, copies every image referenced in `book.html`. Missing images throw by default; `--tolerate-missing-images` downgrades to a warning. --- -## Phase 7: `offline.mjs` +## Module export tables -Mirrors `/` to `-offline/`, rewriting every URL to a page-relative path so the tree opens under `file://`. +The same modules as above, with the full export list per file. -**Entry point** +### `discover.mjs` -```js -writeOffline( - pages: Page[], - staticFiles: StaticFile[], - site: object, - destRoot: string, - { - auxStats?: object, - profileOffline?: boolean, - precomputed?: boolean, - sitePaths?: Set - } -): Promise<{ html, css, redirects, statics, assets, excluded, unresolved }> -``` +| Symbol | Signature | Description | +|---|---|---| +| `discover` | `(srcRoot, ignore) → Promise<{ pages, staticFiles }>` | Walks the source tree, classifies pages vs static files, returns the two sorted arrays. | -Reads every file written by Phases 5 and 6, rewrites absolute URLs to relative paths, and writes to `-offline/`. When `precomputed: true` (set by the scheduler), skips per-page CPU rewriting and writes `page.offlineHtml` directly (I/O only); when absent or false, derives offline HTML on the main thread (legacy path, used by diff tools). When `sitePaths` is provided, skips the `_site/assets/` walk in `buildOfflineState`. Patches `just-the-docs.js` via AST (acorn) to replace `navLink` and `initSearch` with offline-compatible implementations. Writes `search-data.js`, which wraps the search index as a `window.SEARCH_DATA` assignment so offline search works under `file://` (browsers block `XMLHttpRequest` there). `offline_exclude` patterns apply to pages, static files, and theme assets alike; `search-data.json` is listed in `offline_exclude` and is absent from the offline tree --- only the `.js` wrapper is present. +### `nav.mjs` -**Reads:** all files under `` (online tree), `auxStats.redirects` (redirect stub list from Phase 6). -**Writes:** all files to `-offline/`. +| Symbol | Signature | Description | +|---|---|---| +| `computeNav` | `(pages, config) → { navTree }` | Builds the sidebar tree, runs the integrity check (throws on orphan / ambiguous `parent:`), populates `navPath` / `navLevels` / `breadcrumbs` / `children` on each page. | -**All exports** +### `seo.mjs` | Symbol | Signature | Description | |---|---|---| -| `writeOffline` | `(pages, staticFiles, site, destRoot, opts) → Promise` | Main entry point. `opts.precomputed` (bool) switches to I/O-only mode using `page.offlineHtml`; `opts.sitePaths` skips the `_site/assets/` walk. | -| `buildOfflineState` | `(pages, staticFiles, site, destRoot, { stubs?, sitePaths? }) → Promise` | Constructs the state object (site-path set, resolution caches, per-directory nav caches) used by all offline derivation functions. When `sitePaths` is provided, skips the async `_site/assets/` walk. | -| `deriveOfflinePage` | `(page: Page, state: OfflineState) → string` | Rewrites one page's HTML for offline use. Defined in `offline-rewrite.mjs`; re-exported for backward compatibility. | -| `deriveOfflineRedirect` | `(stub, state: OfflineState) → string` | Rewrites a redirect stub's HTML for offline use. Defined in `offline-rewrite.mjs`; re-exported for backward compatibility. | -| `deriveOfflineCss` | `(cssIn: string, themeRel: string, state: OfflineState) → string` | Rewrites `url()` references in a CSS file to page-relative paths. Defined in `offline-rewrite.mjs`; re-exported for backward compatibility. | -| `deriveOfflineJtdJs` | `(src: string) → string` | Patches `just-the-docs.js` via AST: replaces `navLink` and `initSearch` with offline-compatible implementations. A parse failure at build time is a signal that re-extraction produced unreadable source. | -| `deriveOfflineSearchDataJs` | `(jsonBytes: Buffer) → string` | Wraps `search-data.json` as `window.SEARCH_DATA = …` and minifies it. | +| `computeSiteSeo` | `(config, markdown) → { seoSiteTitle, seoLogoUrl }` | Site-level SEO constants. Called by `markdownInit` on main. Requires a built markdown-it instance. | +| `computeChunkSeo` | `(pages, seoSiteTitle, config, markdown) → void` | Per-page SEO (`seoTitle` / `seoFullTitle` / `seoCanonical` / `seoIsHome`). Mutates pages in place. Called by each render worker between `renderPhase` and `templatePhase`. | +| `precomputeSeo` | `(pages, config, markdown) → { seoSiteTitle, seoLogoUrl }` | Convenience wrapper that runs both halves on the main thread. Used by dev tooling. | +| `renderTitle` | `(text, markdown) → string` | Runs one title through `markdownify → strip_html → normalize_whitespace → escape_once`. | +| `stripHtml` | `(s) → string` | Drops `\n` + - ` `; -} // ---------- §6.2 / §6.3 URL helpers -------------------------------------- diff --git a/docs/Documentation/Builder.md b/docs/Documentation/Builder.md index 66f34b2c..9401dca8 100644 --- a/docs/Documentation/Builder.md +++ b/docs/Documentation/Builder.md @@ -80,7 +80,7 @@ Modules grouped by role. Each entry has one line; deep-dive in [Pipeline Stages] | File | Role | |---|---| -| [`mermaid.mjs`](https://github.com/twinbasic/documentation/blob/main/builder/mermaid.mjs) | Regenerates stale `.mmd` → `.svg` via puppeteer + mermaid. One browser launch per build. | +| [`dot.mjs`](https://github.com/twinbasic/documentation/blob/main/builder/dot.mjs) | Regenerates stale `.dot` → `.svg` via the WASM build of Graphviz (`@hpcc-js/wasm-graphviz`). | | [`scss.mjs`](https://github.com/twinbasic/documentation/blob/main/builder/scss.mjs) | Dart Sass over the vendored just-the-docs SCSS. Split across `scssLight` + `scssDark` worker tasks, joined on main. | **Render hot path** @@ -173,9 +173,16 @@ The complete layout, allocation helper, and the `readTaskMeta` / `writeTaskMeta` ## Task DAG by section -The pipeline has 28 named static tasks plus 2N dynamic ones (N render chunks + N flush tasks). The Gantt chart groups them into four sections that also organise the discussion below: **Seeds**, **Spine**, **Render**, **Write**. +The pipeline has 28 named static tasks plus 2N dynamic ones (N render chunks + N flush tasks). The Gantt chart groups them into four sections that also organise the discussion below: -![Task DAG of the SAB pull scheduler, grouped by Gantt section](/assets/images/mmd/scheduler-dag.svg) +- **Seeds**: `buildInfo`, `scssLight`, `scssDark`, `config`, `warmInit`, `highlighterInit`, `discover`, `loadData` +- **Spine**: `nav`, `dot`, `buildInit`, `markdownInit`, `deriveSitemap`, `deriveRedirects`, `resolveBookChapters` +- **Render**: `dispatch`, `prepDest`, `prepPageDirs`, `renderEnvInit`, `render:i`, `renderJoin` +- **Write**: `scss`, `flush:i`, `flushJoin`, `writeAssets`, `searchData`, `writeAux`, `writeOffline`, `writePdf` + +The full task DAG, with every cross-section edge, follows: + +![Task DAG of the SAB pull scheduler](/assets/images/dot/scheduler-dag.svg) **[M]** runs on the main thread; **[W]** runs on a worker. Solid arrows are normal predecessor edges (`expected`); dotted arrows are per-lane dependencies (`perWorkerDeps`) or implicit data dependencies between tasks that share state through `SharedState`. @@ -188,7 +195,7 @@ Seeds have no predecessors and become claimable as soon as the build starts (wit - `scssLight` (worker) --- compiles `just-the-docs-combined.scss` against the light palette. - `scssDark` (worker) --- same against the dark palette. The two halves were one ~700 ms compile in the old design; splitting them saves about 200 ms. - `scss` (main) --- joins both halves, writes the combined CSS to `_site/` and `_site-offline/`. -- `mermaid` (worker) --- regenerates stale `.mmd` → `.svg`. Seed when more than four workers are available; chained after `buildInfo` on small CI runners so it does not contend with `discover` for the I/O window. +- `dot` (worker) --- regenerates stale `.dot` → `.svg` via the WASM build of Graphviz. WASM init (~50 ms) hides behind the main spine; per-diagram render is synchronous after that. - `highlighterInit` (main) --- loads the `Light.theme` + `Dark.theme` palette, emits `tb-highlight.css`. Does not bring up Shiki on main --- workers each init their own. - `warmInit` (worker, `on_demand` + `unique_per_worker` + `run_when_idle` + `survives_reset`) --- per-lane Shiki bootstrap. The flag combination means workers run it during the main-thread spine if they have no other claimable work, every render-worker needs it on its own lane, and in serve mode the per-lane done flag survives across rebuilds so the second build skips warmup entirely. - `prepDest` (main) --- cleans and recreates the three destination trees. Deferred to after `dispatch` so the wipe does not contend with `discover`'s reads. @@ -246,7 +253,7 @@ Once `renderJoin` fires the auxiliary writers can run; once `flushJoin` fires th - `searchData` (main) --- concatenates `state.searchChunks` (already populated by each `render:i`'s `submit()`), renumbers the global `i` index, writes `search-data.json`. The heavy work (heading split, content sanitisation, URL encoding) ran on the workers; this task only consolidates. - `writeAux` (main) --- writes redirect stubs + sitemap + robots.txt. Depends on `writeAssets`, `searchData`, `flushJoin`, `deriveRedirects`, `deriveSitemap`. - `writeOffline` (main) --- produces `_site-offline/`. The per-page offline HTML was already computed inside `render:i` and written by `flush:i`, so this task only handles the cross-cutting work: CSS url() rewriting, the just-the-docs.js AST patch, the `search-data.js` wrapper, theme assets, redirect stubs. -- `writePdf` (main) --- assembles `_site-pdf/book.html` and copies the images it references. Depends on `flushJoin` (so `renderedContent` is filled), `resolveBookChapters` (so `bookData._chapters` is wired), and `mermaid` (so diagram SVGs are in `staticFiles`). +- `writePdf` (main) --- assembles `_site-pdf/book.html` and copies the images it references. Depends on `flushJoin` (so `renderedContent` is filled), `resolveBookChapters` (so `bookData._chapters` is wired), and `dot` (so diagram SVGs are in `staticFiles`). ## What runs where @@ -258,7 +265,7 @@ For a one-page reference, every task and its execution locus: | Seeds | `buildInfo` | worker | Two `git` shell-outs in parallel. | | Seeds | `scssLight`, `scssDark` | workers | Light + dark palettes compile concurrently. | | Seeds | `scss` | main | Joins light + dark; writes online + offline CSS. | -| Seeds | `mermaid` | worker | Seed on > 4 cores; chained after `buildInfo` otherwise. | +| Seeds | `dot` | worker | WASM Graphviz; init hides behind the main spine. | | Seeds | `highlighterInit` | main | Palette CSS only. | | Seeds | `warmInit` | worker (per lane) | Per-worker Shiki bootstrap. `on_demand` + `run_when_idle`. | | Seeds | `prepDest`, `prepPageDirs` | main | Deferred to after `dispatch`. | @@ -284,7 +291,7 @@ The scheduler owns a `SharedState` instance with five fields: | Field | Type | Filled by | |---|---|---| | `pages` | `Page[]` | `discover.submit()`. Never reassigned afterwards --- only mutated in place. | -| `staticFiles` | `StaticFile[]` | `discover.submit()`, plus appends from `mermaid.submit()` for freshly-regenerated SVGs. | +| `staticFiles` | `StaticFile[]` | `discover.submit()`, plus appends from `dot.submit()` for freshly-regenerated SVGs. | | `site` | `object` | Populated progressively by every spine task's `submit()`. | | `pageByDest` | `Map` | `discover.submit()`. Used by render `submit()` to merge deltas into the master `Page` objects. | | `searchChunks` | `Array>` | Pre-allocated to length N by `dispatch.submit()`; each `render:i.submit()` writes one slot. | @@ -341,6 +348,7 @@ A single `package.json` at the repo root carries everything --- the static site ```json { "devDependencies": { + "@hpcc-js/wasm-graphviz": "^1.21", "acorn": "^8.0", "acorn-walk": "^8.0", "fast-glob": "^3.3", @@ -352,21 +360,15 @@ A single `package.json` at the repo root carries everything --- the static site "markdown-it-attrs": "^4.3", "markdown-it-deflist": "^3.0", "markdown-it-footnote": "^4.0", - "mermaid": "11.15.0", "pdf-lib": "1.17.1", "puppeteer": "25.0.4", "sass": "^1.0", "shiki": "^1.0" - }, - "scripts": { - "postinstall": "node builder/scripts/patch-dagre.mjs" } } ``` -No template engine, no framework, no bundler. `acorn` + `acorn-walk` parse the upstream `just-the-docs.js` for the AST-based offline patcher; the `markdown-it-*` packages cover the dialect extensions the legacy parser supported; `shiki` is the syntax highlighter; `mermaid` + `puppeteer` regenerate `.mmd` → `.svg` (one headless Chromium per batch); `sass` is Dart Sass for the SCSS compile. `pdf-lib` + `html-entities` + `htmlparser2` are the PDF renderer's toolchain. The `postinstall` runs `builder/scripts/patch-dagre.mjs`, which rewrites mermaid's bundled dagre adapter --- see [Mermaid Dagre Patches](Fixes/Dagre). - -`mermaid` is **exact-pinned** (`"11.15.0"`, not `"^11.15.0"`). The dagre patches target a chunk filename whose hash component is regenerated on each mermaid release, so a floated range could break the postinstall on a transparent patch bump. +No template engine, no framework, no bundler, no postinstall hooks. `acorn` + `acorn-walk` parse the upstream `just-the-docs.js` for the AST-based offline patcher; the `markdown-it-*` packages cover the dialect extensions the legacy parser supported; `shiki` is the syntax highlighter; `@hpcc-js/wasm-graphviz` is the WASM build of Graphviz that renders `.dot` diagram sources; `sass` is Dart Sass for the SCSS compile. `pdf-lib` + `html-entities` + `htmlparser2` + `puppeteer` are the PDF renderer's toolchain (puppeteer drives headless Chromium for the paged.js layout pass). Node 22+ is required: the SAB scheduler uses `Atomics.wait`, `Atomics.notify`, and `SharedArrayBuffer` --- all baseline in Node 22 without flags. @@ -376,7 +378,7 @@ The site's `/assets/` tree at deploy time is assembled from three sources: | Source on disk | What lives there | Phase that delivers it | |---|---|---| -| `docs/assets/` | Project-owned content: the SCSS entry point, project JS (`theme-switch.js`), hand-written stylesheets (`print.css`, `just-the-docs-head-nav.css`), Mermaid diagrams (`.mmd` sources + `.svg` renders), and any content images contributors add. | Discovered by [`discover.mjs`](https://github.com/twinbasic/documentation/blob/main/builder/discover.mjs), copied by `writeAssets`. | +| `docs/assets/` | Project-owned content: the SCSS entry point, project JS (`theme-switch.js`), hand-written stylesheets (`print.css`, `just-the-docs-head-nav.css`), Graphviz/DOT diagrams (`.dot` sources + `.svg` renders), and any content images contributors add. | Discovered by [`discover.mjs`](https://github.com/twinbasic/documentation/blob/main/builder/discover.mjs), copied by `writeAssets`. | | `builder/vendor/just-the-docs/` | Vendored from the just-the-docs theme (v0.10.1): `_sass/` (the theme's SCSS sources, fed into the compilation) and `assets/js/just-the-docs.js` + `assets/js/vendor/lunr.min.js` (the chrome runtime, copied verbatim). See [`builder/vendor/just-the-docs/README.md`](https://github.com/twinbasic/documentation/blob/main/builder/vendor/just-the-docs/README.md) for the inventory, re-vendoring procedure, and the in-tree patches applied to `just-the-docs.js`. | `_sass/` consumed by [`scss.mjs`](https://github.com/twinbasic/documentation/blob/main/builder/scss.mjs); `assets/` copied by `writeAssets`. | | Generated in-process | `just-the-docs-combined.css` (from `scss.mjs`) and `tb-highlight.css` (from `highlight-theme.mjs`). Neither is committed; both are rebuilt every run. | Written by `scss` (combined CSS) and `writeAssets` (highlight CSS). | @@ -386,10 +388,10 @@ CSS files in either copy path get a baseurl rewrite (`url("/path")` → `url(".svg) + ![Diagram](/assets/images/dot/.svg) -`tbdocs` regenerates each `.svg` from its `.mmd` sibling when the SVG is missing or older than its source --- editing a `.mmd` by one character regenerates the SVG on the next build. Both files belong in git; the `.mmd` is the canonical source, the `.svg` is the build artifact that the browser actually loads. +`tbdocs` regenerates each `.svg` from its `.dot` sibling when the SVG is missing or older than its source --- editing a `.dot` by one character regenerates the SVG on the next build. Both files belong in git; the `.dot` is the canonical source, the `.svg` is the build artifact the browser loads. -The renderer drives `puppeteer` + the `mermaid` package directly (both regular dependencies in the repo-root `package.json`). One headless Chromium covers the whole batch --- previously the project shelled out to `@mermaid-js/mermaid-cli` which forked a fresh node + Chrome process per diagram and shipped its own bundled puppeteer-core. The direct path keeps the dependency tree smaller, removes the per-file process startup overhead, and uses the same Chromium cache as `render-book.mjs`. Two failure modes are handled distinctly: +The renderer drives `@hpcc-js/wasm-graphviz` directly: one WASM module load (~50 ms) covers the whole batch, then each diagram is a synchronous `gv.dot(src)` call. No headless browser, no in-tree patches, no Chromium dependency for diagrams. Two failure modes are handled distinctly: -- **Setup failures** (no puppeteer, no Chrome, no mermaid) emit a one-line warning, retain the existing on-disk SVGs, and let the build exit 0 --- a fresh checkout without `npm install` or a sandbox without Chromium doesn't break unrelated work. -- **Content failures** (broken `.mmd` syntax, render exception) emit the parser error verbatim, leave that diagram's previous SVG in place, continue rendering the rest of the batch, and flip `process.exitCode = 1` so CI catches the bad diagram. +- **Setup failures** (`@hpcc-js/wasm-graphviz` not installed, WASM load fails) emit a one-line warning, retain the existing on-disk SVGs, and let the build exit 0 --- a fresh checkout without `npm install` still builds against the committed SVGs. +- **Content failures** (broken DOT syntax, render throws) emit the error verbatim, leave that diagram's previous SVG in place, continue rendering the rest of the batch, and flip `process.exitCode = 1` so CI catches the bad diagram. -In serve mode the watcher ignores writes to `assets/images/mmd/*.svg`. The `.mmd` is the source of truth; the `.svg` is the build artifact mermaid emits back under `srcRoot`. Without the filter, each `.mmd` edit would fire two rebuilds (one on the edit, one on the SVG write) and the browser would reload twice for one user change. +In serve mode the watcher ignores writes to `assets/images/dot/*.svg`. The `.dot` is the source of truth; the `.svg` is the build artifact the renderer emits back under `srcRoot`. Without the filter, each `.dot` edit would fire two rebuilds (one on the edit, one on the SVG write) and the browser would reload twice for one user change. ## Deploying to docs.twinbasic.com diff --git a/docs/Documentation/Extending.md b/docs/Documentation/Extending.md index cf574f2c..bd0bed79 100644 --- a/docs/Documentation/Extending.md +++ b/docs/Documentation/Extending.md @@ -69,7 +69,7 @@ Worker handlers live in `cpu-worker.mjs`. Two edits: // builder/sab-scheduler.mjs export const HANDLERS = { warmInit: 0, renderEnvInit: 1, flush: 2, - scssLight: 3, scssDark: 4, mermaid: 5, + scssLight: 3, scssDark: 4, dot: 5, buildInfo: 6, render: 7, myHandler: 8, // ← new }; diff --git a/docs/Documentation/Fixes-Dagre.md b/docs/Documentation/Fixes-Dagre.md deleted file mode 100644 index 0c4e4e2a..00000000 --- a/docs/Documentation/Fixes-Dagre.md +++ /dev/null @@ -1,174 +0,0 @@ ---- -title: Mermaid Dagre Patches -parent: Library Patches -grand_parent: Documentation Development -nav_order: 3 -permalink: /Documentation/Development/Fixes/Dagre ---- - -# Mermaid Dagre Patches -{: .no_toc } - -`node_modules/mermaid/dist/chunks/mermaid.esm/dagre-ZXKKJJHT.mjs` is mermaid's adapter between the flowchart parser and dagre, the layered-graph layout algorithm. Five patches are applied to it by `builder/scripts/patch-dagre.mjs`, wired in as an npm `postinstall` hook on the repo-root `package.json` so a fresh `npm install` re-applies them automatically. This page documents each patch --- what dagre does upstream, why it broke the build-pipeline diagrams, and what was changed. - -The patches all target the same bundled file. Mermaid (pinned at exactly 11.15.0 to keep the fingerprint hash in the chunk filename stable) ships dagre inlined into `dagre-ZXKKJJHT.mjs` and the npm bundle's load path imports the chunk directly, so patching the original `dagre-d3-es` source under `node_modules/dagre-d3-es/` has no effect at runtime. - -* TOC goes here -{:toc} - -## Cross-cluster edges in any subgraph (Patch A) - -**Problem.** Mermaid's `extractor()` extracts a subgraph into its own layout pass --- but only when the cluster has no edges crossing its boundary. Clusters with a cross-boundary edge stay in the parent graph as compound nodes. Two consequences: - -- A `direction LR` (or `RL`) subgraph with external connections silently ignores its per-cluster `rankdir` and renders top-to-bottom like the parent. -- A direction-less subgraph with external connections crashes dagre's downstream rank step outright with `Cannot set properties of undefined (setting 'rank')`. - -The first case is the layout problem behind the [build-pipeline diagram](../assets/images/mmd/build-phases.svg). `row1` and `row2` are LR subgraphs, but `row1`'s last node connects to `row2`'s first node, which gives both rows external connections, so neither was extracted and both rendered as vertical stacks. The second case is what the [scheduler-dag diagram](../assets/images/mmd/scheduler-dag.svg) hits whenever any of its four subgraphs is left direction-less. - -**Fix.** Extend the else-branch in `extractor()` so any cluster with children and external connections is extracted into its own sub-graph. The sub-graph's direction is taken from the cluster's explicit `clusterData.dir` if set, otherwise inherited from the parent graph's `rankdir`. Before `copy()` moves the children out, every edge that crosses the cluster boundary is rerouted to use the cluster placeholder node itself; the original endpoint node IDs are preserved on the rerouted edge as `_patchOrigV` and `_patchOrigW` so Patch B can fix up the rendered path later. - -```js -for (const { e: _e, child: _c, other: _other } of _patchEdges) { - const _eData = graph.edge(_e); - if (!_eData._patchOrigV) _eData._patchOrigV = _e.v; - if (!_eData._patchOrigW) _eData._patchOrigW = _e.w; - graph.removeEdge(_e.v, _e.w, _e.name); - if (_e.v === _c) { - graph.setEdge(node, _other, _eData, _e.name); - } else { - graph.setEdge(_other, node, _eData, _e.name); - } -} -const _gs = graph.graph(); -const _dir = clusterDb.get(node)?.clusterData?.dir || _gs.rankdir; -``` - -After the rerouting, the parent graph's cross-cluster edge is `Cluster → ExternalCluster`, and the sub-graph carries only its internal edges. Dagre then lays out the parent (top-to-bottom for a `flowchart TB`) and each sub-graph (`LR` if declared, `TB` if not) as separate passes. - -> [!NOTE] -> Patch V1 required an explicit `direction` on the cluster --- direction-less clusters with cross-boundary edges were left untouched and would crash dagre downstream. The V2 form (current) extends extraction to all clusters with cross-cluster edges, defaulting to the parent's `rankdir` when no explicit direction is set. `patch-dagre.mjs` migrates a V1-patched dagre source to V2 in-place when re-run. - -## Cross-cluster edge endpoints (Patch B) - -**Problem.** Once Patch A reroutes a cross-cluster edge from `Child → ExternalNode` to `Cluster → ExternalCluster`, dagre lays it out as a straight stub between the two cluster bounding-box centres. Visually that's a short vertical line in the gap between the two rows of boxes --- the original source and destination nodes are nowhere in sight. - -**Fix.** Inside `recursiveRender()`, after the parent-graph layout positions the cluster boxes, the cross-cluster edge's waypoints are replaced with an L-shape routing computed from the original endpoint nodes' actual rendered positions: - -```js -const _sx = (_srcC.x - _srcC.width / 2 - _srcMx) + _srcN.x; -const _dx = (_dstC.x - _dstC.width / 2 - _dstMx) + _dstN.x; -const _srcEdgeY = (_srcC.y - _srcC.height / 2 - _srcMy) + _srcN.y + _srcN.height / 2; -const _dstEdgeY = (_dstC.y - _dstC.height / 2 - _dstMy) + _dstN.y - _dstN.height / 2; -const _srcBot = _srcC.y + _srcC.height / 2; -const _dstTop = _dstC.y - _dstC.height / 2; -const _gapY = (_srcBot + _dstTop) / 2; -edge.points = [ - { x: _sx, y: _srcEdgeY }, // exit source on its bottom edge - { x: _sx, y: _srcBot }, // straight down past the source cluster rect - { x: _sx, y: _gapY }, // into the gap between cluster rows - { x: _dx, y: _gapY }, // across the gap to the destination column - { x: _dx, y: _dstTop }, // down to the destination cluster rect - { x: _dx, y: _dstEdgeY } // enter destination on its top edge -]; -``` - -The `_Mx` / `_My` subtractions account for a subtle bookkeeping difference in mermaid's `updateNodeBounds`: it stores the cluster node's `x`/`y` as the bounding-box centre but `width`/`height` as the cluster *rect* dimensions (i.e. without the sub-graph's `marginx`/`marginy`), so `(_srcC.x - _srcC.width/2)` gives the rect left edge, not the SVG group origin. Subtracting the sub-graph margins recovers the true group origin so the absolute coordinate maps back correctly. - -With mermaid's default `curveBasis` interpolator, the six waypoints render as a smooth curve that exits the source node's bottom edge, sweeps across the gap between the two cluster rows, and enters the destination node's top edge. - -## Cross-cluster arrow z-order (Patch C) - -**Problem.** Mermaid renders the top-level SVG children in fixed declaration order: `clusters`, `edgePaths`, `edgeLabels`, `nodes`. The cross-cluster edge's path lives in the top-level `edgePaths` group, rendered *before* `nodes`. The cluster sub-graphs (with their cluster rects) live inside `nodes`. So the cluster rect is painted on top of the cross-cluster arrow. - -**Fix.** After `recursiveRender` finishes inserting all edges in the parent graph, if any of those edges is a cross-cluster edge (carries `_patchOrigV`/`_patchOrigW` from Patch A), the entire top-level `edgePaths` group is moved to the end of the parent via d3's `selection.raise()`: - -```js -if (graph.edges().some(_re => { - const _red = graph.edge(_re); - return _red && _red._patchOrigV && _red._patchOrigW; -})) { - edgePaths.raise(); -} -``` - -Internal edges inside cluster sub-graphs are inside their own SVG groups, so they keep their natural in-group ordering and are unaffected. - -## Edge-less LR subgraphs (Patch D) - -**Problem.** Dagre's `rank` step assigns each node a rank based on the edges between them; in an LR layout the rank becomes the x-coordinate column. When a subgraph's children have no edges between them, dagre puts every node in rank 0, and rank 0 is a single column --- so the children stack vertically regardless of the declared `direction LR`. - -The [`pdf-render-pipeline.mmd` PHASE8 subgraph](../assets/images/mmd/pdf-render-pipeline.svg) runs into this: it lists three sibling functions called from `pdf.mjs`, not a sequence, so there are no arrows between `ASM`, `CSS`, and `IMG`. Without the patch they render in a vertical column. - -**Fix.** Immediately before `layout(graph)` runs inside `recursiveRender`, group every node by its parent and inject layout-only chain edges between consecutive *isolated* siblings --- pairs where neither side has any sibling-to-sibling edge: - -```js -const _patchSiblingMap = new Map(); -for (const _n of graph.nodes()) { - const _p = graph.parent(_n) || "__root__"; - if (!_patchSiblingMap.has(_p)) _patchSiblingMap.set(_p, []); - _patchSiblingMap.get(_p).push(_n); -} -for (const _siblings of _patchSiblingMap.values()) { - if (_siblings.length < 2) continue; - const _siblingSet = new Set(_siblings); - const _isolated = new Set(); - for (const _s of _siblings) { - let _hasSiblingEdge = false; - const _ne = graph.nodeEdges(_s) || []; - for (const _e of _ne) { - const _other = _e.v === _s ? _e.w : _e.v; - if (_siblingSet.has(_other)) { _hasSiblingEdge = true; break; } - } - if (!_hasSiblingEdge) _isolated.add(_s); - } - for (let _pi = 0; _pi < _siblings.length - 1; _pi++) { - const _u = _siblings[_pi]; - const _v = _siblings[_pi + 1]; - if (_isolated.has(_u) && _isolated.has(_v)) { - graph.setEdge(_u, _v, { _patchInvisible: true, weight: 1, minlen: 1 }); - } - } -} -``` - -Two design choices worth calling out: - -- **Group by parent, not by leaf filter.** When `recursiveRender` recurses into a sub-graph it re-adds the parent cluster as a node and reparents the children to it, so `graph.nodes()` at this point returns `[ASM, CSS, IMG, PHASE8]`. Grouping by `graph.parent()` puts the leaves into the `"PHASE8"` group and `PHASE8` itself into the `"__root__"` group, so a child can never get chained to its own parent. (An earlier version used a `children().length === 0` leaf filter; that broke dagre's rank step the moment two compound siblings needed chaining inside a nested subgraph.) -- **Isolated pairs only.** A sibling is "isolated" when none of its edges go to another sibling in the same group. Only pairs where *both* siblings are isolated get a chain edge. This preserves fan-out topologies: in `build-phases.mmd` row3, `P7` and `P8` both have an incoming edge from sibling `P6`, so neither is isolated and `P7 → P8` is not added --- the fan-out stays a fan-out. - -**Behaviour by example.** - -| Subgraph (`direction LR`) | What dagre does without Patch D | What Patch D adds | Result | -|---|---|---|---| -| `A; B; C` (no edges) | All rank 0, single column | `A → B`, `B → C` | Three columns, declaration order | -| `A → B; C; D` | A, C, D in rank 0; B in rank 1 | `C → D` only (A and B are not isolated) | A and B in their own row, C and D in the row below, in two columns | -| `P6 → P7; P6 → P8` | P6 in rank 0; P7, P8 share rank 1 | Nothing (P7 and P8 each have a sibling edge from P6) | Fan-out: P6 in column 0, P7 above P8 in column 1 | - -> [!NOTE] -> The chain reflects the order mermaid parsed the children in. For fine-grained control or complex topologies the author should still write explicit `-->` edges (or `~~~` invisible edges); Patch D only auto-orders strictly-orphan adjacent siblings. - -## Invisible edges at render time (Patch E) - -**Problem.** Patch D's chain edges exist for dagre's benefit only --- they have no visual meaning and would draw as confusing arrows between sibling boxes. - -**Fix.** A guard at the top of the post-layout edges loop in `recursiveRender` skips any edge tagged `_patchInvisible`: - -```js -graph.edges().forEach(function(e) { - const edge = graph.edge(e); - if (edge._patchInvisible) return; - log.info("Edge " + e.v + " -> " + e.w + ": " + JSON.stringify(edge), edge); - edge.points.forEach((point) => point.y += subGraphTitleTotalMargin / 2); - ... -}); -``` - -`processEdges()` runs *before* `layout()` in `recursiveRender`, so the invisible edges injected by Patch D are not yet in the graph when `insertEdgeLabel` iterates --- they only exist between Patch D's setEdge calls and Patch E's skip, with `layout()` in the middle. Patches D and E together detect a topology dagre would mishandle and patch the layout without altering the user's visible diagram. - -## Patch application - -`builder/scripts/patch-dagre.mjs` runs as the repo-root npm `postinstall` hook (single `package.json` at the repo root after the dependency consolidation; there is no per-`builder/` install any more). On a fresh `npm install`, the script applies all five patches in order; on a re-run it detects already-applied patches (by checking for marker strings unique to each one) and skips them. The script also carries migration paths from earlier patch versions: it spots an in-progress upgrade and transforms the prior version's text into the current one rather than failing the `postinstall`. - -The exact-pin on `mermaid` in the root `package.json` keeps the `ZXKKJJHT` fingerprint stable: mermaid regenerates its bundle hashes on each release, so a floated `^11.15.0` could drift the chunk filename on a patch bump and break the postinstall target path. The pin trades the small lockfile churn of manual mermaid bumps for build determinism. - -If mermaid is upgraded to a release that changes the structure of `dagre-ZXKKJJHT.mjs`, the script will fail loudly with `target not found` and the patch text in `patch-dagre.mjs` needs to be regenerated against the new source --- the patches are precise string replacements, not regex matches. diff --git a/docs/Documentation/Fixes.md b/docs/Documentation/Fixes.md index 0d229ba5..b9f65dc3 100644 --- a/docs/Documentation/Fixes.md +++ b/docs/Documentation/Fixes.md @@ -10,10 +10,9 @@ permalink: /Documentation/Development/Fixes # Library Patches {: .no_toc } -Several third-party libraries carry in-tree modifications. `book/lib/paged.browser.js` is a patched copy of paged.js v0.4.3 (MIT); the thirteen `fast-*.mjs` files there are side-effecting shims applied to pdf-lib's live exports before each PDF process phase; and `builder/scripts/patch-dagre.mjs` is a `postinstall` hook that rewrites mermaid's bundled dagre adapter to fix per-cluster layout. This section documents every change: what the upstream behaviour was, why it was unsuitable for the build pipeline, and what was changed. +Two third-party libraries carry in-tree modifications. `book/lib/paged.browser.js` is a patched copy of paged.js v0.4.3 (MIT); the thirteen `fast-*.mjs` files there are side-effecting shims applied to pdf-lib's live exports before each PDF process phase. This section documents every change: what the upstream behaviour was, why it was unsuitable for the build pipeline, and what was changed. ## Sub-pages - [Paged.js Patches](Fixes/PagedJS) --- changes to `book/lib/paged.browser.js`: the synchronous execution chain, hook dispatch fast-paths, DOM lookup optimizations, layout correctness fixes, and miscellaneous headless-specific changes. - [pdf-lib Patches](Fixes/PDFLib) --- the thirteen `fast-*.mjs` shims and `parallel-deflate.mjs` that retune pdf-lib's parser, object model, and serializer for the process phase. -- [Mermaid Dagre Patches](Fixes/Dagre) --- five patches to `node_modules/mermaid/dist/chunks/mermaid.esm/dagre-ZXKKJJHT.mjs` that make `direction LR` subgraphs work correctly when they have cross-cluster edges or no internal edges at all. diff --git a/docs/Documentation/PDF-Generation.md b/docs/Documentation/PDF-Generation.md index 1e08ebf4..29ea1ebc 100644 --- a/docs/Documentation/PDF-Generation.md +++ b/docs/Documentation/PDF-Generation.md @@ -15,7 +15,7 @@ Internals of the two-stage PDF pipeline: `tbdocs` Phase 8 assembles a sparse `_s ## Data flow -![PDF render pipeline](/assets/images/mmd/pdf-render-pipeline.svg) +![PDF render pipeline](/assets/images/dot/pdf-render-pipeline.svg) The two stages are decoupled: `tbdocs` builds `_site-pdf/` as part of its normal run; `render-book.mjs` runs only when `book.bat` calls it explicitly. This keeps `puppeteer` and `pdf-lib` --- both large --- out of the site generator's dependency tree. diff --git a/docs/Documentation/Pipeline-Stages.md b/docs/Documentation/Pipeline-Stages.md index 94fddb60..5526d23c 100644 --- a/docs/Documentation/Pipeline-Stages.md +++ b/docs/Documentation/Pipeline-Stages.md @@ -74,7 +74,7 @@ In addition, `SharedState` itself carries two non-`site` fields that downstream ### Static files (`staticFiles[]`) -Also produced by `discover`. Every file that is not a page --- images, fonts, prebuilt CSS/JS, and any `.md`/`.html` file without frontmatter --- becomes a static file object. `mermaid.submit()` appends additional SVG descriptors for any freshly-regenerated diagrams. +Also produced by `discover`. Every file that is not a page --- images, fonts, prebuilt CSS/JS, and any `.md`/`.html` file without frontmatter --- becomes a static file object. `dot.submit()` appends additional SVG descriptors for any freshly-regenerated diagrams. | Field | Type | Description | |---|---|---| @@ -113,7 +113,7 @@ Each worker handler is registered in `HANDLERS` (in `sab-scheduler.mjs`) as a na ```js export const HANDLERS = { warmInit: 0, renderEnvInit: 1, flush: 2, - scssLight: 3, scssDark: 4, mermaid: 5, + scssLight: 3, scssDark: 4, dot: 5, buildInfo: 6, render: 7, }; ``` @@ -154,9 +154,9 @@ scss.expected = ["scssLight", "scssDark", "prepDest"] Joins the two CSS strings, writes `assets/css/just-the-docs-combined.css` to both `_site/` and `_site-offline/` (the offline copy includes the page-relative URL rewrite via `deriveOfflineCss`). Depends on `prepDest` so the destination directories exist. -### `mermaid` (worker) +### `dot` (worker) -Handler calls `regenerateMermaid(srcRoot)`. Walks `/assets/images/mmd/*.mmd`, compares mtimes against `.svg` siblings, drives one puppeteer + mermaid batch over the stale ones. Returns `mermaidStats` (`processed`, `regenerated`, `failed`, `setupSkipped?`, `svgFiles[]`); `submit()` appends new SVG descriptors to `state.staticFiles`. Seed when `os.availableParallelism() > 4`; chained after `buildInfo` otherwise so it does not contend with `discover` on small CI runners. +Handler calls `regenerateDot(srcRoot)`. Walks `/assets/images/dot/*.dot`, compares mtimes against `.svg` siblings, calls `Graphviz.load()` once then `gv.dot(src)` per stale source. Returns `dotStats` (`processed`, `regenerated`, `failed`, `setupSkipped?`, `svgFiles[]`); `submit()` appends new SVG descriptors to `state.staticFiles`. The WASM render is fast (sub-millisecond per diagram after the ~50 ms one-time `Graphviz.load()`); runs on a worker so the init hides behind the main spine. ### `highlighterInit` (main) @@ -214,7 +214,7 @@ buildInit.expected = ["discover"] buildInit.execute() → { initData } ``` -Calls `buildInitConfig(state.site)` from `template.mjs`. Pre-renders the config-only chrome (SVG sprites, header, search footer, mermaid bootstrap script, favicon, GA). Runs in parallel with `nav`; `dispatch` merges the two outputs. +Calls `buildInitConfig(state.site)` from `template.mjs`. Pre-renders the config-only chrome (SVG sprites, header, search footer, favicon, GA). Runs in parallel with `nav`; `dispatch` merges the two outputs. ### `markdownInit` (main) @@ -262,7 +262,7 @@ Calls `resolveBookChapters(state.site.bookData, state.pages)` from `book.mjs`. M ```js dispatch.expected = [ - "nav", "buildInit", "buildInfo", "mermaid", + "nav", "buildInit", "buildInfo", "dot", "deriveRedirects", "markdownInit", ] dispatch.execute() → { chunks, sharedSAB } @@ -349,7 +349,7 @@ Main-thread tasks that materialise the rest of the output after the render fan-o ### `writeAssets` (main) ```js -writeAssets.expected = ["mermaid", "prepPageDirs", "highlighterInit"] +writeAssets.expected = ["dot", "prepPageDirs", "highlighterInit"] ``` Calls `writePhase(state.pages, state.staticFiles, { destRoot, dryRun, generatedAssets, baseurl, skipPages: true })` from `write.mjs`. Copies vendored theme JS, copies project static files, writes generated CSS (`tb-highlight.css` from `state.site.highlightCss`). **Does not** write page HTML --- the per-chunk `flush:i` tasks already did that. The CSS baseurl rewrite (`url("/path")` → `url("/path")`) applies to both copy paths and to generated assets. @@ -386,7 +386,7 @@ Calls `writeOffline(state.pages, state.staticFiles, state.site, destRoot, { auxS ### `writePdf` (main, terminal) ```js -writePdf.expected = ["flushJoin", "mermaid", "resolveBookChapters"] +writePdf.expected = ["flushJoin", "dot", "resolveBookChapters"] ``` Calls `writePdf(state.pages, state.staticFiles, state.site, destRoot, { tolerateMissingImages, highlightCss })` from `pdf.mjs`. Internally calls `assembleBook(site, pages)` from `book.mjs` for the `book.html` HTML string, writes `tb-highlight.css` from the highlight string passed in, copies `print.css` via the `staticFiles` inventory, copies every image referenced in `book.html`. Missing images throw by default; `--tolerate-missing-images` downgrades to a warning. @@ -445,11 +445,11 @@ The same modules as above, with the full export list per file. |---|---|---| | `loadData` | `(srcRoot) → Promise` | Returns `{ book: }`, or `{}` when `_book.yml` is absent. | -### `mermaid.mjs` +### `dot.mjs` | Symbol | Signature | Description | |---|---|---| -| `regenerateMermaid` | `(srcRoot) → Promise<{ processed, regenerated, failed, setupSkipped?, svgFiles }>` | Regenerates stale `.mmd` → `.svg` via puppeteer + mermaid. `svgFiles` is the appendable static-file descriptor list. | +| `regenerateDot` | `(srcRoot) → Promise<{ processed, regenerated, failed, setupSkipped?, svgFiles }>` | Regenerates stale `.dot` → `.svg` via the WASM build of Graphviz (`@hpcc-js/wasm-graphviz`). `svgFiles` is the appendable static-file descriptor list. | ### `scss.mjs` @@ -487,7 +487,7 @@ The same modules as above, with the full export list per file. | Symbol | Signature | Description | |---|---|---| | `templatePhase` | `(pages, site, initData) → Promise` | Wraps each page's `renderedContent` in the just-the-docs layout, runs `compressHtml`, stores the result in `page.html`. Skips `layout: book-combined`. | -| `buildInitConfig` | `(site) → object` | Pre-renders the config-only chrome (SVG sprites, header, search footer, mermaid script, favicon, GA). Called by the `buildInit` task. | +| `buildInitConfig` | `(site) → object` | Pre-renders the config-only chrome (SVG sprites, header, search footer, favicon, GA). Called by the `buildInit` task. | | `buildInitFn` | (alias of internal `buildInit`) | Available for harnesses; combines `buildInitConfig` + `renderSidebar` in one call. | | `renderSidebar` | `(site) → string` | Pre-renders the sidebar HTML. Called by the `nav` task; the output is folded into the shared payload by `dispatch`. | | `navActivationCss` | `(page) → string` | Per-page `

writeAssets +
searchData +
writeAux

writeOffline → _site-offline/
writePdf → _site-pdf/
(also _site/ from flush)

render:0..N-1
renderPhase + computeChunkSeo +
templatePhase + offline rewrite +
deriveSearchEntries

flush:0..N-1
per-lane pinned;
writes page HTML

Seeds
config + buildInfo +
scssLight/Dark + mermaid +
highlighterInit + warmInit

Spine
discover → nav + buildInit +
markdownInit + deriveRedirects +
loadData; resolveBookChapters

dispatch
fan-out point

 \ No newline at end of file diff --git a/docs/assets/images/mmd/pdf-render-pipeline.mmd b/docs/assets/images/mmd/pdf-render-pipeline.mmd deleted file mode 100644 index 297dca3c..00000000 --- a/docs/assets/images/mmd/pdf-render-pipeline.mmd +++ /dev/null @@ -1,22 +0,0 @@ -%%{init: {"themeVariables": {"lineColor": "#7a8090"}}}%% -flowchart TD - subgraph PHASE8 ["tbdocs writePdf (pdf.mjs + book.mjs)"] - direction LR - ASM["assembleBook()
combine chapter HTML"] - CSS["copyPdfCss()
print.css + tb-highlight.css"] - IMG["copyPdfImages()
referenced images"] - end - - PHASE8 --> PDFSRC["_site-pdf/book.html
(~5 MB)
print.css, tb-highlight.css,
images"] - - PDFSRC --> R1 - - subgraph RENDER ["render-book.mjs"] - direction LR - R1["Phase 1: Render
puppeteer
→ headless Chromium
paged.js
CSS Paged Media layout
→ one .pagedjs_page
per output page"] - R2["Phase 2: Generate
extract metadata
+ outline tree
page.pdf() → raw PDF buffer"] - R3["Phase 3: Process
measureRawPdf()
→ pre-size shim arrays
PDFDocument.load() with fast-* shims
setMetadata() + setOutline()
parallelSave() → async deflate"] - R1 --> R2 --> R3 - end - - R3 --> PDF["_pdf/twinBASIC Book.pdf"] diff --git a/docs/assets/images/mmd/pdf-render-pipeline.svg b/docs/assets/images/mmd/pdf-render-pipeline.svg deleted file mode 100644 index a8a9c94a..00000000 --- a/docs/assets/images/mmd/pdf-render-pipeline.svg +++ /dev/null @@ -1 +0,0 @@ -

render-book.mjs

Phase 1: Render
puppeteer
→ headless Chromium
paged.js
CSS Paged Media layout
→ one .pagedjs_page
per output page

Phase 2: Generate
extract metadata
+ outline tree
page.pdf() → raw PDF buffer

Phase 3: Process
measureRawPdf()
→ pre-size shim arrays
PDFDocument.load() with fast-* shims
setMetadata() + setOutline()
parallelSave() → async deflate

tbdocs writePdf (pdf.mjs + book.mjs)

assembleBook()
combine chapter HTML

copyPdfCss()
print.css + tb-highlight.css

copyPdfImages()
referenced images

_site-pdf/book.html
(~5 MB)
print.css, tb-highlight.css,
images

_pdf/twinBASIC Book.pdf

 \ No newline at end of file diff --git a/docs/assets/images/mmd/scheduler-dag.mmd b/docs/assets/images/mmd/scheduler-dag.mmd deleted file mode 100644 index 2693ad78..00000000 --- a/docs/assets/images/mmd/scheduler-dag.mmd +++ /dev/null @@ -1,106 +0,0 @@ -%%{init: {"themeVariables": {"clusterBkg": "transparent", "clusterBorder": "#7a8090", "lineColor": "#7a8090"}}}%% -flowchart TB - subgraph seeds [Seeds] - direction LR - CF["config
[M]"] - BI["buildInfo
[W]"] - SL["scssLight
[W]"] - SD["scssDark
[W]"] - HI["highlighterInit
[M]"] - LD["loadData
[M]"] - WI["warmInit
[W] per-lane
on_demand, idle"] - end - - subgraph spine [Spine] - direction LR - DI["discover
[M]"] - NV["nav
[M]"] - BU["buildInit
[M]"] - MI["markdownInit
[M]"] - DR["deriveRedirects
[M]"] - MM["mermaid
[W]"] - DSi["deriveSitemap
[M]"] - RBC["resolveBookChapters
[M]"] - end - - subgraph render [Render] - direction LR - DP["dispatch
[M]"] - PD["prepDest
[M]"] - PPD["prepPageDirs
[M]"] - REI["renderEnvInit
[W] per-lane"] - Ri["render:i
[W]"] - RJ["renderJoin
[M]"] - end - - subgraph writes [Write] - direction LR - SCSS["scss
[M]"] - Fi["flush:i
[W] pinned"] - FJ["flushJoin
[M]"] - WA["writeAssets
[M]"] - SDt["searchData
[M]"] - WX["writeAux
[M]"] - WO["writeOffline
[M]"] - WP["writePdf
[M]"] - end - - CF --> DI - CF --> HI - HI --> LD - DI --> NV - DI --> BU - DI --> MI - DI --> DR - - NV --> DP - BU --> DP - BI --> DP - MM --> DP - DR --> DP - MI --> DP - - DP --> PD - DP --> DSi - DP --> REI - DP --> Ri - - PD --> PPD - PD --> SCSS - SL --> SCSS - SD --> SCSS - - DSi --> RBC - - WI -.-> REI - REI -.-> Ri - - Ri --> Fi - Ri --> RJ - PPD --> Fi - Fi --> FJ - - MM --> WA - PPD --> WA - HI --> WA - - RJ --> SDt - PD --> SDt - - WA --> WX - SDt --> WX - FJ --> WX - DR --> WX - DSi --> WX - - WX --> WO - WA --> WO - - FJ --> WP - MM --> WP - RBC --> WP - - classDef worker fill:#e8f0fe,stroke:#4285f4,color:#1a1a2e - classDef main fill:#fef7e0,stroke:#f9ab00,color:#1a1a2e - class BI,SL,SD,MM,WI,REI,Ri,Fi worker - class CF,DI,NV,BU,MI,LD,DR,DSi,RBC,DP,HI,RJ,FJ,SCSS,PD,PPD,WA,SDt,WX,WO,WP main diff --git a/docs/assets/images/mmd/scheduler-dag.svg b/docs/assets/images/mmd/scheduler-dag.svg deleted file mode 100644 index 0ff4d522..00000000 --- a/docs/assets/images/mmd/scheduler-dag.svg +++ /dev/null @@ -1 +0,0 @@ -

Write

scss
[M]

flush:i
[W] pinned

flushJoin
[M]

writeAssets
[M]

writeAux
[M]

searchData
[M]

writeOffline
[M]

writePdf
[M]

Render

dispatch
[M]

prepDest
[M]

renderEnvInit
[W] per-lane

render:i
[W]

prepPageDirs
[M]

renderJoin
[M]

Spine

discover
[M]

nav
[M]

buildInit
[M]

markdownInit
[M]

deriveRedirects
[M]

deriveSitemap
[M]

resolveBookChapters
[M]

mermaid
[W]

Seeds

config
[M]

highlighterInit
[M]

loadData
[M]

buildInfo
[W]

scssLight
[W]

scssDark
[W]

warmInit
[W] per-lane
on_demand, idle

 \ No newline at end of file diff --git a/docs/assets/images/mmd/toolchain-overview.mmd b/docs/assets/images/mmd/toolchain-overview.mmd deleted file mode 100644 index 7a55ed86..00000000 --- a/docs/assets/images/mmd/toolchain-overview.mmd +++ /dev/null @@ -1,13 +0,0 @@ -%%{init: {"themeVariables": {"lineColor": "#7a8090"}}}%% -flowchart TD - SRC["docs/ source tree
*.md + _config.yml + _book.yml"] - SRC --> SERVE["serve.bat"] --> SERVED["_serve/"] --> BROWSER["local
web
browser"] - SRC --> BUILD["build.bat
tbdocs.mjs"] - BUILD --> SITE["_site/
online HTML"] - BUILD --> OFFLINE["_site-offline/
file:// mirror"] - BUILD --> PDFSRC["_site-pdf/
book.html + CSS + images"] - SITE --> CHECK["check.bat
check_links.mjs"] - OFFLINE --> CHECK - PDFSRC -.-> CHECK - PDFSRC --> BOOK["book.bat
render-book.mjs"] - BOOK --> PDF["_pdf/twinBASIC Book.pdf"] diff --git a/docs/assets/images/mmd/toolchain-overview.svg b/docs/assets/images/mmd/toolchain-overview.svg deleted file mode 100644 index d2a2390b..00000000 --- a/docs/assets/images/mmd/toolchain-overview.svg +++ /dev/null @@ -1 +0,0 @@ -

docs/ source tree
*.md + _config.yml + _book.yml

serve.bat

_serve/

local
web
browser

build.bat
tbdocs.mjs

_site/
online HTML

_site-offline/
file:// mirror

_site-pdf/
book.html + CSS + images

check.bat
check_links.mjs

book.bat
render-book.mjs

_pdf/twinBASIC Book.pdf

 \ No newline at end of file diff --git a/package-lock.json b/package-lock.json index 777cefcb..c6669292 100644 --- a/package-lock.json +++ b/package-lock.json @@ -7,8 +7,8 @@ "": { "name": "twinbasic-docs", "version": "0.0.0", - "hasInstallScript": true, "devDependencies": { + "@hpcc-js/wasm-graphviz": "^1.21", "acorn": "^8.0", "acorn-walk": "^8.0", "fast-glob": "^3.3", @@ -20,27 +20,12 @@ "markdown-it-attrs": "^4.3", "markdown-it-deflist": "^3.0", "markdown-it-footnote": "^4.0", - "mermaid": "11.15.0", "pdf-lib": "1.17.1", "puppeteer": "25.0.4", "sass": "^1.0", "shiki": "^1.0" } }, - "node_modules/@antfu/install-pkg": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/@antfu/install-pkg/-/install-pkg-1.1.0.tgz", - "integrity": "sha512-MGQsmw10ZyI+EJo45CdSER4zEb+p31LpDAFp2Z3gkSd1yqVZGi0Ebx++YTEMonJy4oChEMLsxZ64j8FH6sSqtQ==", - "dev": true, - "license": "MIT", - "dependencies": { - "package-manager-detector": "^1.3.0", - "tinyexec": "^1.0.1" - }, - "funding": { - "url": "https://github.com/sponsors/antfu" - } - }, "node_modules/@babel/code-frame": { "version": "7.29.0", "resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.29.0.tgz", @@ -66,49 +51,13 @@ "node": ">=6.9.0" } }, - "node_modules/@braintree/sanitize-url": { - "version": "7.1.2", - "resolved": "https://registry.npmjs.org/@braintree/sanitize-url/-/sanitize-url-7.1.2.tgz", - "integrity": "sha512-jigsZK+sMF/cuiB7sERuo9V7N9jx+dhmHHnQyDSVdpZwVutaBu7WvNYqMDLSgFgfB30n452TP3vjDAvFC973mA==", - "dev": true, - "license": "MIT" - }, - "node_modules/@chevrotain/types": { - "version": "11.1.2", - "resolved": "https://registry.npmjs.org/@chevrotain/types/-/types-11.1.2.tgz", - "integrity": "sha512-U+HFai5+zmJCkK86QsaJtoITlboZHBqrVketcO2ROv865xfCMSFpELQoz1GkX5GzME8pTa+3kbKrZHQtI0gdbw==", + "node_modules/@hpcc-js/wasm-graphviz": { + "version": "1.22.0", + "resolved": "https://registry.npmjs.org/@hpcc-js/wasm-graphviz/-/wasm-graphviz-1.22.0.tgz", + "integrity": "sha512-/l6m+ev4TzYwiSm2pklMALiKejuheJwFkOHKrTAvFei+US+MVv6Vgk06xMc4Mjn57WiqzheMVGnmNu1iqkcBdA==", "dev": true, "license": "Apache-2.0" }, - "node_modules/@iconify/types": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/@iconify/types/-/types-2.0.0.tgz", - "integrity": "sha512-+wluvCrRhXrhyOmRDJ3q8mux9JkKy5SJ/v8ol2tu4FVjyYvtEzkc/3pK15ET6RKg4b4w4BmTk1+gsCUhf21Ykg==", - "dev": true, - "license": "MIT" - }, - "node_modules/@iconify/utils": { - "version": "3.1.3", - "resolved": "https://registry.npmjs.org/@iconify/utils/-/utils-3.1.3.tgz", - "integrity": "sha512-LPKOXPn/zV+zis1oOfGWogaXVpqUybF3ZS6SCZIsz8vg0ivVp9+fVqyYB7xq0aiST/VhUQYGO1qo6uoYSiEJqw==", - "dev": true, - "license": "MIT", - "dependencies": { - "@antfu/install-pkg": "^1.1.0", - "@iconify/types": "^2.0.0", - "import-meta-resolve": "^4.2.0" - } - }, - "node_modules/@mermaid-js/parser": { - "version": "1.1.1", - "resolved": "https://registry.npmjs.org/@mermaid-js/parser/-/parser-1.1.1.tgz", - "integrity": "sha512-VuHdsYMK1bT6X2JbcAaWAhugTRvRBRyuZgd+c22swUeI9g/ntaxF7CY7dYarhZovofCbUNO0G7JesfmNtjYOCw==", - "dev": true, - "license": "MIT", - "dependencies": { - "@chevrotain/types": "~11.1.1" - } - }, "node_modules/@nodelib/fs.scandir": { "version": "2.1.5", "resolved": "https://registry.npmjs.org/@nodelib/fs.scandir/-/fs.scandir-2.1.5.tgz", @@ -595,297 +544,6 @@ "dev": true, "license": "MIT" }, - "node_modules/@types/d3": { - "version": "7.4.3", - "resolved": "https://registry.npmjs.org/@types/d3/-/d3-7.4.3.tgz", - "integrity": "sha512-lZXZ9ckh5R8uiFVt8ogUNf+pIrK4EsWrx2Np75WvF/eTpJ0FMHNhjXk8CKEx/+gpHbNQyJWehbFaTvqmHWB3ww==", - "dev": true, - "license": "MIT", - "dependencies": { - "@types/d3-array": "*", - "@types/d3-axis": "*", - "@types/d3-brush": "*", - "@types/d3-chord": "*", - "@types/d3-color": "*", - "@types/d3-contour": "*", - "@types/d3-delaunay": "*", - "@types/d3-dispatch": "*", - "@types/d3-drag": "*", - "@types/d3-dsv": "*", - "@types/d3-ease": "*", - "@types/d3-fetch": "*", - "@types/d3-force": "*", - "@types/d3-format": "*", - "@types/d3-geo": "*", - "@types/d3-hierarchy": "*", - "@types/d3-interpolate": "*", - "@types/d3-path": "*", - "@types/d3-polygon": "*", - "@types/d3-quadtree": "*", - "@types/d3-random": "*", - "@types/d3-scale": "*", - "@types/d3-scale-chromatic": "*", - "@types/d3-selection": "*", - "@types/d3-shape": "*", - "@types/d3-time": "*", - "@types/d3-time-format": "*", - "@types/d3-timer": "*", - "@types/d3-transition": "*", - "@types/d3-zoom": "*" - } - }, - "node_modules/@types/d3-array": { - "version": "3.2.2", - "resolved": "https://registry.npmjs.org/@types/d3-array/-/d3-array-3.2.2.tgz", - "integrity": "sha512-hOLWVbm7uRza0BYXpIIW5pxfrKe0W+D5lrFiAEYR+pb6w3N2SwSMaJbXdUfSEv+dT4MfHBLtn5js0LAWaO6otw==", - "dev": true, - "license": "MIT" - }, - "node_modules/@types/d3-axis": { - "version": "3.0.6", - "resolved": "https://registry.npmjs.org/@types/d3-axis/-/d3-axis-3.0.6.tgz", - "integrity": "sha512-pYeijfZuBd87T0hGn0FO1vQ/cgLk6E1ALJjfkC0oJ8cbwkZl3TpgS8bVBLZN+2jjGgg38epgxb2zmoGtSfvgMw==", - "dev": true, - "license": "MIT", - "dependencies": { - "@types/d3-selection": "*" - } - }, - "node_modules/@types/d3-brush": { - "version": "3.0.6", - "resolved": "https://registry.npmjs.org/@types/d3-brush/-/d3-brush-3.0.6.tgz", - "integrity": "sha512-nH60IZNNxEcrh6L1ZSMNA28rj27ut/2ZmI3r96Zd+1jrZD++zD3LsMIjWlvg4AYrHn/Pqz4CF3veCxGjtbqt7A==", - "dev": true, - "license": "MIT", - "dependencies": { - "@types/d3-selection": "*" - } - }, - "node_modules/@types/d3-chord": { - "version": "3.0.6", - "resolved": "https://registry.npmjs.org/@types/d3-chord/-/d3-chord-3.0.6.tgz", - "integrity": "sha512-LFYWWd8nwfwEmTZG9PfQxd17HbNPksHBiJHaKuY1XeqscXacsS2tyoo6OdRsjf+NQYeB6XrNL3a25E3gH69lcg==", - "dev": true, - "license": "MIT" - }, - "node_modules/@types/d3-color": { - "version": "3.1.3", - "resolved": "https://registry.npmjs.org/@types/d3-color/-/d3-color-3.1.3.tgz", - "integrity": "sha512-iO90scth9WAbmgv7ogoq57O9YpKmFBbmoEoCHDB2xMBY0+/KVrqAaCDyCE16dUspeOvIxFFRI+0sEtqDqy2b4A==", - "dev": true, - "license": "MIT" - }, - "node_modules/@types/d3-contour": { - "version": "3.0.6", - "resolved": "https://registry.npmjs.org/@types/d3-contour/-/d3-contour-3.0.6.tgz", - "integrity": "sha512-BjzLgXGnCWjUSYGfH1cpdo41/hgdWETu4YxpezoztawmqsvCeep+8QGfiY6YbDvfgHz/DkjeIkkZVJavB4a3rg==", - "dev": true, - "license": "MIT", - "dependencies": { - "@types/d3-array": "*", - "@types/geojson": "*" - } - }, - "node_modules/@types/d3-delaunay": { - "version": "6.0.4", - "resolved": "https://registry.npmjs.org/@types/d3-delaunay/-/d3-delaunay-6.0.4.tgz", - "integrity": "sha512-ZMaSKu4THYCU6sV64Lhg6qjf1orxBthaC161plr5KuPHo3CNm8DTHiLw/5Eq2b6TsNP0W0iJrUOFscY6Q450Hw==", - "dev": true, - "license": "MIT" - }, - "node_modules/@types/d3-dispatch": { - "version": "3.0.7", - "resolved": "https://registry.npmjs.org/@types/d3-dispatch/-/d3-dispatch-3.0.7.tgz", - "integrity": "sha512-5o9OIAdKkhN1QItV2oqaE5KMIiXAvDWBDPrD85e58Qlz1c1kI/J0NcqbEG88CoTwJrYe7ntUCVfeUl2UJKbWgA==", - "dev": true, - "license": "MIT" - }, - "node_modules/@types/d3-drag": { - "version": "3.0.7", - "resolved": "https://registry.npmjs.org/@types/d3-drag/-/d3-drag-3.0.7.tgz", - "integrity": "sha512-HE3jVKlzU9AaMazNufooRJ5ZpWmLIoc90A37WU2JMmeq28w1FQqCZswHZ3xR+SuxYftzHq6WU6KJHvqxKzTxxQ==", - "dev": true, - "license": "MIT", - "dependencies": { - "@types/d3-selection": "*" - } - }, - "node_modules/@types/d3-dsv": { - "version": "3.0.7", - "resolved": "https://registry.npmjs.org/@types/d3-dsv/-/d3-dsv-3.0.7.tgz", - "integrity": "sha512-n6QBF9/+XASqcKK6waudgL0pf/S5XHPPI8APyMLLUHd8NqouBGLsU8MgtO7NINGtPBtk9Kko/W4ea0oAspwh9g==", - "dev": true, - "license": "MIT" - }, - "node_modules/@types/d3-ease": { - "version": "3.0.2", - "resolved": "https://registry.npmjs.org/@types/d3-ease/-/d3-ease-3.0.2.tgz", - "integrity": "sha512-NcV1JjO5oDzoK26oMzbILE6HW7uVXOHLQvHshBUW4UMdZGfiY6v5BeQwh9a9tCzv+CeefZQHJt5SRgK154RtiA==", - "dev": true, - "license": "MIT" - }, - "node_modules/@types/d3-fetch": { - "version": "3.0.7", - "resolved": "https://registry.npmjs.org/@types/d3-fetch/-/d3-fetch-3.0.7.tgz", - "integrity": "sha512-fTAfNmxSb9SOWNB9IoG5c8Hg6R+AzUHDRlsXsDZsNp6sxAEOP0tkP3gKkNSO/qmHPoBFTxNrjDprVHDQDvo5aA==", - "dev": true, - "license": "MIT", - "dependencies": { - "@types/d3-dsv": "*" - } - }, - "node_modules/@types/d3-force": { - "version": "3.0.10", - "resolved": "https://registry.npmjs.org/@types/d3-force/-/d3-force-3.0.10.tgz", - "integrity": "sha512-ZYeSaCF3p73RdOKcjj+swRlZfnYpK1EbaDiYICEEp5Q6sUiqFaFQ9qgoshp5CzIyyb/yD09kD9o2zEltCexlgw==", - "dev": true, - "license": "MIT" - }, - "node_modules/@types/d3-format": { - "version": "3.0.4", - "resolved": "https://registry.npmjs.org/@types/d3-format/-/d3-format-3.0.4.tgz", - "integrity": "sha512-fALi2aI6shfg7vM5KiR1wNJnZ7r6UuggVqtDA+xiEdPZQwy/trcQaHnwShLuLdta2rTymCNpxYTiMZX/e09F4g==", - "dev": true, - "license": "MIT" - }, - "node_modules/@types/d3-geo": { - "version": "3.1.0", - "resolved": "https://registry.npmjs.org/@types/d3-geo/-/d3-geo-3.1.0.tgz", - "integrity": "sha512-856sckF0oP/diXtS4jNsiQw/UuK5fQG8l/a9VVLeSouf1/PPbBE1i1W852zVwKwYCBkFJJB7nCFTbk6UMEXBOQ==", - "dev": true, - "license": "MIT", - "dependencies": { - "@types/geojson": "*" - } - }, - "node_modules/@types/d3-hierarchy": { - "version": "3.1.7", - "resolved": "https://registry.npmjs.org/@types/d3-hierarchy/-/d3-hierarchy-3.1.7.tgz", - "integrity": "sha512-tJFtNoYBtRtkNysX1Xq4sxtjK8YgoWUNpIiUee0/jHGRwqvzYxkq0hGVbbOGSz+JgFxxRu4K8nb3YpG3CMARtg==", - "dev": true, - "license": "MIT" - }, - "node_modules/@types/d3-interpolate": { - "version": "3.0.4", - "resolved": "https://registry.npmjs.org/@types/d3-interpolate/-/d3-interpolate-3.0.4.tgz", - "integrity": "sha512-mgLPETlrpVV1YRJIglr4Ez47g7Yxjl1lj7YKsiMCb27VJH9W8NVM6Bb9d8kkpG/uAQS5AmbA48q2IAolKKo1MA==", - "dev": true, - "license": "MIT", - "dependencies": { - "@types/d3-color": "*" - } - }, - "node_modules/@types/d3-path": { - "version": "3.1.1", - "resolved": "https://registry.npmjs.org/@types/d3-path/-/d3-path-3.1.1.tgz", - "integrity": "sha512-VMZBYyQvbGmWyWVea0EHs/BwLgxc+MKi1zLDCONksozI4YJMcTt8ZEuIR4Sb1MMTE8MMW49v0IwI5+b7RmfWlg==", - "dev": true, - "license": "MIT" - }, - "node_modules/@types/d3-polygon": { - "version": "3.0.2", - "resolved": "https://registry.npmjs.org/@types/d3-polygon/-/d3-polygon-3.0.2.tgz", - "integrity": "sha512-ZuWOtMaHCkN9xoeEMr1ubW2nGWsp4nIql+OPQRstu4ypeZ+zk3YKqQT0CXVe/PYqrKpZAi+J9mTs05TKwjXSRA==", - "dev": true, - "license": "MIT" - }, - "node_modules/@types/d3-quadtree": { - "version": "3.0.6", - "resolved": "https://registry.npmjs.org/@types/d3-quadtree/-/d3-quadtree-3.0.6.tgz", - "integrity": "sha512-oUzyO1/Zm6rsxKRHA1vH0NEDG58HrT5icx/azi9MF1TWdtttWl0UIUsjEQBBh+SIkrpd21ZjEv7ptxWys1ncsg==", - "dev": true, - "license": "MIT" - }, - "node_modules/@types/d3-random": { - "version": "3.0.3", - "resolved": "https://registry.npmjs.org/@types/d3-random/-/d3-random-3.0.3.tgz", - "integrity": "sha512-Imagg1vJ3y76Y2ea0871wpabqp613+8/r0mCLEBfdtqC7xMSfj9idOnmBYyMoULfHePJyxMAw3nWhJxzc+LFwQ==", - "dev": true, - "license": "MIT" - }, - "node_modules/@types/d3-scale": { - "version": "4.0.9", - "resolved": "https://registry.npmjs.org/@types/d3-scale/-/d3-scale-4.0.9.tgz", - "integrity": "sha512-dLmtwB8zkAeO/juAMfnV+sItKjlsw2lKdZVVy6LRr0cBmegxSABiLEpGVmSJJ8O08i4+sGR6qQtb6WtuwJdvVw==", - "dev": true, - "license": "MIT", - "dependencies": { - "@types/d3-time": "*" - } - }, - "node_modules/@types/d3-scale-chromatic": { - "version": "3.1.0", - "resolved": "https://registry.npmjs.org/@types/d3-scale-chromatic/-/d3-scale-chromatic-3.1.0.tgz", - "integrity": "sha512-iWMJgwkK7yTRmWqRB5plb1kadXyQ5Sj8V/zYlFGMUBbIPKQScw+Dku9cAAMgJG+z5GYDoMjWGLVOvjghDEFnKQ==", - "dev": true, - "license": "MIT" - }, - "node_modules/@types/d3-selection": { - "version": "3.0.11", - "resolved": "https://registry.npmjs.org/@types/d3-selection/-/d3-selection-3.0.11.tgz", - "integrity": "sha512-bhAXu23DJWsrI45xafYpkQ4NtcKMwWnAC/vKrd2l+nxMFuvOT3XMYTIj2opv8vq8AO5Yh7Qac/nSeP/3zjTK0w==", - "dev": true, - "license": "MIT" - }, - "node_modules/@types/d3-shape": { - "version": "3.1.8", - "resolved": "https://registry.npmjs.org/@types/d3-shape/-/d3-shape-3.1.8.tgz", - "integrity": "sha512-lae0iWfcDeR7qt7rA88BNiqdvPS5pFVPpo5OfjElwNaT2yyekbM0C9vK+yqBqEmHr6lDkRnYNoTBYlAgJa7a4w==", - "dev": true, - "license": "MIT", - "dependencies": { - "@types/d3-path": "*" - } - }, - "node_modules/@types/d3-time": { - "version": "3.0.4", - "resolved": "https://registry.npmjs.org/@types/d3-time/-/d3-time-3.0.4.tgz", - "integrity": "sha512-yuzZug1nkAAaBlBBikKZTgzCeA+k1uy4ZFwWANOfKw5z5LRhV0gNA7gNkKm7HoK+HRN0wX3EkxGk0fpbWhmB7g==", - "dev": true, - "license": "MIT" - }, - "node_modules/@types/d3-time-format": { - "version": "4.0.3", - "resolved": "https://registry.npmjs.org/@types/d3-time-format/-/d3-time-format-4.0.3.tgz", - "integrity": "sha512-5xg9rC+wWL8kdDj153qZcsJ0FWiFt0J5RB6LYUNZjwSnesfblqrI/bJ1wBdJ8OQfncgbJG5+2F+qfqnqyzYxyg==", - "dev": true, - "license": "MIT" - }, - "node_modules/@types/d3-timer": { - "version": "3.0.2", - "resolved": "https://registry.npmjs.org/@types/d3-timer/-/d3-timer-3.0.2.tgz", - "integrity": "sha512-Ps3T8E8dZDam6fUyNiMkekK3XUsaUEik+idO9/YjPtfj2qruF8tFBXS7XhtE4iIXBLxhmLjP3SXpLhVf21I9Lw==", - "dev": true, - "license": "MIT" - }, - "node_modules/@types/d3-transition": { - "version": "3.0.9", - "resolved": "https://registry.npmjs.org/@types/d3-transition/-/d3-transition-3.0.9.tgz", - "integrity": "sha512-uZS5shfxzO3rGlu0cC3bjmMFKsXv+SmZZcgp0KD22ts4uGXp5EVYGzu/0YdwZeKmddhcAccYtREJKkPfXkZuCg==", - "dev": true, - "license": "MIT", - "dependencies": { - "@types/d3-selection": "*" - } - }, - "node_modules/@types/d3-zoom": { - "version": "3.0.8", - "resolved": "https://registry.npmjs.org/@types/d3-zoom/-/d3-zoom-3.0.8.tgz", - "integrity": "sha512-iqMC4/YlFCSlO8+2Ii1GGGliCAY4XdeG748w5vQUbevlbDu0zSjH/+jojorQVBK/se0j6DUFNPBGSqD3YWYnDw==", - "dev": true, - "license": "MIT", - "dependencies": { - "@types/d3-interpolate": "*", - "@types/d3-selection": "*" - } - }, - "node_modules/@types/geojson": { - "version": "7946.0.16", - "resolved": "https://registry.npmjs.org/@types/geojson/-/geojson-7946.0.16.tgz", - "integrity": "sha512-6C8nqWur3j98U6+lXDfTUWIfgvZU+EumvpHKcYjujKH7woYyLj2sUmff0tRhrqM7BohUw7Pz3ZB1jj2gW9Fvmg==", - "dev": true, - "license": "MIT" - }, "node_modules/@types/hast": { "version": "3.0.4", "resolved": "https://registry.npmjs.org/@types/hast/-/hast-3.0.4.tgz", @@ -906,14 +564,6 @@ "@types/unist": "*" } }, - "node_modules/@types/trusted-types": { - "version": "2.0.7", - "resolved": "https://registry.npmjs.org/@types/trusted-types/-/trusted-types-2.0.7.tgz", - "integrity": "sha512-ScaPdn1dQczgbl0QFTeTOmVHFULt394XJgOQNoyVhZ6r2vLnMLJfBPd53SB52T/3G36VI1/g2MZaX0cwDuXsfw==", - "dev": true, - "license": "MIT", - "optional": true - }, "node_modules/@types/unist": { "version": "3.0.3", "resolved": "https://registry.npmjs.org/@types/unist/-/unist-3.0.3.tgz", @@ -928,17 +578,6 @@ "dev": true, "license": "ISC" }, - "node_modules/@upsetjs/venn.js": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/@upsetjs/venn.js/-/venn.js-2.0.0.tgz", - "integrity": "sha512-WbBhLrooyePuQ1VZxrJjtLvTc4NVfpOyKx0sKqioq9bX1C1m7Jgykkn8gLrtwumBioXIqam8DLxp88Adbue6Hw==", - "dev": true, - "license": "MIT", - "optionalDependencies": { - "d3-selection": "^3.0.0", - "d3-transition": "^3.0.1" - } - }, "node_modules/acorn": { "version": "8.16.0", "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.16.0.tgz", @@ -1144,704 +783,134 @@ "url": "https://github.com/sponsors/wooorm" } }, - "node_modules/character-entities-html4": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/character-entities-html4/-/character-entities-html4-2.1.0.tgz", - "integrity": "sha512-1v7fgQRj6hnSwFpq1Eu0ynr/CDEw0rXo2B61qXrLNdHZmPKgb7fqS1a2JwF0rISo9q77jDI8VMEHoApn8qDoZA==", - "dev": true, - "license": "MIT", - "funding": { - "type": "github", - "url": "https://github.com/sponsors/wooorm" - } - }, - "node_modules/character-entities-legacy": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/character-entities-legacy/-/character-entities-legacy-3.0.0.tgz", - "integrity": "sha512-RpPp0asT/6ufRm//AJVwpViZbGM/MkjQFxJccQRHmISF/22NBtsHqAWmL+/pmkPWoIUJdWyeVleTl1wydHATVQ==", - "dev": true, - "license": "MIT", - "funding": { - "type": "github", - "url": "https://github.com/sponsors/wooorm" - } - }, - "node_modules/chokidar": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/chokidar/-/chokidar-5.0.0.tgz", - "integrity": "sha512-TQMmc3w+5AxjpL8iIiwebF73dRDF4fBIieAqGn9RGCWaEVwQ6Fb2cGe31Yns0RRIzii5goJ1Y7xbMwo1TxMplw==", - "dev": true, - "license": "MIT", - "dependencies": { - "readdirp": "^5.0.0" - }, - "engines": { - "node": ">= 20.19.0" - }, - "funding": { - "url": "https://paulmillr.com/funding/" - } - }, - "node_modules/chromium-bidi": { - "version": "16.0.1", - "resolved": "https://registry.npmjs.org/chromium-bidi/-/chromium-bidi-16.0.1.tgz", - "integrity": "sha512-J63PGu/9PpeCwLIcKYyzWP6yaVL5pxuBc0shlYCYM8BaAkmlwiQboXO1iNbOgSDbVklEyYFfNEcHD8oOAWacUA==", - "dev": true, - "license": "Apache-2.0", - "dependencies": { - "mitt": "^3.0.1", - "zod": "^3.24.1" - }, - "engines": { - "node": ">=20.19.0 <22.0.0 || >=22.12.0" - }, - "peerDependencies": { - "devtools-protocol": "*" - } - }, - "node_modules/cliui": { - "version": "8.0.1", - "resolved": "https://registry.npmjs.org/cliui/-/cliui-8.0.1.tgz", - "integrity": "sha512-BSeNnyus75C4//NQ9gQt1/csTXyo/8Sb+afLAkzAptFuMsod9HFokGNudZpi/oQV73hnVK+sR+5PVRMd+Dr7YQ==", - "dev": true, - "license": "ISC", - "dependencies": { - "string-width": "^4.2.0", - "strip-ansi": "^6.0.1", - "wrap-ansi": "^7.0.0" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/color-convert": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz", - "integrity": "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==", - "dev": true, - "license": "MIT", - "dependencies": { - "color-name": "~1.1.4" - }, - "engines": { - "node": ">=7.0.0" - } - }, - "node_modules/color-name": { - "version": "1.1.4", - "resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.4.tgz", - "integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==", - "dev": true, - "license": "MIT" - }, - "node_modules/comma-separated-tokens": { - "version": "2.0.3", - "resolved": "https://registry.npmjs.org/comma-separated-tokens/-/comma-separated-tokens-2.0.3.tgz", - "integrity": "sha512-Fu4hJdvzeylCfQPp9SGWidpzrMs7tTrlu6Vb8XGaRGck8QSNZJJp538Wrb60Lax4fPwR64ViY468OIUTbRlGZg==", - "dev": true, - "license": "MIT", - "funding": { - "type": "github", - "url": "https://github.com/sponsors/wooorm" - } - }, - "node_modules/commander": { - "version": "7.2.0", - "resolved": "https://registry.npmjs.org/commander/-/commander-7.2.0.tgz", - "integrity": "sha512-QrWXB+ZQSVPmIWIhtEO9H+gwHaMGYiF5ChvoJ+K9ZGHG/sVsa6yiesAD1GC/x46sET00Xlwo1u49RVVVzvcSkw==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">= 10" - } - }, - "node_modules/cose-base": { - "version": "1.0.3", - "resolved": "https://registry.npmjs.org/cose-base/-/cose-base-1.0.3.tgz", - "integrity": "sha512-s9whTXInMSgAp/NVXVNuVxVKzGH2qck3aQlVHxDCdAEPgtMKwc4Wq6/QKhgdEdgbLSi9rBTAcPoRa6JpiG4ksg==", - "dev": true, - "license": "MIT", - "dependencies": { - "layout-base": "^1.0.0" - } - }, - "node_modules/cosmiconfig": { - "version": "9.0.1", - "resolved": "https://registry.npmjs.org/cosmiconfig/-/cosmiconfig-9.0.1.tgz", - "integrity": "sha512-hr4ihw+DBqcvrsEDioRO31Z17x71pUYoNe/4h6Z0wB72p7MU7/9gH8Q3s12NFhHPfYBBOV3qyfUxmr/Yn3shnQ==", - "dev": true, - "license": "MIT", - "dependencies": { - "env-paths": "^2.2.1", - "import-fresh": "^3.3.0", - "js-yaml": "^4.1.0", - "parse-json": "^5.2.0" - }, - "engines": { - "node": ">=14" - }, - "funding": { - "url": "https://github.com/sponsors/d-fischer" - }, - "peerDependencies": { - "typescript": ">=4.9.5" - }, - "peerDependenciesMeta": { - "typescript": { - "optional": true - } - } - }, - "node_modules/cytoscape": { - "version": "3.33.4", - "resolved": "https://registry.npmjs.org/cytoscape/-/cytoscape-3.33.4.tgz", - "integrity": "sha512-HIN5Pmd9MrX9BkV7tDwnOcEJCSFvCpc8X97h3f508J6I5FsqAY65wKOCvgH2CuP42CaahWaz4tuh32SOOIH7ww==", - "dev": true, - "license": "MIT", - "peer": true, - "engines": { - "node": ">=0.10" - } - }, - "node_modules/cytoscape-cose-bilkent": { - "version": "4.1.0", - "resolved": "https://registry.npmjs.org/cytoscape-cose-bilkent/-/cytoscape-cose-bilkent-4.1.0.tgz", - "integrity": "sha512-wgQlVIUJF13Quxiv5e1gstZ08rnZj2XaLHGoFMYXz7SkNfCDOOteKBE6SYRfA9WxxI/iBc3ajfDoc6hb/MRAHQ==", - "dev": true, - "license": "MIT", - "dependencies": { - "cose-base": "^1.0.0" - }, - "peerDependencies": { - "cytoscape": "^3.2.0" - } - }, - "node_modules/cytoscape-fcose": { - "version": "2.2.0", - "resolved": "https://registry.npmjs.org/cytoscape-fcose/-/cytoscape-fcose-2.2.0.tgz", - "integrity": "sha512-ki1/VuRIHFCzxWNrsshHYPs6L7TvLu3DL+TyIGEsRcvVERmxokbf5Gdk7mFxZnTdiGtnA4cfSmjZJMviqSuZrQ==", - "dev": true, - "license": "MIT", - "dependencies": { - "cose-base": "^2.2.0" - }, - "peerDependencies": { - "cytoscape": "^3.2.0" - } - }, - "node_modules/cytoscape-fcose/node_modules/cose-base": { - "version": "2.2.0", - "resolved": "https://registry.npmjs.org/cose-base/-/cose-base-2.2.0.tgz", - "integrity": "sha512-AzlgcsCbUMymkADOJtQm3wO9S3ltPfYOFD5033keQn9NJzIbtnZj+UdBJe7DYml/8TdbtHJW3j58SOnKhWY/5g==", - "dev": true, - "license": "MIT", - "dependencies": { - "layout-base": "^2.0.0" - } - }, - "node_modules/cytoscape-fcose/node_modules/layout-base": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/layout-base/-/layout-base-2.0.1.tgz", - "integrity": "sha512-dp3s92+uNI1hWIpPGH3jK2kxE2lMjdXdr+DH8ynZHpd6PUlH6x6cbuXnoMmiNumznqaNO31xu9e79F0uuZ0JFg==", - "dev": true, - "license": "MIT" - }, - "node_modules/d3": { - "version": "7.9.0", - "resolved": "https://registry.npmjs.org/d3/-/d3-7.9.0.tgz", - "integrity": "sha512-e1U46jVP+w7Iut8Jt8ri1YsPOvFpg46k+K8TpCb0P+zjCkjkPnV7WzfDJzMHy1LnA+wj5pLT1wjO901gLXeEhA==", - "dev": true, - "license": "ISC", - "dependencies": { - "d3-array": "3", - "d3-axis": "3", - "d3-brush": "3", - "d3-chord": "3", - "d3-color": "3", - "d3-contour": "4", - "d3-delaunay": "6", - "d3-dispatch": "3", - "d3-drag": "3", - "d3-dsv": "3", - "d3-ease": "3", - "d3-fetch": "3", - "d3-force": "3", - "d3-format": "3", - "d3-geo": "3", - "d3-hierarchy": "3", - "d3-interpolate": "3", - "d3-path": "3", - "d3-polygon": "3", - "d3-quadtree": "3", - "d3-random": "3", - "d3-scale": "4", - "d3-scale-chromatic": "3", - "d3-selection": "3", - "d3-shape": "3", - "d3-time": "3", - "d3-time-format": "4", - "d3-timer": "3", - "d3-transition": "3", - "d3-zoom": "3" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-array": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/d3-array/-/d3-array-3.2.4.tgz", - "integrity": "sha512-tdQAmyA18i4J7wprpYq8ClcxZy3SC31QMeByyCFyRt7BVHdREQZ5lpzoe5mFEYZUWe+oq8HBvk9JjpibyEV4Jg==", - "dev": true, - "license": "ISC", - "dependencies": { - "internmap": "1 - 2" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-axis": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/d3-axis/-/d3-axis-3.0.0.tgz", - "integrity": "sha512-IH5tgjV4jE/GhHkRV0HiVYPDtvfjHQlQfJHs0usq7M30XcSBvOotpmH1IgkcXsO/5gEQZD43B//fc7SRT5S+xw==", - "dev": true, - "license": "ISC", - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-brush": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/d3-brush/-/d3-brush-3.0.0.tgz", - "integrity": "sha512-ALnjWlVYkXsVIGlOsuWH1+3udkYFI48Ljihfnh8FZPF2QS9o+PzGLBslO0PjzVoHLZ2KCVgAM8NVkXPJB2aNnQ==", - "dev": true, - "license": "ISC", - "dependencies": { - "d3-dispatch": "1 - 3", - "d3-drag": "2 - 3", - "d3-interpolate": "1 - 3", - "d3-selection": "3", - "d3-transition": "3" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-chord": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/d3-chord/-/d3-chord-3.0.1.tgz", - "integrity": "sha512-VE5S6TNa+j8msksl7HwjxMHDM2yNK3XCkusIlpX5kwauBfXuyLAtNg9jCp/iHH61tgI4sb6R/EIMWCqEIdjT/g==", - "dev": true, - "license": "ISC", - "dependencies": { - "d3-path": "1 - 3" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-color": { - "version": "3.1.0", - "resolved": "https://registry.npmjs.org/d3-color/-/d3-color-3.1.0.tgz", - "integrity": "sha512-zg/chbXyeBtMQ1LbD/WSoW2DpC3I0mpmPdW+ynRTj/x2DAWYrIY7qeZIHidozwV24m4iavr15lNwIwLxRmOxhA==", - "dev": true, - "license": "ISC", - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-contour": { - "version": "4.0.2", - "resolved": "https://registry.npmjs.org/d3-contour/-/d3-contour-4.0.2.tgz", - "integrity": "sha512-4EzFTRIikzs47RGmdxbeUvLWtGedDUNkTcmzoeyg4sP/dvCexO47AaQL7VKy/gul85TOxw+IBgA8US2xwbToNA==", - "dev": true, - "license": "ISC", - "dependencies": { - "d3-array": "^3.2.0" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-delaunay": { - "version": "6.0.4", - "resolved": "https://registry.npmjs.org/d3-delaunay/-/d3-delaunay-6.0.4.tgz", - "integrity": "sha512-mdjtIZ1XLAM8bm/hx3WwjfHt6Sggek7qH043O8KEjDXN40xi3vx/6pYSVTwLjEgiXQTbvaouWKynLBiUZ6SK6A==", - "dev": true, - "license": "ISC", - "dependencies": { - "delaunator": "5" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-dispatch": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/d3-dispatch/-/d3-dispatch-3.0.1.tgz", - "integrity": "sha512-rzUyPU/S7rwUflMyLc1ETDeBj0NRuHKKAcvukozwhshr6g6c5d8zh4c2gQjY2bZ0dXeGLWc1PF174P2tVvKhfg==", - "dev": true, - "license": "ISC", - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-drag": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/d3-drag/-/d3-drag-3.0.0.tgz", - "integrity": "sha512-pWbUJLdETVA8lQNJecMxoXfH6x+mO2UQo8rSmZ+QqxcbyA3hfeprFgIT//HW2nlHChWeIIMwS2Fq+gEARkhTkg==", - "dev": true, - "license": "ISC", - "dependencies": { - "d3-dispatch": "1 - 3", - "d3-selection": "3" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-dsv": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/d3-dsv/-/d3-dsv-3.0.1.tgz", - "integrity": "sha512-UG6OvdI5afDIFP9w4G0mNq50dSOsXHJaRE8arAS5o9ApWnIElp8GZw1Dun8vP8OyHOZ/QJUKUJwxiiCCnUwm+Q==", - "dev": true, - "license": "ISC", - "dependencies": { - "commander": "7", - "iconv-lite": "0.6", - "rw": "1" - }, - "bin": { - "csv2json": "bin/dsv2json.js", - "csv2tsv": "bin/dsv2dsv.js", - "dsv2dsv": "bin/dsv2dsv.js", - "dsv2json": "bin/dsv2json.js", - "json2csv": "bin/json2dsv.js", - "json2dsv": "bin/json2dsv.js", - "json2tsv": "bin/json2dsv.js", - "tsv2csv": "bin/dsv2dsv.js", - "tsv2json": "bin/dsv2json.js" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-ease": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/d3-ease/-/d3-ease-3.0.1.tgz", - "integrity": "sha512-wR/XK3D3XcLIZwpbvQwQ5fK+8Ykds1ip7A2Txe0yxncXSdq1L9skcG7blcedkOX+ZcgxGAmLX1FrRGbADwzi0w==", - "dev": true, - "license": "BSD-3-Clause", - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-fetch": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/d3-fetch/-/d3-fetch-3.0.1.tgz", - "integrity": "sha512-kpkQIM20n3oLVBKGg6oHrUchHM3xODkTzjMoj7aWQFq5QEM+R6E4WkzT5+tojDY7yjez8KgCBRoj4aEr99Fdqw==", - "dev": true, - "license": "ISC", - "dependencies": { - "d3-dsv": "1 - 3" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-force": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/d3-force/-/d3-force-3.0.0.tgz", - "integrity": "sha512-zxV/SsA+U4yte8051P4ECydjD/S+qeYtnaIyAs9tgHCqfguma/aAQDjo85A9Z6EKhBirHRJHXIgJUlffT4wdLg==", - "dev": true, - "license": "ISC", - "dependencies": { - "d3-dispatch": "1 - 3", - "d3-quadtree": "1 - 3", - "d3-timer": "1 - 3" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-format": { - "version": "3.1.2", - "resolved": "https://registry.npmjs.org/d3-format/-/d3-format-3.1.2.tgz", - "integrity": "sha512-AJDdYOdnyRDV5b6ArilzCPPwc1ejkHcoyFarqlPqT7zRYjhavcT3uSrqcMvsgh2CgoPbK3RCwyHaVyxYcP2Arg==", - "dev": true, - "license": "ISC", - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-geo": { - "version": "3.1.1", - "resolved": "https://registry.npmjs.org/d3-geo/-/d3-geo-3.1.1.tgz", - "integrity": "sha512-637ln3gXKXOwhalDzinUgY83KzNWZRKbYubaG+fGVuc/dxO64RRljtCTnf5ecMyE1RIdtqpkVcq0IbtU2S8j2Q==", - "dev": true, - "license": "ISC", - "dependencies": { - "d3-array": "2.5.0 - 3" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-hierarchy": { - "version": "3.1.2", - "resolved": "https://registry.npmjs.org/d3-hierarchy/-/d3-hierarchy-3.1.2.tgz", - "integrity": "sha512-FX/9frcub54beBdugHjDCdikxThEqjnR93Qt7PvQTOHxyiNCAlvMrHhclk3cD5VeAaq9fxmfRp+CnWw9rEMBuA==", - "dev": true, - "license": "ISC", - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-interpolate": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/d3-interpolate/-/d3-interpolate-3.0.1.tgz", - "integrity": "sha512-3bYs1rOD33uo8aqJfKP3JWPAibgw8Zm2+L9vBKEHJ2Rg+viTR7o5Mmv5mZcieN+FRYaAOWX5SJATX6k1PWz72g==", - "dev": true, - "license": "ISC", - "dependencies": { - "d3-color": "1 - 3" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-path": { - "version": "3.1.0", - "resolved": "https://registry.npmjs.org/d3-path/-/d3-path-3.1.0.tgz", - "integrity": "sha512-p3KP5HCf/bvjBSSKuXid6Zqijx7wIfNW+J/maPs+iwR35at5JCbLUT0LzF1cnjbCHWhqzQTIN2Jpe8pRebIEFQ==", - "dev": true, - "license": "ISC", - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-polygon": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/d3-polygon/-/d3-polygon-3.0.1.tgz", - "integrity": "sha512-3vbA7vXYwfe1SYhED++fPUQlWSYTTGmFmQiany/gdbiWgU/iEyQzyymwL9SkJjFFuCS4902BSzewVGsHHmHtXg==", - "dev": true, - "license": "ISC", - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-quadtree": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/d3-quadtree/-/d3-quadtree-3.0.1.tgz", - "integrity": "sha512-04xDrxQTDTCFwP5H6hRhsRcb9xxv2RzkcsygFzmkSIOJy3PeRJP7sNk3VRIbKXcog561P9oU0/rVH6vDROAgUw==", - "dev": true, - "license": "ISC", - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-random": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/d3-random/-/d3-random-3.0.1.tgz", - "integrity": "sha512-FXMe9GfxTxqd5D6jFsQ+DJ8BJS4E/fT5mqqdjovykEB2oFbTMDVdg1MGFxfQW+FBOGoB++k8swBrgwSHT1cUXQ==", - "dev": true, - "license": "ISC", - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-sankey": { - "version": "0.12.3", - "resolved": "https://registry.npmjs.org/d3-sankey/-/d3-sankey-0.12.3.tgz", - "integrity": "sha512-nQhsBRmM19Ax5xEIPLMY9ZmJ/cDvd1BG3UVvt5h3WRxKg5zGRbvnteTyWAbzeSvlh3tW7ZEmq4VwR5mB3tutmQ==", - "dev": true, - "license": "BSD-3-Clause", - "dependencies": { - "d3-array": "1 - 2", - "d3-shape": "^1.2.0" - } - }, - "node_modules/d3-sankey/node_modules/d3-array": { - "version": "2.12.1", - "resolved": "https://registry.npmjs.org/d3-array/-/d3-array-2.12.1.tgz", - "integrity": "sha512-B0ErZK/66mHtEsR1TkPEEkwdy+WDesimkM5gpZr5Dsg54BiTA5RXtYW5qTLIAcekaS9xfZrzBLF/OAkB3Qn1YQ==", - "dev": true, - "license": "BSD-3-Clause", - "dependencies": { - "internmap": "^1.0.0" - } - }, - "node_modules/d3-sankey/node_modules/d3-path": { - "version": "1.0.9", - "resolved": "https://registry.npmjs.org/d3-path/-/d3-path-1.0.9.tgz", - "integrity": "sha512-VLaYcn81dtHVTjEHd8B+pbe9yHWpXKZUC87PzoFmsFrJqgFwDe/qxfp5MlfsfM1V5E/iVt0MmEbWQ7FVIXh/bg==", - "dev": true, - "license": "BSD-3-Clause" - }, - "node_modules/d3-sankey/node_modules/d3-shape": { - "version": "1.3.7", - "resolved": "https://registry.npmjs.org/d3-shape/-/d3-shape-1.3.7.tgz", - "integrity": "sha512-EUkvKjqPFUAZyOlhY5gzCxCeI0Aep04LwIRpsZ/mLFelJiUfnK56jo5JMDSE7yyP2kLSb6LtF+S5chMk7uqPqw==", + "node_modules/character-entities-html4": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/character-entities-html4/-/character-entities-html4-2.1.0.tgz", + "integrity": "sha512-1v7fgQRj6hnSwFpq1Eu0ynr/CDEw0rXo2B61qXrLNdHZmPKgb7fqS1a2JwF0rISo9q77jDI8VMEHoApn8qDoZA==", "dev": true, - "license": "BSD-3-Clause", - "dependencies": { - "d3-path": "1" + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" } }, - "node_modules/d3-sankey/node_modules/internmap": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/internmap/-/internmap-1.0.1.tgz", - "integrity": "sha512-lDB5YccMydFBtasVtxnZ3MRBHuaoE8GKsppq+EchKL2U4nK/DmEpPHNH8MZe5HkMtpSiTSOZwfN0tzYjO/lJEw==", + "node_modules/character-entities-legacy": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/character-entities-legacy/-/character-entities-legacy-3.0.0.tgz", + "integrity": "sha512-RpPp0asT/6ufRm//AJVwpViZbGM/MkjQFxJccQRHmISF/22NBtsHqAWmL+/pmkPWoIUJdWyeVleTl1wydHATVQ==", "dev": true, - "license": "ISC" + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } }, - "node_modules/d3-scale": { - "version": "4.0.2", - "resolved": "https://registry.npmjs.org/d3-scale/-/d3-scale-4.0.2.tgz", - "integrity": "sha512-GZW464g1SH7ag3Y7hXjf8RoUuAFIqklOAq3MRl4OaWabTFJY9PN/E1YklhXLh+OQ3fM9yS2nOkCoS+WLZ6kvxQ==", + "node_modules/chokidar": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/chokidar/-/chokidar-5.0.0.tgz", + "integrity": "sha512-TQMmc3w+5AxjpL8iIiwebF73dRDF4fBIieAqGn9RGCWaEVwQ6Fb2cGe31Yns0RRIzii5goJ1Y7xbMwo1TxMplw==", "dev": true, - "license": "ISC", + "license": "MIT", "dependencies": { - "d3-array": "2.10.0 - 3", - "d3-format": "1 - 3", - "d3-interpolate": "1.2.0 - 3", - "d3-time": "2.1.1 - 3", - "d3-time-format": "2 - 4" + "readdirp": "^5.0.0" }, "engines": { - "node": ">=12" + "node": ">= 20.19.0" + }, + "funding": { + "url": "https://paulmillr.com/funding/" } }, - "node_modules/d3-scale-chromatic": { - "version": "3.1.0", - "resolved": "https://registry.npmjs.org/d3-scale-chromatic/-/d3-scale-chromatic-3.1.0.tgz", - "integrity": "sha512-A3s5PWiZ9YCXFye1o246KoscMWqf8BsD9eRiJ3He7C9OBaxKhAd5TFCdEx/7VbKtxxTsu//1mMJFrEt572cEyQ==", + "node_modules/chromium-bidi": { + "version": "16.0.1", + "resolved": "https://registry.npmjs.org/chromium-bidi/-/chromium-bidi-16.0.1.tgz", + "integrity": "sha512-J63PGu/9PpeCwLIcKYyzWP6yaVL5pxuBc0shlYCYM8BaAkmlwiQboXO1iNbOgSDbVklEyYFfNEcHD8oOAWacUA==", "dev": true, - "license": "ISC", + "license": "Apache-2.0", "dependencies": { - "d3-color": "1 - 3", - "d3-interpolate": "1 - 3" + "mitt": "^3.0.1", + "zod": "^3.24.1" }, "engines": { - "node": ">=12" - } - }, - "node_modules/d3-selection": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/d3-selection/-/d3-selection-3.0.0.tgz", - "integrity": "sha512-fmTRWbNMmsmWq6xJV8D19U/gw/bwrHfNXxrIN+HfZgnzqTHp9jOmKMhsTUjXOJnZOdZY9Q28y4yebKzqDKlxlQ==", - "dev": true, - "license": "ISC", - "peer": true, - "engines": { - "node": ">=12" + "node": ">=20.19.0 <22.0.0 || >=22.12.0" + }, + "peerDependencies": { + "devtools-protocol": "*" } }, - "node_modules/d3-shape": { - "version": "3.2.0", - "resolved": "https://registry.npmjs.org/d3-shape/-/d3-shape-3.2.0.tgz", - "integrity": "sha512-SaLBuwGm3MOViRq2ABk3eLoxwZELpH6zhl3FbAoJ7Vm1gofKx6El1Ib5z23NUEhF9AsGl7y+dzLe5Cw2AArGTA==", + "node_modules/cliui": { + "version": "8.0.1", + "resolved": "https://registry.npmjs.org/cliui/-/cliui-8.0.1.tgz", + "integrity": "sha512-BSeNnyus75C4//NQ9gQt1/csTXyo/8Sb+afLAkzAptFuMsod9HFokGNudZpi/oQV73hnVK+sR+5PVRMd+Dr7YQ==", "dev": true, "license": "ISC", "dependencies": { - "d3-path": "^3.1.0" + "string-width": "^4.2.0", + "strip-ansi": "^6.0.1", + "wrap-ansi": "^7.0.0" }, "engines": { "node": ">=12" } }, - "node_modules/d3-time": { - "version": "3.1.0", - "resolved": "https://registry.npmjs.org/d3-time/-/d3-time-3.1.0.tgz", - "integrity": "sha512-VqKjzBLejbSMT4IgbmVgDjpkYrNWUYJnbCGo874u7MMKIWsILRX+OpX/gTk8MqjpT1A/c6HY2dCA77ZN0lkQ2Q==", + "node_modules/color-convert": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz", + "integrity": "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==", "dev": true, - "license": "ISC", + "license": "MIT", "dependencies": { - "d3-array": "2 - 3" + "color-name": "~1.1.4" }, "engines": { - "node": ">=12" + "node": ">=7.0.0" } }, - "node_modules/d3-time-format": { - "version": "4.1.0", - "resolved": "https://registry.npmjs.org/d3-time-format/-/d3-time-format-4.1.0.tgz", - "integrity": "sha512-dJxPBlzC7NugB2PDLwo9Q8JiTR3M3e4/XANkreKSUxF8vvXKqm1Yfq4Q5dl8budlunRVlUUaDUgFt7eA8D6NLg==", + "node_modules/color-name": { + "version": "1.1.4", + "resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.4.tgz", + "integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==", "dev": true, - "license": "ISC", - "dependencies": { - "d3-time": "1 - 3" - }, - "engines": { - "node": ">=12" - } + "license": "MIT" }, - "node_modules/d3-timer": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/d3-timer/-/d3-timer-3.0.1.tgz", - "integrity": "sha512-ndfJ/JxxMd3nw31uyKoY2naivF+r29V+Lc0svZxe1JvvIRmi8hUsrMvdOwgS1o6uBHmiz91geQ0ylPP0aj1VUA==", + "node_modules/comma-separated-tokens": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/comma-separated-tokens/-/comma-separated-tokens-2.0.3.tgz", + "integrity": "sha512-Fu4hJdvzeylCfQPp9SGWidpzrMs7tTrlu6Vb8XGaRGck8QSNZJJp538Wrb60Lax4fPwR64ViY468OIUTbRlGZg==", "dev": true, - "license": "ISC", - "engines": { - "node": ">=12" + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" } }, - "node_modules/d3-transition": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/d3-transition/-/d3-transition-3.0.1.tgz", - "integrity": "sha512-ApKvfjsSR6tg06xrL434C0WydLr7JewBB3V+/39RMHsaXTOG0zmt/OAXeng5M5LBm0ojmxJrpomQVZ1aPvBL4w==", + "node_modules/cosmiconfig": { + "version": "9.0.1", + "resolved": "https://registry.npmjs.org/cosmiconfig/-/cosmiconfig-9.0.1.tgz", + "integrity": "sha512-hr4ihw+DBqcvrsEDioRO31Z17x71pUYoNe/4h6Z0wB72p7MU7/9gH8Q3s12NFhHPfYBBOV3qyfUxmr/Yn3shnQ==", "dev": true, - "license": "ISC", + "license": "MIT", "dependencies": { - "d3-color": "1 - 3", - "d3-dispatch": "1 - 3", - "d3-ease": "1 - 3", - "d3-interpolate": "1 - 3", - "d3-timer": "1 - 3" + "env-paths": "^2.2.1", + "import-fresh": "^3.3.0", + "js-yaml": "^4.1.0", + "parse-json": "^5.2.0" }, "engines": { - "node": ">=12" + "node": ">=14" + }, + "funding": { + "url": "https://github.com/sponsors/d-fischer" }, "peerDependencies": { - "d3-selection": "2 - 3" - } - }, - "node_modules/d3-zoom": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/d3-zoom/-/d3-zoom-3.0.0.tgz", - "integrity": "sha512-b8AmV3kfQaqWAuacbPuNbL6vahnOJflOhexLzMMNLga62+/nh0JzvJ0aO/5a5MVgUFGS7Hu1P9P03o3fJkDCyw==", - "dev": true, - "license": "ISC", - "dependencies": { - "d3-dispatch": "1 - 3", - "d3-drag": "2 - 3", - "d3-interpolate": "1 - 3", - "d3-selection": "2 - 3", - "d3-transition": "2 - 3" + "typescript": ">=4.9.5" }, - "engines": { - "node": ">=12" - } - }, - "node_modules/dagre-d3-es": { - "version": "7.0.14", - "resolved": "https://registry.npmjs.org/dagre-d3-es/-/dagre-d3-es-7.0.14.tgz", - "integrity": "sha512-P4rFMVq9ESWqmOgK+dlXvOtLwYg0i7u0HBGJER0LZDJT2VHIPAMZ/riPxqJceWMStH5+E61QxFra9kIS3AqdMg==", - "dev": true, - "license": "MIT", - "dependencies": { - "d3": "^7.9.0", - "lodash-es": "^4.17.21" + "peerDependenciesMeta": { + "typescript": { + "optional": true + } } }, - "node_modules/dayjs": { - "version": "1.11.21", - "resolved": "https://registry.npmjs.org/dayjs/-/dayjs-1.11.21.tgz", - "integrity": "sha512-98IT+HOahAisibz/yjKbzuOBwYcjJ7BCLPzARyHiyEBmRz4fatF+KPJszEHXsGYjUG234aH/cOjW1wwTbKUZlA==", - "dev": true, - "license": "MIT" - }, "node_modules/debug": { "version": "4.4.3", "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", @@ -1860,16 +929,6 @@ } } }, - "node_modules/delaunator": { - "version": "5.1.0", - "resolved": "https://registry.npmjs.org/delaunator/-/delaunator-5.1.0.tgz", - "integrity": "sha512-AGrQ4QSgssa1NGmWmLPqN5NY2KajF5MqxetNEO+o0n3ZwZZeTmt7bBnvzHWrmkZFxGgr4HdyFgelzgi06otLuQ==", - "dev": true, - "license": "ISC", - "dependencies": { - "robust-predicates": "^3.0.2" - } - }, "node_modules/dequal": { "version": "2.0.3", "resolved": "https://registry.npmjs.org/dequal/-/dequal-2.0.3.tgz", @@ -1965,16 +1024,6 @@ "url": "https://github.com/fb55/domhandler?sponsor=1" } }, - "node_modules/dompurify": { - "version": "3.4.7", - "resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.4.7.tgz", - "integrity": "sha512-2jBxDJY4RR06tQNy4w5FlFH7kfxsQZlufd0sbv+chfHCxeJwrFw2baUDsSwvBISD4K4RDbd0PTfy3uNXsR6siA==", - "dev": true, - "license": "(MPL-2.0 OR Apache-2.0)", - "optionalDependencies": { - "@types/trusted-types": "^2.0.7" - } - }, "node_modules/domutils": { "version": "4.0.2", "resolved": "https://registry.npmjs.org/domutils/-/domutils-4.0.2.tgz", @@ -2051,17 +1100,6 @@ "is-arrayish": "^0.2.1" } }, - "node_modules/es-toolkit": { - "version": "1.47.0", - "resolved": "https://registry.npmjs.org/es-toolkit/-/es-toolkit-1.47.0.tgz", - "integrity": "sha512-n1GuoD0WEQZMBk5tttoZSqwgyLx01oqa5XsBmCHwPyNe1S9jPBEmtR2pSgp2kJuWE3ciFZ6yRHmY4pM4C3OOkw==", - "dev": true, - "license": "MIT", - "workspaces": [ - "docs", - "benchmarks" - ] - }, "node_modules/escalade": { "version": "3.2.0", "resolved": "https://registry.npmjs.org/escalade/-/escalade-3.2.0.tgz", @@ -2219,13 +1257,6 @@ "js-yaml": "bin/js-yaml.js" } }, - "node_modules/hachure-fill": { - "version": "0.5.2", - "resolved": "https://registry.npmjs.org/hachure-fill/-/hachure-fill-0.5.2.tgz", - "integrity": "sha512-3GKBOn+m2LX9iq+JC1064cSFprJY4jL1jCXTcpnfER5HYE2l/4EfWSGzkPa/ZDBmYI0ZOEj5VHV/eKnPGkHuOg==", - "dev": true, - "license": "MIT" - }, "node_modules/hast-util-to-html": { "version": "9.0.5", "resolved": "https://registry.npmjs.org/hast-util-to-html/-/hast-util-to-html-9.0.5.tgz", @@ -2315,19 +1346,6 @@ "node": ">=20.19.0" } }, - "node_modules/iconv-lite": { - "version": "0.6.3", - "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.6.3.tgz", - "integrity": "sha512-4fCk79wshMdzMp2rH06qWrJE4iolqLhCUH+OiuIgU++RB0+94NlDL81atO7GX55uUKueo0txHNtvEyI6D7WdMw==", - "dev": true, - "license": "MIT", - "dependencies": { - "safer-buffer": ">= 2.1.2 < 3.0.0" - }, - "engines": { - "node": ">=0.10.0" - } - }, "node_modules/immutable": { "version": "5.1.6", "resolved": "https://registry.npmjs.org/immutable/-/immutable-5.1.6.tgz", @@ -2352,27 +1370,6 @@ "url": "https://github.com/sponsors/sindresorhus" } }, - "node_modules/import-meta-resolve": { - "version": "4.2.0", - "resolved": "https://registry.npmjs.org/import-meta-resolve/-/import-meta-resolve-4.2.0.tgz", - "integrity": "sha512-Iqv2fzaTQN28s/FwZAoFq0ZSs/7hMAHJVX+w8PZl3cY19Pxk6jFFalxQoIfW2826i/fDLXv8IiEZRIT0lDuWcg==", - "dev": true, - "license": "MIT", - "funding": { - "type": "github", - "url": "https://github.com/sponsors/wooorm" - } - }, - "node_modules/internmap": { - "version": "2.0.3", - "resolved": "https://registry.npmjs.org/internmap/-/internmap-2.0.3.tgz", - "integrity": "sha512-5Hh7Y1wQbvY5ooGgPbDaL5iYLAPzMTUrjMulskHLH6wnv/A+1q5rgEaiuqEjB+oxGXIVZs1FF+R/KPN3ZSQYYg==", - "dev": true, - "license": "ISC", - "engines": { - "node": ">=12" - } - }, "node_modules/is-arrayish": { "version": "0.2.1", "resolved": "https://registry.npmjs.org/is-arrayish/-/is-arrayish-0.2.1.tgz", @@ -2460,39 +1457,6 @@ "dev": true, "license": "MIT" }, - "node_modules/katex": { - "version": "0.16.47", - "resolved": "https://registry.npmjs.org/katex/-/katex-0.16.47.tgz", - "integrity": "sha512-Eeo8Ys1doU1z+x8AZsPpQu+p/QcZBI5PeOo7QGQdy2x2m0MU/hYagBbGOmXwr5KVbEfVuWv9LpnQWeehogurjg==", - "dev": true, - "funding": [ - "https://opencollective.com/katex", - "https://github.com/sponsors/katex" - ], - "license": "MIT", - "dependencies": { - "commander": "^8.3.0" - }, - "bin": { - "katex": "cli.js" - } - }, - "node_modules/katex/node_modules/commander": { - "version": "8.3.0", - "resolved": "https://registry.npmjs.org/commander/-/commander-8.3.0.tgz", - "integrity": "sha512-OkTL9umf+He2DZkUq8f8J9of7yL6RJKI24dVITBmNfZBmri9zYZQrKkuXiKhyfPSu8tUhnVBB1iKXevvnlR4Ww==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">= 12" - } - }, - "node_modules/khroma": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/khroma/-/khroma-2.1.0.tgz", - "integrity": "sha512-Ls993zuzfayK269Svk9hzpeGUKob/sIgZzyHYdjQoAdQetRKpOLj+k/QQQ/6Qi0Yz65mlROrfd+Ev+1+7dz9Kw==", - "dev": true - }, "node_modules/kind-of": { "version": "6.0.3", "resolved": "https://registry.npmjs.org/kind-of/-/kind-of-6.0.3.tgz", @@ -2503,13 +1467,6 @@ "node": ">=0.10.0" } }, - "node_modules/layout-base": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/layout-base/-/layout-base-1.0.2.tgz", - "integrity": "sha512-8h2oVEZNktL4BH2JCOI90iD1yXwL6iNW7KcCKT2QZgQJR2vbqDsldCTPRU9NifTCqHZci57XvQQ15YTu+sTYPg==", - "dev": true, - "license": "MIT" - }, "node_modules/lines-and-columns": { "version": "1.2.4", "resolved": "https://registry.npmjs.org/lines-and-columns/-/lines-and-columns-1.2.4.tgz", @@ -2537,13 +1494,6 @@ "uc.micro": "^2.0.0" } }, - "node_modules/lodash-es": { - "version": "4.18.1", - "resolved": "https://registry.npmjs.org/lodash-es/-/lodash-es-4.18.1.tgz", - "integrity": "sha512-J8xewKD/Gk22OZbhpOVSwcs60zhd95ESDwezOFuA3/099925PdHJ7OFHNTGtajL3AlZkykD32HykiMo+BIBI8A==", - "dev": true, - "license": "MIT" - }, "node_modules/markdown-it": { "version": "14.2.0", "resolved": "https://registry.npmjs.org/markdown-it/-/markdown-it-14.2.0.tgz", @@ -2612,19 +1562,6 @@ "url": "https://github.com/fb55/entities?sponsor=1" } }, - "node_modules/marked": { - "version": "16.4.2", - "resolved": "https://registry.npmjs.org/marked/-/marked-16.4.2.tgz", - "integrity": "sha512-TI3V8YYWvkVf3KJe1dRkpnjs68JUPyEa5vjKrp1XEEJUAOaQc+Qj+L1qWbPd0SJuAdQkFU0h73sXXqwDYxsiDA==", - "dev": true, - "license": "MIT", - "bin": { - "marked": "bin/marked.js" - }, - "engines": { - "node": ">= 20" - } - }, "node_modules/mdast-util-to-hast": { "version": "13.2.1", "resolved": "https://registry.npmjs.org/mdast-util-to-hast/-/mdast-util-to-hast-13.2.1.tgz", @@ -2664,36 +1601,6 @@ "node": ">= 8" } }, - "node_modules/mermaid": { - "version": "11.15.0", - "resolved": "https://registry.npmjs.org/mermaid/-/mermaid-11.15.0.tgz", - "integrity": "sha512-pTMbcf3rWdtLiYGpmoTjHEpeY8seiy6sR+9nD7LOs8KfUbHE4lOUAprTRqRAcWSQ6MQpdX+YEsxShtGsINtPtw==", - "dev": true, - "license": "MIT", - "dependencies": { - "@braintree/sanitize-url": "^7.1.1", - "@iconify/utils": "^3.0.2", - "@mermaid-js/parser": "^1.1.1", - "@types/d3": "^7.4.3", - "@upsetjs/venn.js": "^2.0.0", - "cytoscape": "^3.33.1", - "cytoscape-cose-bilkent": "^4.1.0", - "cytoscape-fcose": "^2.2.0", - "d3": "^7.9.0", - "d3-sankey": "^0.12.3", - "dagre-d3-es": "7.0.14", - "dayjs": "^1.11.19", - "dompurify": "^3.3.1", - "es-toolkit": "^1.45.1", - "katex": "^0.16.25", - "khroma": "^2.1.0", - "marked": "^16.3.0", - "roughjs": "^4.6.6", - "stylis": "^4.3.6", - "ts-dedent": "^2.2.0", - "uuid": "^11.1.0 || ^12 || ^13 || ^14.0.0" - } - }, "node_modules/micromark-util-character": { "version": "2.1.1", "resolved": "https://registry.npmjs.org/micromark-util-character/-/micromark-util-character-2.1.1.tgz", @@ -2846,13 +1753,6 @@ "regex-recursion": "^5.1.1" } }, - "node_modules/package-manager-detector": { - "version": "1.6.0", - "resolved": "https://registry.npmjs.org/package-manager-detector/-/package-manager-detector-1.6.0.tgz", - "integrity": "sha512-61A5ThoTiDG/C8s8UMZwSorAGwMJ0ERVGj2OjoW5pAalsNOg15+iQiPzrLJ4jhZ1HJzmC2PIHT2oEiH3R5fzNA==", - "dev": true, - "license": "MIT" - }, "node_modules/pako": { "version": "1.0.11", "resolved": "https://registry.npmjs.org/pako/-/pako-1.0.11.tgz", @@ -2892,13 +1792,6 @@ "url": "https://github.com/sponsors/sindresorhus" } }, - "node_modules/path-data-parser": { - "version": "0.1.0", - "resolved": "https://registry.npmjs.org/path-data-parser/-/path-data-parser-0.1.0.tgz", - "integrity": "sha512-NOnmBpt5Y2RWbuv0LMzsayp3lVylAHLPUTut412ZA3l+C4uw4ZVkQbjShYCQ8TCpUMdPapr4YjUqLYD6v68j+w==", - "dev": true, - "license": "MIT" - }, "node_modules/pdf-lib": { "version": "1.17.1", "resolved": "https://registry.npmjs.org/pdf-lib/-/pdf-lib-1.17.1.tgz", @@ -2932,24 +1825,6 @@ "url": "https://github.com/sponsors/jonschlinkert" } }, - "node_modules/points-on-curve": { - "version": "0.2.0", - "resolved": "https://registry.npmjs.org/points-on-curve/-/points-on-curve-0.2.0.tgz", - "integrity": "sha512-0mYKnYYe9ZcqMCWhUjItv/oHjvgEsfKvnUTg8sAtnHr3GVy7rGkXCb6d5cSyqrWqL4k81b9CPg3urd+T7aop3A==", - "dev": true, - "license": "MIT" - }, - "node_modules/points-on-path": { - "version": "0.2.1", - "resolved": "https://registry.npmjs.org/points-on-path/-/points-on-path-0.2.1.tgz", - "integrity": "sha512-25ClnWWuw7JbWZcgqY/gJ4FQWadKxGWk+3kR/7kD0tCaDtPPMj7oHu2ToLaVhfpnHrZzYby2w6tUA0eOIuUg8g==", - "dev": true, - "license": "MIT", - "dependencies": { - "path-data-parser": "0.1.0", - "points-on-curve": "0.2.0" - } - }, "node_modules/progress": { "version": "2.0.3", "resolved": "https://registry.npmjs.org/progress/-/progress-2.0.3.tgz", @@ -3127,26 +2002,6 @@ "node": ">=0.10.0" } }, - "node_modules/robust-predicates": { - "version": "3.0.3", - "resolved": "https://registry.npmjs.org/robust-predicates/-/robust-predicates-3.0.3.tgz", - "integrity": "sha512-NS3levdsRIUOmiJ8FZWCP7LG3QpJyrs/TE0Zpf1yvZu8cAJJ6QMW92H1c7kWpdIHo8RvmLxN/o2JXTKHp74lUA==", - "dev": true, - "license": "Unlicense" - }, - "node_modules/roughjs": { - "version": "4.6.6", - "resolved": "https://registry.npmjs.org/roughjs/-/roughjs-4.6.6.tgz", - "integrity": "sha512-ZUz/69+SYpFN/g/lUlo2FXcIjRkSu3nDarreVdGGndHEBJ6cXPdKguS8JGxwj5HA5xIbVKSmLgr5b3AWxtRfvQ==", - "dev": true, - "license": "MIT", - "dependencies": { - "hachure-fill": "^0.5.2", - "path-data-parser": "^0.1.0", - "points-on-curve": "^0.2.0", - "points-on-path": "^0.2.1" - } - }, "node_modules/run-parallel": { "version": "1.2.0", "resolved": "https://registry.npmjs.org/run-parallel/-/run-parallel-1.2.0.tgz", @@ -3171,20 +2026,6 @@ "queue-microtask": "^1.2.2" } }, - "node_modules/rw": { - "version": "1.3.3", - "resolved": "https://registry.npmjs.org/rw/-/rw-1.3.3.tgz", - "integrity": "sha512-PdhdWy89SiZogBLaw42zdeqtRJ//zFd2PgQavcICDUgJT5oW10QCRKbJ6bg4r0/UY2M6BWd5tkxuGFRvCkgfHQ==", - "dev": true, - "license": "BSD-3-Clause" - }, - "node_modules/safer-buffer": { - "version": "2.1.2", - "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz", - "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==", - "dev": true, - "license": "MIT" - }, "node_modules/sass": { "version": "1.100.0", "resolved": "https://registry.npmjs.org/sass/-/sass-1.100.0.tgz", @@ -3343,13 +2184,6 @@ "node": ">=0.10.0" } }, - "node_modules/stylis": { - "version": "4.4.0", - "resolved": "https://registry.npmjs.org/stylis/-/stylis-4.4.0.tgz", - "integrity": "sha512-5Z9ZpRzfuH6l/UAvCPAPUo3665Nk2wLaZU3x+TLHKVzIz33+sbJqbtrYoC3KD4/uVOr2Zp+L0LySezP9OHV9yA==", - "dev": true, - "license": "MIT" - }, "node_modules/tar-fs": { "version": "3.1.2", "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-3.1.2.tgz", @@ -3398,16 +2232,6 @@ "b4a": "^1.6.4" } }, - "node_modules/tinyexec": { - "version": "1.2.2", - "resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-1.2.2.tgz", - "integrity": "sha512-M/Q0B2cp4K7kynaT/vnED1j8TlLY+Pp7C6Wl2bl/7u/F0mUVwdyOpwomQb8JpYLitHUssAJRmLZdMCGsrx7i+g==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=18" - } - }, "node_modules/to-regex-range": { "version": "5.0.1", "resolved": "https://registry.npmjs.org/to-regex-range/-/to-regex-range-5.0.1.tgz", @@ -3432,16 +2256,6 @@ "url": "https://github.com/sponsors/wooorm" } }, - "node_modules/ts-dedent": { - "version": "2.2.0", - "resolved": "https://registry.npmjs.org/ts-dedent/-/ts-dedent-2.2.0.tgz", - "integrity": "sha512-q5W7tVM71e2xjHZTlgfTDoPF/SmqKG5hddq9SzR49CH2hayqRKJtQ4mtRlSxKaJlR/+9rEM+mnBHf7I2/BQcpQ==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=6.10" - } - }, "node_modules/tslib": { "version": "1.14.1", "resolved": "https://registry.npmjs.org/tslib/-/tslib-1.14.1.tgz", @@ -3536,20 +2350,6 @@ "url": "https://opencollective.com/unified" } }, - "node_modules/uuid": { - "version": "14.0.0", - "resolved": "https://registry.npmjs.org/uuid/-/uuid-14.0.0.tgz", - "integrity": "sha512-Qo+uWgilfSmAhXCMav1uYFynlQO7fMFiMVZsQqZRMIXp0O7rR7qjkj+cPvBHLgBqi960QCoo/PH2/6ZtVqKvrg==", - "dev": true, - "funding": [ - "https://github.com/sponsors/broofa", - "https://github.com/sponsors/ctavan" - ], - "license": "MIT", - "bin": { - "uuid": "dist-node/bin/uuid" - } - }, "node_modules/vfile": { "version": "6.0.3", "resolved": "https://registry.npmjs.org/vfile/-/vfile-6.0.3.tgz", diff --git a/package.json b/package.json index 83ad5d9a..5a05ac24 100644 --- a/package.json +++ b/package.json @@ -3,10 +3,8 @@ "version": "0.0.0", "private": true, "description": "twinBASIC documentation source, tbdocs static-site builder, and PDF book pipeline.", - "scripts": { - "postinstall": "node builder/scripts/patch-dagre.mjs" - }, "devDependencies": { + "@hpcc-js/wasm-graphviz": "^1.21", "acorn": "^8.0", "acorn-walk": "^8.0", "fast-glob": "^3.3", @@ -18,7 +16,6 @@ "markdown-it-attrs": "^4.3", "markdown-it-deflist": "^3.0", "markdown-it-footnote": "^4.0", - "mermaid": "11.15.0", "sass": "^1.0", "pdf-lib": "1.17.1", "puppeteer": "25.0.4",