Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
86fea06
Plan a new scheduler + dataflow DAG.
KubaO May 30, 2026
c4670c8
Scheduler Phase 0: skeleton + small refactors
KubaO May 30, 2026
88b1f88
Scheduler Phase 1: seeds + main-thread spine
KubaO May 30, 2026
ee2f52d
Scheduler Phase 2: render fan-out across worker threads
KubaO May 30, 2026
f3e8d9f
Scheduler Phase 3: write + post-write tasks
KubaO May 30, 2026
9a4d679
Scheduler Phase 4: SAB broadcast for render fan-out
KubaO May 30, 2026
6b7f1ba
Close out the scheduler work.
KubaO May 30, 2026
699ec79
Update the scheduler documentation.
KubaO May 30, 2026
5a3a818
Factor out prepDest and start it immediately. Factor out writePdf.
KubaO May 30, 2026
6968a2a
SearchData really requires renderJoin and prepDest, not Write.
KubaO May 30, 2026
2b55c5c
Add the plan to move the offline rewriter to workers.
KubaO May 30, 2026
7623c36
Phase I: extract pure-compute offline rewrite into offline-rewrite.mjs
KubaO May 30, 2026
855b8eb
Phase II: expand dispatch deps, compute sitePaths, expand SAB payload
KubaO May 30, 2026
831d0f3
Phase III: run offline rewrite in render workers
KubaO May 30, 2026
824992b
Phase IV: switch writeOffline to pre-computed HTML
KubaO May 30, 2026
3a6d574
Phase V: documentation for offline-rewrite-in-workers
KubaO May 30, 2026
efd08a3
Make the build console output colorful.
KubaO May 30, 2026
53605b8
Output a Gantt chart of the build.
KubaO May 30, 2026
022f4ee
Move Shiki WASM init to a no-dep seed task (highlighterInit)
KubaO May 30, 2026
657dfe8
Split scss into scssLight + scssDark workers to parallelize compilation
KubaO May 30, 2026
6e57a40
Hide ...Join tasks from Gantt chart
KubaO May 30, 2026
3e82bdd
Write serve Gantt to serve-gantt.mmd, separate from build-gantt.mmd
KubaO May 30, 2026
bea0cb6
Annotate render Gantt bars with compute efficiency percentage
KubaO May 30, 2026
295526a
Extend compute efficiency annotation to all worker tasks
KubaO May 30, 2026
680deb7
Show queue+compute split in worker task Gantt labels.
KubaO May 30, 2026
ebd1043
Move loadData/resolveBookChapters earlier in task DAG
KubaO May 30, 2026
6c6bbbf
highlighterInit stores CSS directly; markdownInit depends only on dis…
KubaO May 30, 2026
d50822e
Move resolveBookChapters dependency from dispatch to writePdf
KubaO May 30, 2026
fcc0d8a
Defer deriveSitemap to after dispatch to reduce spine contention
KubaO May 30, 2026
af8e100
Defer resolveBookChapters to after deriveSitemap (idle window)
KubaO May 30, 2026
b88d211
Split buildInit: sidebar into nav, config chrome runs after discover
KubaO May 30, 2026
314e321
Chain config->highlighterInit->loadData to fill discover I/O window
KubaO May 30, 2026
38a57d9
Centralise dest-dir wipe in prepareDestinations for all three output …
KubaO May 31, 2026
ffd1a85
Defer prepDest to after dispatch to avoid I/O contention with discover
KubaO May 31, 2026
98299cf
Chain mermaid after buildInfo to reduce seed-phase CPU contention on CI
KubaO May 31, 2026
6222b26
Slice render chunks 10x finer for balanced worker utilisation
KubaO May 31, 2026
ee89d55
Defer Shiki init so critical-path seeds run without contention
KubaO May 31, 2026
e9ace9c
Prefer Shiki-warm workers for render dispatch; seed mermaid when core…
KubaO May 31, 2026
23a6cc5
Log changed files at rebuild time in serve mode
KubaO May 31, 2026
e6425ae
Draw the Gantt chart ourselves without Mermaid.
KubaO May 31, 2026
2d19d30
Make task time accounting finer-grained.
KubaO May 31, 2026
5f6d35c
Fix a bug in serve mode that made it detect its own writes as file ch…
KubaO May 31, 2026
55ec068
Stop writing the Gantt .mmd file; the SVG is the only output.
KubaO Jun 1, 2026
fc1b1f0
Plan the SharedArrayBuffer (SAB)-based scheduler.
KubaO Jun 1, 2026
309c03d
Add SAB scheduler skeleton (Phase 5).
KubaO Jun 1, 2026
0c7a9f7
Implement SAB worker pull loop (Phase 6).
KubaO Jun 1, 2026
70ea89c
Integrate main-thread tasks into the SAB scheduler (Phase 7).
KubaO Jun 1, 2026
090493f
Remove push-model dead code from WorkerPool (Phase 8).
KubaO Jun 1, 2026
7f8f450
Add speculative idle execution for warmInit (Phase 9).
KubaO Jun 1, 2026
3d3cf1a
Plan phases 10 and 11 of the SAB scheduler.
KubaO Jun 1, 2026
3598bed
Extract renderEnvInit as explicit per-worker task (Phase 10).
KubaO Jun 1, 2026
e8559c2
Update the plan - phase 11 is a no-go.
KubaO Jun 1, 2026
63edd0c
Update the Gantt chart formatting.
KubaO Jun 1, 2026
de524e0
Pre-create page output dirs during render window (prepPageDirs).
KubaO Jun 1, 2026
6c87922
Per-worker page flush, replacing renderJoin barrier (Phase 12).
KubaO Jun 1, 2026
30d13d6
Gantt formatting changes.
KubaO Jun 1, 2026
64a6c19
Move the flushJoin dependency to writeAux.
KubaO Jun 1, 2026
05b3aa8
Plan phase 13 to capture uniform task timing.
KubaO Jun 1, 2026
be4ffd6
scss task writes combined CSS directly to both site trees.
KubaO Jun 1, 2026
a370c5f
Reintroduce renderJoin barrier; searchData depends on it, not flushJoin.
KubaO Jun 1, 2026
46363db
Plan a deferred phase 14 for moving writePdf off the main thread.
KubaO Jun 1, 2026
9b26a53
Add more timing information to the Gannt and make it zoomable/saveabl…
KubaO Jun 1, 2026
7a18ed3
Make the Gantt chart also available as a PNG.
KubaO Jun 1, 2026
d66a17e
Add more phases to the SAB scheduler plan.
KubaO Jun 1, 2026
4a47bef
Implement Phase 15: generic dynamic tasks and per-chunk flush.
KubaO Jun 1, 2026
02d0501
Update phases 16, 17 and 18 based on the just finished phase 16.
KubaO Jun 1, 2026
5e7ea7c
Implement Phase 16: persistent worker pool and survives_reset.
KubaO Jun 2, 2026
ba73d5f
Implement Phase 17: distribute search-data derivation to render workers.
KubaO Jun 2, 2026
ca67909
Implement Phase 18: move per-page SEO to render workers.
KubaO Jun 2, 2026
3c7b6bc
Update the Dagre patch to make layout direction optional in subgraphs.
KubaO Jun 2, 2026
83204e6
Update the `tbdocs` documentation.
KubaO Jun 2, 2026
969b66e
Add dot as a dependency for graphs, remove Mermaid.
KubaO Jun 2, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions WIP.md
Original file line number Diff line number Diff line change
Expand Up @@ -432,6 +432,8 @@ Python scripts are reserved for non-render concerns: one-off content conversion

The site builds via [builder/](builder/), a custom Node.js static site generator (`tbdocs`). See [builder/PLAN.md](builder/PLAN.md) for the architecture overview, [builder/README.md](builder/README.md) for the quickstart, and the [tbdocs Internals](docs/Documentation/Builder.md) site page for the high-level tour.

A task-graph scheduler / parallelisation pass is designed in [builder/PLAN-scheduler.md](builder/PLAN-scheduler.md) and has been implemented (Phases 0--4).

Historical engineering notes from the Jekyll era --- the original build pipeline, the HTML-compress plugin, the per-phase optimisation passes that preceded the JS port, the migration notes, and the Phase 11 parity-update retrospective --- live in [WIP.OldJekyll.md](WIP.OldJekyll.md).

## Build / preview
Expand Down
61 changes: 32 additions & 29 deletions builder/PLAN-5.md
Original file line number Diff line number Diff line change
Expand Up @@ -307,21 +307,22 @@ size. `fs.writeFile(path, buffer)` is the right primitive.
[1] assertNoDestinationCollisions(pages, staticFiles) ← §6.4
(throws on any page.destPath == staticFile.destRel;
runs BEFORE prepareDestination so a collision aborts
without wiping the previous destination)
a collision aborts the build without any destructive I/O)
[2] prepareDestination(destRoot, dryRun) ← §5.1
(delete + recreate, or skip if dry-run)
NOTE: prepareDestination (§5.1) is now a separate scheduler
task (`prepDest`) that runs as a seed -- it overlaps with
the entire main-thread spine and joins only at `write`.
writePhase assumes the destination is already prepared.
[3] In parallel (Promise.all):
[2] In parallel (Promise.all):
writePages(pages, destRoot, limit) ← §5.2
copyTheme(builderAssetsRoot, destRoot, limit) ← §5.3
copyStaticFiles(staticFiles, destRoot, limit) ← §5.4
[4] summarise(totals) ← §5.5
[3] summarise(totals) ← §5.5
(file counts, byte counts, timing; one log line)
```

Expand Down Expand Up @@ -356,23 +357,22 @@ constrained systems; on the dev machine, no cap at all also works
If profiling shows the cap is too low (write throughput < expected),
bump it. The arg lives at the top of `write.mjs` as a constant.

### Why prepare-destination is sequential before the parallel writes
### Why prepare-destination is a separate seed task

Two reasons:
`prepareDestination` (§5.1) deletes and recreates the destination
directory. It now runs as the `prepDest` scheduler seed --- no
dependencies, so it overlaps with the entire main-thread spine and
worker seeds. The `write` task joins on `prepDest` alongside
`renderJoin`, `scss`, and `mermaid`, guaranteeing the destination is
clean before any file is written.

1. **Correctness.** The clean step deletes the existing tree. The
parallel writers would race the delete if it ran concurrently --
a page write could land before the matching directory is removed,
then the delete would either fail (`ENOTEMPTY`) or destroy the
freshly-written file.
2. **Predictability.** A user-facing error from `prepareDestination`
(e.g. "destination is locked by another process") has a clean
single-source point. If it raced with writes, the error message
would be one of dozens of `EBUSY`s with no obvious culprit.
The same two invariants from the original design hold:

The prepare step is ~50 ms (recursive delete of a tree with ~1,080
files + recreate). Sequencing it costs that 50 ms; parallelising
would save it but risk the failure modes above.
1. **Correctness.** The DAG edge `prepDest → write` ensures the
clean step finishes before the parallel writers start.
2. **Predictability.** A `prepareDestination` failure (e.g. locked
directory) surfaces as a single clear error before any write
begins.

### Phase 5 init order (one-time)

Expand Down Expand Up @@ -886,12 +886,12 @@ function assertNoDestinationCollisions(pages, staticFiles) {
}
```

Called from `writePhase` **before** `prepareDestination`. The order
matters: a collision detected after the clean step would have
already wiped the previous `_site-new/` contents, leaving the user
no way to investigate the previous state. Running the assertion
first means a collision aborts the build without any destructive
I/O. Fast (set membership check over ~1,080 entries; <1 ms).
Called from `writePhase` before any file writes. Because
`prepareDestination` now runs as a separate seed task (`prepDest`),
it may have already cleaned the destination by the time the
collision check runs --- but the check still fires before `write`
begins writing any pages, so a collision still aborts cleanly.
Fast (set membership check over ~1,080 entries; <1 ms).

---

Expand Down Expand Up @@ -1614,10 +1614,13 @@ async function main() {
`writePhase(pages, staticFiles, { destRoot, dryRun })`:

1. Calls `assertNoDestinationCollisions(pages, staticFiles)` (§6.4).
2. Calls `prepareDestination(destRoot, dryRun)` (§5.1).
3. `Promise.all`-fans `writePages`, `copyTheme`, `copyStaticFiles`
2. `Promise.all`-fans `writePages`, `copyTheme`, `copyStaticFiles`
(§5.2 / §5.3 / §5.4).
4. Returns `{ pages: {written, skipped}, theme: {copied}, staticFiles: {copied} }`.
3. Returns `{ pages: {written, skipped}, theme: {copied}, staticFiles: {copied} }`.

Note: `prepareDestination` (§5.1) is no longer called inside
`writePhase`. The scheduler's `prepDest` seed task handles it
before `write` runs.

The orchestrator's return value gains `destRoot` so Phase 6 / Phase 7
/ Phase 8 know where to write or read.
Expand Down
Loading