Implement a new multi-threaded SAB pull scheduler by KubaO · Pull Request #171 · twinbasic/documentation

KubaO · 2026-06-02T21:04:59Z

SAB stands for SharedArrayBuffer used to sync task state between threads).

DAG = Directed [Edge] Acyclic Graph

- Add worker-pool.mjs (WorkerPool class) - Add scheduler.mjs (Scheduler + SharedState) - Add cpu-worker.mjs (dispatcher skeleton, empty handlers) - Re-export buildInit from template.mjs as buildInitFn - Export serializeLinkTables from render.mjs - Change deriveRedirectStubs filter to frontmatter.layout check - Add precomputedStubs/precomputedUrls passthrough params - Add svgFiles to regenerateMermaid return value

Move Shiki WASM init to a no-dep seed task (highlighterInit)

…cover

Having the dependency in writeAssets unnecessarily delayed the write.

…e/copyable.

The `scheduler-dag` diagram is awful. This is a known issue.

KubaO added 30 commits May 30, 2026 21:52

Plan a new scheduler + dataflow DAG.

86fea06

DAG = Directed [Edge] Acyclic Graph

Scheduler Phase 1: seeds + main-thread spine

88b1f88

Scheduler Phase 2: render fan-out across worker threads

ee2f52d

Scheduler Phase 3: write + post-write tasks

f3e8d9f

Scheduler Phase 4: SAB broadcast for render fan-out

9a4d679

Close out the scheduler work.

6b7f1ba

Update the scheduler documentation.

699ec79

Factor out prepDest and start it immediately. Factor out writePdf.

5a3a818

SearchData really requires renderJoin and prepDest, not Write.

6968a2a

Add the plan to move the offline rewriter to workers.

2b55c5c

Phase I: extract pure-compute offline rewrite into offline-rewrite.mjs

7623c36

Phase II: expand dispatch deps, compute sitePaths, expand SAB payload

855b8eb

Phase III: run offline rewrite in render workers

831d0f3

Phase IV: switch writeOffline to pre-computed HTML

824992b

Phase V: documentation for offline-rewrite-in-workers

3a6d574

Make the build console output colorful.

efd08a3

Output a Gantt chart of the build.

53605b8

Move Shiki WASM init to a no-dep seed task (highlighterInit)

022f4ee

Split scss into scssLight + scssDark workers to parallelize compilation

657dfe8

Hide ...Join tasks from Gantt chart

6e57a40

Write serve Gantt to serve-gantt.mmd, separate from build-gantt.mmd

3e82bdd

Annotate render Gantt bars with compute efficiency percentage

bea0cb6

Extend compute efficiency annotation to all worker tasks

295526a

Show queue+compute split in worker task Gantt labels.

680deb7

Move Shiki WASM init to a no-dep seed task (highlighterInit)

Move loadData/resolveBookChapters earlier in task DAG

ebd1043

highlighterInit stores CSS directly; markdownInit depends only on dis…

6c6bbbf

…cover

Move resolveBookChapters dependency from dispatch to writePdf

d50822e

Defer deriveSitemap to after dispatch to reduce spine contention

fcc0d8a

Defer resolveBookChapters to after deriveSitemap (idle window)

af8e100

KubaO added 29 commits June 1, 2026 16:07

Plan the SharedArrayBuffer (SAB)-based scheduler.

fc1b1f0

Add SAB scheduler skeleton (Phase 5).

309c03d

Implement SAB worker pull loop (Phase 6).

0c7a9f7

Integrate main-thread tasks into the SAB scheduler (Phase 7).

70ea89c

Remove push-model dead code from WorkerPool (Phase 8).

090493f

Add speculative idle execution for warmInit (Phase 9).

7f8f450

Plan phases 10 and 11 of the SAB scheduler.

3d3cf1a

Extract renderEnvInit as explicit per-worker task (Phase 10).

3598bed

Update the plan - phase 11 is a no-go.

e8559c2

Update the Gantt chart formatting.

63edd0c

Pre-create page output dirs during render window (prepPageDirs).

de524e0

Per-worker page flush, replacing renderJoin barrier (Phase 12).

6c87922

Gantt formatting changes.

30d13d6

Move the flushJoin dependency to writeAux.

64a6c19

Having the dependency in writeAssets unnecessarily delayed the write.

Plan phase 13 to capture uniform task timing.

05b3aa8

scss task writes combined CSS directly to both site trees.

be4ffd6

Reintroduce renderJoin barrier; searchData depends on it, not flushJoin.

a370c5f

Plan a deferred phase 14 for moving writePdf off the main thread.

46363db

Add more timing information to the Gannt and make it zoomable/saveabl…

9b26a53

…e/copyable.

Make the Gantt chart also available as a PNG.

7a18ed3

Add more phases to the SAB scheduler plan.

d66a17e

Implement Phase 15: generic dynamic tasks and per-chunk flush.

4a47bef

Update phases 16, 17 and 18 based on the just finished phase 16.

02d0501

Implement Phase 16: persistent worker pool and survives_reset.

5e7ea7c

Implement Phase 17: distribute search-data derivation to render workers.

ba73d5f

Implement Phase 18: move per-page SEO to render workers.

ca67909

Update the Dagre patch to make layout direction optional in subgraphs.

3c7b6bc

Update the tbdocs documentation.

83204e6

The `scheduler-dag` diagram is awful. This is a known issue.

Add dot as a dependency for graphs, remove Mermaid.

969b66e

KubaO merged commit 3a2534d into twinbasic:main Jun 2, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement a new multi-threaded SAB pull scheduler#171

Implement a new multi-threaded SAB pull scheduler#171
KubaO merged 72 commits into
twinbasic:mainfrom
KubaO:staging

KubaO commented Jun 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

KubaO commented Jun 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant