Skip to content

Load Node.js in-process on first use: libnode bridge and runtime loader - #1685

Open
cramforce wants to merge 11 commits into
mainfrom
ni/1-libnode-loader
Open

cramforce wants to merge 11 commits into
mainfrom
ni/1-libnode-loader

Conversation

@cramforce

Copy link
Copy Markdown
Contributor

First half of scope-B milestone NI-1 (Node islands): the runtime loads Node.js into the scriptc process the first time a program calls into it, from a prebuilt libnode pack. The second half, the @scriptc/node call API and its tests, is the stacked PR on top of this one.

What changes

  • scr_node_island.c (new optional runtime unit) dlopens the bridge library of a libnode pack on the first call, with RTLD_NOW | RTLD_GLOBAL. It checks the bridge ABI and that the bridge links exactly the Node version recorded at build time, starts one Node environment and runs the embedded bootstrap (scr_node_island_host.cjs, embedded as scr_node_island_host.h by scripts/node-island-host.mjs). A run that never calls into Node never maps libnode.
  • scr_node_bridge.h is the C ABI between the runtime and the bridge. The runtime includes no Node or V8 header.
  • The bridge (packages/libnode-common/bridge/scr_node_bridge.cc) is built against one exact Node release and ships next to libnode in its pack. It:
    • initializes Node once, without default signal handling, the debug-signal handler, the inspector or stdio reinitialization, and with one platform worker thread;
    • creates the environment from Node's built-in snapshot;
    • routes Node's process exit (process.exit, uncaught exceptions) to scriptc's exit sequence;
    • promotes libnode to the global scope so native addons resolve napi_*;
    • runs each call in a CallbackScope with a nested TryCatch.
  • build-pack.mjs assembles a pack (libnode, the bridge, manifest.json, Node's license) from a Node --shared build and the matching headers. Published @scriptc/libnode-<platform> packs are milestone NI-13. Until then a pack is found through SCRIPTC_LIBNODE or --libnode.
  • The SCR_NODE_ISLANDS build of scr_async.c adds one loop slot. Until NI-3 moves scriptc's queues onto Node's libuv, scriptc's loop drives Node's through this slot: the program stays alive while Node has ref'd work, the loop sleeps on Node's backend fd up to Node's next timer, and each loop turn runs one non-blocking turn of Node's loop.
  • Compiler plumbing:
    • the nodeIsland runtime feature: rule, link features, unit table, pack matrix row and the scr_async.c variant;
    • the nodeIsland.call and nodeIsland.callSync libCalls;
    • scr_node_island_install(pack, version) in main;
    • node-islands/libnode.ts, which records the pack: --libnode=<pack dir>|beside, or SCRIPTC_LIBNODE at build time. It reports SC8005 when no pack is usable and requires Node >= 24.13.1.
  • Zero-cost enforcement: the zero-cost gate and the bench nm test now require that no good-code executable defines a Node island symbol.

Locating Node, and bundling it

SCRIPTC_LIBNODE overrides the recorded pack directory at run time. A recorded path that starts with $ORIGIN/ resolves against the directory of the running executable. PATH is never searched, and a pack of another Node version or bridge ABI is a fatal error at first use, never a silent fallback.

--libnode=beside copies the pack next to the executable as <output>.libnode/ and records $ORIGIN/<output>.libnode, so the program and its Node travel together. This is the first form of shipping Node with the program, which the maintainer asked the design to support. The $ORIGIN hook is also what the default embedded packaging (NI-4) will use: the pack's compressed libnode and bridge are appended to the executable and extracted on first use to $XDG_CACHE_HOME/scriptc/node-islands/<hash>/, recorded as a cache-relative path. The same pack serves as the in-process backend, as the bundled Node, and as the build-time probe environment (DESIGN §2.13, §2.16.4). No node executable is ever shipped or required.

Zero cost

Programs without a nodeIsland.* libCall are unchanged:

  • no new unit is linked;
  • the plain scr_async.c object is selected (the variant code is entirely under #ifdef SCR_NODE_ISLANDS, so the default object is byte-identical);
  • main gains no line.

Zero-cost gate (scripts/zero-cost-gate.mjs --set=bench,self,corpus --corpus=all --timing-runs=3, Linux Vercel Sandbox, baseline 0237d70df): 3541 of 3542 programs produced byte-identical IR, LLVM and release executables. In the remaining program, corpus/fs-extra-package, the LLVM and the executable are identical. Its IR differs only in source locations inside the fs-extra shim (packages/compiler/shims/fs-extra), which record each checkout's absolute path (/vercel/sandbox/prev/... against /vercel/sandbox/cand/...). That difference comes from the gate setup, not from this change. Compile time: bench geomean −0.68% [95% CI −2.06%, +0.99%]; the self-hosted compiler +0.27%. Both are within tolerance. The bench nm test passes, and no executable links a Node island symbol.

Tests

  • Unit tests: pack recording (libnode.test.ts) and the host header's freshness (node-island-host.test.ts).
  • End-to-end evidence is in the stacked PR, which adds the API: differential corpus programs against Node 24 run as binaries that load libnode (plain and sanitized lanes), plus harness tests for loading on first use, beside packaging, a missing pack, mismatched versions, and the executable's symbols.

Not yet covered: macOS has no libnode pack yet (NI-13). Process-state integration, signal routing and the hosted loop are NI-3.

🤖 Generated with Claude Code

Scope-B milestone NI-1, runtime side. Programs whose IR has a
`nodeIsland.*` operation link scr_node_island.c and the SCR_NODE_ISLANDS
build of scr_async.c; every other program links neither, byte for byte.

- scr_node_island.c dlopens the bridge of a libnode pack on the first call
  (RTLD_NOW | RTLD_GLOBAL), checks its bridge ABI and that it links exactly
  the Node version recorded at build time, starts one Node environment and
  runs the embedded bootstrap (scr_node_island_host.cjs). A run that never
  calls into Node never maps libnode. The pack directory is SCRIPTC_LIBNODE
  at run time, else the recorded path; a `$ORIGIN/` path resolves against
  the executable's directory (the beside packaging, and the hook for
  shipping Node with the program).
- scr_node_bridge.h is the C ABI between the runtime and the bridge; the
  runtime includes no Node or V8 header. The bridge
  (packages/libnode-common/bridge/scr_node_bridge.cc) initializes Node once
  with no default signal handling, no debug-signal handler or inspector,
  no stdio reinitialization and one platform worker, creates the
  environment from the built-in snapshot, routes Node's process exit to
  the runtime's exit sequence, promotes libnode to the global scope for
  native addons, and runs calls in a CallbackScope with a TryCatch.
  build-pack.mjs assembles a pack (libnode, the bridge, manifest.json) from
  a Node --shared build and the matching headers.
- The SCR_NODE_ISLANDS build of scr_async.c adds a loop slot through which
  the runtime's loop drives Node's until NI-3 hosts the runtime's queues on
  Node's libuv: it keeps the program alive while Node has ref'd work,
  sleeps on Node's backend fd up to its next timer, and runs one
  non-blocking turn per loop turn.
- The `nodeIsland` runtime feature (rule, link features, unit table, pack
  matrix row and scr_async.c variant), the nodeIsland.call/callSync
  libCalls, the install call in main, and the build-time record of the
  pack (node-islands/libnode.ts: --libnode=<pack>|beside or
  SCRIPTC_LIBNODE, SC8005 when none is usable, Node >=24.13.1).
- The zero-cost gate and the bench nm test require that no good-code
  executable defines a Node island symbol.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
scriptc Ready Ready Preview, v0 Oct 22, 2026 12:31pm UTC

Comment thread packages/compiler/src/index.ts
Comment thread packages/compiler/src/node-islands/libnode.ts
Comment thread packages/compiler/src/backend/llvm/lib-shared.ts Outdated
…t lanes

Calls into Node now run on one dedicated 64 MB stack with a guard page,
reached by a context switch, and the bridge sets V8's stack limit on it
(bridge ABI 2). Before, a call from an async fiber (512 KB stacks without a
guard page) failed to bootstrap Node, which the dev-mode LLVM differential
showed. The runtime's stack guard follows the switch, and the switch
carries ASan's fiber annotations.

The Node island tests load Node from a libnode pack. Until the NI-13 packs
are published, .github/actions/libnode-dev-pack assembles one for the
Linux x64 test lanes (plain and sanitized): a pinned, checksum-verified
prebuilt libnode of the repository's Node release (24.22.0) and Node's
official headers (packages/libnode-common/dev-pack.json), with the bridge
built against them by build-pack.mjs. The pack is cached by its inputs and
the bridge source, so only a cache miss downloads and builds; the normal
build path never does.

tests/harness/libnode.ts makes a missing pack a failure in those lanes:
island tests skip only off Linux x64 CI without a configured pack. The
bridge smoke test (packages/libnode-common/test/bridge-smoke.c) loads the
pack the way scr_node_island.c does and makes synchronous, throwing and
asynchronous calls through the bridge with the runtime's bootstrap.

IR, LLVM and object outputs of programs that call into Node no longer need
a pack at build time; only executables record one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scope-B milestone NI-1, API side: the explicit call API that replaces the
build container's hand-written node subprocess (api patch 09).

- The new package @scriptc/node exports callSync(specifier, name, ...args)
  and call(...). The module loads with require() from the calling module;
  arguments and results cross as JSON copies; a thrown Error crosses as its
  nearest intrinsic class with its message and string code, other thrown
  values as JSON copies; callSync refuses a function that returns a
  promise. Its Node implementation runs the calls in-process with the same
  copies, so a program prints the same under Node and compiled.
- The compiler lowers calls to the named or namespace imports
  (node-api-import.ts, lower-node-island.ts) to the nodeIsland.callSync
  and nodeIsland.call libCalls: the calling module's real path, the
  specifier, the export name and the boxed argument array, with the result
  checked against the declared type like `JSON.parse(s) as T`. Arguments
  and results whose types are not JSON data are refused at build time, and
  the functions and the namespace refuse as values.
- Corpus programs 6800-6803 (sync and async results, errors, process.exit
  from Node with the program's exit listeners, Node's pending timers
  keeping the program alive) run under Node and as binaries that load
  libnode; `// @requires-libnode` skips them where SCRIPTC_LIBNODE names no
  pack (until the pack CI matrix publishes packs). The node-island harness
  covers loading Node only on the first call, the beside packaging, a
  missing pack, a pack of another Node version or bridge ABI, and the
  executable's symbols.
- Release tooling publishes @scriptc/node with the other packages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel-security-reviewer

Copy link
Copy Markdown

Security review details

…s manifest, keep the loop for synchronous Node calls, and record the Node call programs' costs

- A --libnode=beside build whose executable comes from the executable
  cache places the pack next to it again (compile, index.ts): the pack
  is an output of the build, and a rebuild after the outputs were
  deleted restored the executable without it.
- The cache key names the libnode pack by the option, the pack
  directory and a digest of its manifest.json (libnodeCacheKey), so a
  pack upgraded in place gives cached executables a new key instead of
  their old recorded Node version.
- nodeIsland.callSync marks the program's event loop live, as
  nodeIsland.call does: the called function may leave Node with ref'd
  timers or I/O, which the loop drives before the program exits.
- scr_runtime.h includes scr_node_island.h, so the LLVM backend's
  declarations of the Node island entry points have prototypes there.
- Corpus costs for 6800-6803, from CI's differential and
  llvm-differential timings of the programs.
cramforce added a commit that referenced this pull request Oct 21, 2026
…s in CI

Stack the libnode packs on the in-process loader (#1685) and use its
pack assembler and manifest format instead of a second one:
build-pack.mjs keeps its CLI and fields (schema, nodeVersion,
moduleVersion, bridgeAbi, platform, arch, files), adds the libnode and
bridge names, strips and (on macOS) re-signs the bridge and links it with
-z defs, and exports buildPack(), which the platform packs call with their
package, target and build inputs as extra manifest fields.

libnodeRecord falls back to the installed @scriptc/libnode-<platform>
pack, found from the entry's directory and then next to the compiler,
when neither --libnode nor SCRIPTC_LIBNODE names one.

The Linux test lanes restore the libnode build that the libnode-packs
workflow cached, assemble the linux-x64 pack from it in seconds
(build.mjs --from-cache never builds Node) and export SCRIPTC_LIBNODE, so
the Node island corpus and harness tests run instead of skipping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cramforce added a commit that referenced this pull request Oct 21, 2026
…s in CI

Stack the libnode packs on the in-process loader (#1685) and use its
pack assembler and manifest format instead of a second one:
build-pack.mjs keeps its CLI and fields (schema, nodeVersion,
moduleVersion, bridgeAbi, platform, arch, files), adds the libnode and
bridge names, strips and (on macOS) re-signs the bridge and links it with
-z defs, and exports buildPack(), which the platform packs call with their
package, target and build inputs as extra manifest fields.

libnodeRecord falls back to the installed @scriptc/libnode-<platform>
pack, found from the entry's directory and then next to the compiler,
when neither --libnode nor SCRIPTC_LIBNODE names one.

The Linux test lanes restore the libnode build that the libnode-packs
workflow cached, assemble the linux-x64 pack from it in seconds
(build.mjs --from-cache never builds Node) and export SCRIPTC_LIBNODE, so
the Node island corpus and harness tests run instead of skipping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The community build's v24.22.0 asset was withdrawn when Node 24.22.1
shipped, so every Linux test shard failed to fetch it (HTTP 404). Pin the
v24.22.1 build and Node's 24.22.1 headers by sha256 instead; the island
tests pass with it in the plain and sanitized lanes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prerelease libnode-dev-24.22.1 mirrors the dockfries v24.22.1 libnode,
Node's v24.22.1 headers and Node's license byte for byte, as CI-only inputs
(not a scriptc release). dev-pack.json lists the mirror first and the
original locations after it; fetch-dev-pack.mjs takes the first source
whose bytes match the pinned sha256 and logs which one it used.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cramforce added a commit that referenced this pull request Oct 22, 2026
…s in CI

Stack the libnode packs on the in-process loader (#1685) and use its
pack assembler and manifest format instead of a second one:
build-pack.mjs keeps its CLI and fields (schema, nodeVersion,
moduleVersion, bridgeAbi, platform, arch, files), adds the libnode and
bridge names, strips and (on macOS) re-signs the bridge and links it with
-z defs, and exports buildPack(), which the platform packs call with their
package, target and build inputs as extra manifest fields.

libnodeRecord falls back to the installed @scriptc/libnode-<platform>
pack, found from the entry's directory and then next to the compiler,
when neither --libnode nor SCRIPTC_LIBNODE names one.

The Linux test lanes restore the libnode build that the libnode-packs
workflow cached, assemble the linux-x64 pack from it in seconds
(build.mjs --from-cache never builds Node) and export SCRIPTC_LIBNODE, so
the Node island corpus and harness tests run instead of skipping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	packages/compiler/src/diagnostics/diagnostic.ts
#	packages/compiler/src/index.ts

This branch was successfully deployed

1 active deployment
Preview — d5a2fac9 Deployed Oct 22, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants