Rework skip-already-included-modules serializer#158
Merged
Conversation
The harness serializer used to overwrite its module set on ANY full-bundle serialization (debugger, browser, other entry points), causing over-exclusion and "Requiring unknown module" crashes on device, and it kept a single flat set shared across platforms/dev-prod variants even though their module sets differ. It also built the set with Metro's getAllFiles helper, which expands asset modules to physical variant files on disk instead of the module paths processModuleFilter/createModuleId actually operate on. Only capture when the serialized entry is positively identified as the harness main entry point (matched against the runtime's resolved entry-point module, which both a project's real entryPoint and Expo's virtual entry resolve to), key the captured sets by graph identity (platform/dev/minify/transform profile/custom transform options), and build the set from preModules + graph.dependencies keys. Delegate non-modulesOnly serialization to the user's own customSerializer when present so this stays non-invasive for consumers like Expo. modulesOnly requests fail open (no exclusion) whenever no set was captured for their graph key. Still uses filter-based exclusion for modulesOnly bundles; phase 2 replaces that with blanking to keep source maps and /symbolicate correct.
Metro's .map endpoint and /symbolicate compute line offsets from the full, unfiltered graph using only the config-level processModuleFilter -- they never see a custom serializer's per-call filter override. Omitting already-included modules from the modulesOnly bundle shifted every subsequent module's line offset, producing wrong code frames for failing tests. Instead, serialize the full graph via baseJSBundle with the user's processModuleFilter passed through untouched, then replace each already-included module's code with the same number of blank lines. This keeps the bundle's line count byte-for-byte identical to an unfiltered bundle (so Metro's separately-computed source maps stay correct) while still dropping the __d() definitions that make the test bundle smaller. Also tightens what counts as "already included": the capture pass now mirrors processModules' own isJsModule + config-level processModuleFilter checks, so a module the user's filter rejected from the main bundle (and so never reached the device) is not later blanked out of a test bundle that legitimately needs it.
…hase 3) Add `skipAlreadyIncludedModules`, defaulting to true, as the stable replacement for `unstable__skipAlreadyIncludedModules`. The deprecated alias keeps working (with a deprecation warning) and still serves as the escape hatch (`unstable__skipAlreadyIncludedModules: false`), but the new flag wins whenever both are set. Neither field carries a zod `.default()`: doing so would make "not set" and "explicitly set to the default" indistinguishable, which is exactly the information `resolveSkipAlreadyIncludedModules` needs to apply the new default without breaking the alias's escape-hatch semantics. The default is applied in that resolver instead, which withRnHarness.ts now calls in place of reading the raw config field directly.
…e 4)
Covers: capture only on a matching main entry (including the Expo virtual
entry, resolved via graph.entryPoints); a non-matching full bundle
serialization never clobbers a previously captured set; graph-identity
keying (an iOS capture doesn't leak into an Android modulesOnly request,
which fails open); blanking preserves the bundle's exact total line count
while removing a skipped module's code and leaving other modules'
wrapped code byte-identical; the captured set is built from
graph.dependencies (so an asset module path is captured directly, without
getAllFiles-style expansion); delegation to a user-supplied
customSerializer for main-bundle requests; and the modulesOnly fail-open
path when no set was ever captured for a graph key.
Also fixes a real bug the new tests caught: `metro/private/DeltaBundler/
Serializers/baseJSBundle` and `metro/private/lib/bundleToString` are CJS
modules compiled with `exports.default = ...`, so `require()` returns
`{default: fn}`, not a callable -- the serializer was calling the require()
result directly and would have thrown at runtime the first time the
feature actually ran. Unwrap `.default` for both.
Adds packages/config/vite.config.ts and an `ignoredDependencies` entry in
its eslint config (matching bundler-metro's) so the config package gets a
`test` target (it had none), plus
packages/config/src/__tests__/resolveSkipAlreadyIncludedModules.test.ts
covering the default-true, alias resolution, deprecation warning, and
explicit-flag-wins-over-alias cases.
…ogging Keying the captured "already included" sets by unstable_transformProfile and customTransformOptions broke the feature on Expo: the Expo client's main-bundle request carries transform.engine=hermes, transform.bytecode=1, transform.routerRoot=app, transform.reactCompiler=true, and unstable_transformProfile=hermes-stable (see bundle-url.ts), while the runtime's ?modulesOnly=true test-bundle requests send only modulesOnly + platform, so their graph has default/empty transform options. The lookup therefore always missed, failed open, and silently turned the feature into a permanent no-op on Expo -- a regression vs. the old flat-set behavior. Key by platform, dev, and minify only. Platform keying stays because it is load-bearing (.ios.ts/.android.ts resolve to different paths). Dropping the transform dimensions is still crash-safe: blanking a path only requires that the device holds *a* __d() definition for it, module IDs are path-based via Metro's single server-wide createModuleId counter, and exclusion only applies to paths captured from a main bundle actually served to the device -- transform options change module content, not the presence of a definition. Regression-tested with a hermes-stable/Expo customTransformOptions main graph against a default-options test graph (verified the test fails against the old key). Also add debug logging on capture (graph key + module count) and on a modulesOnly lookup miss (requested key + held keys) so a future entry-point or keying regression degrades with a diagnostic breadcrumb instead of silently serving fat test bundles, and document the capture boundary when serialization is delegated to a user customSerializer (a tree-shaking serializer could invalidate the captured set, but the harness always serves dev-mode bundles, which don't tree-shake).
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Removes the escape-hatch override now that the serializer rework (PR #158) graduated the feature to stable, default-on.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is this?
This PR reworks the experimental
unstable__skipAlreadyIncludedModulesMetro serializer feature and graduates it to a stable, default-onskipAlreadyIncludedModulesconfig option.The feature exists because of how the harness serves tests: the device first loads the main app bundle, then fetches each test file as a separate
?modulesOnly=truebundle. Cross-bundle requires work because Metro's Server uses a singlecreateModuleIdcounter across all graphs, and metro-runtime'sdefine()ignores redefinition of an existing module ID — so re-sending modules the device already has is pure waste. The harness serializer skips them to keep per-test bundles small and cheap to evaluate.The previous implementation was fragile in ways that either crashed the device or silently corrupted symbolication:
Set<string>and overwrote it on every non-modulesOnly serialization. Any full-bundle build for a different entry point or bundle URL (a debugger, a browser tab, another client) replaced the set with the wrong module list, and subsequent test bundles excluded modules the device never received — crashing with "Requiring unknown module". Now capture only happens when the serialized entry is positively identified as the harness main entry (matched against the resolved@react-native-harness/runtime/entry-pointpath, which both a project's realentryPointand Expo's.expo/.virtual-metro-entryresolve to via the harness resolver)..ios.ts/.android.tsresolve to different paths. An iOS capture could poison an Android run. Sets are now stored in aMapkeyed by graph identity (see "Expo consideration" below), and a modulesOnly request with no captured set for its key fails open — it serializes unfiltered rather than guessing.getAllFilesasset mismatch: the set was built with Metro'sgetAllFileshelper, which expands asset modules to their physical variant files on disk (icon@2x.png) instead of the module path Metro uses in the graph — so asset modules never matched and were always re-sent, plus the helper does pointless async fs work. The set is now built directly frompreModules+graph.dependencies, filtered the same way Metro's ownprocessModulesfilters (isJsModule+ the user's config-levelprocessModuleFilter, so a module the user's filter kept out of the main bundle is never wrongly treated as already-delivered)..defaultrequire bug (found by the new tests):metro/private/.../baseJSBundleandmetro/private/lib/bundleToStringare CJS modules compiled withexports.default = fn, sorequire()returns{default: fn}, not a callable. The old code called the require() result directly and would have thrownTypeErrorthe first time the modulesOnly branch actually ran.The serializer also now delegates non-modulesOnly serializations to the user's own
customSerializerwhen one exists (e.g. Expo's) instead of discarding it — capture is a read-only side effect that never changes what gets served.The centerpiece: blanking instead of omission, for correct source maps
The old implementation excluded modules via a per-call
processModuleFilteroverride. That breaks symbolication: Metro's.mapendpoint (Server._processSourceMapRequest) andPOST /symbolicate(Server._explodedSourceMapForBundleOptions) compute line offsets from the full graph using only the config-levelprocessModuleFilter— a custom serializer's per-call override is invisible to them. Omitting a module shifts the line offset of every module after it, so the code frames the harness runtime gets back from/symbolicatefor test failures point at the wrong lines.The fix: serialize the full bundle via
baseJSBundlewith the user'sprocessModuleFilterpassed through untouched, then post-process the returned{pre, post, modules}— for each module whose path is in the captured set, replace its wrapped code with'\n'.repeat(lineCount - 1)beforebundleToString. Line offsets stay byte-for-byte identical to the unfiltered bundle Metro's map endpoints describe, so symbolication is correct with zero server or middleware changes. The skipped__d()definitions still disappear (that's the size/eval win); the blank lines cost almost nothing. One subtlety verified against metro 0.83.3:bundleToStringdrops a zero-length module entry entirely (no trailing newline), so a single-line module blanks to' 'rather than''to preserve its one line.Everything follows one failure-direction rule: under-exclusion (a fatter test bundle) is always safe; over-exclusion crashes the device. Every ambiguous path fails open toward inclusion, and both the capture and the lookup-miss paths emit debug logs so a silent degradation leaves a breadcrumb.
Config graduation
skipAlreadyIncludedModules?: boolean— new stable flag, defaults totrue;falseis the escape hatch.unstable__skipAlreadyIncludedModuleskeeps working as a deprecated alias (logs a deprecation warning); the explicit new flag wins when both are set.resolveSkipAlreadyIncludedModulesrather than the zod schema so "unset" stays distinguishable from "explicitly set" — that distinction is what keeps the alias's escape-hatch semantics working.Expo consideration: graph keying
Captured sets are keyed by
platform/dev/minifyonly — deliberately not byunstable_transformProfile/customTransformOptions. Expo main-bundle requests carrytransform.engine=hermes,transform.bytecode=1,transform.routerRoot=app,transform.reactCompiler=true, andunstable_transformProfile=hermes-stable, while the runtime's?modulesOnly=truerequests send onlymodulesOnly+platform— keying on the transform dimensions would make the lookup miss on every Expo test bundle and silently turn the feature into a permanent no-op. Excluding them is still crash-safe: blanking a path only requires that the device holds a__d()definition for it, module IDs are path-based via the server-widecreateModuleIdcounter, and exclusion only applies to paths captured from a main bundle actually served to the device — transform options change module content, not the presence of a definition. Platform keying stays because it is load-bearing (.ios.ts/.android.tsare different path sets).Test plan
Unit tests (
packages/bundler-metro/src/__tests__/getHarnessSerializer.test.ts, 9 tests, built on fake graphs/modules matching Metro's shapes and exercising the realbaseJSBundle/bundleToString):graph.entryPoints)hermes-stable+ ExpocustomTransformOptionsstill blanks a default-transform-options test bundle (verified this test fails against the old keying)graph.dependenciespaths (asset module paths match directly, no variant-file expansion)customSerializer, with capture still occurringConfig tests (
packages/config/src/__tests__/resolveSkipAlreadyIncludedModules.test.ts, 5 tests): defaulttrue, alias resolution in both directions, deprecation warning emitted only when the alias is set, explicit new flag wins over the alias. The config package previously had notesttarget; this adds itsvite.config.ts.nxtypecheck, lint, and unit tests pass for all affected packages. The runtime package is untouched.Playground validation
The playground app (
apps/playground) previously ran with the escape hatch pinned off (unstable__skipAlreadyIncludedModules: false). This PR removes that override, so the playground now exercises the feature at its new stable default (true) on every CI and local run, alongside the existingunstable__enableMetroCache: truesetting, which is left untouched.To confirm the graduated serializer is actually a win and not just crash-safe, I ran the playground's full
normalsuite (18 test files) on a real Android emulator withRN_HARNESS_DIAGNOSTICS=true, comparing a trace captured with the old pinned-off config against one captured after removing the override, with Metro cache held constant across both runs:Every stage improved, with the largest gains on the device side (
client.collect,client.bundle.module,app.reset) — exactly where smaller per-test bundles should pay off, since the device has less already-known code to receive, parse, and evaluate per test file. This closes the "no device/e2e coverage" gap noted above: the feature is now both correctness-verified (unit tests) and performance-verified (real device run) before graduating to default-on.