Speed up yarn develop on warm restart#3391
Conversation
`yarn develop` previously ran `gatsby clean` before every start, so every run was a cold start: ~150s webpack bundle and ~130s sharp image processing for our ~500 MDX pages. Drop `gatsby clean` from `develop` so the babel-loader, terser, webpack, and sharp caches survive between runs, and add `develop:clean` as an escape hatch when a true reset is needed. The reason `develop` previously required a clean is a Gatsby cache quirk: ExampleFile nodes are derived inside onCreateNode from parent File nodes sourced by gatsby-source-filesystem. On warm restart, Gatsby does not call onCreateNode for cached-unchanged parents, so the derived ExampleFile nodes are never touched and get removed by the stale-node GC. createPages then fails with "Examples result is undefined". Add a sourceNodes hook that touches every ExampleFile each build so they survive. With derived-node GC fixed, GATSBY_EXPERIMENTAL_DEV_SSR is safe to enable; it lazy-compiles pages on demand and is the largest single win on warm restart. Earlier attempts blamed DEV_SSR for the missing nodes, but those failures were the same stale-GC bug presenting earlier. Also restrict gatsby-plugin-sharp formats to ['auto'] in dev so cold image processing doesn't produce WebP variants we never look at locally (production unchanged), and set GATSBY_CPU_COUNT=physical_cores on dev scripts so sharp uses every physical core. Measured locally: cold ~290s -> ~120s; warm restart ~17s end-to-end (previously not viable because of the cache bug).
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
kennethkalmer
left a comment
kennethkalmer
left a comment
There was a problem hiding this comment.
Nice one @zknill, thanks for making everyone's lives better here 🚀
2 participants
yarn developpreviously rangatsby cleanbefore every start, so every run was a cold start: ~150s webpack bundle and ~130s sharp image processing for our ~500 MDX pages. Dropgatsby cleanfromdevelopso the babel-loader, terser, webpack, and sharp caches survive between runs, and adddevelop:cleanas an escape hatch when a true reset is needed.The reason
developpreviously required a clean is a Gatsby cache quirk: ExampleFile nodes are derived inside onCreateNode from parent File nodes sourced by gatsby-source-filesystem. On warm restart, Gatsby does not call onCreateNode for cached-unchanged parents, so the derived ExampleFile nodes are never touched and get removed by the stale-node GC. createPages then fails with "Examples result is undefined". Add a sourceNodes hook that touches every ExampleFile each build so they survive.With derived-node GC fixed, GATSBY_EXPERIMENTAL_DEV_SSR is safe to enable; it lazy-compiles pages on demand and is the largest single win on warm restart. Earlier attempts blamed DEV_SSR for the missing nodes, but those failures were the same stale-GC bug presenting earlier.
Also restrict gatsby-plugin-sharp formats to ['auto'] in dev so cold image processing doesn't produce WebP variants we never look at locally (production unchanged), and set GATSBY_CPU_COUNT=physical_cores on dev scripts so sharp uses every physical core.
Measured locally: cold ~290s -> ~120s; warm restart ~17s end-to-end (previously not viable because of the cache bug).