feat(bench): scale benchmark generator + nightly perf budget (6.2, D3)#58
Merged
Merged
Conversation
Guard against scaling regressions with a large synthetic app and a gated budget. - eval/src/bench.ts deterministically generates a ~2,000-file React app with realistic import depth (page -> 8 sections -> 8 atoms + a hook, direct fetch per section), scans it, and asserts a perf budget (--files / --budget-seconds / --budget-rss-mb), measuring wall time and peak RSS via process.resourceUsage().maxRSS. - Measured ~2,016 files in ~2s / ~400 MB — far under the 300s / 4 GB budget, so the optional worker-thread / tree-sitter fast paths were deliberately not applied (no measured bottleneck). - Root `pnpm bench` script; nightly .github/workflows/perf.yml (schedule + manual dispatch) runs the budget-gated bench off the per-PR path. - eval/bench/ gitignored and self-cleaned after each run. eval 317/0/0/0, determinism 1.000; typecheck + lint clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 6.2 — Scale & performance (failure mode D3)
Guards against scaling regressions with a large synthetic app and a gated perf budget.
Changes
eval/src/bench.tsdeterministically generates a ~2,000-file React app with realistic import depth (feature → page → 8 sections → 8 atoms + a hook, with a directfetchper section so the endpoint-resolution pass is exercised too). It scans the tree and asserts a perf budget, measuring wall time and peak RSS viaprocess.resourceUsage().maxRSS. Flags:--files,--budget-seconds,--budget-rss-mb,--keep,--regenerate.pnpm benchscript..github/workflows/perf.yml(schedule + manual dispatch) runs the budget-gated bench so scaling regressions fail loudly without slowing per-PR CI.eval/bench/is gitignored and self-cleaned after each run.Measured
~2,016 files (1,904 components / 896 data sources / 2,688 instances / 8,176 edges) scanned in ~2 s, ~400 MB RSS — far under the 300 s / 4 GB budget. (The synthetic app has no tsconfig, so no type-resolution cost; the real Grafana field run was 6,461 files in 72 s / 1.5 GB.)
Because ts-morph is nowhere near the bottleneck at target scale, the optional worker-thread / tree-sitter fast paths were deliberately not applied — they'd add complexity for no measured win. Revisit if a nightly budget run regresses.
Verification
node eval/dist/bench.js --files 2000→perf budget: OKpnpm eval→ 317 pass · determinism 1.000; full suite, typecheck, lint green.Tracker updated. This bench repo is what 6.1's incremental-rescan property test (deep-equals full rescan on 20 randomized edits) will run against — the last Phase 6 step.
🤖 Generated with Claude Code