feat(determinism): stable ordering + eval two-run byte-diff gate (6.3, G8)#55
Merged
Merged
Conversation
… gate (6.3, G8) Make scan output reproducible across runs and machines, and enforce it in CI. - scanReact iterates source files in stable path order (new sortedSourceFiles) across all three file passes, and returns nodes sorted by id + edges by a total-order key over every identifying field (edgeSortKey). - resolveHookEdges re-sorts and dedups after rewriting unresolved-hook placeholders (a rewrite can collide two edges onto one hook id). - Candidate ranking gets a final id tiebreak (same-named components rank deterministically instead of by scan order). - Eval runner scans each fixture twice and byte-compares (dropping only generatedAt): new determinismStablePct metric, printed and hard-gated. - 4 new parser-react determinism tests; queryfn.test order-fragile assertion made order-independent. eval 304/0/0/0, determinism 1.000, all metrics 1.000; typecheck + lint clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 6.3 — Determinism (failure mode G8)
Makes the scan output reproducible across runs and machines, and enforces it in the eval gate. First Phase 6 step because a determinism gate de-risks every later lifecycle step.
Changes
scanReactnow iterates files in a stable path order (sortedSourceFiles) across all three file passes; ts-morph otherwise returns glob-enumeration order, which varies by platform/filesystem.id; edges sorted by a total-order key over every identifying field (edgeSortKey). The query side addresses nodes by id, so array order carries no semantics — only reproducibility.resolveHookEdgesre-sorts and dedups after rewritingunresolved-hook:placeholders (a rewrite can collide two edges onto one hook id).idtiebreak so same-named components in different files rank deterministically.generatedAt): newdeterminismStablePctmetric, printed each run and a hard gate — any non-deterministic fixture fails the run.Tests
queryfn.test.ts(relied on node insertion order to find the react-query/api/statssource ahead of the raw fetch it wraps) rewritten to assert the react-query resolution exists, order-independently.Verification
pnpm eval→ 304 pass · 0 fail · determinism 1.000 · all metrics 1.000pnpm typecheck,pnpm lintall green.Bench-repo half of the accept criterion lands with 6.2; vision/OCR image-hash caching deferred (no live OCR on the eval path today). Tracker updated.
🤖 Generated with Claude Code