Repository navigation
Randomized differential testing harness across the Swift and Rust engines - #158
Merged
Merged
Conversation
… engines GER-2566 item 1: seeded op sequences (send, drop, reorder, duplicate, crash-restore, behind-delivery negative) driven on Swift↔Rust pairs with convergence, healing, and ledger-matched divergence checks. Runs against the deployed pin (c501f9d, binding contract 33) via `just differential-deployed`, which swaps the pin binding/xcframework in and restores after; the suite skips without DIFFERENTIAL_DEPLOYED_PIN so the standard build is unaffected. Known un-ledgered engine divergences (offer surfacing, post-restore frame handling) intentionally fail — see the harness README. Co-Authored-By: Claude Code <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 878d3d1 The changes in this PR will be included in the next version bump. This PR includes changesets to release 0 packagesWhen changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Co-Authored-By: Claude Code <noreply@anthropic.com>
The first pin run's fold-legality rejections were the harness folding non-live offers: records kept after nil-offer deliveries, offers parked across the sender's own later commit, superseded rotation candidates, and offers at the previous group epoch (swift-mls verifying rejects wrongEpoch). Five liveness gates now precede every fold, keyed on the engine's own still-verifies answer — fold rejections drop to 0 across 12 seeds, so no engine defect or rule-8 advert gap exists to ledger. FoldGuardReplica re-runs the engine's guard sequence on dumped offers under DIFFERENTIAL_DEBUG_GUARDS/_FOLDS/_ADVERTS for future root-causing. Co-Authored-By: Claude Code <noreply@anthropic.com>
One row per engine divergence from the differential runs, evaluated against: spec status, sender-legality, accept-legality, sender-fix recoverability, and safe leniency (healing without accepting the bugged behavior). Rows 1 and 4 closed/ledgered; rows 2, 3, 6 carry open cells for the remaining investigations. Co-Authored-By: Claude Code <noreply@anthropic.com>
Duplicate delivery is expected protocol behavior; rejection rides on application-message consumption (ratchet generation), not ciphertext-level duplicate detection — so a true duplicate fails to decrypt in both engines and neither re-surfaces its offer. Row 2's correlation is re-aimed at classifying mismatch sites as duplicate-app, fresh-decrypt, or post-restore. Co-Authored-By: Claude Code <noreply@anthropic.com>
Per-frame (offerSurfaced, errClass) matches on every frame in both directions: duplicate-app re-deliveries reject stale with no offer on either engine (app-consumption model validated directly), all fresh decrypts surface on both, zero fresh-decrypt failures. Residual call-count findings are direction-level offer-presence differences from harness engine-state gating, not offer handling. Co-Authored-By: Claude Code <noreply@anthropic.com>
A fold/deliver call-count difference is by construction a presence difference — the op acts on whatever is live — so cross-direction offer/frame-presence divergence is now classified offer-presence-divergence: counted in a per-run summary line, not failed. Per-frame handling, payloads, remoteCommitApplied, errorClass and probe invariants still fail; presence comparison stays in the pure-pair oracles. The 12-seed pin red set collapses to the post-restore class (matrix row 3) — 189 finding lines. Engine-state-independent scheduling filed as follow-up (GER-2583). Co-Authored-By: Claude Code <noreply@anthropic.com>
The matrix and findings are work product tracked in Linear (linked from GER-2583), not repo content; tool documentation stays in the README. Co-Authored-By: Claude Code <noreply@anthropic.com>
Every wedging run carries a restoreBehindDelivery to establishment-era state, and a pure-Rust pair wedges identically — the designed fail-closed consequence of restoring behind delivered state, not an engine divergence. 15 runs heal fully by replay. One non-restore wedge (seed 1, side-band phase, pre-restore) remains open for triage. The probe invariant now fails when quiescent round-trips fail both directions — both-sides-full symmetry previously masked wedges. Co-Authored-By: Claude Code <noreply@anthropic.com>
Per-op HANDOUT/DELIVER dumps with leg first-byte/len/srcOp/lane/openKind and facade state lines. Seed 1's non-restore wedge triaged to the side-band driver never discharging a round: the responder's leg stays pending after bind, is handed out again, and the peer applies a second ratchetCT for the same round. Engine-side acceptance of a superseded ratchetCT stays an open question until rounds are driven correctly. Co-Authored-By: Claude Code <noreply@anthropic.com>
The generator now emits the full bind-discharge committing round (offer, fold, commit, apply) after each A.4 ratchet round as ordinary scheduler-composed ops, so the PQ turn passes and legs are never re-handed. The restore-legality gate now tracks the receiver's own lastStateSeq at delivery time instead of the emitter's dependsOnSeq watermark (incommensurable counters — restores silently no-op'd before); skips are dumped with a reason. 12-seed rerun: the non-restore wedge is gone, no double-apply signature survives, every remaining wedge is restoreBehindDelivery-driven (the designed fail-closed outcome, pure-Rust pairs included), and restores now fire with visible skip accounting. Co-Authored-By: Claude Code <noreply@anthropic.com>
The restoreBehindDelivery negative op now asserts clean classification rather than probe convergence: findings whose op postdates a fired behind-restore are excluded as designed-wedge (counted per run) unless either side's outcome is unclassified, which always fails. Attribution is structural, so divergence in restore-free windows still surfaces. The 12-seed rerun leaves 216 unexcused findings: a pre-restore remoteCommitApplied divergence and 82 behind-restore outcomes failing clean classification — both recorded as engine-side questions in the Linear evaluation matrix. Co-Authored-By: Claude Code <noreply@anthropic.com>
The error-class table now equates Swift aeadOpenFailed with Rust DecryptionFailed (the same AEAD rejection) and treats unsupportedFrameTag as a classified clean rejection, so unclassified means only .other. The designed-wedge exclusion grows 58->104 and the failing set shrinks 216->170. First-divergent-op analysis shows the pre-restore remoteCommitApplied class is NOT a ledgered cascade: three of four exemplars diverge in rotation-free, restore-free, plain-messaging windows — a genuine per-frame handling divergence, recorded in the Linear evaluation matrix. Residue: four post-restore .other sites (truncatedSection, messageFromUnretainedEpoch) pending owner classification. Co-Authored-By: Claude Code <noreply@anthropic.com>
germ-mark
force-pushed
the
llm/ger-2566-differential-harness
branch
from
September 26, 2026 06:57
d9df6ee to
94e2e8f
Compare
The restore-detail dump now runs after the restore attempt (FIRED=), matching the attribution rule's fired-op set exactly; the per-seed summary line prints behindOps so analysis and rule can't drift again. True 12-seed failing set: 169 (presence 303, designed-wedge 104, ledger-cascade 0). Co-Authored-By: Claude Code <noreply@anthropic.com>
Currently inert (dumps nil at the read site - a normalize-path mismatch to diagnose next round). Re-ride ruled out at every remoteCommitApplied divergence site: all divergent deliveries are first deliveries of those bytes; the 9 duplicate deliveries classify stale symmetrically on both engines. Co-Authored-By: Claude Code <noreply@anthropic.com>
The success-path debugDeliver call never passed epoch: (a silently unmatched call-site edit), so every line printed nil - now 216 delivery lines carry epoch=N on both engines. The per-op state line adds recvEpoch (Swift via @testable groupEpochs(of: recvGroup); Rust per-delivery via the decrypted message epoch). Co-Authored-By: Claude Code <noreply@anthropic.com>
The remoteCommitApplied divergence verdict: no rule-5 engine defect. At every divergent site the two directions hold different receiver epochs, and each engine's report matches its own epoch position (the seed 11 control - matching epochs, both engines true - confirms it). The class reclassifies as state divergence; the underlying commit-cadence divergence under plain messaging (recv epochs split 1 vs 2 by op 15/28, pre-rotation) is a new undiagnosed lead, recorded in the Linear evaluation matrix. Co-Authored-By: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements GER-2566 item 1: seeded randomized op sequences — send, drop, reorder, duplicate, crash-restore (legality-gated, with a behind-delivery negative op) — driven on Swift↔Rust session pairs in-process, checking convergence, re-staple healing, and ledger-matched known divergences.
Tests/TwoMLSPQMigrateTests/Differential/(~1800 lines): op DSL + phase-biased seeded generator, two-engine facade with normalized outcome comparison, two-lane delivery scheduler, engine-attributed divergence ledger (JSON), env-gated test suite.two-mls-pq@c501f9d, binding contract 33):just differential-deployedbuilds the pin xcframework + binding, swaps them in (TWOMLSPQ_PIN_BINDINGexcludes the pin-missing migrate target), runs only the differential filter, and restores. The suite skips withoutDIFFERENTIAL_DEPLOYED_PIN, so the standard build/test/lint are unaffected.replayScriptruns the identical comparison path.queuedProposalnon-optional vs Rust optionalproposal, both mismatch directions, identical under pin and main bindings) and a post-restore frame-handling class (remoteCommitApplied). Root-causing is follow-up work; the ledger's c1 entry was deliberately trimmed so these surface.Deferred: full 250-seed sweep (verified at 12 seeds), born-dedicated topology, CI wiring for the pin run (local-only first cut per GER-2566 decision).
Test notes: standard
swift testgreen with the differential suite skipping (4 tests);swift format lint --strictclean; pin run executes at 12 seeds — pure-pair oracles and replay pass, mixed legs red on the known divergences above (by design).🤖 Generated with Claude Code