perf(e2e): warp dead waits in multi-node recovery and proving tests

[spalladino]

PR 4 of the round-4 e2e speedup effort: extend the merged warpWithSequencersPaused primitive (from #24475, proven on pipeline_prune) to the remaining production-sequencer-bound dead waits. One commit per converted site so a CI-hostile one reverts in isolation.

On the fresh merge-train/spartan-v5 base, most of the sites the plan targeted have already been warp-optimized (they use warpToBuildWindowForSlot / warpToEpochStart / waitUntilNextEpochStarts, and pipeline_prune already uses warpWithSequencersPaused). The residual waits in the remaining sites ride real-time consensus/production that a warp cannot safely collapse — warping would skip the checkpoints/attestations/offenses the tests are verifying. After reading each file on the fresh base, only one genuinely-dead wall-clock wait was left to convert; the rest are honest skips (per the round-3 skip rule: an honest skip beats a flaky convert).

Converted (one commit)

• multi-node/recovery/proposal_failure_recovery — missed-L1-publish test. Between "proposed chain reached slotTwo" and the prune, the test waited in wall-clock (waitUntilL1Timestamp) for the L1 clock to roll past slotOne so the archiver prunes the uncheckpointed slotOne/slotTwo blocks. The pipelined slotTwo broadcast has already reached every node and slotThree does not build until slotTwo, so nothing must be produced in that gap — a genuine dead wait, the direct pipeline_prune analog. Replaced with warpWithSequencersPaused(..., { restart: false }): pause across the warp (warping under a running sequencer would interrupt in-flight builds), keep the sequencers stopped until the prune is confirmed so no proposer builds on the unpruned tip, then restart them for recovery. Expected saving: up to ~1 L2 slot (aztecSlot=36s on this suite's cadence) per run of that test.

Skipped, with reasons

• proposal_failure_recovery — orphan-prune test (second it). The prune wait overlaps P2's concurrent S2 checkpoint rebuild inside the same slot (pipelining builds S2 during S1 while the orphan is pruned). Pausing the sequencers to warp would stop the S2 rebuild the test then asserts on. No pausable dead gap.
• multi-node/recovery/equivocation_recovery. The dominant waits are the heal phase (waitUntilCheckpointNumber / waitForAllNodesToReachCheckpoint for baseline+2) which requires B/C to actually produce two checkpoints after A is stopped, plus waitForOffenseOnNodes which requires real slashing-round detection. Warping skips the very production being verified.
• multi-node/slashing/slash_veto_demo. waitForSubmittableRound accrues inactivity offenses to quorum over proven blocks, and the veto/expiry logic rides real slashing-round consensus. This is proof-submission/consensus real time (the prover must actually run) — warp does not help, same as the fees catchUpProvenChain finding from round 3.
• multi-node/slashing/attested_invalid_proposal (now under slashing/, not invalid-attestations/ — base drift). Epoch advances already use cheatcode warps (advanceToEpoch / advanceToNextEpoch); the remaining waits are real-time committee/attestation/offense-detection rounds (bad proposer must build and broadcast, lazy validator must attest, honest node must detect the offense).
• multi-node/block-production/multi_validator_node. Cost is real-time attestation + checkpoint building via an actual deploy tx; the only clock jump (advanceToEpoch past the validator-set lag) is already a cheatcode warp, and this test was just flake-fixed (fix(e2e): resolve flaky checkpoint TypeError in multi_validator_node test #24543) so it is left untouched.
• single-node/proving/optimistic.parallel (wait:epoch tails, pool 2). Already warp-optimized on this base: every epoch-boundary skip uses warpToEpochStart / waitUntilNextEpochStarts / warpToBuildWindowForSlot. The residual wait:epoch is the deliberate ~2-slot real-time tail (the epoch's final checkpoint must land and the sequencer must settle) — pausing across it would suppress that checkpoint. The waitUntilCheckpointNumber(midCheckpoint) waits require the sequencer to produce two mid-epoch checkpoints, and waitUntilProvenCheckpointNumber(..., 240) rides real prover latency. Nothing left to safely convert.
• wait:tx-mined sites (pool 3). These are inclusion waits for in-flight builds, not waits for a future slot, so they do not qualify (as anticipated in the plan). Not converted.

Local verification

• yarn build (full TS project): passes.
• yarn format end-to-end / yarn lint end-to-end: clean.
• The converted test is a 4-validator mock-gossip pipelining suite, too heavy to run on the dev machine, so it was not executed locally. CI (ci-no-fail-fast) is the verifier; per-commit isolation contains any flake.

Measured impact

The conversion landed as scoped: warp:sequencers-paused goes from 1 occurrence run-wide
(pipeline_prune, pre-existing from #24475) to 2 — the new one on the converted
proposal_failure_recovery test.

• Converted test ("prune and recover when proposer fails to publish to L1"): total 130.8s → 122.5s
  (−8.3s), body 114.6s → 109.2s (−5.4s). The added warp:sequencers-paused span is 36.1s.
• The net saving is modest: the pause+warp+drain+deferred-restart cycle itself costs ~36s, only ~8s
  less than the real-time prune-boundary wait it replaced — not a full slot. The clean attribution is
  the span count (1→2); the ~8s test-total delta sits near the multi-node noise floor.
• Untouched sites are flat within noise: the sibling orphan-prune test 93.7s → 91.5s, and
  pipeline_prune 260.4s → 258.7s.

This is consistent with the PR's own finding that the warp lever is largely exhausted on this base —
one convertible site, a single-test gain.

Baseline CI run 1783374468714213 (merge-base e1711d3, full). PR CI run 1783387686257121 (x-fast).

Notes for the rest of round 4

• No new shared helper was added — the converted site reuses the existing warpWithSequencersPaused on SingleNodeTestContext (inherited by MultiNodeTestContext). PRs 1a/1b/6/5/7 have nothing new to reuse from here.
• Finding relevant to the round's premise: on this drifted base the multi-node recovery/slashing suites' large wait:checkpoint/wait:epoch pools are dominated by real production (checkpoint building, slashing rounds, prover latency), not dead clock time. The warp lever is largely exhausted there; PR 5 (shrink slot times) is the remaining cadence lever for those suites.

Fixes A-1183