Skip to content

perf(e2e): shrink e2e slot times#24570

Merged
PhilWindle merged 3 commits into
merge-train/spartan-v5from
spl/e2e-slot-cuts
Jul 8, 2026
Merged

perf(e2e): shrink e2e slot times#24570
PhilWindle merged 3 commits into
merge-train/spartan-v5from
spl/e2e-slot-cuts

Conversation

@spalladino

@spalladino spalladino commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

What

PR 5 of the round-4 e2e speedup effort: shrink e2e slot times, the highest-leverage remaining lever
now that the warp lever is exhausted (PR 4) and the fees setup costs are pure inclusion latency at
CI cadence (PR 6). Every cadence-bound cost — setup txs, inclusion waits, epoch walks — scales
linearly with what these commits cut.

Config-only. Three commits, in ascending risk order so the CI-survey/fix phase can revert from
the top. A fourth candidate (WIDE_SLOT 72s → 48s) was investigated and dropped — see below.

All guard math is recomputed from stdlib/src/timetable/budgets.ts +
stdlib/src/timetable/proposer_timetable.ts:

maxBlocksPerCheckpoint = floor((S - init - D - 2P - prepCp) / D)   must be >= 1

where S = aztecSlotDuration, init = checkpointProposalInitTime (1s, never clamped), D =
blockDurationMs/1000, P = p2pPropagationTime, prepCp = checkpointProposalPrepareTime. Fast-profile
clamping (P -> 0.5, prepCp -> 0.5, minBlock -> 1) applies only when ethereumSlotDuration < 8
(FAST_PROFILE_ETHEREUM_SLOT_DURATION). At or above 8s the production budgets apply verbatim.

Commit 1 (lowest risk) — perf(e2e): cut default CI L1 block time to 8s

DEFAULT_L1_BLOCK_TIME: process.env.CI ? 12 : 88. Local dev already ran at 8, so this only
removes a CI-vs-local cadence asymmetry. Default single-node L2 slot goes 24s → 16s.

Guard (default single-node: D = DEFAULT_BLOCK_DURATION_MS/1000 = 3, production budgets P=2,
prepCp=1, init=1):

  • before (eth 12): S = 2x12 = 24 → floor((24 - 1 - 3 - 4 - 1)/3) = floor(15/3) = 5 blocks/cp
  • after (eth 8): S = 2x8 = 16 → floor((16 - 1 - 3 - 4 - 1)/3) = floor(7/3) = 2 blocks/cp

Both >= 1. eth=8 is AT the fast-profile boundary, so budgets stay at production values (no clamping).

Blast radius is much smaller than the round-4 plan assumed. The plan expected this to cut the
fees / cross-chain / bot families, but the base has drifted: those suites now run on
PIPELINING_SETUP_OPTS / AUTOMINE_E2E_OPTS (both set ethereumSlotDuration: 4 explicitly) and no
longer read DEFAULT_L1_BLOCK_TIME. The genuine default-cadence consumers (eth from
SingleNodeTestContext.getSlotDurations, no explicit eth) are four single-node proving/recovery
suites: proving/default_node, proving/cross_chain_public_message, recovery/sync_after_reorg,
partial-proofs/multi_root. The only other process.env.CI timing coupling
(multi-node/slashing/inactivity_setup.ts, process.env.CI ? 8 : 4) sets its own eth slot and does
not read DEFAULT_L1_BLOCK_TIME, so it is untouched.

Commit 2 (medium risk) — perf(e2e): cut reorg cadence to 24s slots

REORG_TIMING_BASE: aztecSlotDuration 36 → 24, blockDurationMs 8000 → 5000 (epoch stays 4 slots).
Shared by both reorg profiles, both below the fast-profile boundary:

  • FAST_REORG_TIMING (eth 4): proving/optimistic reorg describes, l1-reorgs/
  • MULTI_VALIDATOR_REORG_TIMING (eth 6, attestationPropagationTime 0.5): recovery/proposal_failure_recovery,
    recovery/equivocation_recovery, high-availability/ha_sync, high-availability/ha_checkpoint_handoff

Both run fast-profile: P = 0.5, prepCp = 0.5, init = 1.

Guard, FAST_REORG_TIMING and MULTI_VALIDATOR_REORG_TIMING (identical budgets):

  • before: S = 36, D = 8 → floor((36 - 1 - 8 - 1 - 0.5)/8) = floor(25.5/8) = 3 blocks/cp
  • after: S = 24, D = 5 → floor((24 - 1 - 5 - 1 - 0.5)/5) = floor(16.5/5) = 3 blocks/cp

blockDurationMs drops 8000 → 5000 specifically to preserve 3 blocks/checkpoint. Leaving it at 8000
would give floor(13.5/8) = 1, which would break the l1-reorgs suites' assertMultipleBlocksPerSlot(2)
(they push TX_COUNT=8 with maxTxsPerBlock=1 to force multi-block checkpoints). All other timing scales
off test.constants.slotDuration (waits, warps). equivocation_recovery's manual fit calc still holds
(3 x 5s = 15 + 0.5 init + 5 final block + 2.5 finalization = 23s <= 24s); its comment is updated.

Commit 3 (higher risk) — perf(e2e): cut multi-validator block-production cadence to 24s slots

MULTI_VALIDATOR_BLOCK_PRODUCTION_TIMING: aztecSlotDurationInL1Slots 3 → 2 (36s → 24s at eth 12),
blockDurationMs 6000 → 4000. eth stays 12 (production budgets). Consumers: block-production/simple,
first_slot, proof_boundary.

Guard (eth 12, production budgets prepCp=1, init=1; P = per-test attestationPropagationTime), S=24, D=4:

  • simple (P=default 2): floor((24 - 1 - 4 - 4 - 1)/4) = floor(14/4) = 3 (was 4 at 36/6)
  • proof_boundary (P=default 2): floor((24 - 1 - 4 - 4 - 1)/4) = 3 (was 4)
  • first_slot (P=0.5): floor((24 - 1 - 4 - 1 - 1)/4) = floor(17/4) = 4 (was 4)

All >= 1. simple asserts no block-count (only no-sequencer-failures); proof_boundary asserts
proof-vs-boundary timing that scales off test.constants; first_slot keeps 4 blocks/checkpoint
(its comment stays accurate). So the drop 4 → 3 is harmless for those three.

high_tps is pinned to the old 36s/6s cadence at its call site (aztecSlotDurationInL1Slots: 3,
blockDurationMs: 6000 in its setupOpts, overriding the shared profile). Its checkpoint packing is
2 txs x 2.5s = 5s per block, which needs a 6s block sub-slot; a 4s block cannot hold it, and the suite
hard-asserts max-checkpoint-length == 4 and max-txs-per-block == 2. Pinning keeps high_tps at its
tuned cadence rather than rewriting its whole timing model. The other three suites still take the cut.

Dropped candidate — WIDE_SLOT 72s → 48s

Investigated per the plan's stretch item and dropped. blob_promotion hard-asserts a checkpoint
with >= PIPELINE_EXPECTED_BLOCKS_PER_CHECKPOINT = 8 blocks. At the profile's D = 5.5s:

  • current 72/12: floor((72 - 1 - 5.5 - 4 - 1)/5.5) = floor(60.5/5.5) = 11 blocks/cp (fits 8)
  • 48/12: floor((48 - 1 - 5.5 - 4 - 1)/5.5) = floor(36.5/5.5) = 6 blocks/cp (cannot fit 8)

Forcing 8 blocks would require shrinking D to ~4.5s, which lands the guard at exactly 8 with zero
margin — under the deliberate mockGossipSubNetworkLatency: 500ms the suite runs with, and on top of
the profile's documented A-914 constraint (pipelined multiple-blocks-per-slot starves non-proposer
nodes with CheckpointNumberNotSequentialError when the L2 slot is tightened, worsened by a shorter
slot). This is the "provably can't fit" case, and it cannot be verified locally (heavy multi-node
pipelining), so it is not shipped. proposed_chain / cross_chain_messages (only need 2 blocks/cp) would
survive the guard but share the A-914 risk; not worth the exposure for one profile.

Local verification

  • Commit 1: verified. Ran single-node/proving/default_node.test.ts -t 'returns initial block data'
    locally (eth=8 = the new CI cadence, which local already uses). The node came up at
    ethereumSlotDuration: 8, aztecSlotDuration: 16, blockDurationMs: 3000 and logged
    Sequencer timetable initialized with 2 blocks per slot {"maxNumberOfBlocks":2} — exactly the guard
    value above. No "Invalid timing configuration", no missed-slot warnings, test passed. Budgets stayed
    at production values (attestationPropagationTime: 2, minBlockDuration: 2), confirming eth=8 does not
    trip fast-profile clamping.
  • Commits 2 and 3: guard math + CI only. The reorg and multi-validator block-production suites are
    too heavy for the dev machine (multi-node + real-time waits + proving). The guard math above is the
    verification; CI validates end-to-end. The formula was confirmed exact against commit 1's live run.

Revert protocol for the fix phase

Commits are ordered by ascending risk; revert individual commits from the top on flake rather than
fighting them:

  1. Any block-production suite flake (simple / first_slot / proof_boundary) → revert commit 3 first.
    proof_boundary is the most real-time-sensitive (it warps to N-3 then runs the boundary in real time
    at the tighter 24s cadence); watch it first.
  2. Any reorg / prune / HA flake (l1-reorgs, proving/optimistic reorg cases,
    recovery/proposal_failure_recovery, recovery/equivocation_recovery, high-availability/*) → revert
    commit 2. Watch the l1-reorgs assertMultipleBlocksPerSlot(2) assertions and equivocation_recovery's
    slashing-round timing.
  3. Any of the four default-cadence single-node proving/recovery suites → revert commit 1 (lowest
    probability; the cadence is what local already runs).

Notes for PR 7 (bot suite cadence)

The bot suite (single-node/bot/bot.test.ts) uses setup(0, { ...PIPELINING_SETUP_OPTS, ... }), which
sets ethereumSlotDuration: 4 explicitly. It does not read DEFAULT_L1_BLOCK_TIME, so commit 1 does
not affect the bot suite
— its cadence lever remains its own PIPELINING preset, unchanged here.

Measured impact

The cadence cuts land a run-wide −6.4%: −1,600s summed across shards over the 135 suites present in
both runs (−2,046s of improvements against +446s of regressions). The wall-clock benefit is smaller,
since shards run in parallel across workers.

Controls confirm the deltas are real cadence effects, not run-type artifacts. high_tps (deliberately
pinned at the old cadence) is unchanged (+0.1s), and the WIDE_SLOT / explicit-preset proving suites are
flat — pipeline_prune −1.2s, blob_promotion −0.1s, long_proving_time −1.1s, prune_when_cannot_build
−0.1s. Only the default-cadence and reorg/block-production timings moved.

Largest per-suite wall cuts (default-cadence proving/recovery from commit 1; reorg from commit 2):

  • optimistic.parallel −399.9s (−23.5%, ×8 shards)
  • proof_boundary.parallel −346.0s (−25.6%, ×5 shards)
  • full −202.7s; blocks.parallel −196.3s (−28.8%, ×5 shards)
  • invalidate_block.parallel −118.5s (×11 shards); sync_after_reorg −108.3s
  • data_withholding_slash −72.0s; proposal_failure_recovery.parallel −60.1s;
    cross_chain_public_message −48.2s; equivocation_recovery −46.9s; late_prover_tx_collection −43.3s

Regressions, all on high-variance multi-node suites that are not cadence-cut here (run-to-run noise,
not attributable to this change): multi_root +74.1s, inactivity_slash +64.4s,
attested_invalid_proposal.parallel +32.1s, proposed_chain.parallel +26.1s, bot +22.3s.

The metric is whole-suite wall time (sum of test durations + hooks); a cadence change carries no new
span to attribute against, so these are run-vs-run wall deltas — the flat pinned/preset controls above
are what make the cadence attribution defensible.

Baseline CI run 1783417125211175 (merge-base 13a53f1, full). PR CI run 1783426164237985
(x-fast).

Fixes A-1184

@spalladino spalladino added the ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure label Jul 7, 2026
Set DEFAULT_L1_BLOCK_TIME to a plain 8 instead of `process.env.CI ? 12 : 8`. Local runs already
used 8, so this only removes a CI-vs-local cadence asymmetry: default single-node L2 slots drop from
2 x 12 = 24s to 2 x 8 = 16s (-33%) across the fees / cross-chain / bot single-node families, exactly
where the remaining inclusion-latency cost lives.

eth=8 sits AT the fast-profile boundary (FAST_PROFILE_ETHEREUM_SLOT_DURATION = 8): fast-profile
budget clamping only applies strictly below 8s, so the proposer keeps the production operational
budgets, identical to the old eth=12 path.

Guard math (budgets.ts: maxBlocksPerCheckpoint = floor((S - init - D - 2P - prepCp) / D) >= 1),
default single-node config (D = DEFAULT_BLOCK_DURATION_MS/1000 = 3s, production budgets P=2,
prepCp=1, init=1):
  before (eth=12): S = 2x12 = 24 -> floor((24 - 1 - 3 - 4 - 1)/3) = floor(15/3) = 5 blocks/checkpoint
  after  (eth=8):  S = 2x8  = 16 -> floor((16 - 1 - 3 - 4 - 1)/3) = floor(7/3)  = 2 blocks/checkpoint
Both >= 1. The after-config is what every single-node suite already runs locally.

Only tests that rely on the default eth slot are affected. Tests that set ethereumSlotDuration
explicitly (PIPELINING_SETUP_OPTS/bot at eth 4, the reorg and block-production profiles at eth 4/6/12)
are unchanged. The only other process.env.CI timing coupling in the fixtures
(multi-node/slashing/inactivity_setup.ts, `process.env.CI ? 8 : 4`) sets its own eth slot and does not
read DEFAULT_L1_BLOCK_TIME, so it is untouched.
Shrink REORG_TIMING_BASE from a 36s L2 slot / 8s blocks to a 24s L2 slot / 5s blocks (epoch stays 4
slots). This cuts every reorg / prune / HA suite by a third: proving/optimistic's reorg describes and
l1-reorgs/ (via FAST_REORG_TIMING, eth 4s) and recovery/proposal_failure_recovery,
recovery/equivocation_recovery, high-availability/ha_sync, high-availability/ha_checkpoint_handoff
(via MULTI_VALIDATOR_REORG_TIMING, eth 6s).

blockDurationMs drops 8000 -> 5000 (not left at 8000) specifically to preserve blocks-per-checkpoint.
Both reorg profiles run below the fast-profile boundary (eth 4s and 6s < 8s), so budgets clamp to
p2p 0.5s, prepare 0.5s, min-block 1s; init stays 1s.

Guard math (maxBlocksPerCheckpoint = floor((S - init - D - 2P - prepCp) / D) >= 1):
  FAST_REORG_TIMING (eth 4, P=min(2,0.5)=0.5, prepCp=0.5, init=1):
    before: S=36, D=8 -> floor((36 - 1 - 8 - 1 - 0.5)/8) = floor(25.5/8) = 3
    after:  S=24, D=5 -> floor((24 - 1 - 5 - 1 - 0.5)/5) = floor(16.5/5) = 3
  MULTI_VALIDATOR_REORG_TIMING (eth 6, attestationPropagationTime 0.5 -> P=0.5, prepCp=0.5, init=1):
    before: S=36, D=8 -> floor((36 - 1 - 8 - 1 - 0.5)/8) = 3
    after:  S=24, D=5 -> floor((24 - 1 - 5 - 1 - 0.5)/5) = 3
Both preserve exactly 3 blocks per checkpoint. Keeping blockDurationMs at 8000 would have yielded
floor(13.5/8) = 1 block/checkpoint, breaking the l1-reorgs suites' assertMultipleBlocksPerSlot(2)
(they send TX_COUNT=8 with maxTxsPerBlock=1 to force multi-block checkpoints).

Everything else scales off aztecSlotDuration automatically: the tests wait on checkpoint/epoch/proof
boundaries via test.constants.slotDuration and warp via the same. equivocation_recovery's manual
fit calc still holds at the new cadence (3 blocks x 5s = 15s + 0.5 init + 5s final block + 2.5s
finalization = 23s <= 24s); its comment is updated. Doc-only: the profiles' JSDoc, the l1-reorgs and
recovery/HA cadence comments, and optimistic's now-stale "12s CI / 24s" default-cadence note (left
over from the DEFAULT_L1_BLOCK_TIME=8 change) are refreshed to the new numbers.
Shrink MULTI_VALIDATOR_BLOCK_PRODUCTION_TIMING from aztecSlotDurationInL1Slots 3 (36s L2 slot at
eth 12) to 2 (24s), with blockDurationMs 6000 -> 4000. eth stays 12 (production budgets, not fast
profile). Cuts the block-production/simple, first_slot, and proof_boundary suites by a third.

These tests run at eth=12 (>= 8), so budgets are the production defaults (init 1s, prepare 1s,
min-block 2s) with p2p = per-test attestationPropagationTime.

Guard math (maxBlocksPerCheckpoint = floor((S - init - D - 2P - prepCp) / D) >= 1), S=24, D=4:
  simple (P=default 2):        floor((24 - 1 - 4 - 4 - 1)/4) = floor(14/4) = 3  (was 4 at 36/6)
  proof_boundary (P=default 2): floor((24 - 1 - 4 - 4 - 1)/4) = 3               (was 4)
  first_slot (P=0.5):          floor((24 - 1 - 4 - 1 - 1)/4) = floor(17/4) = 4  (was 4; its
                               "4 blocks per checkpoint" comment stays accurate)
All >= 1. simple and proof_boundary assert no fixed block-count (simple checks no sequencer failures;
proof_boundary checks proof-vs-boundary timing that scales off test.constants.slotDuration), so the
drop from 4 to 3 blocks/checkpoint is harmless.

high_tps is PINNED to the old 36s/6s cadence at its call site (aztecSlotDurationInL1Slots: 3,
blockDurationMs: 6000 in its setupOpts, which override the shared profile). Its checkpoint-packing is
2 txs x 2.5s = 5s per block, which needs a 6s block sub-slot; a 4s block cannot hold it, and the
suite hard-asserts max-checkpoint-length == 4 and max-txs-per-block == 2. Pinning keeps high_tps
byte-for-byte at its tuned cadence rather than rewriting its timing model, so its header timing budget
stays correct. The other three suites still take the cut.

Doc-only: profile JSDoc + simple/proof_boundary cadence comments refreshed to 24s/4s.
@spalladino spalladino force-pushed the spl/e2e-slot-cuts branch from 4222567 to 01cdc99 Compare July 7, 2026 12:09
@spalladino spalladino added wip Work in progress and removed wip Work in progress labels Jul 7, 2026
@PhilWindle PhilWindle merged commit eedf4eb into merge-train/spartan-v5 Jul 8, 2026
32 checks passed
@PhilWindle PhilWindle deleted the spl/e2e-slot-cuts branch July 8, 2026 10:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants