feat: merge-train/spartan-v5#24606
Open
AztecBot wants to merge 14 commits into
Open
Conversation
Updates every README in the e2e test section to describe the suite as it will look once the nine open round-2 consolidation PRs land. This is docs-only — no code, config, or `bootstrap.sh` changes. ## What changed per README - **`end-to-end/README.md`** (root): repointed the `LOG_LEVEL` example off the deleted `proving/empty_blocks.test.ts` to `proving/default_node.test.ts`; linked the new p2p README from the category table (the p2p row now reads "yes" and points at `src/p2p/README.md`); and renamed the `setupBlockProducer` submission-window literal `1024` to the named constant `NO_REORG_SUBMISSION_EPOCHS`. - **`end-to-end/src/single-node/README.md`**: rewrote the `cross-chain/` row for the merged layout (`CrossChainMessagingTest` + `message_test_helpers.ts`, and the `l1_to_l2` / `l1_to_l2_inbox_drift` / `l2_to_l1` / `token_bridge` files); folded `public_payments`/`sponsored_payments` into `private_payments.parallel` and fixed the stale `failures.parallel` name to `failures`; replaced `slasher_config`/`sequencer_config` with the merged `runtime_config`; replaced `world_state_pruning`/`empty_blocks` with the merged `default_node`; added the previously-missing `snapshot_sync` to the `sync/` row; and applied the `NO_REORG_SUBMISSION_EPOCHS` rename. - **`end-to-end/src/multi-node/README.md`**: named `equivocation_offenses` in the slashing row (in place of the separate `duplicate_proposal`/`duplicate_attestation` files) and applied the `NO_REORG_SUBMISSION_EPOCHS` rename in the `MOCK_GOSSIP_MULTI_VALIDATOR_OPTS` preset. - **`end-to-end/src/p2p/README.md`** (new): created with the content added by #24500. - **`end-to-end/src/automine/README.md`** and **`end-to-end/src/infra/README.md`**: untouched — both are behavior-level (no file names) or cover a directory none of the nine PRs change, so they remain accurate as-landed. --------- Co-authored-by: AztecBot <tech@aztec-labs.com>
## What / why
Round-4 e2e speedup, PR 3 of the effort. The `single-node/bot` suite's
file-level `beforeAll` costs
~178s on CI with 97% untagged, and the residual proven-chain waits in
the `private_payments`/`failures`
fees suites (~32s/shard) were also invisible. This PR instruments both
so the cost is attributable, and
diagnoses the bot residual.
Spans are pure passthrough when `TEST_TIMING_FILE` is unset (see
`fixtures/timing.ts`), so this is zero
behavior change for normal runs and CI.
## Commit
- `test(e2e): instrument bot suite setup and fees proven-chain
residuals`
- Bot suite (`single-node/bot/bot.test.ts`): `setup:wallet` around
`EmbeddedWallet.create`,
`wallet:create` around `createSchnorrInitializerlessAccount`, and
`setup:bot` around the four nested
`describe` beforeAll hooks (`Bot.create` / `AmmBot.create` /
`CrossChainBot.create` / `BotStore`).
- Fees harness (`single-node/fees`): `wait:proven-checkpoint` around
`catchUpProvenChain`'s poll loop,
and `warp:proven-checkpoint-epoch` around `advanceToNextEpoch` via a new
instrumented
`FeesTest.advanceToNextEpoch()` wrapper; `waitForEpochProven` and the
direct call sites in
`failures`/`private_payments` now route through it.
- Reuses existing tags where the concept matches (`wallet:create` is
already what
`createFundedInitializerlessAccounts` fires;
`warp:proven-checkpoint-epoch` is already what
`advanceToNextEpoch` fires in `block-production/setup.ts`) rather than
minting near-duplicates.
## Diagnosis
The timing environment (`shared/timing_env.mjs`) folds *every*
`beforeAll` in a file — the file-level
one plus each nested `describe`'s — into a single suite-scoped
`beforeHooksMs`, and every span fired
during any beforeAll lands on that same suite line. The shortlist read
the whole number as the
file-level hook and, seeing only `setup:env:*` tagged, attributed the
rest to the one visible untagged
file-level call (`createSchnorrInitializerlessAccount`). It is actually
the nested hooks.
Local read (`TEST_TIMING_SPANS=1`, `PIPELINING_SETUP_OPTS` = 12s L2
slots, production sequencer, fake
prover), suite `beforeHooksMs = 153,733 ms`:
- `setup:env:*` (file-level `setup()`): ~4.6s — already tagged
- `setup:wallet` (`EmbeddedWallet.create`, ephemeral): **137 ms** — the
PXE has `autoSync:false`, no
genesis walk
- `wallet:create` (`createSchnorrInitializerlessAccount`): **69 ms** —
initializerless, no deploy tx, as
designed
- `setup:bot` (4 nested hooks): **149,582 ms = 97.3%** of the file's
beforeAll
- transaction-bot `Bot.create`: 32.2s (token deploy class+instance, then
private+public mint)
- bridge-resume `new BotStore(...)`: 0.9 ms
- amm-bot `AmmBot.create`: 73.9s (deploys 4 contracts + set_minter +
mint + add_liquidity)
- cross-chain-bot `CrossChainBot.create`: 43.5s (TestContract deploy +
seed 2 L1→L2 msgs + wait ready)
**Root cause: structural, not a bug.** The bot suite runs the production
Sequencer to exercise
CHECKPOINTED/PROPOSED follow modes and L1 fee-juice bridging, so each
`Bot.create` deploys real
contracts and mints serially at slot cadence (`bot/src/factory.ts`). The
two calls the shortlist
suspected are ~0.1s each.
No fix is shipped here because there is no safe one-liner: speeding this
up means batching/parallelizing
the `bot/src/factory.ts` deploy/mint paths (production code,
PR-6-shaped), genesis-seeding (PR 1/2), or
cutting the CI slot time (PR 5) — none belong in the instrumentation PR.
Tracked as a round-5 item; the
new `setup:bot` span makes that win measurable. Full write-up in
`tmp/SPEEDUP_ROUND4_FINDINGS.md`.
## Expected effect
No wall-clock change (passthrough spans). The deliverable is visibility:
the previously-invisible
`setup:bot` / `setup:wallet` / `wallet:create` decomposition of the bot
beforeAll, and the
`wait:proven-checkpoint` / `warp:proven-checkpoint-epoch` residual in
the fees suites.
Measured numbers from a full green CI run will be appended below once
available.
## Measured impact
Both runs are full CI runs. PR run: `ci/x-fast` on head `1f7898cd2b` (CI
1783345614459429, 1,799 test
lines). Baseline: `ci/x-full-no-test-cache` on the exact merge-base
`a8cfa93df5` (CI 1783341726388943,
1,863 test lines). No wall-clock change claimed (spans are passthrough;
suite-level noise floor is
~11s median / ~52s p90) — the deliverable is attribution, measured as
span `busyMs`.
Bot suite (`suite` line, beforeAll = 181.5s on the PR run vs 182.1s
baseline):
- Baseline: 7.4s tagged (`setup:env:*` only) → **174.8s / 96%
invisible**.
- PR run: `setup:bot` **174,203 ms** (4 occurrences, max 86,961 ms = the
amm-bot hook),
`setup:wallet` **250 ms**, `wallet:create` **144 ms**; residual now
**−1.5s ≈ 0** →
**100% of the beforeAll is attributed**.
- This confirms the diagnosis: `EmbeddedWallet.create` +
`createSchnorrInitializerlessAccount` are
~0.4s combined; the "invisible 178s" is the four nested describes'
bot-factory setup
(deploys + mints at production 12s-slot cadence), folded into the file's
suite line by the
hook accounting.
Fees suites (`private_payments.parallel` × 8 shards + `failures`):
- Baseline: avg **32.3s/shard untagged residual** (zero
`wait:proven-checkpoint` /
`warp:proven-checkpoint-epoch` lines on these suites).
- PR run: `wait:proven-checkpoint` fires on all 9 suite hooks, **293,888
ms total**
(avg 32.7s/shard — matches the residual exactly), plus one in-test
occurrence (32.1s);
`warp:proven-checkpoint-epoch` 19 occurrences, 361 ms total (the warp
RPC itself is
near-free — the cost is entirely the proven-chain poll that follows).
Per-shard residual is now
≈ −1s, i.e. fully attributed.
Net: **~469s/run of previously invisible setup/wait cost is now tagged
and attributable**
(174.2s `setup:bot` + 0.4s wallet spans + 294.3s fees proven-chain
waits), with no behavior or
wall-clock change.
Fixes A-1407
…24566) PR 4 of the round-4 e2e speedup effort: extend the merged `warpWithSequencersPaused` primitive (from #24475, proven on `pipeline_prune`) to the remaining production-sequencer-bound dead waits. One commit per converted site so a CI-hostile one reverts in isolation. On the fresh `merge-train/spartan-v5` base, most of the sites the plan targeted have already been warp-optimized (they use `warpToBuildWindowForSlot` / `warpToEpochStart` / `waitUntilNextEpochStarts`, and `pipeline_prune` already uses `warpWithSequencersPaused`). The residual waits in the remaining sites ride real-time consensus/production that a warp cannot safely collapse — warping would skip the checkpoints/attestations/offenses the tests are verifying. After reading each file on the fresh base, only one genuinely-dead wall-clock wait was left to convert; the rest are honest skips (per the round-3 skip rule: an honest skip beats a flaky convert). ## Converted (one commit) - **`multi-node/recovery/proposal_failure_recovery` — missed-L1-publish test.** Between "proposed chain reached slotTwo" and the prune, the test waited in wall-clock (`waitUntilL1Timestamp`) for the L1 clock to roll past slotOne so the archiver prunes the uncheckpointed slotOne/slotTwo blocks. The pipelined slotTwo broadcast has already reached every node and slotThree does not build until slotTwo, so nothing must be produced in that gap — a genuine dead wait, the direct `pipeline_prune` analog. Replaced with `warpWithSequencersPaused(..., { restart: false })`: pause across the warp (warping under a running sequencer would interrupt in-flight builds), keep the sequencers stopped until the prune is confirmed so no proposer builds on the unpruned tip, then restart them for recovery. Expected saving: up to ~1 L2 slot (aztecSlot=36s on this suite's cadence) per run of that test. ## Skipped, with reasons - **`proposal_failure_recovery` — orphan-prune test (second `it`).** The prune wait overlaps P2's concurrent S2 checkpoint rebuild inside the same slot (pipelining builds S2 during S1 while the orphan is pruned). Pausing the sequencers to warp would stop the S2 rebuild the test then asserts on. No pausable dead gap. - **`multi-node/recovery/equivocation_recovery`.** The dominant waits are the heal phase (`waitUntilCheckpointNumber` / `waitForAllNodesToReachCheckpoint` for baseline+2) which requires B/C to actually produce two checkpoints after A is stopped, plus `waitForOffenseOnNodes` which requires real slashing-round detection. Warping skips the very production being verified. - **`multi-node/slashing/slash_veto_demo`.** `waitForSubmittableRound` accrues inactivity offenses to quorum over proven blocks, and the veto/expiry logic rides real slashing-round consensus. This is proof-submission/consensus real time (the prover must actually run) — warp does not help, same as the fees `catchUpProvenChain` finding from round 3. - **`multi-node/slashing/attested_invalid_proposal`** (now under `slashing/`, not `invalid-attestations/` — base drift). Epoch advances already use cheatcode warps (`advanceToEpoch` / `advanceToNextEpoch`); the remaining waits are real-time committee/attestation/offense-detection rounds (bad proposer must build and broadcast, lazy validator must attest, honest node must detect the offense). - **`multi-node/block-production/multi_validator_node`.** Cost is real-time attestation + checkpoint building via an actual deploy tx; the only clock jump (`advanceToEpoch` past the validator-set lag) is already a cheatcode warp, and this test was just flake-fixed (#24543) so it is left untouched. - **`single-node/proving/optimistic.parallel` (`wait:epoch` tails, pool 2).** Already warp-optimized on this base: every epoch-boundary skip uses `warpToEpochStart` / `waitUntilNextEpochStarts` / `warpToBuildWindowForSlot`. The residual `wait:epoch` is the deliberate ~2-slot real-time tail (the epoch's final checkpoint must land and the sequencer must settle) — pausing across it would suppress that checkpoint. The `waitUntilCheckpointNumber(midCheckpoint)` waits require the sequencer to produce two mid-epoch checkpoints, and `waitUntilProvenCheckpointNumber(..., 240)` rides real prover latency. Nothing left to safely convert. - **`wait:tx-mined` sites (pool 3).** These are inclusion waits for in-flight builds, not waits for a future slot, so they do not qualify (as anticipated in the plan). Not converted. ## Local verification - `yarn build` (full TS project): passes. - `yarn format end-to-end` / `yarn lint end-to-end`: clean. - The converted test is a 4-validator mock-gossip pipelining suite, too heavy to run on the dev machine, so it was not executed locally. CI (`ci-no-fail-fast`) is the verifier; per-commit isolation contains any flake. ## Measured impact The conversion landed as scoped: `warp:sequencers-paused` goes from 1 occurrence run-wide (pipeline_prune, pre-existing from #24475) to 2 — the new one on the converted `proposal_failure_recovery` test. - Converted test ("prune and recover when proposer fails to publish to L1"): total 130.8s → 122.5s (−8.3s), body 114.6s → 109.2s (−5.4s). The added `warp:sequencers-paused` span is 36.1s. - The net saving is modest: the pause+warp+drain+deferred-restart cycle itself costs ~36s, only ~8s less than the real-time prune-boundary wait it replaced — not a full slot. The clean attribution is the span count (1→2); the ~8s test-total delta sits near the multi-node noise floor. - Untouched sites are flat within noise: the sibling orphan-prune test 93.7s → 91.5s, and pipeline_prune 260.4s → 258.7s. This is consistent with the PR's own finding that the warp lever is largely exhausted on this base — one convertible site, a single-test gain. Baseline CI run 1783374468714213 (merge-base e1711d3, full). PR CI run 1783387686257121 (x-fast). ## Notes for the rest of round 4 - No new shared helper was added — the converted site reuses the existing `warpWithSequencersPaused` on `SingleNodeTestContext` (inherited by `MultiNodeTestContext`). PRs 1a/1b/6/5/7 have nothing new to reuse from here. - Finding relevant to the round's premise: on this drifted base the multi-node recovery/slashing suites' large `wait:checkpoint`/`wait:epoch` pools are dominated by real production (checkpoint building, slashing rounds, prover latency), not dead clock time. The warp lever is largely exhausted there; PR 5 (shrink slot times) is the remaining cadence lever for those suites. Fixes A-1183
## What
PR 5 of the round-4 e2e speedup effort: shrink e2e slot times, the
highest-leverage remaining lever
now that the warp lever is exhausted (PR 4) and the fees setup costs are
pure inclusion latency at
CI cadence (PR 6). Every cadence-bound cost — setup txs, inclusion
waits, epoch walks — scales
linearly with what these commits cut.
Config-only. Three commits, in **ascending risk order** so the
CI-survey/fix phase can revert from
the top. A fourth candidate (WIDE_SLOT 72s → 48s) was investigated and
**dropped** — see below.
All guard math is recomputed from `stdlib/src/timetable/budgets.ts` +
`stdlib/src/timetable/proposer_timetable.ts`:
maxBlocksPerCheckpoint = floor((S - init - D - 2P - prepCp) / D) must be
>= 1
where `S` = aztecSlotDuration, `init` = checkpointProposalInitTime (1s,
never clamped), `D` =
blockDurationMs/1000, `P` = p2pPropagationTime, `prepCp` =
checkpointProposalPrepareTime. Fast-profile
clamping (`P -> 0.5`, `prepCp -> 0.5`, `minBlock -> 1`) applies only
when `ethereumSlotDuration < 8`
(`FAST_PROFILE_ETHEREUM_SLOT_DURATION`). At or above 8s the production
budgets apply verbatim.
## Commit 1 (lowest risk) — `perf(e2e): cut default CI L1 block time to
8s`
`DEFAULT_L1_BLOCK_TIME`: `process.env.CI ? 12 : 8` → `8`. Local dev
already ran at 8, so this only
removes a CI-vs-local cadence asymmetry. Default single-node L2 slot
goes 24s → 16s.
Guard (default single-node: D = DEFAULT_BLOCK_DURATION_MS/1000 = 3,
production budgets P=2,
prepCp=1, init=1):
- before (eth 12): S = 2x12 = 24 → floor((24 - 1 - 3 - 4 - 1)/3) =
floor(15/3) = **5** blocks/cp
- after (eth 8): S = 2x8 = 16 → floor((16 - 1 - 3 - 4 - 1)/3) =
floor(7/3) = **2** blocks/cp
Both >= 1. eth=8 is AT the fast-profile boundary, so budgets stay at
production values (no clamping).
**Blast radius is much smaller than the round-4 plan assumed.** The plan
expected this to cut the
fees / cross-chain / bot families, but the base has drifted: those
suites now run on
`PIPELINING_SETUP_OPTS` / `AUTOMINE_E2E_OPTS` (both set
`ethereumSlotDuration: 4` explicitly) and no
longer read `DEFAULT_L1_BLOCK_TIME`. The genuine default-cadence
consumers (eth from
`SingleNodeTestContext.getSlotDurations`, no explicit eth) are four
single-node proving/recovery
suites: `proving/default_node`, `proving/cross_chain_public_message`,
`recovery/sync_after_reorg`,
`partial-proofs/multi_root`. The only other `process.env.CI` timing
coupling
(`multi-node/slashing/inactivity_setup.ts`, `process.env.CI ? 8 : 4`)
sets its own eth slot and does
not read `DEFAULT_L1_BLOCK_TIME`, so it is untouched.
## Commit 2 (medium risk) — `perf(e2e): cut reorg cadence to 24s slots`
`REORG_TIMING_BASE`: aztecSlotDuration 36 → 24, blockDurationMs 8000 →
5000 (epoch stays 4 slots).
Shared by both reorg profiles, both below the fast-profile boundary:
- `FAST_REORG_TIMING` (eth 4): proving/optimistic reorg describes,
l1-reorgs/
- `MULTI_VALIDATOR_REORG_TIMING` (eth 6, attestationPropagationTime
0.5): recovery/proposal_failure_recovery,
recovery/equivocation_recovery, high-availability/ha_sync,
high-availability/ha_checkpoint_handoff
Both run fast-profile: P = 0.5, prepCp = 0.5, init = 1.
Guard, FAST_REORG_TIMING and MULTI_VALIDATOR_REORG_TIMING (identical
budgets):
- before: S = 36, D = 8 → floor((36 - 1 - 8 - 1 - 0.5)/8) =
floor(25.5/8) = **3** blocks/cp
- after: S = 24, D = 5 → floor((24 - 1 - 5 - 1 - 0.5)/5) = floor(16.5/5)
= **3** blocks/cp
blockDurationMs drops 8000 → 5000 specifically to preserve 3
blocks/checkpoint. Leaving it at 8000
would give floor(13.5/8) = **1**, which would break the l1-reorgs
suites' `assertMultipleBlocksPerSlot(2)`
(they push TX_COUNT=8 with maxTxsPerBlock=1 to force multi-block
checkpoints). All other timing scales
off `test.constants.slotDuration` (waits, warps).
equivocation_recovery's manual fit calc still holds
(3 x 5s = 15 + 0.5 init + 5 final block + 2.5 finalization = 23s <=
24s); its comment is updated.
## Commit 3 (higher risk) — `perf(e2e): cut multi-validator
block-production cadence to 24s slots`
`MULTI_VALIDATOR_BLOCK_PRODUCTION_TIMING`: aztecSlotDurationInL1Slots 3
→ 2 (36s → 24s at eth 12),
blockDurationMs 6000 → 4000. eth stays 12 (production budgets).
Consumers: block-production/simple,
first_slot, proof_boundary.
Guard (eth 12, production budgets prepCp=1, init=1; P = per-test
attestationPropagationTime), S=24, D=4:
- simple (P=default 2): floor((24 - 1 - 4 - 4 - 1)/4) = floor(14/4) =
**3** (was 4 at 36/6)
- proof_boundary (P=default 2): floor((24 - 1 - 4 - 4 - 1)/4) = **3**
(was 4)
- first_slot (P=0.5): floor((24 - 1 - 4 - 1 - 1)/4) = floor(17/4) =
**4** (was 4)
All >= 1. simple asserts no block-count (only no-sequencer-failures);
proof_boundary asserts
proof-vs-boundary timing that scales off `test.constants`; first_slot
keeps 4 blocks/checkpoint
(its comment stays accurate). So the drop 4 → 3 is harmless for those
three.
`high_tps` is **pinned to the old 36s/6s cadence at its call site**
(`aztecSlotDurationInL1Slots: 3`,
`blockDurationMs: 6000` in its setupOpts, overriding the shared
profile). Its checkpoint packing is
2 txs x 2.5s = 5s per block, which needs a 6s block sub-slot; a 4s block
cannot hold it, and the suite
hard-asserts `max-checkpoint-length == 4` and `max-txs-per-block == 2`.
Pinning keeps high_tps at its
tuned cadence rather than rewriting its whole timing model. The other
three suites still take the cut.
## Dropped candidate — WIDE_SLOT 72s → 48s
Investigated per the plan's stretch item and **dropped**.
`blob_promotion` hard-asserts a checkpoint
with >= `PIPELINE_EXPECTED_BLOCKS_PER_CHECKPOINT = 8` blocks. At the
profile's D = 5.5s:
- current 72/12: floor((72 - 1 - 5.5 - 4 - 1)/5.5) = floor(60.5/5.5) =
**11** blocks/cp (fits 8)
- 48/12: floor((48 - 1 - 5.5 - 4 - 1)/5.5) = floor(36.5/5.5) = **6**
blocks/cp (cannot fit 8)
Forcing 8 blocks would require shrinking D to ~4.5s, which lands the
guard at exactly 8 with zero
margin — under the deliberate `mockGossipSubNetworkLatency: 500ms` the
suite runs with, and on top of
the profile's documented A-914 constraint (pipelined
multiple-blocks-per-slot starves non-proposer
nodes with `CheckpointNumberNotSequentialError` when the L2 slot is
tightened, worsened by a shorter
slot). This is the "provably can't fit" case, and it cannot be verified
locally (heavy multi-node
pipelining), so it is not shipped. proposed_chain / cross_chain_messages
(only need 2 blocks/cp) would
survive the guard but share the A-914 risk; not worth the exposure for
one profile.
## Local verification
- **Commit 1: verified.** Ran `single-node/proving/default_node.test.ts
-t 'returns initial block data'`
locally (eth=8 = the new CI cadence, which local already uses). The node
came up at
`ethereumSlotDuration: 8, aztecSlotDuration: 16, blockDurationMs: 3000`
and logged
`Sequencer timetable initialized with 2 blocks per slot
{"maxNumberOfBlocks":2}` — exactly the guard
value above. No "Invalid timing configuration", no missed-slot warnings,
test passed. Budgets stayed
at production values (`attestationPropagationTime: 2, minBlockDuration:
2`), confirming eth=8 does not
trip fast-profile clamping.
- **Commits 2 and 3: guard math + CI only.** The reorg and
multi-validator block-production suites are
too heavy for the dev machine (multi-node + real-time waits + proving).
The guard math above is the
verification; CI validates end-to-end. The formula was confirmed exact
against commit 1's live run.
## Revert protocol for the fix phase
Commits are ordered by ascending risk; revert individual commits from
the top on flake rather than
fighting them:
1. **Any block-production suite flake** (simple / first_slot /
proof_boundary) → revert commit 3 first.
proof_boundary is the most real-time-sensitive (it warps to N-3 then
runs the boundary in real time
at the tighter 24s cadence); watch it first.
2. **Any reorg / prune / HA flake** (l1-reorgs, proving/optimistic reorg
cases,
recovery/proposal_failure_recovery, recovery/equivocation_recovery,
high-availability/*) → revert
commit 2. Watch the l1-reorgs `assertMultipleBlocksPerSlot(2)`
assertions and equivocation_recovery's
slashing-round timing.
3. **Any of the four default-cadence single-node proving/recovery
suites** → revert commit 1 (lowest
probability; the cadence is what local already runs).
## Notes for PR 7 (bot suite cadence)
The bot suite (`single-node/bot/bot.test.ts`) uses `setup(0, {
...PIPELINING_SETUP_OPTS, ... })`, which
sets `ethereumSlotDuration: 4` explicitly. It does not read
`DEFAULT_L1_BLOCK_TIME`, so **commit 1 does
not affect the bot suite** — its cadence lever remains its own
PIPELINING preset, unchanged here.
## Measured impact
The cadence cuts land a run-wide −6.4%: −1,600s summed across shards
over the 135 suites present in
both runs (−2,046s of improvements against +446s of regressions). The
wall-clock benefit is smaller,
since shards run in parallel across workers.
Controls confirm the deltas are real cadence effects, not run-type
artifacts. `high_tps` (deliberately
pinned at the old cadence) is unchanged (+0.1s), and the WIDE_SLOT /
explicit-preset proving suites are
flat — pipeline_prune −1.2s, blob_promotion −0.1s, long_proving_time
−1.1s, prune_when_cannot_build
−0.1s. Only the default-cadence and reorg/block-production timings
moved.
Largest per-suite wall cuts (default-cadence proving/recovery from
commit 1; reorg from commit 2):
- optimistic.parallel −399.9s (−23.5%, ×8 shards)
- proof_boundary.parallel −346.0s (−25.6%, ×5 shards)
- full −202.7s; blocks.parallel −196.3s (−28.8%, ×5 shards)
- invalidate_block.parallel −118.5s (×11 shards); sync_after_reorg
−108.3s
- data_withholding_slash −72.0s; proposal_failure_recovery.parallel
−60.1s;
cross_chain_public_message −48.2s; equivocation_recovery −46.9s;
late_prover_tx_collection −43.3s
Regressions, all on high-variance multi-node suites that are not
cadence-cut here (run-to-run noise,
not attributable to this change): multi_root +74.1s, inactivity_slash
+64.4s,
attested_invalid_proposal.parallel +32.1s, proposed_chain.parallel
+26.1s, bot +22.3s.
The metric is whole-suite wall time (sum of test durations + hooks); a
cadence change carries no new
span to attribute against, so these are run-vs-run wall deltas — the
flat pinned/preset controls above
are what make the cadence attribution defensible.
Baseline CI run 1783417125211175 (merge-base 13a53f1, full). PR CI
run 1783426164237985
(x-fast).
Fixes A-1184
…24564) ## What / why PR 2 of the round-4 e2e speedup effort. The fees-family suites pay a `setup:bridge` span (~48s ≈ 4 production slots per process) in `FeesTest.applyFPCSetup`, bridging fee juice from L1 to the BananaFPC via the gas portal (`bridgeFromL1ToL2` = prepare-on-L1 + advance 2 blocks + a claim tx). That funding is pure setup plumbing — the fees tests snapshot the FPC's gas balance and assert deltas, they don't assert on the bridge itself. Fee-juice balances are just public-data leaves at `computeFeePayerBalanceLeafSlot(address)`, which `getGenesisValues` (`world-state/src/testing.ts`) already prefills for genesis-funded accounts — the same mechanism that funds the initial accounts and (already, on this base) the sponsored FPC. This PR seeds the BananaFPC's fee juice at genesis instead of bridging it during setup. Note: the plan's premise that `fundSponsoredFPC: true` bridges was stale — the sponsored FPC has been genesis-funded since #19532 (setup pushes its deterministic address into the genesis-funded list). The remaining `setup:bridge` cost in the fees family is the BananaFPC bridge, which this PR removes. ## Mechanism - The BananaFPC address depends on the BananaCoin address and the FPC admin (the first setup account), and the accounts are generated *inside* `setup()` with random secrets — so the FPC address can't be precomputed by the test before genesis. Fixed deploy salts are added for BananaCoin and BananaFPC so both addresses become deterministic once the admin is known, and a `computeExtraGenesisFundedAddresses(defaultAccounts)` hook on `SetupOptions` runs after the accounts are generated and before genesis values are computed. `FeesTest` uses it to derive the BananaFPC address from the first account and add it to the genesis-funded list. - The returned address flows through the existing initial-accounts path (`addressesToFund` → `getGenesisValues`), so it is funded with the same fee juice as an initial account (`10^22`) and automatically included in the L1 `FeeJuicePortal` `feeJuicePortalInitialBalance` (`fundingNeeded`) accounting — the portal's locked L1 balance stays consistent with total L2 fee-juice supply, exactly as for every other genesis-funded balance. - `applyFPCSetup` deploys the FPC with the fixed salt and asserts the deployed address equals the seeded one, so any drift in the deterministic deploy params surfaces as a clear error instead of a downstream "insufficient fee payer balance". The `bridgeFromL1ToL2` call is removed. - The `setup:bridge` span wrapper is kept in place for the bridging that remains, so its disappearance from the fees setup hooks is the measured proof. ## Scope decisions (site by site) - **`FeesTest.applyFPCSetup` BananaFPC funding** → moved to genesis. Used by `account_init`, `private_payments.parallel`, `failures`, `gas_estimation.parallel`. This is the only funding that moved. - **`FeesTest.mintAndBridgeFeeJuice`** (bridges to an arbitrary recipient) → kept. It fires in `account_init` test bodies that test the bridge/claim flow ("pays natively in the Fee Juice after Alice bridges funds") — real behavior under test. - **`account_init` `FeeJuicePaymentMethodWithClaim` / `prepareTokensOnL1`** → kept. Tests the atomic claim-in-deploy flow. - **Sponsored FPC** → already genesis-funded via `fundSponsoredFPC: true` on this base; no change. - **`cross_chain_messaging_test`** (`fundSponsoredFPC: true`) → already genesis-funded; its own cross-chain bridging harness is the behavior under test, kept. - **`bench/client_flows_benchmark`** bridges to its BananaFPC → out of scope (benchmark harness, not in the fees e2e timing family), kept. - Production/sandbox funding paths (cli, `aztec sandbox`) → untouched. The new hook defaults to unset; only the e2e fixtures opt in. ## Expected effect ~48s × the number of fees processes that ran the FPC bridge in setup (account_init, private_payments, failures, gas_estimation). No C++, no new config, no change to production genesis roots. ## Local verification - `single-node/fees/account_init -t 'pays privately through an FPC'` passes; the span leaderboard for the run shows **zero `setup:bridge`** occurrences (was ~48s), while `deploy:fpc`/`deploy:token` remain. The FPC-paying test asserts the FPC's gas balance decreases by the fee, so the genesis funding is exercised end to end, and the deployed-address assertion held. - `single-node/fees/account_init -t 'after Alice bridges funds'` passes with `setup:bridge` present in the test body, confirming the retained bridge/claim harness still works. ## Measured impact The `setup:bridge` span is eliminated across the run: 17 occurrences totalling 646.4s of bridge setup time in the baseline drop to zero in the PR run. The 13 fees-hook bridges (~48s each) are the bulk of it. - Fees beforeHooks per shard: `private_payments.parallel` −51.1s (×8 shards), `gas_estimation.parallel` −48.1s (×3), `failures` −47.8s, `account_init` −47.8s — each matching the ~48s (4-slot) bridge cost. - `setup:bridge` busyMs removed per suite: account_init 48.1s, failures 48.2s, gas_estimation ×3 = 144.6s, private_payments ×8 = 384.7s (13 fees-hook bridges ≈ 625.5s), plus the ~5s sponsored-FPC bridges on `amm` and `storage_proof`. - Total beforeHooks reduction over the suites present in both runs: −907.4s summed across shards (wall-clock benefit is smaller, since the fees shards run in parallel). - One suite moved the wrong way within noise: `fee_juice_payments` +11.0s. It keeps its real bridging/claim txs (no `setup:bridge` span removed), so this is run-to-run variance, not a regression from this change. Baseline CI run 1783374468714213 (merge-base e1711d3, full). PR CI run 1783375878950264 (x-fast — the comparison is per-suite over the suites present in both runs). Fixes A-1405
Parallelizes the independent setup steps in the bot factory (`@aztec/bot`, `bot/src/factory.ts`). This is production bot code that runs against live networks, so the changes are strictly behavior-preserving: same contracts at the same (salt-derived) addresses, same amounts, same final state. Only the ordering/concurrency of provably-independent setup steps changes. The bot factory is the single largest cost in the `single-node/bot` e2e suite: each `*.create` runs the production sequencer at 12s L2 slots (`PIPELINING_SETUP_OPTS`) and deploys/mints one tx per slot, fully serial. `AmmBot.create` alone was ~74s (7 serial txs). ## Dependency graph per bot type Edges below are "must be mined before", verified against the Noir contract source (`token_contract`, `amm_contract`). ### Transaction bot -- `setup()` (unchanged) `setupAccount` -> register recipient -> `ensureFeeJuiceBalance` -> deploy token -> mint. This chain is fully data-dependent: funding gates the deploy, the deploy gates the mint. The recipient (`createSchnorrAccount`) is register-only (no tx), and the private+public mints are already batched into one tx. Nothing is independent, so `setup()` is left serial. ### AMM bot -- `setupAmm()` Steps: deploy token0 (D0), token1 (D1), LP token (DL), AMM (DA); `set_minter` granting the AMM rights over the LP token (SM); mint token0+token1 to the provider (M, one batched tx); `add_liquidity` (AL). - D0, D1, DL: independent -- distinct contracts, no cross-references. - DA: the AMM constructor only stores the three token addresses (no calls into the tokens). Those addresses are salt-derived, so DA needs only the (pre-derived) addresses, not the token deploys mined. - SM: `Token::set_minter` just writes `minters[amm] = true`; it does not call or validate the AMM contract. It needs the LP token deployed and the (derivable) AMM address only -- not the AMM deployed. The deployer is authorized because the Token constructor sets `minters[admin] = true`. - M: `mint_to_private`/`mint_to_public` require the caller be a minter; the deployer is admin=minter from the constructor, so the token0/token1 mints need only those tokens deployed (no `set_minter`). - AL: `add_liquidity` transfers token0/token1 from the provider (needs M for balances), mints LP tokens (needs SM so the AMM is an LP minter), and targets the AMM (needs DA). Restructured from 7 serial txs into 3 slot-phases: - Phase 1: `Promise.all([D0, D1, DL])` - Phase 2: `Promise.all([DA, SM, M])` - Phase 3: `AL` ### Cross-chain bot -- `setupCrossChain()` Steps: deploy TestContract (an L2 tx), seed N L1->L2 messages (L1 txs), wait for the first message ready. - The seeds reference only the L2 recipient (TestContract) address, which is salt-derived. L1->L2 messages are queued on L1 and do not require the L2 contract to exist yet; they are consumed later in `bot.run()`, after setup completes. So the L2 deploy and the L1 seeding are independent. - The L2 deploy pays via the L2 wallet/PXE account; the seeds use the bot's L1 client -- different accounts, so there is no L1 nonce interaction between them. Restructured to overlap the deploy with the seed loop: `Promise.all([deploy TestContract, seed loop])`, then the message-ready wait. ## What stayed serial, and why - The whole transaction-bot `setup()` (data-dependent chain, nothing independent). - `ensureFeeJuiceBalance` before every deploy (funding must precede spending). - `add_liquidity` after its mints + minter grant + AMM deploy. - The individual L1->L2 seeds inside the loop: they share the bot's single L1 account, so concurrent sends would race on the L1 nonce. The suite already documents this nonce hazard. Only the loop as a whole overlaps the (L2) TestContract deploy. ## Production safety / idempotent re-entry - Every deploy still goes through `registerOrDeployContract`, which checks `getContractMetadata(address).isContractPublished` per contract and only registers (no tx) if already deployed. If one parallel deploy fails and the others succeed, the next `*.create` recovers: the succeeded contracts register-only, the failed one redeploys. Addresses are salt-derived and identical across runs. - `set_minter` is idempotent (writes `true` again). The AMM path's mint and `add_liquidity` always run (no balance guard) -- unchanged from the previous `fundAmm`. - Same-sender concurrent `.send()`s are safe by construction: the PXE serializes simulate/prove through its `SerialQueue` and setup txs pay with random tx nonces. The bot runs the same `EmbeddedWallet` -> PXE path in the e2e suite and in production. - Deliberate, benign failure-path change: because Phase 2 runs DA/SM/M concurrently, a failure of one no longer prevents the others from being sent. Their effects (a deployed-but-unused AMM, a minter grant, an extra mint) are all idempotent-safe, and `add_liquidity` uses fixed amounts, so re-entry converges to the same final state. - `withNoMinTxsPerBlock` (the `flushSetupTransactions` save/zero/restore wrapper around each setup tx) was not safe to re-enter concurrently: interleaved save/zero/restore could read the already-zeroed value and "restore" `minTxsPerBlock` to 0 permanently. It now reference-counts entrants -- the first saves and zeroes, the last restores -- with unit tests covering the overlapping and failure paths (`bot/src/factory.test.ts`). Serial callers see the exact same RPC sequence as before. Final state is identical in every path: same contract addresses, same minter grant, same minted amounts, same liquidity. ## Local evidence Two full local runs of `single-node/bot/bot.test.ts` (production sequencer, 12s L2 slots), all 10 tests passing in both. Hook durations from factory log timestamps, against the round-4 local baseline measured on the same machine at the same cadence (findings from #24534): - `Bot.create` (transaction-bot hook, untouched): 31.8s vs 32.2s baseline -- unchanged, as intended. - `AmmBot.create`: 57.2s vs 73.9s baseline (-16.7s). The three token deploy txs go out within 300ms of each other; the AMM deploy, `set_minter` and the mint batch all go out within the following slot. - `CrossChainBot.create`: 24.0s vs 43.5s baseline (-19.5s). The TestContract deploy overlaps both L1 seeds; the first message is ready almost immediately after the deploy is mined. - Suite-level `beforeHooksMs`: 116.9s vs 153.7s baseline (-36.8s), matching the per-hook deltas. - Unit: `bot/src/factory.test.ts` (4 tests) covering the `withNoMinTxsPerBlock` reentrancy fix. ## Expected effect on CI - `AmmBot.create`: 7 serial txs -> 3 slot-phases (~87s on CI per #24534's spans -> ~55-60s). - `CrossChainBot.create`: deploy overlaps the seed pipeline (~43s -> ~25s). - Transaction bot: unchanged. - Aggregate: roughly -35 to -45s on the bot suite's beforeAll wall time. - Fix-phase proof metric (once #24534's `setup:bot` instrumentation is on this base): the amm-bot and cross-chain `setup:bot` occurrences drop per the numbers above. Measured CI numbers to be appended when a full run lands. ## Measured impact The bot suite (a single shard) drops 216.3s → 118.3s total (**−98.0s, −45%**); beforeHooks 215.9s → 118.0s. - No `setup:bot` spans exist on this base (that instrumentation lives in the unmerged #24534), so the metric is the suite-level hook fold, which covers all three nested bot factory setups (Bot, AmmBot, CrossChainBot). - Noise context from the same run pair: untouched light suites move −2 to −3s per shard (private_initialization −2.9s/shard over 20 shards, deploy_method −2.1s/shard over 14), and the largest counter-move is validators_sentinel +24s (multi-node variance). A −98s single-shard delta is far outside that envelope. - The CI delta exceeds the local −36.8s because the CI baseline pays more missed-slot penalties per serial tx (baseline beforeHooks 215.9s on CI vs 153.7s locally at the same 12s cadence); collapsing 7 serial txs into 3 slot-phases removes proportionally more where slots are more often missed. Baseline CI run 1783393028773172 (merge-base 30966a4, full). PR CI run 1783427250198215 (x-fast). Fixes A-1408
) ## What Adds opt-in plumbing to seed nullifier leaves into the genesis nullifier tree, threaded through the C++ world state and the TypeScript world-state layer. The field is optional and defaults to empty, so **production genesis is unchanged** — every existing genesis root stays bit-identical to today. - **C++**: `WorldState` constructors, the napi binding, `create_canonical_fork`, and the standalone `aztec-wsdb` IPC server gain a `prefilled_nullifiers` input, inserted as the nullifier tree's initial leaves. The indexed nullifier tree requires those leaves to be unique and strictly increasing (and distinct from the padding leaves), which the tree already enforces; the TS side additionally checks it before the native call for a friendlier error. - **TS**: `GenesisData.prefilledNullifiers` (optional, defaults to `[]`) threads through `native_world_state_instance.ts`. `getGenesisValues` gains an optional `prefilledNullifiers` param (sorted ascending on the way in), and the `GENESIS_ARCHIVE_ROOT` fast-path in `generateGenesisValues` now also checks the nullifier list is empty, so the short-circuit only fires for the canonical empty genesis. ## Why Mechanism for round-4 e2e speedup PR 1b, which will seed the standard-contract (AuthRegistry / PublicChecks / HandshakeRegistry) registration nullifiers at genesis in the e2e fixtures so their publish txs short-circuit. This PR carries no consumers and is intentionally behavior- and timing-neutral: with the field empty (the only value ever passed in production), the native tree, all genesis roots, and the archive root are identical to before. ## Relationship to #24254 This is a deliberate subset port of #24254 ("seed protocol contract registration nullifiers at genesis"), kept shape- and signature-identical wherever it does not diverge, so that when #24254 itself reaches the v5 line the mechanism merges as "already applied". The intentional divergences: - `prefilledNullifiers` is **optional and defaults to empty** here (#24254 made it required and non-empty by default via `DEFAULT_GENESIS_DATA`). - This PR does **not** port `DEFAULT_GENESIS_DATA`, the protocol-contract nullifier generation (`protocol-contracts/src/genesis_data.ts`, `scripts/generate_data.ts`), or **any** constant recomputation (`GENESIS_ARCHIVE_ROOT`, `constants.nr`, `constants.gen.ts`, `ConstantsGen.sol`, `aztec_constants.hpp`, checkpoint / AVM golden fixtures). With the field empty, every genesis root is unchanged, which is the invariant CI must prove. ## Testing (local) - C++ `world_state_tests`: `GetInitialTreeInfoForAllTrees` (unchanged — still asserts the current empty-nullifier root `0x18935581…` and `GENESIS_ARCHIVE_ROOT`, proving empty-field behavior is bit-identical), `GetInitialTreeInfoWithPrefilledPublicData` (updated to pass an empty nullifier vector), and a new `GetInitialTreeInfoWithPrefilledNullifiers` (seeding changes the root, tree size stays 128, seeded leaves are present) — 3/3 pass. - `@aztec/world-state` `testing.test.ts` (7/7): empty genesis returns `GENESIS_ARCHIVE_ROOT` via the fast path; the empty-genesis computed root matches that constant; an explicit empty `prefilledNullifiers` yields the same root as omitting the field; a non-empty sorted list yields a deterministic root that differs from empty; `getGenesisValues` sorts and seeds; and non-strictly-increasing (descending / duplicate) lists are rejected by the defensive check. - Regression: `@aztec/world-state` `native_world_state.test.ts` (53/53), `@aztec/stdlib` `l2_block.test.ts` (3/3, confirms the genesis nullifier root is unchanged), `@aztec/prover-client` `lightweight_checkpoint_builder.test.ts` (7/7). - Full `yarn build`, plus `yarn format`/`yarn lint` on the touched packages, are clean. ## Notes This PR is intentionally behavior- and timing-neutral; no e2e behavior changes until PR 1b adopts the mechanism. ## Measured impact None expected, and none observed. `prefilledNullifiers` defaults to `[]`, so the production genesis payload and roots are bit-identical (asserted by the unchanged C++ empty-nullifier root and `GENESIS_ARCHIVE_ROOT` tests); no fixture opts into a non-empty list in this PR. CI timings confirm neutrality: per-suite beforeHooks deltas scatter in both directions within the per-suite noise floor — largest were runtime_config −12.8s, token_bridge −12.7s, account_init +11.5s, failures +11.0s — with the fees suites essentially flat (private_payments −3.4s, gas_estimation −0.2s) and no systematic direction. The summed beforeHooks delta over all 276 suites present in both runs is −122.8s (~−2%), well inside accumulated run-to-run variance. Baseline CI run 1783374468714213 (merge-base e1711d3, full). PR CI run 1783389062016888 (x-fast). Part of A-1404
Round-4 e2e speedup, PR 1b (adoption half of the AuthRegistry prize). **Stacked on #24567** (`spl/genesis-prefilled-nullifiers`, the genesis-nullifier mechanism). This PR targets that branch; retarget to `merge-train/spartan-v5` once #24567 merges. ## What Make e2e environments start with the standard contracts (AuthRegistry, PublicChecks, HandshakeRegistry) already "published", so `ensureAuthRegistryPublished` and its two siblings skip their two publish txs each. At production cadence `setup:auth-registry` alone was ~16 min/run (2 sequential txs per process, paid per-process across the suite). This collapses that span to sub-second. ## Mechanism (three parts, all default-off outside e2e) - **Genesis nullifiers.** Per standard contract, seed `siloNullifier(ContractClassRegistry, classId)` and `siloNullifier(ContractInstanceRegistry, instanceAddress)` (the real derived address — standard contracts are not magic-address protocol contracts) into the fixtures' `getGenesisValues` call, via PR 1a's new 5th param. These are exactly the nullifiers the publish txs would emit, so the AVM's deployment-nullifier check passes when the contracts are called publicly. A single helper (`getStandardContractGenesisNullifiers`) feeds all four e2e genesis builders (`fixtures/setup.ts`, `e2e_prover_test.ts`, `p2p/p2p_network.ts`, `multi-node/governance/add_rollup.test.ts`) — the latter three recompute a genesis that must reproduce the L1-deployed archive root, so they must seed the identical set. Non-e2e callers (cli, sandbox/local-network) are untouched. - **Archiver preload.** `registerStandardContracts` mirrors `registerProtocolContracts`: it seeds each contract's class (with recomputed `publicBytecodeCommitment`), instance, and public-function signatures into the contract store at block 0, reading the bundled `@aztec/standard-contracts` artifacts. It is idempotent (skips already-registered classes on restart) and gated behind a new archiver config flag `testPreloadStandardContracts` (env `TEST_PRELOAD_STANDARD_CONTRACTS`, default **false**). The flag is set from the e2e node config so every spawned node (validators, prover nodes) picks it up through the normal config path. This adds `@aztec/standard-contracts` as an archiver dependency. - **Guard short-circuit.** The `ensure*Published` helpers are unchanged. Their guards read `wallet.getContractClassMetadata(id).isContractClassPubliclyRegistered` (to `aztecNode.getContractClass`) and `wallet.getContractMetadata(addr).isContractPublished` (to `aztecNode.getContract`) — both are archiver-store reads, not nullifier-tree reads. The store preload alone makes both short-circuit; the genesis nullifiers are what make the contract actually callable in the AVM afterwards. Keeping the helpers doubles as a regression check: if seeding ever breaks, the tx path re-engages and tests still pass, just slow. ## Why the flag is test-only (A-1257 rationale) Preloading unconditionally in production would recreate the A-1257 collision #24254 fixed: a real on-chain publish of a preloaded class would collide with the block-0 preload, because production genesis will **not** carry the matching nullifiers. The flag is only set by the e2e fixtures, which also seed the nullifiers, keeping the store and the nullifier tree consistent. A single source of truth (`getPublishableStandardContracts`) drives both the preloaded set and the seeded-nullifier set so they cannot drift. ## State-write verification Inspected the two publish txs on this base: - `ContractClassRegistry.publish` pushes one nullifier (`classId`, siloed to the class-registry address) and broadcasts a `ContractClassPublished` contract-class log carrying the packed bytecode. No public-data writes. - `ContractInstanceRegistry.publish_for_public_execution` asserts the class-registration nullifier exists, then pushes one nullifier (`address`, siloed to the instance-registry address) and broadcasts a `ContractInstancePublished` private log. No public-data writes. So the full state-write set is two nullifiers + two broadcast logs. The nullifiers are replicated via genesis `prefilledNullifiers`; the archiver normally learns the class/instance from those two logs (`data_store_updater.ts`), and the block-0 preload replaces that path exactly. No `genesisPublicData` is required. PXE-side registration (`wallet.registerContract`) is unaffected and still runs in the helpers. ## Local verification - `automine/accounts/authwit.test.ts` (public-authwit path against the standard AuthRegistry), timing on: passes; `setup:auth-registry` span = **10 ms** (was ~33 s at production cadence). No on-chain AuthRegistry publish (the only `ContractClassRegistry.publish` executions are the test's own AuthWitTest/GenericProxy deploys); the public AuthRegistry contract executes fine against the seeded nullifier. `testPreloadStandardContracts: true` confirmed in the node config dump. - `single-node/fees/account_init.test.ts` (production sequencer + simulated prover node), timing on: 5/5 pass; `setup:auth-registry` span = **10 ms**. Prover node syncs against the seeded genesis (no root divergence), and the flag propagates to it. No duplicate-nullifier errors. - New unit test in `archiver/src/modules/data_store_updater.test.ts`: `registerStandardContracts` preloads each publishable standard contract's class + instance into the store and is idempotent on a second call. ## Expected impact ~14–16 min/run from auth-registry alone, plus more from the two untagged siblings (PublicChecks / HandshakeRegistry) wherever used. ## Measured impact `setup:auth-registry` collapses from 702.2s of setup time run-wide (39 occurrences, 5–35s each) to 482ms total (max 87ms per occurrence) — sub-second run-wide, every shard clean, no seeding or flag-propagation slow path. - The measured beforeHooks reduction is larger than the auth-registry span alone, because the two untagged sibling publishes (PublicChecks, HandshakeRegistry) are removed too. Example: account_init beforeHooks −49.6s vs a 30.9s tagged auth-registry span; the extra ~19s is the two siblings. - Per-suite beforeHooks deltas: account_init −49.6s, fee_settings −49.5s, failures −48.7s, private_payments.parallel −41.8s/shard (×8), gas_estimation.parallel −41.5s/shard (×3), l1_to_l2 −38.1s, l2_to_l1 −37.5s, l1_to_l2_inbox_drift −38.2s, token_bridge −37.7s, fee_juice_payments −37.4s, bot −34.3s; automine/parallel suites shed 5–15s each (their fast-cadence publishes). - Total beforeHooks reduction over the suites present in both runs: −1,649.7s summed across shards (wall-clock benefit is smaller, since shards run in parallel across workers). Baseline CI run 1783389062016888 (#24567 `ci/x-fast`; mechanism-only, so timing-neutral vs merge-base). PR CI run 1783391194852197 (#24568 `ci/x-fast`). ## Notes for downstream - **PR 6 (fees harness):** `applyEnsureAuthRegistryPublished` now short-circuits; the fees setup chain loses the ~32 s/shard auth-registry step, so rebase the overlap/batch shape onto what remains (token deploy, FPC, mints). - **Fix phase / timing capture:** expected span assertion is `setup:auth-registry` around sub-second across the run (10 ms observed locally per process). Any shard still showing tens of seconds means the guards took the slow path (a seeding or flag-propagation bug), not noise. Fixes A-1404
…state (#24578) v5 version of #24558 (same change, based on `merge-train/spartan-v5`). ## Problem When the prover broker cancelled a proving job it settled the job as a cached `{ status: 'rejected', reason: 'Aborted' }` result — in memory and in the database. Because a job's id is a deterministic hash of its inputs, any later enqueue of the same job returned that cached abort (`Cached proving job … Not enqueuing again`) and the facade failed the job with `Aborted`. So once a proof was aborted it stayed aborted, and a retry — a fresh epoch attempt after a reorg/failure, or a re-orchestration after a restart — got the cached abort instead of re-proving. A planned mid-epoch redeploy hit this and turned into a testnet epoch prune. ## Fix Cancellation is now its own first-class, **revivable** state: - A new `aborted` proving-job status (`ProvingJobStatus` / `ProvingJobSettledResult`). `cancelProvingJob` records it, notifies the current waiter, and **persists** it (`ProvingBrokerDatabase.setProvingJobAborted`). - `enqueueProvingJob` **revives** an aborted job: when the producer re-requests it, the broker clears the aborted state — in memory and, via the new `deleteProvingJobResult`, in the database — and re-enqueues it as if new. So a restart mid-revival keeps the job pending rather than resurrecting the abort. Genuine agent errors and timeouts still settle as terminal `rejected` results. Net effect: cancelling frees the job cleanly and records why, but re-requesting the same proof always brings it back to life — in the same process or after a restart — so an abort can never permanently block an epoch from proving. The stacked PR (prover-node clean-shutdown, v5) stops a clean prover-node shutdown from cancelling its in-flight jobs in the first place. ## Tests `proving_broker.test.ts` (both DB variants): cancel leaves a job `aborted`; re-request revives it and it completes; the aborted state persists across a restart; and a revived job survives a restart mid-revival as pending (not aborted). Note: not run locally in this session — the prover-client suite needs a full noir/wasm bootstrap that wasn't available here — so relied on CI. --- *Created by [claudebox](https://claudebox.work/v2/sessions/6fa5e242ed9ceae7) · group: `slackbot`* --------- Co-authored-by: Phil Windle <philip.windle@gmail.com>
Adds `yarn-project/.claude/skills/writing-e2e-tests/SKILL.md`: a skill
that gives an agent everything it needs to place, structure, and write a
robust e2e test with little guidance. Written against the v5-next e2e
layout (category directories with per-category READMEs and context
classes).
## What it covers
- **Step 0 decision ladder**: unit test → new expectation in an existing
test → new `it` in an existing suite → new file on an existing
context/harness → new standalone test.
- **Where to place the test**: recap of the category directories
(`automine/` via `AutomineTestContext`, `single-node/` via
`setupWithProver`/`setupBlockProducer`, `multi-node/` via
`MultiNodeTestContext` on the mock-gossip bus, `p2p/` via
`P2PNetworkTest` on real libp2p, plus `composed/`, `infra/`, `spartan/`,
`bench/`), the multi-node-vs-p2p decision rule, pointers to the
per-category READMEs as the authoritative reference, file naming
(`describe` matches path, header comment), automatic CI registration
through `bootstrap.sh` globs, `.parallel.test.ts` semantics, and
jest/bash timeout sync.
- **Setup reuse**: the three layers (category contexts → domain
harnesses → root `SetupOptions`), the standard suite file shape,
one-environment-per-file, preset spread order, guarded teardown.
- **Readability**: intent-only test bodies, the named-waiter surface
(`fixtures/wait_helpers.ts`, context waiters, `ChainMonitor`), shared
helpers and co-located `*_test_helpers.ts`, simulators with `afterEach`
checks, shared error constants.
- **Speed**: themes distilled from the recent
`test(e2e)`/`perf(e2e)`/`chore(e2e)` speedup PRs and their tracking
issues — avoid unneeded setup, genesis seeding over setup txs,
`BatchCall` batching over `Promise.all` overlap (with the
PXE-serialization caveat), warping dead waits with
`markProvenAndWarp`/`warpWithSequencersPaused` ("an honest wait beats a
flaky warp"), named timing profiles over ad-hoc cadences, and measuring
with the `testSpan` instrumentation + `track-e2e-times` skill before
optimizing.
- **Flakiness**: twelve golden rules synthesized from six months of
deflake PRs and the accumulated flaky-test gotchas (poll-don't-sleep via
named waiters, receipt-anchored assertions, mined ≠ checkpointed ≠
proven, `syncChainTip` tag awareness, pipelining timing margins +
`findSlotsWithProposers`, invariant-not-exact-value assertions, L1
account/nonce and port hygiene, freeze-L1-across-restarts, gossip-mesh
readiness via `runGossipScenario`, fee padding, determinism, honest
timeouts), plus `deflaker.sh` validation and `.test_patterns.yml` as a
last resort.
- Final pre-ship checklist.
All helper/API names cited were verified to exist on
`merge-train/spartan-v5` (`AutomineTestContext.setup`,
`setupWithProver`/`setupBlockProducer`,
`MOCK_GOSSIP_MULTI_VALIDATOR_OPTS`, `wait_helpers.ts` waiters,
`warpWithSequencersPaused`, `markProvenAndWarp`,
`testSpan`/`TEST_TIMING_FILE`, `runGossipScenario`,
`waitForP2PMeshConnectivity`, `findSlotsWithProposers`,
`getPaddedMaxFeesPerGas`, etc.). Genesis-prefill techniques that only
exist in still-open speedup PRs are described as themes to look for
rather than as concrete APIs.
Stacked on #24564; retarget to `merge-train/spartan-v5` after it merges. (Unifies former #24584 into this PR — the two touched the same mint path in `fees_test.ts`, so they now land together.) Restructures the serial setup-tx chains in the fees and cross-chain e2e harnesses so independent setup transactions stop paying one production slot each: independent txs run concurrently, and same-sender calls are batched into a single tx (one proof, one tx, one slot). Part of the round-4/5 e2e speedup effort. ## Commits - **perf(e2e): overlap fees harness setup txs** — after `FeesTest.setup` deploys BananaCoin, the BananaFPC deploy, the SponsoredFPC registration (a PXE registration, no L2 tx), and Alice's banana mints each depend only on the token, so the fees suites (`account_init`, `gas_estimation`, `private_payments`) now run them under a `Promise.all`. The cross-chain harness (`deployAndInitializeTokenAndBridgeContracts`) deploys the L2 token and its bridge concurrently: the bridge takes the token address as a constructor arg, but that address is deterministic before the deploy tx mines (the deploy is now constructed with an explicit `deployer`, same address derivation as the previous send-time lock), and the bridge constructor only stores it without calling into the token. - **perf(e2e): parallelize dual token mints** + **perf(e2e): batch same-sender setup txs into single txs** — `applyFundAliceWithBananas` sent Alice's private and public banana mints back-to-back (measured 100% serial: `tx:mint` busyMs == totalMs on the base). The first commit made them concurrent; the second replaces the concurrency with a single `BatchCall([mint_to_private, mint_to_public])` tx, because measurement showed concurrent same-sender txs still pay *consecutive* slots: the PXE serializes `simulateTx`/`proveTx` on a single queue, so the second tx is only proven after the first and misses the first's slot. Overlapping cannot remove whole slots — only batching can. Both mints are Alice sends on `BananaCoin` touching disjoint balances; the private-balance before/after assertion is preserved, and the public mint's effect is checked by the downstream tests that spend Alice's public bananas. Precedent for the pattern already lives in this tree (`mintNotes` in `fixtures/token_utils.ts`). ## Dependency analysis - FPC deploy needs the token address (constructor arg) → runs after the token deploy, concurrent with the mint batch. - Mints need the token → concurrent with the FPC deploy, and the private/public mints are mutually independent (disjoint balances), so they share one tx. - SponsoredFPC setup is a wallet registration only → free to join the `Promise.all`. - TokenBridge's initializer stores `{token, portal}` in its own storage and never calls the token, so the bridge deploy does not need the token to be mined first. - #24564 invariants preserved: `BANANA_COIN_SALT`/`BANANA_FPC_SALT` + `deployer: alice` are untouched, and the genesis-funded-address assertion in `applyFPCSetup` still runs (and passed locally). ## Same-sender concurrency Concurrent `.send()` calls from the same sender through the same TestWallet are safe: the PXE serializes `simulateTx`/`proveTx` through a `SerialQueue` (`#putInJobQueue`), so simulations never interleave and only the block-inclusion waits overlap. The setup txs pay fees from Alice's public fee-juice balance (`PREEXISTING_FEE_JUICE`) with random tx nonces, so there are no nullifier or note conflicts between the concurrent txs. Verified locally with full `Promise.all` of the send+wait calls; no NO_WAIT fallback or cross-sender split was needed. ## Surveyed and skipped - Batching token class+instance publication: not applicable on this base — `DeployMethod.request()` already merges class publication, instance publication, and the constructor call into a single execution payload sent as one tx. Measured locally, `deploy:token` is a single ~12s (1 slot) occurrence; the ~24s occurrences in CI span data are inclusion latency at CI cadence, not a second tx. - `applyFPCSetup`: a single `deploy:fpc` tx — no same-sender pair inside the phase. Folding the FPC deploy into the mint batch (one class publication + app calls is within the class-log limit) would remove the remaining consecutive slot, but it merges two harness phases that four test files compose differently via `Promise.all`, and deploy-inside-BatchCall is not a proven pattern here — a restructure, not a mechanical merge. Left as a possible follow-up. - `applySponsoredFPCSetup`: no on-chain tx — it only calls `wallet.registerContract` locally; the sponsored FPC is genesis-funded and its class preloaded. Nothing to batch. - Two different contract deploys cannot share a tx (`MAX_CONTRACT_CLASS_LOGS_PER_TX = 1`), which rules out any two-deploy batch (e.g. cross-chain token+bridge) — those stay concurrent rather than batched. ## Local verification (12s slots; CI runs at its own cadence) Overlap commits (fees + cross-chain): - Base (`account_init`, spans on): 5/5 passed; beforeAll 83.5s = `setup:auth-registry` 32.0s + `deploy:token` 12.0s + `tx:mint` 23.9s (busyMs == totalMs, fully serial) + `deploy:fpc` 11.8s. - After overlap: 5/5 passed; beforeAll 72.2s (−11.3s); `deploy:fpc` fully overlapped with the mint window. - `gas_estimation`: 3/3 passed; `private_payments`: 8/8 passed; `token_bridge` (cross-chain overlap): 7/7 passed — `deploy:token` 12.1s and `deploy:bridge` 23.7s ran concurrently (union ≈ 23.7s vs ~48s serial on the CI baseline); `l2_to_l1`: 6/6 passed (regression check on the untouched path). Batch commit (`account_init`, 5/5 green on all runs): | run | tx:mint count | tx:mint busyMs | deploy:fpc busyMs | beforeHooksMs | |-----|---------------|----------------|-------------------|---------------| | before batch | 2 | 23618 | 23713 | 71607 | | run 1 (after) | 1 | 11834 | 24065 | 72372 | | run 2 (after) | 1 | 11807 | 23743 | 71678 | The mint phase drops from two consecutive slots (~23.6s) to one (~11.8s), one proof and one tx fewer per shard. No intermittent failures across any run. ## Measured impact Measured on CI in two steps (each commit group had its own PR run before unification). **Overlap** (PR run 1783394912998666 vs #24564's run 1783375878950264): −162s summed across shards. Per-suite beforeHooks: private_payments.parallel −16.0s (×8), gas_estimation.parallel −15.5s (×3), token_bridge −12.0s, fee_juice_payments −12.0s, account_init −11.5s. The overlap is real but partial: `tx:mint` busyMs/totalMs = 0.68 run-wide — concurrent txs land in consecutive slots because the PXE serializes proving (the finding that motivated the batch commit). Cross-chain `token_bridge` collapses its deploy phase from serial 35.9s to concurrent ~23.4s (union ≈ max, as designed). **Batch** (PR run 1783432285938536 vs 1783394912998666): `tx:mint` 42 → 29 occurrences (one merged per fees shard); busyMs 523s → 431s (−92s of tx/proving work; busyMs == totalMs in the PR run — no overlapping mint txs left to union). Wall-clock lands where the mint is the setup critical path: `failures` −12.0s, `fee_juice_payments` −11.5s (≈ one 12s slot each). Suites where the mint overlaps `applyFPCSetup` are flat (`account_init` −0.4s, `private_payments.parallel` +2s, `gas_estimation.parallel` +3s): `deploy:fpc` is unchanged (13 occurrences, ~2 slots each) and remains their critical path, so removing a slot from the mint branch of the `Promise.all` doesn't move the union. Run-wide sum over common suites: −8s; noise envelope on untouched suites ±20–37s. Net vs #24564's baseline: roughly −185s summed across shards, plus 13 fewer txs/proofs per run (less proving contention through the sequencer). The remaining ~12s/shard lever on the overlapped fees suites is folding the FPC deploy into the same batch, noted above as a follow-up. Fixes A-1406 Fixes A-1409
Removes `yarn-project/THREAT_MODEL.md`, added in [#24465](#24465). Per discussion in #team-alpha: detailed threat models and invariants may make it easier for attackers to find bugs, so the team wants this content pulled from the public repo for now. It will be re-added to LabsBox/ClaudeBox (private/internal) — see [claudebox#1357](AztecProtocol/claudebox#1357) — and can come back here once the team is finding fewer bugs. Only the threat model doc itself is removed here — the incidental doc cleanups bundled into the same squashed commit (stale BLOCK protocol references in `yarn-project/p2p/README.md` and `yarn-project/p2p/src/services/reqresp/README.md`, and the missing `DUPLICATE_ATTESTATION` entry in `yarn-project/slasher/README.md`) are kept as-is.
Collaborator
Author
Flakey Tests🤖 says: This CI run detected 1 tests that failed, but were tolerated due to a .test_patterns.yml entry. |
Collaborator
Author
|
🤖 Auto-merge enabled after 4 hours of inactivity. This PR will be merged automatically once all checks pass. |
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BEGIN_COMMIT_OVERRIDE
docs(e2e): update READMEs for the consolidated suite layout (#24518)
test(e2e): instrument and diagnose bot suite setup cost (#24534)
perf(e2e): warp dead waits in multi-node recovery and proving tests (#24566)
perf(e2e): shrink e2e slot times (#24570)
perf(e2e): seed BananaFPC fee juice at genesis instead of bridging (#24564)
perf(bot): parallelize independent bot factory setup steps (#24581)
feat(world-state): support prefilled nullifiers in genesis state (#24567)
perf(e2e): seed standard contracts at genesis (#24568)
feat(prover): revive cancelled proving jobs from a persisted aborted state (#24578)
chore: add writing-e2e-tests skill (#24597)
perf(e2e): overlap and batch setup txs in e2e harnesses (#24569)
docs: revert threat model for node (#24465) (#24610)
END_COMMIT_OVERRIDE