chore: add writing-e2e-tests skill#24597
Draft
AztecBot wants to merge 493 commits into
Draft
Conversation
No longer used after removing fastMsgIdFn. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes [F-669](https://linear.app/aztec-labs/issue/F-669/aztec-nr-calculate-secret-and-index-helper-for-constrained-delivery) Fixes [F-670](https://linear.app/aztec-labs/issue/F-670/aztec-nr-send-constrained-msg-helper-that-emits-a-constrained) ## The change - Adds constrained-delivery helpers that resolve or bootstrap the app-siloed handshake secret, seed same-tx index reuse, validate index `0` against the standard HandshakeRegistry, and constrain `index > 0` through the previous chain nullifier. - Wires constrained private message delivery to derive constrained log tags from the resolved `(secret, index)` pair and emit the current constrained-message nullifier. - Keeps constrained logs from being squashed with note hashes so recipient discovery can advance the per-secret index chain. - Authorizes the standard HandshakeRegistry utility reads needed by the flow. - Adds TXE, snapshot, and e2e coverage for bootstrap, reuse, index advancement, missing prior-nullifier failure, invalid API combinations, and standard-registry constrained delivery. Large portion of the diff is tests. [F-741](https://linear.app/aztec-labs/issue/F-741/getsharedsecrets-throws-no-public-key-during-recipient-discovery-for-a) follow-up for the test skipped in [fd07996](fd07996). ## Concurrency / batching constraint (worth a look) Pinning the e2e tests surfaced a sequencing constraint in constrained delivery that reviewers should sanity-check: - **Parallel sends on one `(sender, recipient, secret)` chain collide.** Each send is keyed on an incrementing index, so two sends fired as separate parallel txs read the same index and one tx is rejected. Distinct recipients are distinct chains and parallelize fine. Pinned as `it.failing` (documents the limit; flips green if parallel sends ever become supported). - **Same-chain sends _can_ be batched into one tx**, but only onto an already-committed handshake. A later send discharges its predecessor check against a same-tx pending nullifier. Works both via a single contract call (`emit_two_events`) and client-side `BatchCall`. - **Batching onto a brand-new chain does not reuse, it re-handshakes.** The reuse-vs-bootstrap decision in `get_or_create_app_siloed_handshake_secret` is a utility call reading _committed_ state, so a bootstrap earlier in the same tx is invisible and the later send mints a fresh secret on a separate chain (next index lands at 1, not 2). A new recipient therefore needs one landed tx to establish the chain before sends can be batched onto it. This is why the batched tests seed the handshake first. All four behaviors are pinned in `e2e_constrained_delivery.test.ts`; the rationale lives in the module doc and on `get_or_create_app_siloed_handshake_secret`. Follow-up: allowing utility functions to execute against combined committed + pending transaction state is tracked in [F-238](https://linear.app/aztec-labs/issue/F-238/allow-execution-of-utility-functions-touched-by-pending-transactions). --------- Co-authored-by: AztecBot <tech@aztec-labs.com> Co-authored-by: Nicolas Chamo <nicolas@chamo.com.ar>
… EpochSession (A-1212) (#24216) Fixes A-1212. ## Problem `prover_getJobs` exposes each `EpochSession`'s state, but `EpochSession` only ever assigned `initialized`, `awaiting-checkpoints`, and the terminal states. The intermediate phases that exist to describe active work were never set, so a session read `awaiting-checkpoints` continuously while it was actually (a) proving the top tree or (b) publishing to L1. Observed on staging-internal: every `startProof` session sat in `awaiting-checkpoints` even with logs showing `Starting top-tree prove…`. This is a regression vs the old `EpochProvingJob`, which advanced through distinct phases. ## Fix - Add a new **non-terminal** `awaiting-root` state to `EpochProvingJobState` (the zod schema is derived from the same array, so it updates automatically). Not added to `EpochProvingJobTerminalState`. - Set `awaiting-root` when the top-tree (root) prove begins, via the `TopTreeJob` `beforeProve` hook — which fires once the sub-tree (checkpoint block) proofs are ready and the root prove is about to start. `toTopTreeHooks()` now always wires this transition and layers any test-provided hooks on top (previously it returned `undefined` when no test hooks were set). - Set `publishing-proof` at the top of `submitProof()`, before the L1 submit. Both transitions are guarded by `isTerminal()` so a concurrent `cancel()` still wins (the post-submit `isTerminal()` check relies on this). Phase progression is now: `initialized → awaiting-checkpoints → awaiting-root → publishing-proof → terminal`. ## Tests - `epoch-session.test.ts`: new "state reporting" test asserts `getState()` is `awaiting-root` during the prove and `publishing-proof` during the submit, then `completed`. Existing hook-ordering test (`before`/`prove`/`after`) still holds. - `prover-node.test.ts`: the JSON-RPC schema round-trip mock now includes an `awaiting-root` job. Targets the v5 line — the `EpochSession`/`TopTreeJob`/`awaiting-checkpoints` code (post-#23552) exists only on `merge-train/spartan-v5`. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolves conflict in e2e_nested_utility_calls.test.ts (keeps the train's doc comment above the 'denies...private function' test; the msg_sender test from v5-next is already present). Real merge commit so v5-next becomes an ancestor of the train.
…empotent The archiver preloads bundled protocol contract classes and instances at synthetic block 0. When a bundled protocol class id (or instance address) was later published on chain, the contract stores threw "already exists" while re-adding the already-present key, stalling L1 sync. Make the stores idempotent for protocol-preloaded entries: - addContractClass / addContractInstance now skip (no-op) when the key already exists and it belongs to a bundled protocol contract, keeping the existing block-0 entry untouched. Genuine non-protocol redefinitions still throw. - deleteContractClass / deleteContractInstance never delete protocol entries, so they survive reorgs of the publishing block. Adds isProtocolContractClass to protocol-contracts as a sibling of isProtocolContract, plus store unit tests and an integration test that publishes a bundled protocol class id through ArchiverDataStoreUpdater. Fixes A-1257
…ting (A-1258) (#24213) Fixes A-1258. ## Problem `DataTxValidator` decodes contract-class publication logs **before** proof validation (data validation runs ahead of proof validation on the req/resp, block-proposal, RPC, and gossip paths). The decode trusts the declared public-bytecode length field: - `bufferFromFields` (`stdlib/src/abi/buffer.ts`) reads the declared `byteLength` and, when the payload is shorter, does `Buffer.alloc(byteLength)` — trusting the declared length over the bytes actually present. - `ContractClassPublishedEvent.fromLog` decodes the bytecode via `bufferFromFields`, and `DataTxValidator.#hasCorrectContractClassIds` calls `fromLog` before any class-id / proof rejection. A contract-class log is a fixed `CONTRACT_CLASS_LOG_SIZE_IN_FIELDS` array, so the real packed bytecode is capped at ~93 KiB. But the declared length is unbounded: a ~96 KiB log declaring 16 MiB forces a 16 MiB allocation on every node that validates the tx, before the proof check rejects it — a memory-pressure / validation-amplification path. ## Fix - Add an optional `maxByteLength` parameter to `bufferFromFields`; when provided, it throws (before allocating) if the declared length exceeds it. Callers that don't pass it keep the documented blob-reconstruction padding behaviour unchanged. - `fromLog` passes the fixed log's payload capacity (`payloadFields × 31`), so an over-declared length is rejected before allocation. `DataTxValidator` already wraps `fromLog` in try/catch, so the throw becomes a clean pre-proof `TX_ERROR_MALFORMED_CONTRACT_CLASS_LOG` rejection. ## Tests - `data_validator.test.ts`: a log declaring a 16 MiB length is rejected, and `Buffer.alloc` is never called with the declared size (test time drops from ~1.2s to ~40ms — no large allocation). Run against the pre-fix compiled code, the allocation still happened (red); green after the fix. - Existing `contract_class_published_event.test.ts` "fits a max-size public bytecode" and the `buffer.test.ts` padding tests confirm legitimate max-size bytecode and the padding contract are unaffected.
…24214) Fixes A-1256. ## Problem The gossipsub full message id was `SHA256(topic || data)[0..20]` with **no framing** between the topic string and the message bytes, and `LibP2PService` set no `allowedTopics`. Raw concatenation isn't injective: for a real message `(T, D)`, a peer can craft `T' = T + D[0]`, `D' = D[1:]` so `T' || D'` is byte-identical to `T || D` → same msgId. ChainSafe gossipsub transforms the arbitrary-topic message and inserts that id into `seenCache` **before** the subscription check (it isn't delivered to us, since we're not subscribed to `T'`). When the genuine proposal/attestation arrives on `T`, gossipsub drops it as a **duplicate** before application validation, peer scoring, or handling — suppressing a time-sensitive consensus message within the slot. ## Fix (defense in depth — either alone breaks the attack) 1. **Frame the msgId input** as `uint32be(topicLen) || topic || data` (`encoding.ts`). The topic length pins the `(topic, data)` boundary, so a boundary-shifted pair no longer collides. (`getMsgIdFn`'s parameter is narrowed to `Pick<Message, 'topic' | 'data'>` — the only fields it reads — which stays assignable to gossipsub's `msgIdFn` slot.) 2. **Set exact `allowedTopics`** (`libp2p_service.ts`) to the subscribed Aztec topic strings. Verified against the installed gossipsub: the allowlist is enforced in `handleReceivedRpc` *before* `handleReceivedMessage`/`seenCache.put`, so an unsubscribed-topic message is dropped before transform / msgId / seenCache. ## Test `encoding.test.ts`: builds a real `P2PMessage.toMessageData()` buffer (confirming `data[0] === 0x00`), constructs the shifted `(T', D')`, and asserts the two msg ids now differ (they collided before the framing change); plus a determinism check. ## Compatibility `msgId` is computed locally for dedup; changing the function doesn't change any on-wire format. During a rolling upgrade, mixed nodes briefly compute different ids for the same message (minor IHAVE/IWANT inefficiency), with no correctness impact.
## Merge `v5-next` into `merge-train/spartan-v5` — MUST merge as a merge commit (do not squash) ### Why the conflicts came back The previous stack (#24221/#24222) resolved the conflict correctly, but #24221 was **squash-merged** onto the train. A squash creates a brand-new single commit and does **not** record `v5-next` as a parent, so git still sees the train and `v5-next` as diverged (`git merge-base --is-ancestor origin/v5-next origin/merge-train/spartan-v5` → false). The merge-train auto-pull then re-attempts `v5-next → train` and hits the same conflict again. ### What this PR does This is a real **merge commit** of `v5-next` into the train (two parents: train tip `eb7d64d` + `v5-next` `4df7243`), with the one conflict resolved. The resulting tree is **identical to the current train** (0 file changes) because the content already landed via the earlier squash — this PR exists purely to record `v5-next` as an ancestor and advance the merge base. After this lands, `v5-next` is an ancestor of the train and the long-lived `merge-train/spartan-v5` → `v5-next` PR is conflict-free. ### Conflict resolved (1) `yarn-project/end-to-end/src/e2e_nested_utility_calls.test.ts` — kept the train's doc comment above the `denies…private function` test; the `msg_sender` test from `v5-next` is already present. ###⚠️ Merge method **Merge this with "Create a merge commit"** (or `merge_pr merge_method=merge`). A squash or rebase merge will recreate the same divergence and the conflicts will return. --- *Created by [claudebox](https://claudebox.work/v2/sessions/51d528a3438f2849) · group: `slackbot`*
…255) (#24215) Fixes A-1255. ## Problem A discovery peer can advertise a signed ENR with valid `ip`/`udp` contact data and the expected Aztec ENR version key, but a **malformed** `tcp` field (e.g. 3 bytes instead of a 2-byte port). `DiscV5Service.validateEnr` only checked the aztec key + version, so upstream discv5 accepted the session and emitted `enrAdded`. The unguarded async `onEnrAdded` listener then awaited `enr.getFullMultiaddr('tcp')`, which throws `RangeError` on the malformed value; as an async EventEmitter listener the rejection is unhandled and **the node process exits**. One malformed, UDP-contactable, same-version ENR crashes any p2p-enabled node. ## Audit A sweep of every `getFullMultiaddr` / `getLocationMultiaddr` call in `p2p/src` found the same unguarded-parse pattern in more than one place: | Site | ENR source | Crash today | Action | |------|-----------|-------------|--------| | `discV5_service.ts` `onEnrAdded` | remote | yes | guard + drop | | `bootstrap.ts` `discovered` listener | remote | yes (crashes the **bootnode**) | guard + drop | | `peer_manager.ts` `handleDiscoveredPeer` | remote | no (caught by wrapper) | covered by `validateEnr` rejecting malformed-addr ENRs | | `peer_manager.ts` / `libp2p_service.ts` `preferredPeers` | config | aborts setup | skip + log | ## Fix - Guard `onEnrAdded` and the bootnode `discovered` listener so a parser exception is caught, logged, and the ENR dropped — no unhandled rejection. - `validateEnr` rejects an ENR whose TCP multiaddr won't parse, so it's never emitted as a full peer (`PeerEvent.DISCOVERED`) or re-parsed by `handleDiscoveredPeer`. A *missing* TCP addr stays allowed — those peers are simply skipped at dial time; only an *unparseable* one is rejected. - `preferredPeers` ENR parsing skips and logs a malformed configured ENR instead of aborting node / preferred-peer setup. The self-ENR log line (`discV5_service.ts:199-200`) is intentionally left unguarded — it reads our own ENR, which we build and which can't be attacker-malformed. ## Tests `discv5_service.test.ts`: builds a same-version ENR with valid `ip`/`udp` and a 3-byte `tcp` value, and asserts (a) `validateEnr` rejects it while a valid control built from the same identity passes — isolating the TCP check — and (b) `onEnrAdded` resolves (no unhandled rejection, previously a `RangeError`) and never emits `DISCOVERED`. Note: `discV5_service.ts` overlapped with the already-merged PR #24169; this branch is off the post-merge `merge-train/spartan-v5`, so no conflict.
…nt state The archive tree is an append-only accumulator of block-header hashes, so a single bad leaf (e.g. from a mishandled reorg) is never self-corrected: every later root stays noncanonical while the other state trees can re-converge from block effects. sync_block only checked the four non-archive trees, so a self-consistent orphan block — commonly an empty one — could silently fork the archive root from canonical. Verify the archive root against canonical both before appending (the committed root must equal the block's lastArchive) and after (the resulting root must equal the block's archive), failing before commit so the divergence is never persisted. The checks are optional std::optional parameters; the napi and wsdb transports forward the canonical roots from the block being synced.
Fixes [F-664](https://linear.app/aztec-labs/issue/F-664/e2e-test-constrained-delivery-via-non-interactive-handshake) Add e2e tests for discovering events and notes sent with constrained delivery. --------- Co-authored-by: Nicolas Chamo <nicolas@chamo.com.ar>
BEGIN_COMMIT_OVERRIDE fix(p2p): re-seed discovery from persisted peer ENRs after restart (#24169) docs(e2e): annotate e2e tests with setup/category notes (#24191) chore: merge v5-next into merge-train/spartan-v5 (raw, conflict markers) (#24221) fix(archiver): index zero-field logs under empty tag instead of throwing (A-1253) (#24212) fix(prover-node): report awaiting-root and publishing-proof phases in EpochSession (A-1212) (#24216) fix(p2p): bound declared contract-class bytecode length before allocating (A-1258) (#24213) fix(p2p): frame gossipsub msgId and restrict allowedTopics (A-1256) (#24214) chore: merge v5-next into merge-train/spartan-v5 (#24226) fix(p2p): guard ENR address parsing against malformed TCP fields (A-1255) (#24215) END_COMMIT_OVERRIDE
…nt state (#24229) Fixes A-1235 # A-1235 reviewer notes: archive-root divergence and the v4/v5 block-stream race ## Short version The A-1235 symptom was a mainnet fisherman rejecting every proposal with `ReExInitialStateMismatchError`. The proposal/canonical `lastArchive.root` was correct; the local world-state archive root was wrong. The important distinction for review is: - v4.3.0 appears vulnerable to a specific block-stream/cache race that can seed this state. - v5 has already hardened that specific race. - v5 still lacks the world-state archive-root invariant, so if any other path ever writes a bad archive leaf, the corruption can still be committed silently. This change is therefore defense in depth at the only layer that can conclusively reject a bad archive tree before it is persisted: `WorldState::sync_block`. ## What went wrong World state maintains the archive tree as an append-only accumulator of block-header hashes. During `sync_block`, v4.3.0 appended the new block header hash and then checked: - `is_archive_tip`: the just-appended header hash is the tip leaf. - `is_same_state_reference`: nullifier, note hash, public data, and L1-to-L2 message trees match the block state reference. Neither check verifies the archive root against canonical block data: - before append: local archive root must equal the block's `lastArchive.root`; - after append: computed archive root must equal the block's `archive.root`. That means an orphan block can be internally self-consistent and still poison the archive tree. If the orphan and canonical block at the same height have the same effects, commonly because both are empty, the four non-archive trees can reconverge while the archive tree remains permanently forked. The resulting sequence is: 1. World state syncs an orphan block at height `F`. 2. The archiver later canonicalizes a different block at height `F`. 3. The block stream should prune world state to `F - 1`. 4. In the suspected v4 race, it prunes only to `F`, so the orphan archive leaf remains. 5. World state syncs canonical descendants starting at `F + 1` on top of the orphan leaf. 6. The four-tree state-reference check passes, but every archive root from `F` onward is off-chain. ## The v4 block-stream/cache race The original explainer says roughly: `getL2Tips` had processed the reorg, while the archiver had not yet swapped orphan `F` for canonical `F`. More precisely, this is not about committed DB state being partially updated. The block/checkpoint mutation is one LMDB writer transaction. The race is that v4's `L2TipsCache` can publish a tip computed from the uncommitted writer view, while separate block/header reads still use committed read transactions. In v4.3.0: - `ArchiverDataStoreUpdater.addCheckpoints()` wraps prune, checkpoint insertion, logs, contract data, and `l2TipsCache.refresh()` in one `store.transactionAsync(...)`. - `L2TipsCache.refresh()` assigns `#tipsPromise = this.loadFromStore()` before the writer transaction commits. - The LMDB wrapper reuses the active write transaction for reads inside that callback, so `loadFromStore()` can see the post-reorg uncommitted view. - A concurrent consumer calling `archiver.getL2Tips()` can receive that cached future/post-reorg tip. - The same consumer's later `getBlockHeader(F)` call is outside the writer context, so it opens a normal committed read transaction and can still see the pre-reorg orphan at `F`. The v4 `L2BlockStream` performs exactly this mixed read pattern: ```ts const sourceTips = await this.l2BlockSource.getL2Tips(); const localTips = await this.localData.getL2Tips(); let latestBlockNumber = localTips.proposed.number; const sourceCache = new BlockHashCache([sourceTips.proposed]); while (!(await this.areBlockHashesEqualAt(latestBlockNumber, { sourceCache }))) { latestBlockNumber--; } ``` `areBlockHashesEqualAt()` then asks the source for a per-height hash via `getBlockHeader(blockNumber)`. So a single pass can be planned from a future/post-reorg tip but compare old committed per-height headers. If both local world state and the committed archiver still have the orphan at `F`, the walk can falsely conclude `F` is the common ancestor. Pruning to `F` keeps block `F`; the correct target was `F - 1`. That is the suspected seed path for A-1235. The live evidence proves the archive tree forked; the historical interleaving predates retained logs, so treat this as the best source-grounded mechanism, not as directly logged fact. ## What v5 changes in this area v5 has multiple changes that make the specific v4 race much less plausible. ### 1. Tip cache refresh is post-commit In v5, `L2TipsCache` says refresh should happen after the writer transaction has committed, and `ArchiverDataStoreUpdater` does exactly that: ```ts const result = await this.stores.db.transactionAsync(async () => { // mutate blocks/checkpoints/logs/etc. return ...; }); await this.l2TipsCache?.refresh(); return result; ``` So the cache is loaded from committed DB state, and an aborted writer cannot replace the cache with a future view. ### 2. `getL2TipsData()` is one DB snapshot v5 moves chain-tip construction into `BlockStore.getL2TipsData(genesisBlockHash)` and wraps it in a single `db.transactionAsync(...)`. It also validates the resulting tier ordering: - finalized <= proven <= checkpointed <= proposed; - checkpointed block <= proposed block. That is materially stronger than v4's `L2TipsCache.loadFromStore()`, which assembled tip numbers and block data through several independent store calls. ### 3. The block stream fails closed on incoherent source reads v5's `L2BlockStream` no longer uses the old checkpoint-prefetch path in the same way. It drives from a source tips snapshot, compares per-height hashes via `getBlockData({ number })`, and adds guards: - missing local hash compares unequal rather than accidentally stopping the walk; - missing source data at or below the advertised source proposed tip aborts the pass; - source tips are re-read after a prune before downstream reconciliation; - the download pass verifies the delivered proposed block hash matches the snapshot's proposed hash; - tier advancement is skipped if the block download plan did not complete. These are all aimed at preventing a stale or mixed source snapshot from becoming an under-deep prune. Over-deep or skipped reconciliation is recoverable; under-deep pruning is dangerous because it can leave the losing fork's block at the divergence height. ## Why this change still matters on v5 The v5 block stream fixes the known/suspected seed path, but it does not add the missing invariant to world state. Without this patch, `sync_block` can still commit a divergent archive tree if any future path presents it with a self-consistent block whose non-archive state matches: - a different reorg bug; - a cache/snapshot bug elsewhere; - operator/datadir corruption; - a future refactor that bypasses one of the v5 block-stream protections. Once a bad archive leaf is committed, appending more canonical leaves does not repair it. The archive root remains noncanonical forever while the four non-archive trees can look healthy. The fix makes `sync_block` verify the archive tree at the source of truth: ```cpp // Before append: committed local root must be the block's parent archive root. actual_previous_archive_root == expected_previous_archive_root // After append, before commit: uncommitted computed root must be the block's archive root. actual_archive_root == expected_archive_root ``` These checks turn silent permanent corruption into a loud sync failure before commit. The node may need a datadir resync, but it will not write and seal in the divergent archive leaf. ## Reviewer checklist - Confirm the TS/native message path passes both canonical roots: - `expectedPreviousArchiveRoot = l2Block.header.lastArchive.root` - `expectedArchiveRoot = l2Block.archive.root` - Confirm native `sync_block` checks the previous root before `add_value`. - Confirm native `sync_block` checks the resulting root before `commit`. - Confirm errors clearly tell operators that local world state diverged and must be resynced. - Confirm tests cover the archive-only divergence case, not just non-archive state-reference mismatch. ## Expected operator behavior This patch prevents recurrence; it does not repair an already-poisoned archive tree. A node that already has the bad historical leaf must wipe/resync its world-state datadir.
…24202) ## Motivation The sentinel is required for the network to reach consensus on slashing offenses — it's a core duty of participating in consensus, not an optional add-on. Yet `SENTINEL_ENABLED` defaulted to `false`, so a validator could run without one and silently degrade the network's ability to enforce its own rules. While fixing that, two adjacent issues surfaced: - Whether the node runs the sentinel and the slashing-detection watchers was gated on a *prover-only* flag (`enableProverNode && disableValidator`). That coupling is wrong: those subsystems should depend on **validator status**, not on whether the prover is enabled. - A non-validator (e.g. an RPC or full node) had no way to collect and inspect offenses, because the only component that persists them lives inside the slasher client, which was created only for validators. ## Approach - **Force the sentinel for validators.** `createSentinel` now self-gates: if the node runs a validator it always creates the sentinel (ignoring `SENTINEL_ENABLED`); otherwise it respects `SENTINEL_ENABLED` (default `false`). It logs which case applies so operators understand why a `SENTINEL_ENABLED=false` is being overridden. - **Decouple subsystem gating from the prover.** Removed the `proverOnly` guard. The slashing-detection watchers and the slasher are now gated on `collectOffenses = !disableValidator || enableOffenseCollection`. - **Opt-in offense collection for non-validators.** A new `OFFENSE_COLLECTION_ENABLED` flag lets a non-validator run the watchers plus a read-only slasher client (it never writes to L1 — voting/execution flow through the sequencer publisher, which a non-validator lacks) to collect and serve offenses over the existing `getSlashOffenses` admin RPC. The spartan deploy enables it by default, mirroring `SENTINEL_ENABLED`. - **Make the attested-invalid-proposal watcher work without a validator.** Lifted the invalid-proposal / equivocation slot tracking out of `ValidatorClient` into `ProposalHandler`, which now implements `InvalidProposalSlotSource`. The handler populates this from its all-nodes proposal handlers, so any node that re-executes proposals (the default) can feed the watcher. ## API changes - New node config `enableOffenseCollection` (env `OFFENSE_COLLECTION_ENABLED`, default `false`). Named distinctly from the existing L1-deploy `AZTEC_SLASHER_ENABLED` to avoid confusion. - Validator nodes now always run the sentinel regardless of `SENTINEL_ENABLED`. ## Changes - **aztec-node**: `createSentinel` forces the sentinel for validators and logs the reason; `server.ts` removes `proverOnly`, gates watchers + a split-out read-only slasher on `collectOffenses`, and wires the attested-invalid-proposal watcher to the proposal handler. - **aztec-node (config) / foundation**: new `enableOffenseCollection` / `OFFENSE_COLLECTION_ENABLED` flag and env-var entry. - **validator-client**: `ProposalHandler` now owns the invalid-proposal / equivocation slot tracking and implements `InvalidProposalSlotSource`; `ValidatorClient` delegates to it, preserving validator behavior. - **spartan**: `OFFENSE_COLLECTION_ENABLED` wired through the chart value, pod template, network defaults, terraform variable, and deploy script, defaulting on (mirrors `SENTINEL_ENABLED`). Fixes A-1242
…empotent (#24227) ## Motivation The archiver preloads every bundled protocol contract class (and its canonical instance) into its local store at synthetic block 0, before L1 sync. World-state genesis, however, seeds no registration nullifiers for those classes/instances. As a result a *first* on-chain `ContractClassRegistry.publish` of a bundled protocol class id is protocol-valid (fresh class-id nullifier + `ContractClassPublished` log). On replay the archiver recomputes the same class id and unconditionally re-inserts it, but the store throws on the pre-existing block-0 key (`Contract class <id> already exists, cannot add again`). Because that insert runs inside the block/checkpoint store transaction, the throw aborts persistence, and L1 sync retries the same valid checkpoint indefinitely — a sync stall. ## Approach Make the archiver treat protocol-preloaded entries as idempotent and immutable, guarded at the store layer (the single chokepoint for both the add-throw and the block-gated delete): - `addContractClass` / `addContractInstance`: when the key already exists and it is a protocol class id / magic protocol address, treat the (re-)publish as a no-op and keep the existing block-0 entry — crucially **without** bumping its recorded block number. Genuine non-protocol duplicates still throw unchanged. - `deleteContractClass` / `deleteContractInstance`: skip deletion for protocol entries, so a reorg of the publishing block can never roll out the preload. Keeping the preload at block 0 is deliberate: `deleteContractClass` only deletes when the stored `l2BlockNumber >= blockNumber`, so block 0 makes protocol entries survive any reorg; bumping to the publish block would let a deep reorg delete them. The instance-side guards are defensive only: on-chain `publish_for_public_execution` always emits the *derived* address, never a magic address, so the instance store is not reached with a magic address via the on-chain replay path today — the guards exist for symmetry and to protect future code paths. This is a non-breaking, node-local resilience fix. A follow-up PR (PR2) seeds the protocol registration nullifiers into world-state genesis so the on-chain re-publish is rejected at the protocol level — the root-cause fix for the class path. ## Changes - **protocol-contracts**: add `isProtocolContractClass(classId)` (sibling of the existing `isProtocolContract(address)`), backed by a set of the generated `ProtocolContractClassId` values. - **archiver**: idempotent add + protected delete for protocol classes and instances in `ContractClassStore` / `ContractInstanceStore`. - **archiver (tests)**: store unit tests (idempotent re-add stays queryable with block-0 / bytecode-commitment preserved, protected delete, non-protocol duplicate still throws — for both classes and instances) and an A-1257 integration test that preloads protocol contracts, builds an `L2Block` carrying a `ContractClassPublished` log for a bundled class id, and asserts `addProposedBlock` commits and the class stays queryable. Fixes A-1257
…24065) Fixes a race in `e2e_token_bridge_tutorial_test` where L1 setup transactions were submitted but not awaited before dependent bridge calls. - Waits for the `TestERC20.addMinter` transaction receipt before minting through the handler. - Waits for the `TokenPortal.initialize` transaction receipt before simulating `depositToAztecPublic`. - Prevents intermittent `SafeERC20FailedOperation(address token) (0x0)` when the portal deposit simulation runs before `underlying` is initialized. See http://ci.aztec-labs.com/e100f59864b9c18c for sample failed run.
Backports the source-doc corrections from two already-merged `next` docs PRs to the `v5-next` release line, where the same pages were stale: - #24005 — `fix(docs): fable review` (merged Jun 22) - #23830 — `docs: complete and correct the proving historic state page` (merged Jun 22) `v5-next` has no `version-v5.0.0-rc.1` versioned snapshot (its versioned docs are v4.3.0-era), so this touches **source docs only** — the authored pages under `docs/docs-developers/` and `docs/docs-operate/`. ## What's included (12 files) Each fix was re-verified against `v5-next` source, not blindly copied from `next`/v6: **From #23830** - `aztec-nr/.../advanced/how_to_prove_history.md` — rewrite. Every history fn it documents (`assert_note_existed_by`, `assert_note_was_nullified_by`, `assert_note_was_valid_by`, `assert_contract_bytecode_was_published_by`, `assert_contract_was_initialized_by`, `public_storage_historical_read`) exists verbatim in v5-next's `aztec/src/history/`. **From #24005** - `aztec-nr/.../how_to_retrieve_filter_notes.md`, `state_variables.md` — `RetrievedNote`/`HintedNote` → `ConfirmedNote` (type confirmed present on v5-next). - `aztec-nr/.../events_and_logs.md` — nonexistent `self.context.emit_public_log(...)` → `emit_public_log_unsafe` (confirmed at `public_context.nr:110` on v5-next). - `aztec-nr/.../contract_structure.md` — storage struct must be named `Storage` (version-agnostic). - `foundational-topics/contract_creation.md` — `aztec::history::contract_inclusion` → `aztec::history::deployment` (the `deployment` module exists on v5-next). - `foundational-topics/.../outbox.md` — checkpoint-based `IOutbox` (`getRootData(Epoch, uint256)` confirmed in v5-next's `IOutbox.sol`). - `cli/aztec_cli_reference.md` — fills the empty `aztec start` section; all documented module flags (`--network`, `--node`, `--sequencer`, `--prover-node`, `--prover-broker`, `--prover-agent`, `--p2p-bootstrap`, `--txe`, `--bot`, `--admin-port`) exist in v5-next's `aztec_start_options.ts`. - `cli/aztec_wallet_cli_reference.md` — removes generator-machine leakage in default values (`/home/josh/.aztec/wallet` → `~/.aztec/wallet`, `host.docker.internal` → `localhost`). - `operators/.../governance-participation.md`, `useful-commands.md` — stale governance calls fixed: `M()`→`ROUND_SIZE()`, `N()`→`QUORUM_SIZE()`, `yeaCount(...)`→`signalCount(...)`, `proposals(uint256)`→`getProposal(uint256)`. Confirmed: v5-next `Governance.sol` exposes `getProposal`, `EmpireBase.sol` exposes `ROUND_SIZE`/`QUORUM_SIZE`/`signalCount`; old names are gone. - `tutorials/js_tutorials/token_bridge.md` — Hardhat artifact-path / run-command corrections; the `@aztec/l1-contracts` version note uses the `#include_aztec_version` macro, so it resolves to the v5 version automatically. ## Deliberately excluded — needs a v5-specific numeric pass - **`slashing-configuration.md`** — #24005's numbers are `next`/mainnet-framed and **do not match the live v5 testnet**. The v5 upgrade deploy (`DeployRollupForUpgradeV5.s.sol`, Sepolia branch) sets `localEjectionThreshold = 199,000e18` and slash amounts **100k / 250k / 250k** (AZIP-16 full-stake), versus the doc's 190k + 2,000/5,000. Slot duration (72s) and round size (128 slots) do check out for v5 (confirmed via live testnet block spacing of 72s). This page should be rewritten with verified v5-testnet values in a follow-up rather than copied. Source PRs: [#24005](#24005), [#23830](#23830). No tracked issue is closed by this backport. --- *Created by [claudebox](https://claudebox.work/v2/sessions/59cd7d98b38b9d90) · group: `slackbot`*
Backports the counter-tutorial half of #23517 (`docs: align counter tutorial example with the aztec init template and add TXE tests`, merged to `next` Jun 22) to `v5-next`. ## Why this is the right scope #23517's core intent was to **align the counter tutorial example with what `aztec init` generates**. On `v5-next` the scaffold template is *already* updated — `yarn-project/aztec/scripts/templates/counter/contract/src/main.nr` already defines `fn constructor(initial_value: u128, ...)`. But the tutorial prose and the inlined example contract are still on the old `initialize(headstart)` naming, so the v5 tutorial currently contradicts the code `aztec init` produces. This backport closes that gap. ## Files (2) - `docs/docs-developers/docs/tutorials/contract_tutorials/counter_contract.md` — `initialize`→`constructor`, `headstart`→`initial_value`, intro reworded (encrypted private state), heading casing. - `docs/examples/contracts/counter_contract/src/main.nr` — same rename, so the tutorial's `#include_code constructor` block matches both the prose and the v5 scaffold template. ## Deliberately excluded - **TXE test harness** (`counter_contract_test`, the `docs/examples/bootstrap.sh test-contracts` step, `Nargo.toml` member, `logging_example_test` tweak) — that's CI tooling, not docs, and `counter_contract_test` doesn't exist on `v5-next`. - **Scaffold template** — already correct on `v5-next`; no change needed. ## Safety check `recursive_verification_contract` also uses a `headstart` parameter, but it's an independent contract with its own `constructor(headstart: Field, ...)` (untouched by #23517 even on `next`), so the counter rename doesn't affect it. Source PR: [#23517](#23517). No tracked issue is closed by this backport. --- *Created by [claudebox](https://claudebox.work/v2/sessions/59cd7d98b38b9d90) · group: `slackbot`*
(cherry picked from commit ef43cb7)
) ## Human-written summary Invalid numeric values in config were silently ignored and replaced with the default. Also, config entries defined with `numberConfigHelper` accepted non-integer values but silently truncated them. ## Context The numeric config helpers mishandled bad input in two ways: - `numberConfigHelper` / `optionalNumberConfigHelper` parsed with `parseInt`, which silently truncated decimals — an integer config set to `0.8` became `0` with no warning. In the worst case (A-1312) that turned a fractional retention window into `0`, i.e. "delete immediately". - `numberConfigHelper`, `floatConfigHelper` and `percentageConfigHelper` swallowed any unparseable value and returned the config default (the JSDoc even documented "...or is invalid"), so a typo'd env var ran silently with an unexpected value. ## Approach The config default is for an **unset** env var only — empty/unset is already resolved to the default before `parseEnv` runs, so the only thing `parseEnv` returning a default achieved was hiding bad input. The helpers now parse strictly and throw on any set-but-invalid value: - `numberConfigHelper` / `optionalNumberConfigHelper`: parse with `parseFloat` and require a safe integer (no more silent truncation). - `floatConfigHelper` / `percentageConfigHelper`: require a finite number; percentage additionally enforces 0–1. `bigint`, `enum` and the secret helpers already threw on invalid input. `booleanConfigHelper` is intentionally unchanged: `parseBooleanEnv` is a total function (unrecognized tokens map to `false`), it never substitutes the configured default. Auditing all ~150 `numberConfigHelper` call sites then surfaced options that legitimately take fractional values and would now wrongly reject them; these were moved to `floatConfigHelper`: - p2p gossipsub tx scoring — topic weight, invalid-message-delivery weight, and decay (default `0.5`), which feed libp2p's float `TopicScoreParams`. - sequencer `perBlockDAAllocationMultiplier` (default `1.5`); its sibling `perBlockAllocationMultiplier` already used `floatConfigHelper`. - L1 tx fee percentages — `gasLimitBufferPercentage`, `priorityFeeBumpPercentage`, `priorityFeeRetryBumpPercentage`. - bot `minFeePadding`, consumed as the fractional overpay factor `1 + padding`; its zod schema no longer pins it to an integer. ## Impact A node whose numeric config env var (or CLI flag) is set to a non-numeric, fractional (for integer options), or out-of-range value now fails at startup with a clear error instead of silently running with a truncated or default value. Operators relying on the old lenient behavior must correct the value. Fixes A-1398
…(A-1401) (#24537) ## Context A-1351 (#24515) fixed the honest-proposer and relay path for non-canonical yParity attestation signatures, but a malicious selected proposer can still hand-craft `propose()` calldata carrying a non-proposer signature slot with `v ∈ {0,1}` (or an all-zero `(r,s,v)` in a bitmap-marked signature slot). L1 accepts this at propose time (it only recovers the proposer's own slot) and stores the attestations hash over the raw bytes. Afterwards, either: - the slot recovers to the correct member, so honest nodes deem the checkpoint valid and nobody invalidates it — but epoch proving reverts at `ECDSA.recover` (which rejects `v ∉ {27,28}`), a silent stall; or - the slot is detected as invalid, but the invalidation repack via `packAttestations` is not byte-faithful, so the recomputed hash diverges from the stored one and `InvalidateLib` reverts — wedging the chain until prune. This is the malicious-proposer sibling of A-1351; it survives that fix. Tracks aztec-claude#650. ## Approach TS-only — no L1 changes, since this targets a fresh deployment. Two parts: - **Detection** — `getAttestationInfoFromPayload` no longer passes `allowYParityAsV: true`, so TS signature validity matches L1's `ECDSA.recover`. A non-canonical slot is now classified as an invalid attestation, so honest nodes detect the bad checkpoint instead of silently accepting it. - **Byte-faithful invalidation** — the raw packed `CommitteeAttestations` tuple from the checkpoint calldata is threaded verbatim (as `verbatimAttestations`) through the negative `ValidateCheckpointResult` into `buildInvalidateCheckpointRequest`, which submits those exact bytes rather than repacking. With the original bytes, `invalidateBadAttestation` reproduces the stored hash and L1's `tryRecover` returns `address(0) ≠ member`, so invalidation succeeds. `verbatimAttestations` is a required field on the negative result and is serialized unconditionally — a repack fallback is intentionally absent, since silently repacking would re-introduce the wedge. The gossip sender-recovery path (`recoverCoordinationSigner`) deliberately stays lenient (`allowYParityAsV`) so an honest node can still attribute a yParity-encoded message and canonicalize it on ingress; only the on-chain checkpoint validation is made strict. A comment documents the split so it is not accidentally unified. ## Tests An e2e regression under the invalidation suite (`invalidate_block.parallel.test.ts`, "proposer invalidates checkpoint with a yParity attestation slot") demonstrates the attack red→green: a malicious proposer lands a checkpoint whose non-proposer attestation slots carry raw yParity bytes, and the honest next proposer invalidates it — which only succeeds when the invalidation submits the verbatim calldata bytes (a repack diverges from the on-chain `attestationsHash` and reverts). Plus unit coverage for the strict attestation-info classification, the byte-faithful passthrough, and the result round-trip. An attester-side variant (a committee member emitting yParity attestations) is not separately tested: the honest proposer's `orderAttestations` normalization (A-1351) canonicalizes the byte before the L1 bundle, so it never reaches L1 and produces the same on-chain outcome as — and is subsumed by — the malicious-proposer case above. Fixes A-1401
Closes F-771 --------- Co-authored-by: Nicolas Chamo <nicolas@chamo.com.ar>
## Problem `l1_tx_utils.test.ts` flaked in CI ([log](http://ci.aztec-labs.com/390c881d64e094b6), seen on [#24454](#24454 (comment)), which is unrelated to the failure). The failing test was `monitors all sent txs`: ``` ● L1TxUtils › L1TxUtils with blobs › monitors all sent txs TimeoutError: L1 transaction 0x82cf...aa3a timed out at TestL1TxUtils.monitorTransaction (l1_tx_utils/l1_tx_utils.ts:601:11) ``` ## Root cause The test creates the monitor promise and only attaches its rejection expectation after several intervening awaits: ```ts const monitorPromise = gasUtils.monitorTransaction(state); await sleep(100); await cheatCodes.mineEmptyBlock(); await expect(monitorPromise).rejects.toThrow('timed out'); // handler attached here ``` Anvil timestamps have 1s granularity, so when the wall clock crosses a second boundary between the send and the monitor's timeout check, the monitor (with `txTimeoutMs: 200`, `checkIntervalMs: 100`) can observe the L1 timestamp already 1s past `sentAt` and reject **before** `mineEmptyBlock()` completes. In the failing run the timeout fired at `16:41:20.355`, before the empty block was mined at `.356` and before the `.rejects` handler attached at `.357`. Node emitted `unhandledRejection` in that window and jest 30 reported it as the test failure — which is why the failure stack has no test-file frame and the test's own assertions all passed in the log. The same file already uses the safe idiom in a dozen newer tests: attach the handler synchronously with `.catch(err => err)` and assert via `resolves.toBeInstanceOf(TimeoutError)`. ## Fix Apply that established pattern to the three remaining tests that expect a monitor timeout but attach the handler only after intervening awaits: - `stops trying after timeout once block is mined` - `attempts to cancel timed out transactions` - `monitors all sent txs` (the observed flake) Test-only change; no product code touched. `TimeoutError` is exactly what `monitorTransaction` throws on timeout, so the assertions are equivalent-or-stricter than the previous message matching. Evidence: CI log http://ci.aztec-labs.com/390c881d64e094b6 (search for `monitors all sent txs`). --- *Created by [claudebox](https://claudebox.work/v2/sessions/0959caa9f03f59fa) · group: `slackbot`*
…24546) ## Problem `e2e_ha_full.parallel.test.ts` ("should produce blocks with HA coordination and attestations") failed in CI on an unrelated PR (#24492) before the test even started: Docker Hub returned a transient 502 while `docker compose up` pulled `postgres:16-alpine` for the HA compose stack. CI log: http://ci.aztec-labs.com/cac25b9251c99496 ``` postgres Error unknown: failed to resolve reference "docker.io/library/postgres:16-alpine": unexpected status from HEAD request to https://registry-1.docker.io/v2/library/postgres/manifests/16-alpine: 502 Bad Gateway ``` This can hit any compose-based test (`run_compose_test` is shared by the e2e compose/web3signer/ha suites and docs examples) whenever a required image is not in the local Docker cache and the registry blips. ## Fix Wrap the `docker compose up -d --force-recreate` in `ci3/run_compose_test` with the existing `ci3/retry -p <regex>` helper, retrying only failures that match transient registry/network errors (5xx, HEAD-request failures, TLS/connection timeouts, rate limiting). This reuses the same pattern-gated retry convention already used for network flakes in `l1-contracts/bootstrap.sh`, `barretenberg/sol/bootstrap.sh`, and `noir-projects/aztec-nr/bootstrap.sh`, so genuine failures (e.g. an image tag that doesn't exist) still fail immediately without retrying. Retrying `up -d --force-recreate` is idempotent: a retry recreates any containers from a partially failed attempt. Verified the exact error message from the CI log matches the retry pattern (retried, then succeeds), while a `manifest not found` failure exits immediately without retries. No tracked issue exists for this flake. --- *Created by [claudebox](https://claudebox.work/v2/sessions/0959caa9f03f59fa) · group: `slackbot`* --------- Co-authored-by: Santiago Palladino <santiago@aztec-labs.com>
…uplicate_proposal test (#24557) ## Problem `multi-node/slashing/duplicate_proposal › slashes validator who sends duplicate proposals` (`yarn-project/end-to-end/src/multi-node/slashing/equivocation_offenses.test.ts`) flaked in CI on the unrelated PR #24556 with: ``` Target proposer 0x90f79bf6eb2c4f870365e785982e1f101e93b906 not found in any slot after 20 epoch attempts ``` Evidence: http://ci.aztec-labs.com/eb0b4e658e28440c (the sibling `duplicate_attestation` case in the same file passed in the same run). ## Root cause Pure statistics — no logic bug. `advanceToEpochBeforeProposer` (`yarn-project/end-to-end/src/multi-node/slashing/setup.ts` on this branch) scans upcoming epochs for one where the target validator is the proposer, giving up after `maxAttempts = 20` epochs. The test calls it with the default. This suite uses a 2-slot epoch with `warmupSlots = 1`, so each epoch attempt inspects exactly **one** candidate slot. The proposer for a slot is `keccak256(abi.encode(epoch, slot, seed)) % committeeSize`, and with the suite's 4-member committee (`COMMITTEE_SIZE = NUM_VALIDATORS = 4` in `setup.ts`) each attempt is an independent 1/4 draw. Miss probability over 20 attempts: `(3/4)^20 ≈ 0.32%`, i.e. roughly 1 in 315 runs — matching the observed occasional flake. ## Fix Raise the default `maxAttempts` from 20 to 50, bounding the miss probability at `(3/4)^50 ≈ 5.7e-7` (~1 in 1.8M). Each attempt is only a committee query plus an anvil timestamp warp, so the expected attempt count stays ~4 and the extra headroom costs essentially nothing. The raised default also covers this helper's other callers with the same exposure (`broadcasted_invalid_block_proposal_slash`, `broadcasted_invalid_checkpoint_proposal_slash`), none of which override `maxAttempts`. ## Relationship to PR #24548 This is the same failure mode fixed by #24548 for the `duplicate_attestation` flake, but #24548 does **not** cover this call site: its patch touches `yarn-project/end-to-end/src/e2e_p2p/shared.ts`, a file that does not exist on `merge-train/spartan-v5` (on this branch the helper lives in `multi-node/slashing/setup.ts`, and there is no `e2e_p2p/` directory). #24548's head branch was cut from a `next`-layout tree while targeting `merge-train/spartan-v5`, which is why GitHub reports it as dirty with ~4k changed files — it needs a rebase onto the branch it targets, at which point its fix would land in this same file. This PR applies the fix directly to the `merge-train/spartan-v5` copy of the helper. No tracked issue exists for this flake. --- *Created by [claudebox](https://claudebox.work/v2/sessions/0959caa9f03f59fa) · group: `slackbot`*
…test (#24543) ## Flake `multi-node/block-production/multi_validator_node.parallel.test.ts`, case "should build blocks & attest with multiple validator keys", failed on [#24537](#24537) (unrelated PR) with: ``` TypeError: Cannot read properties of undefined (reading 'checkpoint') at checkpoint (multi-node/block-production/multi_validator_node.parallel.test.ts:92:73) ``` CI log: http://ci.aztec-labs.com/0f321ffb9fa923af ## Root cause `deployContractAndGetAttestedCheckpoint` reads the published checkpoint from the archiver immediately after the deploy tx receipt turns mined: ```ts const [publishedCheckpoint] = await dataStore.getCheckpoints({ from: blockData!.checkpointNumber, limit: 1 }); const payload = ConsensusPayload.fromCheckpoint(publishedCheckpoint.checkpoint, ...); // undefined here ``` The receipt turns mined as soon as the block is built locally, but the archiver only stores the *published* checkpoint (with its attestations, which come from the L1 `propose` calldata) once its L1 sync downloads that tx. In the window between the two, `getCheckpoints` returns `[]` and `publishedCheckpoint` is `undefined`. The CI log shows the race directly: the test node published checkpoint 1 to L1 at `15:17:57.694` (`sequencer:publisher ... Published checkpoint 1 at slot 14`), and its archiver logged `Downloaded checkpoint 1` at `15:17:57.795` — by which point the test had already failed and teardown was running. This is a test-side race, not a production bug: the archiver API is behaving as documented for a not-yet-synced checkpoint. ## Fix Poll for the checkpoint with `retryUntil` instead of reading it once, mirroring the existing pattern in `single-node/cross-chain/cross_chain_messaging_test.ts` (`advanceToEpochProven`), which wraps the same lookup in `retryUntil(..., 'archiver indexes checkpoint N', 120, 0.5)`. Note: the `next`-line variant of this test (`e2e_multi_validator/e2e_multi_validator_node.test.ts`) has the same unguarded lookup in two places and could flake the same way; that file is on a different base branch, so it is not touched here. Based on `merge-train/spartan-v5` because the flaking file only exists on the v5 line (it was consolidated into `multi-node/` there), and the failing run was on a PR targeting that branch. --- *Created by [claudebox](https://claudebox.work/v2/sessions/0959caa9f03f59fa) · group: `slackbot`*
Closes #5837 This PR makes it so we no longer derive signing keys from the master privacy key. If there's no user-supplied privacy key seed, we instead derive it _from_ the signing key, which is a safe derivation: PXE never sees the signing key nor any value it is derived from. This has a large knock-on effect on wallets, CLI, tutorials, etc., but there's ultimately no large decisions being made there, it is mostly mechanical changes. In some cases I reworked some internals of wallets to make this distinction clearer.
…during shutdown (v5 line, partial) (#24551) ## Flake `multi-node/slashing/sentinel_status_slash.parallel.test.ts` ("slashes an attestor that gets stopped after the network is running") flaked on PR #24507 (unrelated change). CI log: http://ci.aztec-labs.com/9bb09d2f904e290a ## What actually failed The test itself **passed**. Immediately after, the node process crashed with an **unhandled rejection** (`ECONNREFUSED` on anvil after teardown) — something was still polling L1 after shutdown. Root cause: `L1TxUtils.monitorTransaction`'s loop calls `getL1Timestamp()` at the top of its loop, outside the loop's own try/catch, and nothing waited for in-flight monitor iterations to actually stop during shutdown. ## Scope of this PR (rebased onto `merge-train/spartan-v5`) This PR was originally written against `next` and touched 4 files. After rebasing onto `merge-train/spartan-v5`, 2 of those files cherry-picked cleanly and are included here: - `yarn-project/ethereum/src/l1_tx_utils/l1_tx_utils.ts` — guards the loop-top `getL1Timestamp()` call to break quietly when interrupted. - `yarn-project/sequencer-client/src/publisher/sequencer-publisher.ts` — adds `.catch` to a bare fire-and-forget `backupDroppedInSim(...)` call. The other 2 files (`publisher_manager.ts`, `checkpoint_proposal_job.ts`) have diverged in structure on this branch and needed their own investigation rather than a blind cherry-pick — that's covered separately in [#24560](#24560), which found `publisher_manager.ts` had the same defect (fixed there) and `checkpoint_proposal_job.ts` did not (this branch already routes fire-and-forget L1 requests through a `RequestsTracker` that attaches its own rejection handler). No tracked issue exists for this flake; reference only. --- *Created by [claudebox](https://claudebox.work/v2/sessions/0959caa9f03f59fa) · group: `slackbot`*
…ts during shutdown (v5 line) (#24560) Ports the remaining part of the `sentinel_status_slash` flake fix to the v5 line. The flake crashed the jest process after all tests passed: an in-flight `eth_getBlockByNumber` against the torn-down anvil surfaced as an unhandled `L1RpcError` (`ECONNREFUSED`). CI log: http://ci.aztec-labs.com/9bb09d2f904e290a This complements #24551, which (after retargeting to `merge-train/spartan-v5`) now carries only the two files that cherry-picked cleanly from `next` — `l1_tx_utils.ts` (guard the loop-top `getL1Timestamp()` in `monitorTransaction`) and `sequencer-publisher.ts` (`.catch` on the fire-and-forget `backupDroppedInSim`). The other two files had diverged on this branch and are addressed here; #24551's title/body should be updated to reflect that it covers 2 of the original 4 files, with this PR covering the rest. ### `yarn-project/ethereum/src/publisher_manager.ts` — fixed This branch's `stop()` (which, unlike `next`, also clears the `started` flag and supports restart via `start()`) interrupted all publishers and the funder but returned immediately, so an in-flight tx monitor loop could still be issuing L1 RPC calls after test teardown. Applied the equivalent of the `next` fix, fitted to this branch's structure: `stop()` now awaits `waitMonitoringStopped()` on every publisher and the funder before returning. `L1TxUtils.waitMonitoringStopped(timeoutSeconds = 10)` already exists on this branch, is self-bounded, and swallows its own timeout with a warning, so shutdown cannot hang. ### `yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.ts` — no change needed On `next`, the votes-only path assigned a bare `pendingL1Submission = publisher.sendRequestsAt(...)` with no rejection handler, and only the last job's submission was awaited at shutdown, so an orphaned job's rejection could crash the process. On this branch that field no longer exists: both fire-and-forget paths go through `this.pendingRequests.trackRequest(promise, () => this.interrupt())`, backed by the sequencer's shared `RequestsTracker` (`requests_tracker.ts`). `trackRequest` attaches a rejection handler synchronously at track time (`promise.then(delete, delete)`), so a tracked promise can never surface as an unhandled rejection regardless of whether shutdown awaits it. Additionally, `Sequencer.stop()` calls `pendingRequests.interruptRequests()` followed by `awaitRequests()`, so all tracked requests — not just the last one — are interrupted and drained at shutdown. The `next` bug does not exist here, so no change was forced. (Minor observability note, not a bug: a rejection on the votes-only path is swallowed silently by the tracker rather than logged.) Verified `publisher_manager.ts` parses cleanly; the full workspace type-check runs in CI (this container lacks the bootstrapped bb.js/noir/l1-artifacts toolchain). No tracked issue. --- *Created by [claudebox](https://claudebox.work/v2/sessions/0959caa9f03f59fa) · group: `slackbot`*
BEGIN_COMMIT_OVERRIDE fix(sqlite3mc-wasm): restore bundler-visible wasm resolution (#24529) refactor(aztec-nr): rename get_handshakes to non_interactive variant (#24511) fix(pxe): UNFINALIZED_TAGGING_INDEXES_WINDOW_LEN should be MAX_PRIVATE_LOGS_PER_TX + headroom, not exactly MAX_PRIVATE_LOGS_PER_TX (#24437) refactor(aztec-nr): keep handshake secrets internal to TagSecretSource (#24508) test(txe): extend oracle roundtrip coverage to more scalar oracles (#24550) refactor!: reimplement partial notes on FactStore (#24369) feat!: stop deriving signing key from privacy keys (#24439) END_COMMIT_OVERRIDE
BEGIN_COMMIT_OVERRIDE perf(e2e): warp proven-checkpoint waits, tighten in-process polling, and instrument setup spans (#24452) fix: reject aztecSlotDuration not a multiple of ethereumSlotDuration (#24481) fix(config): fail loudly on invalid or fractional numeric config (#24536) fix(sequencer): invalidate malicious yParity attestation checkpoints (A-1401) (#24537) fix(ethereum): deflake monitor timeout tests in l1_tx_utils (#24545) fix(ci): retry docker compose up on transient registry pull failures (#24546) fix(e2e): resolve flaky proposer-not-found in equivocation_offenses duplicate_proposal test (#24557) fix(e2e): resolve flaky checkpoint TypeError in multi_validator_node test (#24543) fix(ethereum): prevent unhandled L1 rejections from tx monitor loops during shutdown (v5 line, partial) (#24551) fix(sequencer): prevent unhandled rejections from in-flight L1 requests during shutdown (v5 line) (#24560) END_COMMIT_OVERRIDE
…24563) ## Context `getWorldState` backs most node RPC tree/witness queries, and had three related problems: - World-state sync failures were swallowed (logged and ignored), so queries were served from whatever state the node had — stale reads with no signal to the caller. - For tag and number queries, the target block was resolved *after* syncing, so an archiver tip advancing mid-request (routine during catch-up) made valid queries fail spuriously with "not yet synced". - Resolution pinned only a block *height*, never a *fork*: a reorg replacing the block at that height between resolution, sync, and snapshot read was served silently — e.g. a `proven` query could be answered with a same-height block from a new fork that is not proven. ## Approach - Every query variant (number, hash, archive, tag) now resolves up front to a concrete (block number, block hash) via the archiver's `getBlockData`, and the sync is driven to that exact block and hash so the synchronizer barriers on it and detects fork mismatches. - A new `WorldStateSynchronizer.getVerifiedSnapshot(blockNumber, blockHash)` verifies the snapshot's own archive view matches the requested fork before handing it out (block 0 is checked against the initial header hash, since its snapshot predates the genesis archive leaf), closing the remaining window between sync and snapshot read. - Sync and fork-verification failures surface as `WorldStateSynchronizerError` and are retried a few times with the query re-resolved each time, so prunes and fork flips heal onto the current chain instead of failing or serving wrong-fork state. Hash and archive resolution misses stay terminal with a clear reorg error; tag and number misses are treated as transient. - `proposed`/`latest` queries keep their "latest on the current fork" semantics: plain sync plus the committed db, unverified. ## API changes - `WorldStateSynchronizer` gains `getVerifiedSnapshot(blockNumber, blockHash)`, implemented by `ServerWorldStateSynchronizer` (fork-verified) and `TXESynchronizer` (passthrough, no reorgs). Other `getSnapshot` callers are unchanged. - Node RPC queries at a block now fail instead of silently returning stale state when world state cannot sync, and the error messages for unknown or unsyncable blocks have changed accordingly. Fixes A-1339
…ndle pg pool errors (A-1313, A-1314) (#24556) Hardens the validator HA signing path against an honest-validator double-sign and a Postgres-induced crash. Both issues live entirely in the HA signer. ## Context **A-1313 (stuck-duty cleanup race → slashable double-sign).** `signWithProtection` discarded the boolean from `recordSuccess` and returned the signature regardless. If the background cleanup loop deleted our `SIGNING` protection row while the (remote) signer was slow, the signature was broadcast with no protection record in place — so a later proposal for the same slot with different data could sign freely, i.e. slashable equivocation by an honest validator. The race is live today: `SequencerClient.start()` starts the validator client, which starts the cleanup loop. (The issue text claimed this was masked by #1111 never starting the cleanup loop; that is stale and no longer true.) **A-1314 (pg.Pool without an 'error' listener crashes HA validators).** `createHASigner` built a `new Pool(...)` with no `pool.on('error')`. pg re-emits idle-client errors (e.g. a Postgres restart severing an idle connection) on the pool; with no listener Node escalates to `uncaughtException` and crashes the process — taking down every HA replica sharing the DB at once. ## Approach Four defense-in-depth fixes for A-1313 (part 1 alone closes the slashable broadcast; parts 2-3 remove the ways the race arises) plus the independent A-1314 liveness fix: - Treat `recordSuccess === false` as a signing failure: throw a new `SigningLockLostError` and never return/broadcast the signature. The duty is not deleted, since we no longer own the row. This is the hard safety backstop: even under pathological timer behavior the worst case is a failed duty, never a broadcast double-sign. - Bound each signer call with a hard timeout (`executeTimeout`), configured via `signerCallTimeoutMs` (default 30s) and clamped at construction to `maxStuckDutiesAgeMs / 2` (default 144s / 2). Since `signWithProtection` is the only writer of `SIGNING` rows and every path through it is bounded by this clamped timeout, an in-flight signing always times out and releases its row well before stuck-duty cleanup could reclaim it — so cleanup can never race a live signing. A timeout takes the existing failure path (release the lock, then throw), and the orphaned signer promise is discarded. - Add a request timeout (`AbortSignal.timeout`) to both Web3Signer key-store fetches, surfaced as a clear timeout error instead of hanging. The keystore is only constructed directly in tests today (production uses node-keystore's remote signer), so the timeout is a constructor option defaulting to 30s. - Register a `pool.on('error', ...)` structured-log handler in `createHASigner` (covering both the created and injected pool), and add the same handler to the e2e HA fixture pools. Only message/code is logged, never the raw error. ## API changes - New env var `VALIDATOR_SIGNER_CALL_TIMEOUT_MS` (default 30000) controls the per-call signing timeout (effective value clamped to `maxStuckDutiesAgeMs / 2`), exposed as config field `signerCallTimeoutMs`. - The confusable existing config field `signingTimeoutMs` (how long to wait for a *peer* node's in-progress signing, default 3s) is renamed to `peerSigningTimeoutMs` so each name says whose wait it bounds. Its env var `VALIDATOR_HA_SIGNING_TIMEOUT_MS` is already shipped and is kept unchanged; the derived CLI flag follows the field rename (`--sequencer.signingTimeoutMs` → `--sequencer.peerSigningTimeoutMs`). Each fix follows red/green with unit tests in `validator-ha-signer` (errors, signer, slashing-protection service, LMDB + Postgres backends, factory) and `validator-client` (Web3Signer key store). Fixes A-1313 Fixes A-1314
## Context `readMessage` on the requesting side accumulated **all** response chunks into memory before the snappy `maxSizeKb` validation ran (that guard lives in `SnappyTransform.inboundTransformData`, which only executes after the full compressed stream has been buffered). A peer we dialed could send a valid `SUCCESS` status chunk and then stream data up to the request timeout, forcing the requesting node to buffer arbitrarily much (~1.2 GB/response at 1 Gbps within the 10s timeout) → remote-triggerable OOM, amplified by concurrent in-flight requests. ## Approach - Track a running byte total in the `readMessage` loop and abort as soon as it exceeds a bound derived from `maxSizeKb`, before buffering more chunks. The bound is `maxSizeKb * 1024 * 2`: snappy can expand incompressible input to ~1.17x, so 2x sits comfortably above the worst case (a legitimate max-size response is never rejected) while capping buffered memory at twice the permitted post-decompression size. - Uses `chunk.byteLength` rather than `chunk.subarray().length` — calling `subarray()` on a `Uint8ArrayList` consolidates its backing buffers into a copy, which would materialize the very chunk we are trying to reject. - Throws a dedicated `ResponseSizeLimitExceededError`, penalized as a `LowToleranceError` via the existing `categorizeResponseError`/`handleResponseError` path, so the offending peer's stream is torn down and it is scored down. - The existing decompressed-size preamble check in `inboundTransformData` still guards the decompression-bomb variant; this closes the residual reception-side buffering gap. Per-request bound only — aggregate in-flight amplification across peers is tracked separately. Fixes A-1399
…24554) A reqresp request payload must fit in a single muxer frame: yamux splits writes larger than 64KiB (minus the 12-byte frame header) into multiple frames, each of which arrives at the responder as a separate chunk, and the responder never reassembles a request from multiple chunks. A request type growing past that limit would surface as a confusing decoding error on the responder instead of failing at the sender. - Assert the payload size in `sendRequestToPeer` before dialing, throwing a descriptive `OversizedReqRespRequestError` locally. The check runs before the generic error handling so the remote peer is not penalized for a local bug. - Pin `maxMessageSize` on the yamux muxer to its library default (64KiB) so a dependency upgrade cannot silently change the limit the assertion relies on. - Tests: an oversized payload is rejected without dialing the peer and without penalizing it; a payload at exactly the limit round-trips successfully. Related to #24552. ## No current subprotocol can exceed the limit The limit is 65,524 bytes (64KiB yamux frame minus the 12-byte header). **No reqresp subprotocol today can produce a request anywhere near it**, so this assertion cannot fire on legitimate traffic: - `GOODBYE`: 1 byte (the reason code). - `PING`: a few bytes. - `STATUS`: a `StatusMessage` (component versions string, block numbers, block hash) — tens of bytes. - `AUTH`: an `AuthRequest` (`StatusMessage` plus a 32-byte challenge) — tens of bytes. - `TX`: the request type is `TxHashArray` (4 + 32·n bytes), but this subprotocol currently has no sender — only the server-side handler remains. - `BLOCK_TXS`: the only request that scales with anything. Serialized as `archiveRoot` (32 bytes) + tx-indices `BitVector` (4 + ⌈N/8⌉ bytes, N = tx count of the proposal) + tx-hashes commitment (32 bytes) + optional full-hash vector (4 + 32·H bytes). Pinned-peer and smart-peer requests send indices only (H = 0). Dumb-peer requests include full hashes but chunk them to `txBatchSize`, which is always the default 8 in production (`BatchTxRequester` is constructed without opts, and the option is not wired to any config). N is capped at deserialization by `MAX_TXS_PER_BLOCK = 2^16`, so even a maliciously large proposal yields a worst-case request of ~8.5KB — about 13% of the limit. A realistic 32-tx block produces requests of a few hundred bytes. Breaching the limit would take a proposal with ~460k txs (which `BitVector.fromBuffer` rejects at 2^16 anyway) or a `txBatchSize` over ~1,800 with full hashes enabled — neither is reachable today. The assertion exists so that if a future change makes a request type scale past the limit, it fails loudly at the sender instead of as a decoding error at the responder.
#24552) ## Context The reqresp rate limiter is consulted once per inbound stream (`streamHandler`), but the sub-protocol handler was invoked once per chunk read from that stream (`processStream`). Req/resp is one-request-one-response and an honest sender writes a single payload before half-closing, but a malicious peer can open one stream (costing a single rate-limit token) and then push many request frames on it — each frame arrives as its own chunk and drives a full handler invocation (mempool / block / tree lookups). Per-peer and global rate limits are bypassed by the fan-out factor. ## Approach Make `processStream` handle exactly one request per stream: after emitting the response for the first chunk, the pipeline generator returns instead of looping over the rest of the source. Extra frames a peer queued on the same stream are discarded when the stream closes, so work is bounded to one request per token. This does not regress legitimate traffic — no code path pipelines multiple requests on a single stream, and the one-chunk-per-request framing is already in force. The alternative (per-invocation rate checks inside the loop) was rejected: nothing pipelines, and mid-stream status signalling is broken on the requester side, which parses the first chunk as status and concatenates the rest as data. Adds a unit test that drives `processStream` with three frames on one stream and asserts the handler runs once and the sink receives a single SUCCESS + response pair. Fixes A-1324
…-1315, A-1317, A-1318) (#24565) ## Context A store that must never silently become empty — the single-node signing-protection LMDB, and (for the ordering fix) world-state — could be wiped or bypassed by the `DatabaseVersionManager` / store-creation path. - **A-1315**: `writeVersion()` ran *before* `onOpen()`, and was a plain non-atomic `writeFile`. A crash in between left a "valid" marker over an empty/partial data dir; on restart the self-healing reset was skipped forever (a stable wedge for world-state, which opens several native stores in `onOpen`). - **A-1318**: any non-ENOENT read/parse failure of the version file (EACCES, EIO, truncation) fell back to `DatabaseVersion.empty()`, which unconditionally triggered a reset — so a transient permissions/disk error at startup rm-rf'd the signing-protection DB. The `schemaVersionMismatchPolicy: 'throw'` added by A-1029 only guarded the numeric-mismatch branches, not this one. - **A-1317**: a missing `dataDirectory` silently selected a fresh ephemeral tmp store on every start, giving no double-signing protection across restarts, with only a `debug` log. ## Approach - **A-1315**: `DatabaseVersionManager.open()` now opens the database *before* writing the version marker, making the marker a post-commit record — a crash before a durable open leaves no marker, so the next start re-runs the reset. `writeVersion` is now an atomic durable write (temp file → fsync → rename → best-effort directory fsync). The marker is only (re)written when it would actually change (first boot, reset, upgrade), which also avoids leaking a freshly opened DB if the write fails and drops the per-boot fsync. - **A-1318**: new `versionFileReadFailurePolicy` (`'reset'` default, preserving existing behavior for archiver/p2p/world-state; `'throw'` for signing protection). On `'throw'`, an unreadable version file fails startup with an operator-actionable error and leaves data untouched. Threaded through the kv-store `createStore` options. - **A-1317**: `createLocalSignerWithProtection` now throws when no data directory is configured, unless `allowEphemeralSigningProtection` (env `VALIDATOR_ALLOW_EPHEMERAL_SIGNING_PROTECTION`, default false) is set, in which case it warns loudly. The local network sets the flag by default; the production default is strict fail-fast. The rollup-address-change reset is intentionally left as-is: the LMDB slashing DB relies on it (see the `cleanupOutdatedRollupDuties` no-op). Reviewed by Codex and a second model; both confirmed the atomic-write mechanics, ENOENT branching, migration idempotency, and the schema-composition decision (a cross-field Zod refine was avoided because it would break the downstream `.merge()`/`.extend()` chain, so the invariant is enforced at the factory). Operator-facing changes are documented in the v5 changelog. Fixes A-1315 Fixes A-1317 Fixes A-1318
…#24329) ## What Adds `keepFinalizedTxsForSlots` to the v2 tx pool. Instead of deleting a finalized tx's data at the finalized tip, the pool keeps it for a configurable number of slots behind finality. Default is `0` (current behaviour). Prover nodes raise it automatically. ## Why A prover node fetches a checkpoint's txs from its tx pool (`TxProvider` → pool first, reqresp fallback) and re-reads them for failure upload. ## How - New `keepFinalizedTxsForSlots` config (`P2PConfig` + env `P2P_KEEP_FINALIZED_TXS_FOR_SLOTS`, default `0`). - `handleFinalizedBlock` resolves the slot margin to a block cutoff: target slot = `finalizedSlot − margin`, find the checkpoint at or before it, and use that checkpoint's last block as the deletion cutoff. Applied to both the active-pool deletion and the `DeletedPool` finalize path. Rounds to a checkpoint boundary, so it can retain slightly more than the configured margin but never less, and never deletes past the finalized block. A margin of `0` short-circuits to the previous behaviour. - The "checkpoint at or before the target slot" lookup is a single reverse range query, not a slot-by-slot scan: a new `{ fromSlot, limit, reverse }` variant on `CheckpointsQuery` (backed by `BlockStore.getCheckpointsBySlot`, a one-pass walk of the existing slot index) returns the nearest checkpoint at or before the slot with `limit: 1, reverse: true`. All `CheckpointsQuery` resolution now lives in `getCheckpointsData`. - Prover nodes: `createAztecNodeService` floors `keepFinalizedTxsForSlots` at `(proofSubmissionEpochs + 1) × epochDuration` (read from the rollup), taking the max with any operator-configured value and warning when it raises it. This matches the prover-node catch-up window. ## Testing - New unit tests in `tx_pool_v2.test.ts`: the retain/delete boundary and the checkpoint resolution when the target slot falls in a gap. - New `getCheckpointsBySlot` tests in `block_store.test.ts`: reverse exact-hit, gap walk-back, before-genesis, multi nearest-first, and the forward direction. - `fromSlot` schema round-trip added to the archiver and aztec-node interface tests. - Full `tx_pool_v2` (254), `block_store` (159), and interface (120) suites pass. Closes A-1274.
BEGIN_COMMIT_OVERRIDE fix(node): fork-aware getWorldState that fails closed on sync errors (#24563) fix(validator): prevent double-sign on stuck-duty cleanup race and handle pg pool errors (A-1313, A-1314) (#24556) fix(p2p): bound reqresp response buffering before size check (#24553) fix(p2p): reject reqresp requests larger than a single muxer frame (#24554) fix(p2p): process one request per reqresp stream to enforce rate limit (#24552) fix(validator): fail closed on signing-protection persistence gaps (A-1315, A-1317, A-1318) (#24565) feat(p2p): retain finalized txs a configurable margin behind finality (#24329) END_COMMIT_OVERRIDE
…24593) `CheckpointSubTreeOrchestrator.addTxs` logs `Provided no txs to addTxs.` at **warn** level, but this path is expected control flow, not an anomaly: `checkpoint-prover.ts` calls `this.subTree.addTxs(processed)` unconditionally for every block in the checkpoint, so every empty block hits this early return (the empty block is then finalized via `setBlockCompleted`). On low-traffic networks this fires constantly — ~100 times in 4h on staging-internal — polluting warn-level log reviews with noise. Demote it to `verbose`. No behavior change; no tests assert on this message. Replaces #24591 and #24592 (closed), per request to land this as a single PR via the spartan-v5 merge train. Spotted during a staging-internal log review requested by @spalladino in Slack. --- *Created by [claudebox](https://claudebox.work/v2/sessions/ac050bae673edac4) · group: `slackbot`*
Adds a new `THREAT_MODEL.md` doc. This document describes the threat model of the Aztec L2 network: how transactions flow from users into the proven chain, what each participant can and cannot do, and the properties the implementation must uphold. It is intended as a guideline for the security of the node implementation (`yarn-project`) and its interaction with the L1 rollup contracts (`l1-contracts`). **In scope**: transaction dissemination and the mempool, the p2p layer, block and checkpoint production, committee attestation, L1 checkpoint submission and sync, epoch proving, slashing, and the escape hatch. **Out of scope**: client-side private execution and proving (PXE, wallets), hardening of a node's public RPC interface, the cryptographic soundness of the proving system itself (treated as an assumption below), and L1 governance internals.
AztecBot
added
ci-draft
AztecBot
force-pushed
the
cb/writing-e2e-tests-skill
branch
from
6057fdb to
1dcf5bc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
yarn-project/.claude/skills/writing-e2e-tests/SKILL.md: a skill that gives an agent everything it needs to place, structure, and write a robust e2e test with little guidance. Written against the v5-next e2e layout (category directories with per-category READMEs and context classes).What it covers
itin an existing suite → new file on an existing context/harness → new standalone test.automine/viaAutomineTestContext,single-node/viasetupWithProver/setupBlockProducer,multi-node/viaMultiNodeTestContexton the mock-gossip bus,p2p/viaP2PNetworkTeston real libp2p, pluscomposed/,infra/,spartan/,bench/), the multi-node-vs-p2p decision rule, pointers to the per-category READMEs as the authoritative reference, file naming (describematches path, header comment), automatic CI registration throughbootstrap.shglobs,.parallel.test.tssemantics, and jest/bash timeout sync.SetupOptions), the standard suite file shape, one-environment-per-file, preset spread order, guarded teardown.fixtures/wait_helpers.ts, context waiters,ChainMonitor), shared helpers and co-located*_test_helpers.ts, simulators withafterEachchecks, shared error constants.test(e2e)/perf(e2e)/chore(e2e)speedup PRs and their tracking issues — avoid unneeded setup, genesis seeding over setup txs,BatchCallbatching overPromise.alloverlap (with the PXE-serialization caveat), warping dead waits withmarkProvenAndWarp/warpWithSequencersPaused("an honest wait beats a flaky warp"), named timing profiles over ad-hoc cadences, and measuring with thetestSpaninstrumentation +track-e2e-timesskill before optimizing.syncChainTiptag awareness, pipelining timing margins +findSlotsWithProposers, invariant-not-exact-value assertions, L1 account/nonce and port hygiene, freeze-L1-across-restarts, gossip-mesh readiness viarunGossipScenario, fee padding, determinism, honest timeouts), plusdeflaker.shvalidation and.test_patterns.ymlas a last resort.All helper/API names cited were verified to exist on
merge-train/spartan-v5(AutomineTestContext.setup,setupWithProver/setupBlockProducer,MOCK_GOSSIP_MULTI_VALIDATOR_OPTS,wait_helpers.tswaiters,warpWithSequencersPaused,markProvenAndWarp,testSpan/TEST_TIMING_FILE,runGossipScenario,waitForP2PMeshConnectivity,findSlotsWithProposers,getPaddedMaxFeesPerGas, etc.). Genesis-prefill techniques that only exist in still-open speedup PRs are described as themes to look for rather than as concrete APIs.