txmgr: enable spamoor TxBatcher in NewWalletPoolByPrivkey#171
Merged
Conversation
The recent eoa-transactions-test failures on Erigon-paired runs in ethpandaops/assertoor-test (latest-releases-1, -7, -9 since 2026-04-29) trace back to here: NewWalletPoolByPrivkey creates the RootWallet but never calls InitTxBatcher, so processFundingRequests in spamoor's walletpool.go takes the fall-back path of one EOA tx per recipient with sequential nonces. When funding 100 child wallets (the eoa-transactions-test default), that produces 100 sequential-nonce txs from the root wallet that arrive at Erigon's HTTP server in random order on parallel TCP connections. Erigon v3.4.1 (PR #19449, commit 93e5c77719) immediately evicts any queued tx with nonce gap > MaxNonceGap (default 64) from the on-chain nonce, rejecting ~33 of them with "INTERNAL_ERROR: nonce gap too large". Funding stalls, child wallets stay at 0 ETH, the 30-min foreground task times out. The batcher contract path bundles all funding recipients into one outer tx — funding 100 wallets uses 1-2 root-wallet nonces total, well below any per-account threshold on any EL. Both spamoor entry points (cmd/spamoor/main.go:223, cmd/spamoor-daemon/main.go:258) already call InitTxBatcher right after InitRootWallet. This makes assertoor consistent with that pattern.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
NewWalletPoolByPrivkeycreates theRootWalletbut never callsInitTxBatcher, so spamoor'sprocessFundingRequestsfalls back to one tx per recipient with sequential nonces.eoa-transactions-testdefault) therefore emits 100 sequential-nonce txs from the root wallet.MaxNonceGap=64zombie-eviction landed inrelease/3.4via erigontech/erigon#19449 (commit93e5c77719) — reject ~33 of those txs as nonce-gapped zombies when they arrive out-of-order on parallel TCP connections (INTERNAL_ERROR: nonce gap too large).rootWallet.InitTxBatcher(ctx, s.txpool)— matches the pattern in both spamoor entry points (cmd/spamoor/main.go:223,cmd/spamoor-daemon/main.go:258). All 100 fundings now ride in a single outer batcher-contract tx, so the root wallet only uses 1–2 nonces total.Why this manifested now in ethpandaops/assertoor-test
The CI
Run scheduled testworkflow has been failing daily since 2026-04-29 on Erigon-paired matrix entries (latest-releases-1,-7,-9). The post-failure Claude analysis in those runs flagged Erigon's txpool log[txpool] stat pending=0 baseFee=0 queued=44andeth_sendRawTransaction err=\"INTERNAL_ERROR: nonce gap too large\"as the immediate signal — but the upstream reason that signal fires is the missingInitTxBatchercall here.Note the 5–10 tx/slot the test phase emits is well below
MaxNonceGap=64. The 64-gap is hit only during the funding burst, and only because the batcher path was disabled.Reproduction (Erigon v3.4.1, shuffled-order burst)
```
$ python3 repro_nonce_gap.py --rpc http://127.0.0.1:32894 --count 100 --shuffle
Funder addr 0x8943545177806ED17B9F23F0a21ee5948eCaa776
On-chain nonce: 100
Submitting 100 txs to http://127.0.0.1:32894 in parallel
Submitted in 0.39s
Accepted: 67 Rejected: 33
Rejection reasons:
33 {'code': -32000, 'message': "INTERNAL_ERROR: nonce gap too large..."}
```
Matches the production failure logs exactly.
End-to-end validation
2-pair Kurtosis devnet (reth+lighthouse, erigon+lighthouse), running the same
eoa-transactions-testplaybook the CI uses, with the patched image:latest-releases-7)eoa-transactions-testoutcomenonce gap too largerejectionstxs > 0[txpool] statpending=0 queued=44pending=0 queued=0Both Erigon-specific sub-tests now pass:
Wait for block proposal with >= 5 transactions from 2-erigon-lighthouse✅Check if legacy EOA transactions can be sent via 2-erigon-lighthouse✅Check if dynfee EOA transactions can be sent via 2-erigon-lighthouse✅Test plan
go build ./...cleaneoa-transactions-testpasses end-to-end (35/35 sub-tasks).nonce gap too largerejections over the full run.ethpandaops/assertoor-test'slatest-releases-*matrix once the assertoor image is rebuilt.Follow-ups (not blocking, separate issues)
MaxNonceGapeviction in theaddTxns(RPC-submit) path is over-aggressive even outside this test pattern: a legitimate burst of out-of-order parallel consecutive-nonce submissions will still trip it. The eviction makes sense on real on-chain state changes (addTxnsOnNewBlock) but should be deferred while a burst is still arriving. Worth a separate upstream PR.execution_payload_availablebeacon-event topic — assertoor's subscription returnsBAD_REQUEST: unable to parse queryevery 10s. Non-blocking warning, but noisy.