fix(replication): stop the first-audit scheduler from targeting the local peer#182
fix(replication): stop the first-audit scheduler from targeting the local peer#182grumbach wants to merge 2 commits into
Conversation
The payment verifier walks every quote in a verified payment, and the local node's own quote is among them, so each verified payment emitted a monetized-pin event targeting the local peer. The first-audit scheduler launched these audits without a self check, so the node dialed itself, which fails instantly (no dialable address for the local peer) and was miscounted as a subtree audit timeout. On 400-node testnets these self-targets were ~8% of all first-monetized audit starts. Drop self-targets at queue ingress: never queued, never launched, never marked first-audited, counted under a new self_target_skipped stat in the scheduler summary line. The pin still receives its deterministic first audit from the payment's other payees, which schedule their own audits of this node.
There was a problem hiding this comment.
Pull request overview
This PR fixes first-audit scheduling in the replication engine by preventing “first monetized” audit events that target the local node from being queued and launched, which previously caused instant dial failures (“Peer not found”) and inflated subtree-audit timeout metrics. The fix is implemented at the single queue ingress point and adds observability to confirm the filter is active during testnet runs.
Changes:
- Add a self-target filter to
queue_first_audit_eventso events targeting the local peer are dropped before entering the pending LRU queue. - Introduce and report a
self_target_skippedcounter in the first-audit scheduler’s periodic summary log line. - Add a unit test verifying self-target events are dropped while remote-peer events still queue normally, and update the existing queue/coalescing test for the new parameter.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Add a test-utils-gated first_audit_stats() snapshot of the scheduler counters (previously created inside the drainer task and unobservable from tests), and two e2e tests against a running multi-node testnet: - a self-targeting monetized-pin event injected into the live drainer is ingested and dropped (received=1, self_target_skipped=1, queued=0, launched=0); - a remote peer's real commitment pin passes the filter and completes a passing live-wire subtree audit (queued=1, launched=1, passed=1). With the guard ablated the self event reproduces the production bug exactly (queued=1, launched=1, timed_out=1: the node dials itself and the instant failure is miscounted as an audit timeout), so both the unit and e2e layers go red on a regression. Also soften the queue helper's comment: other payees' first audits of the pin are best-effort, not guaranteed.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (3)
tests/e2e/subtree_audit_testnet.rs:233
- These assertions use absolute counter values (
== 1/== 0). Since the drainer counters are cumulative for the lifetime of the node, the robust check is that only the expected counters changed relative to the pre-injection snapshot (and that queued/launched did not increase).
assert_eq!(stats.received, 1, "drainer must have ingested the event");
assert_eq!(
stats.self_target_skipped, 1,
"a self-targeting event must be counted as skipped, got {stats:?}"
);
assert_eq!(
stats.queued, 0,
"a self-targeting event must never enter the pending queue, got {stats:?}"
);
assert_eq!(
stats.launched, 0,
"the scheduler must never launch an audit against the local peer, got {stats:?}"
);
tests/e2e/subtree_audit_testnet.rs:273
- The wait condition (
passed >= 1 || timed_out >= 1 || failed >= 1) can be satisfied by prior activity since these counters are cumulative, causing the poller to return immediately and making the subsequent assertions flaky. Take a baseline snapshot before injection and wait for one of the terminal outcome counters to increase relative to it.
b_engine
.monetized_pin_sender()
.send(MonetizedPinEvent {
peer: a_peer,
pin: a_pin,
tests/e2e/subtree_audit_testnet.rs:299
- Like the self-target test, these assertions assume the counters start at 0 and will end at exactly 1. Since the counters are lifetime totals, assert on deltas from the baseline snapshot instead (at least +1 queued/launched/passed, and no increase in self-target skips).
assert_eq!(
stats.self_target_skipped, 0,
"a remote event must not be misfiltered, got {stats:?}"
);
assert_eq!(
stats.queued, 1,
"remote event must be queued, got {stats:?}"
);
assert_eq!(
stats.launched, 1,
"remote event must launch an audit, got {stats:?}"
);
assert_eq!(
stats.passed, 1,
"the live-wire first audit of an honest holder must pass, got {stats:?}"
);
| let before = engine.first_audit_stats(); | ||
| assert_eq!(before.received, 0, "no events expected before injection"); | ||
|
|
||
| engine | ||
| .monetized_pin_sender() | ||
| .send(MonetizedPinEvent { | ||
| peer: self_peer, | ||
| pin: [0x42; 32], | ||
| key_count: 8, | ||
| quote_ts: SystemTime::now(), | ||
| }) | ||
| .expect("drainer alive"); | ||
|
|
||
| let stats = wait_for_first_audit_stats(engine, Duration::from_secs(10), |s| { | ||
| s.received >= 1 && s.self_target_skipped >= 1 | ||
| }) | ||
| .await; |
Problem
On every payment a node verifies, the payment's quote list includes the node's own quote, so the verifier emits a monetized-pin first-audit event targeting the local peer. The first-audit scheduler launched these audits without a self check, so the node tried to send an audit challenge to itself. The dial fails instantly (
Peer not found, 0-1 ms, the local peer has no dialable address) and was counted as a subtree audit timeout.Reported by @mickvandijke from two independent 400-node testnet runs: 396/396 and 382/423
Peer not foundfirst-audit failures targeted the requester's own peer ID, about 8% of all first-monetized subtree audit starts, inflating the timeout metrics.Fix
Filter self-targets at the single ingress choke point (
queue_first_audit_event) that both receive arms of the drainer feed through:self_target_skippedcounter is reported in the periodic scheduler summary line, so a testnet run can confirm the filter is catching events (zero self-targeted outbound audit requests, non-zero skip counter) rather than the events silently not occurring.test-utils-gatedfirst_audit_stats()snapshot exposes the scheduler counters to e2e tests (they were previously created inside the drainer task and unobservable).Audit coverage of the node's own pin is not reduced: a self-dial can never verify anything, while the payment's other payees each run the same cross-check on the same quote list and schedule their own first audit of this node (best-effort, as before).
Evidence
Live-testnet e2e (multi-node in-process testnet, real QUIC wire, real LMDB, live drainer task):
first_audit_drainer_drops_self_targeting_monetized_pin: a self event injected into the running drainer is ingested and dropped:received: 1, self_target_skipped: 1, queued: 0, launched: 0.first_audit_drainer_launches_and_passes_remote_monetized_pin: a remote peer's real commitment pin passes the filter and completes a passing live-wire subtree audit:queued: 1, launched: 1, passed: 1.received: 1, queued: 1, launched: 1, timed_out: 1— the node dialed itself and the instantPeer not foundwas recorded as an audit timeout.An independent adversarial model review of the commit confirmed the filter is complete: the only production insertion into the pending queue is
queue_first_audit_event(both receive arms), the cooldown re-put only recycles already-filtered events, and every other audit path (gossip-triggered, possession checks, prune confirmation) already self-excludes. Two low-severity notes from that review are addressed in this PR (live-drainer test coverage; comment wording on other-payee coverage being best-effort).Not included
Remote-peer
Peer not foundresults are still classified as timeouts. Splitting those into a distinct unreachable outcome touches audit failure classification and is left for a follow-up.Semver: patch, internal behavior only (one new field in an internal summary log line; the stats accessor is test-utils-gated).