Skip to content

fix(replication): stop the first-audit scheduler from targeting the local peer#182

Open
grumbach wants to merge 2 commits into
WithAutonomi:mainfrom
grumbach:fix/first-audit-self-target
Open

fix(replication): stop the first-audit scheduler from targeting the local peer#182
grumbach wants to merge 2 commits into
WithAutonomi:mainfrom
grumbach:fix/first-audit-self-target

Conversation

@grumbach

@grumbach grumbach commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Problem

On every payment a node verifies, the payment's quote list includes the node's own quote, so the verifier emits a monetized-pin first-audit event targeting the local peer. The first-audit scheduler launched these audits without a self check, so the node tried to send an audit challenge to itself. The dial fails instantly (Peer not found, 0-1 ms, the local peer has no dialable address) and was counted as a subtree audit timeout.

Reported by @mickvandijke from two independent 400-node testnet runs: 396/396 and 382/423 Peer not found first-audit failures targeted the requester's own peer ID, about 8% of all first-monetized subtree audit starts, inflating the timeout metrics.

Fix

Filter self-targets at the single ingress choke point (queue_first_audit_event) that both receive arms of the drainer feed through:

  • A self-targeting event is never queued, never launched, and never marks its pin first-audited.
  • A new self_target_skipped counter is reported in the periodic scheduler summary line, so a testnet run can confirm the filter is catching events (zero self-targeted outbound audit requests, non-zero skip counter) rather than the events silently not occurring.
  • A test-utils-gated first_audit_stats() snapshot exposes the scheduler counters to e2e tests (they were previously created inside the drainer task and unobservable).

Audit coverage of the node's own pin is not reduced: a self-dial can never verify anything, while the payment's other payees each run the same cross-check on the same quote list and schedule their own first audit of this node (best-effort, as before).

Evidence

Live-testnet e2e (multi-node in-process testnet, real QUIC wire, real LMDB, live drainer task):

  • first_audit_drainer_drops_self_targeting_monetized_pin: a self event injected into the running drainer is ingested and dropped: received: 1, self_target_skipped: 1, queued: 0, launched: 0.
  • first_audit_drainer_launches_and_passes_remote_monetized_pin: a remote peer's real commitment pin passes the filter and completes a passing live-wire subtree audit: queued: 1, launched: 1, passed: 1.
  • Ablation (guard temporarily disabled, not committed): both test layers go red, and the e2e test reproduces the production bug exactly: received: 1, queued: 1, launched: 1, timed_out: 1 — the node dialed itself and the instant Peer not found was recorded as an audit timeout.
  • Unit tests cover the filter outcome directly (self dropped + pending untouched, remote unaffected, coalescing/eviction semantics unchanged).

An independent adversarial model review of the commit confirmed the filter is complete: the only production insertion into the pending queue is queue_first_audit_event (both receive arms), the cooldown re-put only recycles already-filtered events, and every other audit path (gossip-triggered, possession checks, prune confirmation) already self-excludes. Two low-severity notes from that review are addressed in this PR (live-drainer test coverage; comment wording on other-payee coverage being best-effort).

Not included

Remote-peer Peer not found results are still classified as timeouts. Splitting those into a distinct unreachable outcome touches audit failure classification and is left for a follow-up.

Semver: patch, internal behavior only (one new field in an internal summary log line; the stats accessor is test-utils-gated).

The payment verifier walks every quote in a verified payment, and the
local node's own quote is among them, so each verified payment emitted a
monetized-pin event targeting the local peer. The first-audit scheduler
launched these audits without a self check, so the node dialed itself,
which fails instantly (no dialable address for the local peer) and was
miscounted as a subtree audit timeout. On 400-node testnets these
self-targets were ~8% of all first-monetized audit starts.

Drop self-targets at queue ingress: never queued, never launched, never
marked first-audited, counted under a new self_target_skipped stat in
the scheduler summary line. The pin still receives its deterministic
first audit from the payment's other payees, which schedule their own
audits of this node.
Copilot AI review requested due to automatic review settings July 23, 2026 03:52

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes first-audit scheduling in the replication engine by preventing “first monetized” audit events that target the local node from being queued and launched, which previously caused instant dial failures (“Peer not found”) and inflated subtree-audit timeout metrics. The fix is implemented at the single queue ingress point and adds observability to confirm the filter is active during testnet runs.

Changes:

  • Add a self-target filter to queue_first_audit_event so events targeting the local peer are dropped before entering the pending LRU queue.
  • Introduce and report a self_target_skipped counter in the first-audit scheduler’s periodic summary log line.
  • Add a unit test verifying self-target events are dropped while remote-peer events still queue normally, and update the existing queue/coalescing test for the new parameter.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Add a test-utils-gated first_audit_stats() snapshot of the scheduler
counters (previously created inside the drainer task and unobservable
from tests), and two e2e tests against a running multi-node testnet:

- a self-targeting monetized-pin event injected into the live drainer is
  ingested and dropped (received=1, self_target_skipped=1, queued=0,
  launched=0);
- a remote peer's real commitment pin passes the filter and completes a
  passing live-wire subtree audit (queued=1, launched=1, passed=1).

With the guard ablated the self event reproduces the production bug
exactly (queued=1, launched=1, timed_out=1: the node dials itself and
the instant failure is miscounted as an audit timeout), so both the unit
and e2e layers go red on a regression.

Also soften the queue helper's comment: other payees' first audits of
the pin are best-effort, not guaranteed.
Copilot AI review requested due to automatic review settings July 23, 2026 04:18

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (3)

tests/e2e/subtree_audit_testnet.rs:233

  • These assertions use absolute counter values (== 1 / == 0). Since the drainer counters are cumulative for the lifetime of the node, the robust check is that only the expected counters changed relative to the pre-injection snapshot (and that queued/launched did not increase).
    assert_eq!(stats.received, 1, "drainer must have ingested the event");
    assert_eq!(
        stats.self_target_skipped, 1,
        "a self-targeting event must be counted as skipped, got {stats:?}"
    );
    assert_eq!(
        stats.queued, 0,
        "a self-targeting event must never enter the pending queue, got {stats:?}"
    );
    assert_eq!(
        stats.launched, 0,
        "the scheduler must never launch an audit against the local peer, got {stats:?}"
    );

tests/e2e/subtree_audit_testnet.rs:273

  • The wait condition (passed >= 1 || timed_out >= 1 || failed >= 1) can be satisfied by prior activity since these counters are cumulative, causing the poller to return immediately and making the subsequent assertions flaky. Take a baseline snapshot before injection and wait for one of the terminal outcome counters to increase relative to it.
    b_engine
        .monetized_pin_sender()
        .send(MonetizedPinEvent {
            peer: a_peer,
            pin: a_pin,

tests/e2e/subtree_audit_testnet.rs:299

  • Like the self-target test, these assertions assume the counters start at 0 and will end at exactly 1. Since the counters are lifetime totals, assert on deltas from the baseline snapshot instead (at least +1 queued/launched/passed, and no increase in self-target skips).
    assert_eq!(
        stats.self_target_skipped, 0,
        "a remote event must not be misfiltered, got {stats:?}"
    );
    assert_eq!(
        stats.queued, 1,
        "remote event must be queued, got {stats:?}"
    );
    assert_eq!(
        stats.launched, 1,
        "remote event must launch an audit, got {stats:?}"
    );
    assert_eq!(
        stats.passed, 1,
        "the live-wire first audit of an honest holder must pass, got {stats:?}"
    );

Comment on lines +203 to +219
let before = engine.first_audit_stats();
assert_eq!(before.received, 0, "no events expected before injection");

engine
.monetized_pin_sender()
.send(MonetizedPinEvent {
peer: self_peer,
pin: [0x42; 32],
key_count: 8,
quote_ts: SystemTime::now(),
})
.expect("drainer alive");

let stats = wait_for_first_audit_stats(engine, Duration::from_secs(10), |s| {
s.received >= 1 && s.self_target_skipped >= 1
})
.await;
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants