From 969fb797a35d13f36e95995fd1a1e06f1e4659b1 Mon Sep 17 00:00:00 2001 From: grumbach Date: Tue, 14 Jul 2026 11:47:50 +0900 Subject: [PATCH 01/21] fix(replication)!: bound monetized first-audit scheduling (ADR-0004 Amendment 2) Production v0.14.3 showed the deterministic first-audit path amplifying fleet-wide: launches scaled as uploads x pinned-quotes-per-proof x verifying-storers, with hourly pin rotation defeating dedup and no aggregate launch bound. The overflow tripped responder admission caps and was recorded as peer Timeouts (545k/24h), degrading downloads ~31%. Payments now NOMINATE pins; the clock LAUNCHES them: - verifier: nominate only the settlement-verified paid candidate (single-node median issuer; merkle paid_node_addresses indices), not every quote in the bundle / candidate in the pool - drainer: launch limiter with a token bucket (burst 2, one launch per 5 min), an in-flight cap (2), and 0-30s launch jitter so one chunk's storers do not challenge the paid peer simultaneously - per-peer re-audit window (2h) that survives pin rotation, with a >1.5x committed-count jump override so inflated sidecar-only pins are still re-nominated immediately - budget deferrals are penalty-free and consume nothing; newest-monetized pins get budget first - scheduler summary gains rate_deferred_attempts / window_deduped / tokens; outcome labels stay append-only vs #173 - ADR-0004 Amendment 2 documents first audits as bounded best-effort sampling and restates the coverage guarantee - tests/e2e/first_audit_ab.rs: env-gated production-shaped A/B workload driver (real Anvil settlement, quote sidecars, median payment, close-group PUTs, periodic rotation) for baseline-vs-fix comparisons --- ...ADR-0004-commitment-bound-quote-pricing.md | 61 ++ src/payment/verifier.rs | 211 ++++++- src/replication/config.rs | 67 +++ src/replication/mod.rs | 426 +++++++++++++- tests/e2e/first_audit_ab.rs | 537 ++++++++++++++++++ tests/e2e/mod.rs | 3 + 6 files changed, 1266 insertions(+), 39 deletions(-) create mode 100644 tests/e2e/first_audit_ab.rs diff --git a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md index 97411d16..138d8fc4 100644 --- a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md +++ b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md @@ -334,3 +334,64 @@ attributing malicious silence/transient conditions network-wide is the (out-of-scope) distributed non-response problem. This supersedes the "unanswerable quoted pin is graced, never confirmed" rule and deletes `RejectKind::is_graced`; the regression tests are updated accordingly. + +--- + +## Amendment 2 (2026-07-14): first audits are bounded best-effort sampling — payments nominate, the clock launches + +The original decision made the monetized first audit deterministic: every pinned +quote in every verified client-put proof entered the first-audit queue at every +verifying storer, gated only by the per-peer 30-minute cooldown. That per-peer +gate bounds one observer against one peer but places no bound on the aggregate: +fleet-wide launch pressure scaled as +`uploads x pinned-quotes-per-proof x verifying-storers`, with hourly commitment +rotation re-arming pin-level dedup every rotation. In the v0.14.3 production +rollout this amplification saturated the audit-responder admission pools; +overflow challenges were dropped and recorded by auditors as Timeout failures +(545k Timeout errors/24h fleet-wide, ~31% slower downloads). A matched staging +ablation attributed 97.7% of storage-commitment audit traffic and 99.955% of +one-key timeout failures to this path, and a per-service concurrency cap alone +(without a launch-rate budget) reproduced the storm. Amendment 1's answerability +guarantees are unaffected; this amendment changes only who is nominated and how +fast nominations launch. + +**Nomination narrows to paid pins.** Only the candidate whose on-chain +settlement actually verified is nominated: the single-node path nominates the +settlement-verified median candidate (not the whole bundle), and the merkle +path nominates the contract-paid indices (not the whole pool). Unpaid quotes +earned nothing; their gossiped commitments remain under the ADR-0002 lottery. +The storer-side cross-check (arithmetic re-check and mismatch evidence) still +runs on every quote and candidate. + +**Launches are budgeted, not deterministic.** Each node's first-audit drainer +runs a launch limiter: a token bucket (burst 2, one token per 5 minutes — a +hard per-node ceiling of 12 launches/hour), an in-flight cap (2), and a uniform +0-30s launch jitter so the storers of one chunk, which all verify the same +payment at the same instant, do not challenge the paid peer simultaneously. +Fleet-wide first-audit pressure is therefore `nodes x refill-rate`, independent +of upload volume. Budget deferral is penalty-free: a deferred pin stays pending +(newest-per-peer, bounded) and launches when tokens allow; only audits that +actually launch have consequences. + +**Per-peer re-audit window survives pin rotation.** After a first audit +launches at a peer, further nominations for that peer are dropped for 2 hours +(inside the 3h answerability TTL) — unless the new pin's committed key count +exceeds the audited one by more than 1.5x, which re-nominates immediately. The +jump override preserves the anti-inflation property this queue exists for: an +inflated commitment delivered only as a quote sidecar is visible to payment +verifiers alone, so no gossip-lottery audit can ever select it; a peer that +passes an audit on an honest count and then mints a much larger sidecar-only +commitment is re-audited at once. Ordinary rotations with a stable count stop +re-arming the fleet. + +**Coverage restated.** "The latest commitment earning money for a peer always +faces an audit soon" becomes: the FIRST commitment earning money for a peer, +and any later commitment whose claimed count materially jumps, faces an audit +soon (minutes, from multiple independent storers, each within its own budget); +commitments rotated without material count change rely on the ADR-0002 lottery +for re-audit. First-audit coverage is best-effort supplementary sampling under +an explicit load budget, not a per-payment guarantee. The scheduler's funnel +(received / queued / coalesced / duplicates / window_deduped / rate_deferred / +cooldown_deferred / launched / terminal outcomes, plus tokens and in-flight +gauges) is exported in the periodic scheduler summary so this coverage is +measurable in production. diff --git a/src/payment/verifier.rs b/src/payment/verifier.rs index 41aa5e33..7e116b0e 100644 --- a/src/payment/verifier.rs +++ b/src/payment/verifier.rs @@ -975,15 +975,23 @@ impl PaymentVerifier { // The paid (median) price and settled on-chain amount of the winning // candidate, kept for the receiver-side price-floor policy below. let mut verified_paid_quote: Option<(Amount, Amount)> = None; + // ADR-0004 Amendment 2: remember WHICH candidate's on-chain settlement + // verified — only that peer's commitment actually earned money, so only + // its pin is nominated for a deterministic first audit below. The + // non-median quotes merely locate the median and stay covered by the + // gossip-lottery audit path. + let mut paid_peer: Option<[u8; 32]> = None; for candidate in candidates { let paid_price = candidate.quote.price; + let candidate_peer = *candidate.encoded_peer_id.as_bytes(); match self .verify_legacy_median_candidate(xorname, candidate) .await { Ok(settled_amount) => { verified_paid_quote = Some((paid_price, settled_amount)); + paid_peer = Some(candidate_peer); break; } Err(err) => failures.push(err.to_string()), @@ -1022,7 +1030,8 @@ impl PaymentVerifier { // bundle can never enqueue audits or drive pin fetches — closing the // free-amplification path. if context.is_store_admission() { - self.cross_check_quotes(payment, commitment_sidecars).await; + self.cross_check_quotes(payment, commitment_sidecars, paid_peer) + .await; } if crate::logging::enabled!(crate::logging::Level::INFO) { @@ -1635,7 +1644,12 @@ impl PaymentVerifier { /// missing cache or absent `P2PNode` degrades to "resolve nothing", never an /// error on the payment path — the synchronous arithmetic gate and the /// later audit remain the load-bearing checks. - async fn cross_check_quotes(&self, payment: &ProofOfPayment, commitment_sidecars: &[Vec]) { + async fn cross_check_quotes( + &self, + payment: &ProofOfPayment, + commitment_sidecars: &[Vec], + paid_peer: Option<[u8; 32]>, + ) { let now = std::time::Instant::now(); let ttl = crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL; let p2p = self.p2p_node.read().as_ref().map(Arc::clone); @@ -1659,17 +1673,24 @@ impl PaymentVerifier { }; let peer_id = PeerId::from_bytes(*encoded_peer_id.as_bytes()); - // ADR-0004: this commitment backed a payment — route it for a - // deterministic first audit (the drainer dedups by pin and respects - // the cooldown). Best-effort: a closed channel just means no first - // audit is scheduled, never an error on the payment path. - if let Some(ref tx) = monetized_pin_tx { - let _ = tx.send(crate::replication::MonetizedPinEvent { - peer: peer_id, - pin, - key_count: quote.committed_key_count, - quote_ts: quote.timestamp, - }); + // ADR-0004 Amendment 2: only the PAID candidate's commitment earned + // money, so only its pin is routed for a deterministic first audit + // (the drainer dedups, rate-budgets, and respects the cooldown). + // Pre-amendment this fired for every pinned quote in the bundle — + // up to CLOSE_GROUP_SIZE audits per proof for peers that were never + // paid — a 7x term in the fleet-wide amplification that degraded + // v0.14.3. Best-effort: a closed channel just means no first audit + // is scheduled, never an error on the payment path. + let is_paid = paid_peer.is_some_and(|paid| paid == *encoded_peer_id.as_bytes()); + if is_paid { + if let Some(ref tx) = monetized_pin_tx { + let _ = tx.send(crate::replication::MonetizedPinEvent { + peer: peer_id, + pin, + key_count: quote.committed_key_count, + quote_ts: quote.timestamp, + }); + } } // Resolution order: sidecar (synchronous, no state) -> gossip cache // (fresh within TTL) -> fetch fallback (collected as unresolved). @@ -2730,10 +2751,19 @@ impl PaymentVerifier { // fetches). Store admissions only (direct PUT + immediate fresh // replication, the paths that previously verified under `ClientPut`) — // a paid-list receipt's pins have aged out. + // Amendment 2: only the candidates the contract actually PAID + // (`paid_node_addresses` indices, verified above) are nominated for + // first audits; the rest of the pool merely established the median. if context.is_store_admission() { + let paid_indices: std::collections::HashSet = payment_info + .paid_node_addresses + .iter() + .map(|(_, idx, _)| *idx) + .collect(); self.cross_check_merkle_candidates( &merkle_proof.winner_pool, &merkle_proof.commitment_sidecars, + &paid_indices, ) .await; } @@ -2752,6 +2782,7 @@ impl PaymentVerifier { &self, pool: &evmlib::merkle_payments::MerklePaymentCandidatePool, commitment_sidecars: &[Vec], + paid_indices: &std::collections::HashSet, ) { let now = std::time::Instant::now(); let ttl = crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL; @@ -2763,23 +2794,29 @@ impl PaymentVerifier { let sidecar_map = Self::index_valid_sidecars(commitment_sidecars); let mut unresolved: Vec<(PeerId, [u8; 32], u32, Vec)> = Vec::new(); - for candidate in &pool.candidate_nodes { + for (idx, candidate) in pool.candidate_nodes.iter().enumerate() { let Some(pin) = candidate.commitment_pin else { continue; // baseline candidate pins nothing }; let peer_id = PeerId::from_bytes(*blake3::hash(&candidate.pub_key).as_bytes()); - if let Some(ref tx) = monetized_pin_tx { - let _ = tx.send(crate::replication::MonetizedPinEvent { - peer: peer_id, - pin, - key_count: candidate.committed_key_count, - quote_ts: std::time::UNIX_EPOCH - .checked_add(std::time::Duration::from_secs( - candidate.merkle_payment_timestamp, - )) - .unwrap_or(std::time::UNIX_EPOCH), - }); + // ADR-0004 Amendment 2: nominate only the candidates the contract + // actually paid — the unpaid pool members earned nothing and stay + // covered by the gossip-lottery audit path. Pre-amendment every + // pool candidate (16) was nominated per verified proof. + if paid_indices.contains(&idx) { + if let Some(ref tx) = monetized_pin_tx { + let _ = tx.send(crate::replication::MonetizedPinEvent { + peer: peer_id, + pin, + key_count: candidate.committed_key_count, + quote_ts: std::time::UNIX_EPOCH + .checked_add(std::time::Duration::from_secs( + candidate.merkle_payment_timestamp, + )) + .unwrap_or(std::time::UNIX_EPOCH), + }); + } } let resolved = match sidecar_map.get(&(peer_id, pin)) { @@ -4714,6 +4751,130 @@ mod tests { (merkle_proof, pool_hash, xorname, timestamp) } + /// ADR-0004 Amendment 2: only the PAID (settlement-verified) quote's pin is + /// nominated for a deterministic first audit; the unpaid bundle quotes are + /// cross-checked but never enqueue audits. + #[tokio::test] + async fn adr0004_first_audit_nominates_only_paid_single_node_quote() { + use evmlib::{EncodedPeerId, RewardsAddress}; + + let verifier = create_test_verifier(); + let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel(); + verifier.attach_monetized_pin_sender(tx); + + let ids: Vec<[u8; 32]> = (1..=3u8).map(|b| [b; 32]).collect(); + let payment = ProofOfPayment { + peer_quotes: ids + .iter() + .enumerate() + .map(|(i, id)| { + let pin_byte = u8::try_from(i).expect("small bundle") + 1; + let mut quote = make_fake_quote( + [0xD0; 32], + SystemTime::now(), + RewardsAddress::new([pin_byte; 20]), + ); + quote.commitment_pin = Some([pin_byte; 32]); + quote.committed_key_count = 100; + (EncodedPeerId::new(*id), quote) + }) + .collect(), + }; + + // The middle quote is the candidate whose on-chain settlement verified. + let paid = ids.get(1).copied().expect("paid id"); + verifier.cross_check_quotes(&payment, &[], Some(paid)).await; + + let event = rx.try_recv().expect("the paid quote must be nominated"); + assert_eq!(event.peer, PeerId::from_bytes(paid)); + assert_eq!(event.pin, [2u8; 32]); + assert!( + rx.try_recv().is_err(), + "unpaid bundle quotes must not be nominated for first audits" + ); + } + + /// ADR-0004 Amendment 2: with no verified paid candidate, nothing is + /// nominated (defensive: the caller never reaches the cross-check on a + /// failed verification, but the emission itself must also fail closed). + #[tokio::test] + async fn adr0004_first_audit_nominates_nothing_without_paid_peer() { + use evmlib::{EncodedPeerId, RewardsAddress}; + + let verifier = create_test_verifier(); + let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel(); + verifier.attach_monetized_pin_sender(tx); + + let mut quote = + make_fake_quote([0xD1; 32], SystemTime::now(), RewardsAddress::new([1; 20])); + quote.commitment_pin = Some([1u8; 32]); + let payment = ProofOfPayment { + peer_quotes: vec![(EncodedPeerId::new([1u8; 32]), quote)], + }; + + verifier.cross_check_quotes(&payment, &[], None).await; + assert!(rx.try_recv().is_err(), "no paid peer, no nomination"); + } + + /// ADR-0004 Amendment 2, merkle path: only the candidates at the + /// contract-verified PAID indices are nominated; the rest of the pool + /// merely established the median and must not enqueue audits. + #[tokio::test] + async fn adr0004_first_audit_nominates_only_paid_merkle_candidates() { + use evmlib::merkle_payments::{MerklePaymentCandidatePool, MerkleTree}; + + let verifier = create_test_verifier(); + let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel(); + verifier.attach_monetized_pin_sender(tx); + + let timestamp = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .expect("system time") + .as_secs(); + let mut candidate_nodes = make_candidate_nodes(timestamp); + for (i, candidate) in candidate_nodes.iter_mut().enumerate() { + candidate.commitment_pin = Some([u8::try_from(i & 0xFF).expect("byte"); 32]); + candidate.committed_key_count = 50; + } + let addresses: Vec = (0..4u8) + .map(|i| xor_name::XorName::from_content(&[i])) + .collect(); + let tree = MerkleTree::from_xornames(addresses).expect("tree"); + let midpoint_proof = tree + .reward_candidates(timestamp) + .expect("reward candidates") + .first() + .expect("at least one candidate") + .clone(); + let pool = MerklePaymentCandidatePool { + midpoint_proof, + candidate_nodes, + }; + + let paid_indices: std::collections::HashSet = [1usize, 3].into_iter().collect(); + verifier + .cross_check_merkle_candidates(&pool, &[], &paid_indices) + .await; + + let mut nominated = Vec::new(); + while let Ok(event) = rx.try_recv() { + nominated.push(event); + } + assert_eq!( + nominated.len(), + 2, + "only the PAID candidates are nominated, not the whole pool" + ); + for (event, idx) in nominated.iter().zip([1usize, 3]) { + let candidate = pool.candidate_nodes.get(idx).expect("paid candidate"); + assert_eq!( + event.peer, + PeerId::from_bytes(*blake3::hash(&candidate.pub_key).as_bytes()) + ); + assert_eq!(event.pin, [u8::try_from(idx).expect("byte"); 32]); + } + } + /// Helper: build a minimal valid `MerklePaymentProof` with real ML-DSA-65 /// signatures. Returns `(xorname, serialized_tagged_proof, pool_hash, timestamp)`. fn make_valid_merkle_proof_bytes() -> ( diff --git a/src/replication/config.rs b/src/replication/config.rs index ed42f470..815964c6 100644 --- a/src/replication/config.rs +++ b/src/replication/config.rs @@ -404,6 +404,73 @@ pub const FIRST_AUDIT_SUMMARY_INTERVAL: Duration = Duration::from_secs(5 * 60); /// events the drainer processes audits, then loops back to drain more. pub const FIRST_AUDIT_DRAIN_BATCH: usize = 64; +/// ADR-0004 Amendment 2: token-bucket refill interval for monetized first-audit +/// launches — one launch token per interval, per node. +/// +/// This is the load-bearing bound: fleet-wide first-audit pressure becomes +/// `nodes x (1 / interval)` regardless of upload volume, where the pre-amendment +/// scheduler scaled with `uploads x pinned-quotes-per-proof x verifying-storers` +/// (measured at ~110 storage-commitment audit events/hour/service in the +/// DEV-01 staging run, with 77.7% of launches timing out in the DEV-03 +/// per-service-concurrency attempt). At 5 minutes the hard ceiling is +/// 12 launches/hour/node; steady-state demand sits far below it because +/// nomination is paid-pin-only and re-nominations are suppressed by +/// [`FIRST_AUDIT_PEER_REAUDIT_INTERVAL`]. +pub const FIRST_AUDIT_LAUNCH_INTERVAL: Duration = Duration::from_secs(5 * 60); + +/// ADR-0004 Amendment 2: token-bucket capacity for monetized first-audit +/// launches. +/// +/// Lets a node absorb a small burst of NEW earning peers (several distinct +/// paid pins verified back-to-back) without waiting a full refill interval per +/// launch, while keeping the sustained rate pinned to +/// [`FIRST_AUDIT_LAUNCH_INTERVAL`]. +pub const FIRST_AUDIT_BUDGET_BURST: u32 = 2; + +/// ADR-0004 Amendment 2: max concurrently in-flight monetized first audits per +/// node. +/// +/// A first audit can hold a slot for the full size-scaled two-round deadline, +/// so without this cap a burst of slow/timing-out targets would pile up +/// concurrent subtree audits even under the launch-rate budget. Deferral is +/// penalty-free: the pin stays pending and is retried when a slot frees. +pub const FIRST_AUDIT_MAX_INFLIGHT: u64 = 2; + +/// ADR-0004 Amendment 2: max uniform random delay applied before sending a +/// monetized first-audit challenge. +/// +/// Every storer of a chunk verifies the same payment at the same instant, so +/// unjittered first audits from the whole close group would hit the paid peer +/// simultaneously and trip its per-peer responder admission cap (drops that +/// auditors then record as Timeout). The jitter decorrelates the observers. +pub const FIRST_AUDIT_LAUNCH_JITTER_MAX: Duration = Duration::from_secs(30); + +/// ADR-0004 Amendment 2: per-peer suppression window between monetized first +/// audits, across commitment rotations. +/// +/// Pin-level dedup alone is defeated by the hourly commitment rotation (every +/// rotation mints a fresh pin, re-arming every observer). After a first audit +/// LAUNCHES at a peer, further monetized nominations for that peer are dropped +/// for this window — unless the new pin's committed key count exceeds the +/// audited one by more than [`FIRST_AUDIT_COUNT_JUMP_NUM`]/ +/// [`FIRST_AUDIT_COUNT_JUMP_DEN`] (a peer that passes an audit on an honest +/// count and then mints a much larger commitment must be re-nominated +/// immediately: inflated SIDECAR-ONLY pins are visible to payment verifiers +/// only, so no gossip-lottery audit can ever cover them). Kept comfortably +/// inside the 3h answerability TTL so a re-nomination after the window still +/// lands in-window. Gossip-lottery re-audits are unaffected. +pub const FIRST_AUDIT_PEER_REAUDIT_INTERVAL: Duration = Duration::from_secs(2 * 3600); + +/// ADR-0004 Amendment 2: committed-count jump that overrides the per-peer +/// re-audit window, as a ratio (`new > old * NUM / DEN`, integer math). +/// +/// 3/2: a >1.5x growth in claimed committed keys within the suppression window +/// re-nominates the peer despite a recent first audit. +pub const FIRST_AUDIT_COUNT_JUMP_NUM: u64 = 3; +/// Denominator of the count-jump override ratio. See +/// [`FIRST_AUDIT_COUNT_JUMP_NUM`]. +pub const FIRST_AUDIT_COUNT_JUMP_DEN: u64 = 2; + /// Number of subtree leaves spot-checked against real chunk bytes per audit /// (ADR-0002 real-bytes layer). /// diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 89badb4f..fad0f62a 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -89,6 +89,8 @@ struct FirstAuditObservability { duplicates: AtomicU64, capacity_evicted: AtomicU64, cooldown_deferred_attempts: AtomicU64, + rate_deferred_attempts: AtomicU64, + window_deduped: AtomicU64, launched: AtomicU64, passed: AtomicU64, timed_out: AtomicU64, @@ -166,6 +168,148 @@ fn queue_first_audit_event( } } +/// A first audit the limiter recently launched at a peer: when, and the +/// committed key count that was audited (for the count-jump override). +#[derive(Debug, Clone, Copy)] +struct RecentFirstAudit { + launched_at: Instant, + key_count: u32, +} + +/// ADR-0004 Amendment 2: whether `new_count` exceeds `audited_count` by more +/// than the [`config::FIRST_AUDIT_COUNT_JUMP_NUM`]/ +/// [`config::FIRST_AUDIT_COUNT_JUMP_DEN`] ratio (`new > old * NUM / DEN`, +/// overflow-free integer math). A jump re-nominates a peer despite a recent +/// first audit: an inflated SIDECAR-ONLY pin is visible to payment verifiers +/// only, so no gossip-lottery audit can ever cover it. +const fn first_audit_count_jump(audited_count: u32, new_count: u32) -> bool { + (new_count as u64) * config::FIRST_AUDIT_COUNT_JUMP_DEN + > (audited_count as u64) * config::FIRST_AUDIT_COUNT_JUMP_NUM +} + +/// The launch limiter's verdict for one pending monetized pin. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum LimiterVerdict { + /// Within budget and window: the caller may launch (and MUST call + /// [`FirstAuditLimiter::commit_launch`] if it actually does). + Admit, + /// Launch-rate budget or in-flight cap exhausted. Penalty-free: keep the + /// pin pending and retry on a later tick. + RateDeferred, + /// The peer had a first audit within + /// [`config::FIRST_AUDIT_PEER_REAUDIT_INTERVAL`] and the new pin's count + /// shows no [`first_audit_count_jump`]. Drop the nomination (the + /// gossip-lottery re-audit path is unaffected). + WindowDeduped, +} + +/// ADR-0004 Amendment 2: the first-audit launch limiter — a token bucket +/// (launch rate), an in-flight cap, and a per-peer re-audit window that +/// survives pin rotation. +/// +/// This is the load-bearing aggregate bound the original scheduler lacked: +/// fleet-wide first-audit pressure becomes `nodes x refill-rate` instead of +/// `uploads x pinned-quotes-per-proof x verifying-storers`. Pure over passed +/// `now`/`inflight` values so every decision is unit-testable without a +/// runtime. +struct FirstAuditLimiter { + /// Launch tokens available, at most [`config::FIRST_AUDIT_BUDGET_BURST`]. + tokens: u32, + /// Refill anchor. While the bucket is full this tracks `now` (a full + /// bucket accrues nothing); while below capacity it advances only by + /// whole refill intervals so fractional elapsed time is never lost. + last_refill: Instant, + /// Peers given a first audit recently, with the audited key count. + /// Bounded like the drainer's other per-peer maps. + recent: LruCache, +} + +impl FirstAuditLimiter { + fn new(now: Instant) -> Self { + Self { + tokens: config::FIRST_AUDIT_BUDGET_BURST, + last_refill: now, + recent: LruCache::new( + NonZeroUsize::new(MAX_LAST_COMMITMENT_BY_PEER).unwrap_or(NonZeroUsize::MIN), + ), + } + } + + /// Whether the per-peer re-audit window admits a nomination for `peer` + /// carrying `key_count` at `now`. Read-only (`peek`), so it is safe to + /// call at ENQUEUE time — suppressed nominations never occupy pending + /// slots — without disturbing LRU recency. + fn window_allows(&self, peer: &PeerId, key_count: u32, now: Instant) -> bool { + self.recent.peek(peer).map_or(true, |prev| { + now.saturating_duration_since(prev.launched_at) + >= config::FIRST_AUDIT_PEER_REAUDIT_INTERVAL + || first_audit_count_jump(prev.key_count, key_count) + }) + } + + /// Refill the token bucket for the time elapsed up to `now`. + fn refill(&mut self, now: Instant) { + if self.tokens >= config::FIRST_AUDIT_BUDGET_BURST { + // Full bucket accrues nothing; keep the anchor current so the + // next consumption starts its interval from here. + self.last_refill = now; + return; + } + let interval = config::FIRST_AUDIT_LAUNCH_INTERVAL; + let elapsed = now.saturating_duration_since(self.last_refill); + let earned = elapsed.as_nanos() / interval.as_nanos().max(1); + if earned == 0 { + return; + } + let capacity_gap = u128::from(config::FIRST_AUDIT_BUDGET_BURST - self.tokens); + if earned >= capacity_gap { + self.tokens = config::FIRST_AUDIT_BUDGET_BURST; + self.last_refill = now; + } else { + // earned < capacity_gap <= u32::MAX, so both casts are lossless. + self.tokens = self + .tokens + .saturating_add(u32::try_from(earned).unwrap_or(u32::MAX)); + let advance = interval.saturating_mul(u32::try_from(earned).unwrap_or(u32::MAX)); + self.last_refill = self.last_refill.checked_add(advance).unwrap_or(now); + } + } + + /// Decide whether a pending pin may launch now. Consumes NOTHING: the + /// caller runs the remaining (per-peer cooldown) gate and calls + /// [`Self::commit_launch`] only for a launch that actually happens, so a + /// deferral elsewhere never burns budget or stamps the window. + fn assess( + &mut self, + peer: &PeerId, + key_count: u32, + now: Instant, + inflight: u64, + ) -> LimiterVerdict { + if !self.window_allows(peer, key_count, now) { + return LimiterVerdict::WindowDeduped; + } + self.refill(now); + if inflight >= config::FIRST_AUDIT_MAX_INFLIGHT || self.tokens == 0 { + return LimiterVerdict::RateDeferred; + } + LimiterVerdict::Admit + } + + /// Consume one token and stamp the per-peer window for a launch that is + /// actually happening. + fn commit_launch(&mut self, peer: PeerId, key_count: u32, now: Instant) { + self.tokens = self.tokens.saturating_sub(1); + self.recent.put( + peer, + RecentFirstAudit { + launched_at: now, + key_count, + }, + ); + } +} + // --------------------------------------------------------------------------- // Constants // --------------------------------------------------------------------------- @@ -933,8 +1077,13 @@ impl ReplicationEngine { let mut pending: LruCache = LruCache::new( NonZeroUsize::new(MAX_LAST_COMMITMENT_BY_PEER).unwrap_or(NonZeroUsize::MIN), ); - // Periodic retry tick for pending (cooldown-blocked) pins. Created - // once; `Skip` so a backlog of missed ticks collapses to one. + // ADR-0004 Amendment 2: the launch limiter (token bucket + + // in-flight cap + per-peer re-audit window). This is what makes + // first-audit load independent of upload volume: payments only + // NOMINATE pins, the clock launches them. + let mut limiter = FirstAuditLimiter::new(Instant::now()); + // Periodic retry tick for pending (cooldown/budget-blocked) pins. + // Created once; `Skip` so a backlog of missed ticks collapses to one. let mut tick = tokio::time::interval(config::FIRST_AUDIT_RETRY_INTERVAL); tick.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); let mut last_summary = Instant::now(); @@ -960,6 +1109,15 @@ impl ReplicationEngine { "First-audit scheduler: audit_trigger=first_monetized outcome=duplicate peer={} pin={} key_count={} pending={}", e.peer, hex::encode(e.pin), e.key_count, pending.len() ); + } else if !limiter.window_allows(&e.peer, e.key_count, Instant::now()) { + // Recently first-audited peer, no count jump: + // drop at the door so rotated pins never + // occupy pending slots or re-arm launches. + observability.window_deduped.fetch_add(1, Ordering::Relaxed); + debug!( + "First-audit scheduler: audit_trigger=first_monetized outcome=window_deduped peer={} pin={} key_count={} pending={}", + e.peer, hex::encode(e.pin), e.key_count, pending.len() + ); } else { match queue_first_audit_event(&mut pending, e) { FirstAuditQueueOutcome::Queued => { @@ -997,6 +1155,14 @@ impl ReplicationEngine { observability .duplicates .fetch_add(1, Ordering::Relaxed); + } else if !limiter.window_allows( + &e.peer, + e.key_count, + Instant::now(), + ) { + observability + .window_deduped + .fetch_add(1, Ordering::Relaxed); } else { match queue_first_audit_event(&mut pending, e) { FirstAuditQueueOutcome::Queued => { @@ -1034,13 +1200,15 @@ impl ReplicationEngine { if last_summary.elapsed() >= config::FIRST_AUDIT_SUMMARY_INTERVAL { info!( - "First-audit scheduler summary: audit_trigger=first_monetized received={} queued={} coalesced={} duplicates={} capacity_evicted={} cooldown_deferred_attempts={} launched={} passed={} timeout={} failed={} bootstrap_claims={} idle={} insufficient_keys={} outside_answerability_window={} pending={} inflight={}", + "First-audit scheduler summary: audit_trigger=first_monetized received={} queued={} coalesced={} duplicates={} capacity_evicted={} cooldown_deferred_attempts={} rate_deferred_attempts={} window_deduped={} launched={} passed={} timeout={} failed={} bootstrap_claims={} idle={} insufficient_keys={} outside_answerability_window={} pending={} inflight={} tokens={}", observability.received.load(Ordering::Relaxed), observability.queued.load(Ordering::Relaxed), observability.coalesced.load(Ordering::Relaxed), observability.duplicates.load(Ordering::Relaxed), observability.capacity_evicted.load(Ordering::Relaxed), observability.cooldown_deferred_attempts.load(Ordering::Relaxed), + observability.rate_deferred_attempts.load(Ordering::Relaxed), + observability.window_deduped.load(Ordering::Relaxed), observability.launched.load(Ordering::Relaxed), observability.passed.load(Ordering::Relaxed), observability.timed_out.load(Ordering::Relaxed), @@ -1051,6 +1219,7 @@ impl ReplicationEngine { observability.outside_answerability_window.load(Ordering::Relaxed), pending.len(), observability.inflight.load(Ordering::Relaxed), + limiter.tokens, ); last_summary = Instant::now(); } @@ -1060,15 +1229,19 @@ impl ReplicationEngine { } // Try to launch an audit for each pending peer; keep the ones - // still blocked by cooldown for the next tick. Drain into a vec - // first so we can re-insert the still-blocked ones afterwards - // (LruCache has no drain). `iter()` yields most- to least-recently- - // used; we reverse so re-inserting blocked entries below restores - // their relative recency (oldest re-put first → stays the eviction - // victim, newest stays most-recently-used). + // blocked by cooldown or budget for the next tick. Drain into a + // vec first (LruCache has no drain). `iter()` yields most- to + // least-recently-used, and with the Amendment 2 launch budget + // that order is load-bearing: the NEWEST-monetized peers must + // consume tokens first (freshest answerability windows, per the + // ADR's newest-first coverage rule), with older pins deferred. + // Deferred entries are collected and re-inserted afterwards in + // reverse order so relative recency is restored (oldest re-put + // first → stays the eviction victim). let snapshot: Vec<(PeerId, MonetizedPinEvent)> = - pending.iter().rev().map(|(p, e)| (*p, *e)).collect(); + pending.iter().map(|(p, e)| (*p, *e)).collect(); pending.clear(); + let mut deferred: Vec<(PeerId, MonetizedPinEvent)> = Vec::new(); for (peer, event) in snapshot { // Dedup: a pin already first-audited is dropped (done). if first_audited.contains(&event.pin) { @@ -1092,6 +1265,40 @@ impl ReplicationEngine { ); continue; } + // ADR-0004 Amendment 2 launch limiter: per-peer re-audit + // window (drop), then launch-rate budget + in-flight cap + // (penalty-free deferral: the pin stays pending). Assessed + // BEFORE the cooldown gate so a budget deferral never burns + // the peer's 30-min cooldown stamp; nothing is consumed + // until `commit_launch` below. + let assess_now = Instant::now(); + match limiter.assess( + &peer, + event.key_count, + assess_now, + observability.inflight.load(Ordering::Relaxed), + ) { + LimiterVerdict::WindowDeduped => { + observability.window_deduped.fetch_add(1, Ordering::Relaxed); + debug!( + "First-audit scheduler: audit_trigger=first_monetized outcome=window_deduped peer={peer} pin={} key_count={} pending={}", + hex::encode(event.pin), event.key_count, pending.len() + ); + continue; + } + LimiterVerdict::RateDeferred => { + deferred.push((peer, event)); + observability + .rate_deferred_attempts + .fetch_add(1, Ordering::Relaxed); + debug!( + "First-audit scheduler: audit_trigger=first_monetized outcome=rate_deferred peer={peer} pin={} key_count={} deferred={} tokens={}", + hex::encode(event.pin), event.key_count, deferred.len(), limiter.tokens + ); + continue; + } + LimiterVerdict::Admit => {} + } // Cooldown: if the peer's per-peer audit window is closed, keep // this pin pending and retry on a later tick once it reopens. // We do NOT treat "cooldown closed" as "already audited" (a @@ -1103,18 +1310,20 @@ impl ReplicationEngine { let now = Instant::now(); let mut map = gossip_audit.cooldown.write().await; if !cooldown_allows_audit(&mut map, &peer, now) { - pending.put(peer, event); + deferred.push((peer, event)); observability .cooldown_deferred_attempts .fetch_add(1, Ordering::Relaxed); debug!( - "First-audit scheduler: audit_trigger=first_monetized outcome=cooldown_deferred peer={peer} pin={} key_count={} pending={}", - hex::encode(event.pin), event.key_count, pending.len() + "First-audit scheduler: audit_trigger=first_monetized outcome=cooldown_deferred peer={peer} pin={} key_count={} deferred={}", + hex::encode(event.pin), event.key_count, deferred.len() ); continue; } } - // Audit is launching: now mark the pin first-audited. + // Audit is launching: consume budget, stamp the per-peer + // window, and mark the pin first-audited. + limiter.commit_launch(peer, event.key_count, assess_now); first_audited.put(event.pin, ()); observability.launched.fetch_add(1, Ordering::Relaxed); observability.inflight.fetch_add(1, Ordering::Relaxed); @@ -1126,6 +1335,19 @@ impl ReplicationEngine { let trigger = gossip_audit.clone(); let audit_observability = Arc::clone(&observability); tokio::spawn(async move { + // ADR-0004 Amendment 2: jitter the send. Every storer of + // a chunk verifies the same payment at the same instant; + // unjittered, the whole close group would challenge the + // paid peer simultaneously and trip its per-peer + // responder admission cap (drops that auditors then + // record as Timeout). The in-flight slot is already + // held, so jittered launches cannot pile up beyond the + // cap either. + let jitter_ms = rand::thread_rng().gen_range( + 0..=u64::try_from(config::FIRST_AUDIT_LAUNCH_JITTER_MAX.as_millis()) + .unwrap_or(u64::MAX), + ); + tokio::time::sleep(Duration::from_millis(jitter_ms)).await; let started = Instant::now(); let credit = storage_commitment_audit::AuditCredit { recent_provers: &trigger.recent_provers, @@ -1184,6 +1406,12 @@ impl ReplicationEngine { .await; }); } + // Re-insert deferred entries oldest-first so the LRU's + // relative recency is restored (newest stays most-recently- + // used and is the last to be capacity-evicted). + for (peer, event) in deferred.into_iter().rev() { + pending.put(peer, event); + } } debug!("First-audit drainer shut down"); }); @@ -5442,6 +5670,25 @@ async fn rebuild_and_rotate_commitment( #[allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] mod tests { use super::*; + use super::{ + apply_audit_failure_credit_revocation, audit_failure_clears_bootstrap_claim, + audit_failure_revokes_holder_credit, audit_launch_decision, config, cooldown_allows_audit, + first_audit_count_jump, first_audit_terminal_outcome, first_failed_key_label, + fresh_offer_payment_context, paid_notify_payment_context, queue_first_audit_event, + quote_within_audit_window, FirstAuditLimiter, FirstAuditQueueOutcome, + FirstAuditTerminalOutcome, LimiterVerdict, MonetizedPinEvent, MONETIZED_AUDIT_SKEW_MARGIN, + }; + use crate::payment::VerificationContext; + use crate::replication::audit::AuditTickResult; + use crate::replication::recent_provers::RecentProvers; + use crate::replication::types::{AuditFailureReason, AuditFailureSummary, FailureEvidence}; + use lru::LruCache; + use saorsa_core::identity::PeerId; + use std::collections::HashMap; + use std::num::NonZeroUsize; + use std::time::Duration; + use std::time::Instant; + use std::time::SystemTime; fn test_peer(b: u8) -> PeerId { let mut bytes = [0u8; 32]; @@ -5591,6 +5838,157 @@ mod tests { ); } + // -- ADR-0004 Amendment 2: first-audit launch limiter -------------------- + + #[test] + fn first_audit_limiter_enforces_burst_then_refills() { + let base = Instant::now(); + let mut limiter = FirstAuditLimiter::new(base); + let interval = config::FIRST_AUDIT_LAUNCH_INTERVAL; + + // The full burst is admitted back-to-back (distinct peers). + for i in 0..config::FIRST_AUDIT_BUDGET_BURST { + let peer = test_peer(u8::try_from(i).expect("small burst")); + assert_eq!(limiter.assess(&peer, 10, base, 0), LimiterVerdict::Admit); + limiter.commit_launch(peer, 10, base); + } + // Bucket empty: the next distinct peer is deferred, never dropped. + let extra = test_peer(0xEE); + assert_eq!( + limiter.assess(&extra, 10, base, 0), + LimiterVerdict::RateDeferred + ); + + // One full interval later exactly one token is available again. + let later = base + interval; + assert_eq!(limiter.assess(&extra, 10, later, 0), LimiterVerdict::Admit); + limiter.commit_launch(extra, 10, later); + let extra2 = test_peer(0xEF); + assert_eq!( + limiter.assess(&extra2, 10, later, 0), + LimiterVerdict::RateDeferred + ); + } + + #[test] + fn first_audit_limiter_refill_keeps_fractional_remainder() { + let base = Instant::now(); + let mut limiter = FirstAuditLimiter::new(base); + let interval = config::FIRST_AUDIT_LAUNCH_INTERVAL; + for i in 0..config::FIRST_AUDIT_BUDGET_BURST { + let peer = test_peer(u8::try_from(i).expect("small burst")); + assert_eq!(limiter.assess(&peer, 1, base, 0), LimiterVerdict::Admit); + limiter.commit_launch(peer, 1, base); + } + // 1.5 intervals later one token is earned and the half interval is + // NOT lost to drift... + let at_1_5 = base + interval + interval / 2; + let p = test_peer(0xAA); + assert_eq!(limiter.assess(&p, 1, at_1_5, 0), LimiterVerdict::Admit); + limiter.commit_launch(p, 1, at_1_5); + // ...so the next token arrives at 2.0 intervals, not 2.5. + let at_2_0 = base + interval * 2; + let q = test_peer(0xAB); + assert_eq!(limiter.assess(&q, 1, at_2_0, 0), LimiterVerdict::Admit); + } + + #[test] + fn first_audit_limiter_inflight_cap_defers_until_slot_frees() { + let base = Instant::now(); + let mut limiter = FirstAuditLimiter::new(base); + let peer = test_peer(1); + assert_eq!( + limiter.assess(&peer, 1, base, config::FIRST_AUDIT_MAX_INFLIGHT), + LimiterVerdict::RateDeferred + ); + // A freed slot admits without any clock movement. + assert_eq!( + limiter.assess(&peer, 1, base, config::FIRST_AUDIT_MAX_INFLIGHT - 1), + LimiterVerdict::Admit + ); + } + + #[test] + fn first_audit_limiter_assess_consumes_nothing() { + let base = Instant::now(); + let mut limiter = FirstAuditLimiter::new(base); + let peer = test_peer(3); + // Repeated assessment must not burn budget or stamp the window: only + // `commit_launch` consumes (the cooldown gate between assess and + // commit can defer, and that deferral must be free). + for _ in 0..10 { + assert_eq!(limiter.assess(&peer, 5, base, 0), LimiterVerdict::Admit); + } + for i in 0..config::FIRST_AUDIT_BUDGET_BURST { + let p = test_peer(0x20 + u8::try_from(i).expect("small burst")); + assert_eq!(limiter.assess(&p, 5, base, 0), LimiterVerdict::Admit); + limiter.commit_launch(p, 5, base); + } + } + + #[test] + fn first_audit_limiter_window_dedups_rotated_pins_and_count_jump_overrides() { + let base = Instant::now(); + let mut limiter = FirstAuditLimiter::new(base); + let peer = test_peer(7); + assert_eq!(limiter.assess(&peer, 100, base, 0), LimiterVerdict::Admit); + limiter.commit_launch(peer, 100, base); + + // A rotated pin with a similar count inside the window is dropped... + let soon = base + Duration::from_secs(60); + assert_eq!( + limiter.assess(&peer, 100, soon, 0), + LimiterVerdict::WindowDeduped + ); + // ...even at the exact jump boundary (new*DEN == old*NUM is no jump)... + assert_eq!( + limiter.assess(&peer, 150, soon, 0), + LimiterVerdict::WindowDeduped + ); + // ...but a >1.5x committed-count jump re-nominates immediately (an + // inflated sidecar-only pin is invisible to the gossip lottery, so + // the window must not shield it). + assert_eq!(limiter.assess(&peer, 151, soon, 0), LimiterVerdict::Admit); + + // Window expiry re-admits an unchanged count. + let expired = base + config::FIRST_AUDIT_PEER_REAUDIT_INTERVAL; + assert_eq!( + limiter.assess(&peer, 100, expired, 0), + LimiterVerdict::Admit + ); + } + + #[test] + fn first_audit_limiter_window_verdict_outranks_empty_budget() { + // A window-deduped nomination must be DROPPED, not kept pending as + // rate-deferred, even when the bucket is also empty: re-queuing a + // suppressed rotation would hold a pending slot for two hours. + let base = Instant::now(); + let mut limiter = FirstAuditLimiter::new(base); + let peer = test_peer(9); + assert_eq!(limiter.assess(&peer, 10, base, 0), LimiterVerdict::Admit); + limiter.commit_launch(peer, 10, base); + let other = test_peer(10); + assert_eq!(limiter.assess(&other, 10, base, 0), LimiterVerdict::Admit); + limiter.commit_launch(other, 10, base); + assert_eq!( + limiter.assess(&peer, 10, base, 0), + LimiterVerdict::WindowDeduped + ); + } + + #[test] + fn first_audit_count_jump_boundaries() { + // Exactly 1.5x is NOT a jump; strictly above is. + assert!(!first_audit_count_jump(100, 150)); + assert!(first_audit_count_jump(100, 151)); + // Anything beats an audited zero; zero never jumps. + assert!(first_audit_count_jump(0, 1)); + assert!(!first_audit_count_jump(0, 0)); + // Equal max counts must not jump (and must not overflow). + assert!(!first_audit_count_jump(u32::MAX, u32::MAX)); + } + #[test] fn fresh_offer_runs_store_admission_payment_checks() { let context = fresh_offer_payment_context(); diff --git a/tests/e2e/first_audit_ab.rs b/tests/e2e/first_audit_ab.rs new file mode 100644 index 00000000..62fe141e --- /dev/null +++ b/tests/e2e/first_audit_ab.rs @@ -0,0 +1,537 @@ +//! ADR-0004 first-audit A/B workload driver (NOT a pass/fail regression test). +//! +//! Drives a production-shaped paid-upload workload against a local testnet +//! with real on-chain payment verification (Anvil), so the monetized +//! first-audit pipeline runs end-to-end exactly as in production: +//! quote collection (with commitment sidecars) -> median payment on chain -> +//! `ChunkPutRequest` with the full proof -> storer-side `verify_payment` -> +//! monetized-pin nomination -> first-audit drainer -> subtree audits over QUIC. +//! +//! Run the SAME file against two builds (baseline `main` vs the scheduler-fix +//! branch) and compare the emitted `AB-METRIC` lines plus log-event counts +//! (audit launches, timeouts, responder drops). Skipped entirely unless +//! `FIRST_AUDIT_AB=1` is set, so it never runs in CI. +//! +//! ```bash +//! FIRST_AUDIT_AB=1 RUST_LOG=info,ant_node::replication=debug \ +//! cargo test --test e2e first_audit_ab_workload -- --nocapture +//! ``` +//! +//! Knobs (env): `AB_NODES`, `AB_SEED_UPLOADS`, `AB_UPLOADS`, `AB_GAP_MS`, +//! `AB_REBUILD_EVERY`, `AB_DOWNLOADS`, `AB_TAIL_SECS`, `AB_CHUNK_KIB`. + +#![allow( + clippy::unwrap_used, + clippy::expect_used, + clippy::panic, + clippy::unnested_or_patterns, + clippy::cast_precision_loss, + clippy::redundant_closure_for_method_calls +)] + +use ant_node::ant_protocol::{ + ChunkGetRequest, ChunkGetResponse, ChunkMessage, ChunkMessageBody, ChunkPutRequest, + ChunkPutResponse, ChunkQuoteRequest, ChunkQuoteResponse, CLOSE_GROUP_SIZE, +}; +use ant_node::client::send_and_await_chunk_response; +use ant_node::payment::{serialize_single_node_proof, PaymentProof}; +use evmlib::common::Amount; +use evmlib::{EncodedPeerId, ProofOfPayment}; +use rand::Rng; +use saorsa_core::identity::PeerId; +use saorsa_core::P2PNode; +use serial_test::serial; +use std::sync::Arc; +use std::time::{Duration, Instant}; + +use super::anvil::TestAnvil; +use super::testnet::{TestNetwork, TestNetworkConfig}; + +/// The verifier accepts a settlement of at least 3x the median quote price. +/// Mirrors `PAID_QUOTE_PAYMENT_MULTIPLIER` (private to the verifier). +const PAYMENT_MULTIPLIER: u64 = 3; + +/// Per-request timeout for quote/PUT/GET protocol round trips. +const REQUEST_TIMEOUT: Duration = Duration::from_secs(20); + +fn env_usize(name: &str, default: usize) -> usize { + std::env::var(name) + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(default) +} + +fn env_u64(name: &str, default: u64) -> u64 { + std::env::var(name) + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(default) +} + +/// One collected quote: the issuing peer, the decoded quote, and the raw +/// commitment sidecar blob (when the quote pinned a commitment). +struct CollectedQuote { + peer: PeerId, + quote: evmlib::PaymentQuote, + sidecar: Option>, +} + +/// Request a signed quote (plus ADR-0004 commitment sidecar) from one peer. +async fn request_quote( + client: &Arc, + peer: &PeerId, + address: [u8; 32], + data_size: u64, +) -> Result { + let request_id: u64 = rand::thread_rng().gen(); + let message = ChunkMessage { + request_id, + body: ChunkMessageBody::QuoteRequest(ChunkQuoteRequest { + address, + data_size, + data_type: 0, + }), + }; + let message_bytes = message + .encode() + .map_err(|e| format!("encode quote request: {e}"))?; + + let (quote_bytes, sidecar) = send_and_await_chunk_response( + client, + peer, + message_bytes, + request_id, + REQUEST_TIMEOUT, + &[], + |body| match body { + ChunkMessageBody::QuoteResponse(ChunkQuoteResponse::Success { + quote, + already_stored: _, + commitment, + }) => Some(Ok((quote, commitment))), + ChunkMessageBody::QuoteResponse(ChunkQuoteResponse::Error(e)) => { + Some(Err(format!("quote error: {e:?}"))) + } + _ => None, + }, + |e| format!("send quote request: {e}"), + || "quote request timed out".to_string(), + ) + .await?; + + let quote: evmlib::PaymentQuote = + rmp_serde::from_slice("e_bytes).map_err(|e| format!("decode quote: {e}"))?; + Ok(CollectedQuote { + peer: *peer, + quote, + sidecar, + }) +} + +/// Send a paid PUT to one peer; `Ok(true)` when it stored (or already had it). +async fn paid_put( + client: &Arc, + peer: &PeerId, + address: [u8; 32], + data: &[u8], + proof_bytes: Vec, +) -> Result { + let request_id: u64 = rand::thread_rng().gen(); + let message = ChunkMessage { + request_id, + body: ChunkMessageBody::PutRequest(ChunkPutRequest::with_payment( + address, + bytes::Bytes::copy_from_slice(data), + proof_bytes, + )), + }; + let message_bytes = message + .encode() + .map_err(|e| format!("encode put request: {e}"))?; + + send_and_await_chunk_response( + client, + peer, + message_bytes, + request_id, + REQUEST_TIMEOUT, + &[], + |body| match body { + ChunkMessageBody::PutResponse(ChunkPutResponse::Success { .. }) + | ChunkMessageBody::PutResponse(ChunkPutResponse::AlreadyExists { .. }) => { + Some(Ok(true)) + } + ChunkMessageBody::PutResponse(ChunkPutResponse::PaymentRequired { message }) => { + Some(Err(format!("payment required: {message}"))) + } + ChunkMessageBody::PutResponse(ChunkPutResponse::Error(e)) => { + Some(Err(format!("put error: {e:?}"))) + } + _ => None, + }, + |e| format!("send put: {e}"), + || "put timed out".to_string(), + ) + .await +} + +/// Fetch a chunk from one peer; `Ok(true)` when the bytes came back. +async fn get_chunk( + client: &Arc, + peer: &PeerId, + address: [u8; 32], +) -> Result { + let request_id: u64 = rand::thread_rng().gen(); + let message = ChunkMessage { + request_id, + body: ChunkMessageBody::GetRequest(ChunkGetRequest::new(address)), + }; + let message_bytes = message + .encode() + .map_err(|e| format!("encode get request: {e}"))?; + + send_and_await_chunk_response( + client, + peer, + message_bytes, + request_id, + REQUEST_TIMEOUT, + &[], + |body| match body { + ChunkMessageBody::GetResponse(ChunkGetResponse::Success { .. }) => Some(Ok(true)), + ChunkMessageBody::GetResponse(ChunkGetResponse::NotFound { .. }) => Some(Ok(false)), + ChunkMessageBody::GetResponse(ChunkGetResponse::Error(e)) => { + Some(Err(format!("get error: {e:?}"))) + } + _ => None, + }, + |e| format!("send get: {e}"), + || "get timed out".to_string(), + ) + .await +} + +/// Outcome of one full paid upload. +struct UploadOutcome { + duration: Duration, + pinned_quotes: usize, + total_quotes: usize, + stored_acks: usize, +} + +/// One production-shaped paid upload: collect quotes from the close group, +/// pay the median issuer 3x on chain, then PUT the chunk (with the full quote +/// bundle + sidecars) to every close-group peer. +async fn paid_upload( + client: &Arc, + client_peer: &PeerId, + wallet: &evmlib::wallet::Wallet, + data: &[u8], +) -> Result<(UploadOutcome, [u8; 32]), String> { + let address = ant_node::compute_address(data); + let started = Instant::now(); + + let closest = client + .dht() + .find_closest_nodes(&address, CLOSE_GROUP_SIZE + 1) + .await + .map_err(|e| format!("find closest: {e}"))?; + let storers: Vec = closest + .iter() + .map(|n| n.peer_id) + .filter(|p| p != client_peer) + .take(CLOSE_GROUP_SIZE) + .collect(); + if storers.is_empty() { + return Err("no close-group peers found".to_string()); + } + + let mut quotes: Vec = Vec::with_capacity(storers.len()); + for peer in &storers { + match request_quote(client, peer, address, data.len() as u64).await { + Ok(q) => quotes.push(q), + Err(e) => eprintln!("AB-WARN quote from {peer} failed: {e}"), + } + } + if quotes.is_empty() { + return Err("no quotes collected".to_string()); + } + + // Median selection mirrors the node verifier: sort by price, index len/2, + // settle 3x that price to the median issuer. + quotes.sort_by_key(|q| q.quote.price); + let median = quotes + .get(quotes.len() / 2) + .ok_or_else(|| "median index out of range".to_string())?; + let amount = median + .quote + .price + .checked_mul(Amount::from(PAYMENT_MULTIPLIER)) + .ok_or_else(|| "payment amount overflow".to_string())?; + let (tx_by_quote, _gas) = wallet + .pay_for_quotes(vec![( + median.quote.hash(), + median.quote.rewards_address, + amount, + )]) + .await + .map_err(|e| format!("pay_for_quotes: {e:?}"))?; + let tx_hashes: Vec = tx_by_quote.values().copied().collect(); + + let pinned_quotes = quotes + .iter() + .filter(|q| q.quote.commitment_pin.is_some()) + .count(); + let total_quotes = quotes.len(); + let proof = PaymentProof { + proof_of_payment: ProofOfPayment { + peer_quotes: quotes + .iter() + .map(|q| (EncodedPeerId::new(*q.peer.as_bytes()), q.quote.clone())) + .collect(), + }, + tx_hashes, + commitment_sidecars: quotes.iter().filter_map(|q| q.sidecar.clone()).collect(), + }; + let proof_bytes = + serialize_single_node_proof(&proof).map_err(|e| format!("serialize proof: {e}"))?; + + let mut stored_acks = 0usize; + for peer in &storers { + match paid_put(client, peer, address, data, proof_bytes.clone()).await { + Ok(true) => stored_acks += 1, + Ok(false) => {} + Err(e) => eprintln!("AB-WARN put to {peer} failed: {e}"), + } + } + if stored_acks == 0 { + return Err("no storer accepted the paid put".to_string()); + } + + Ok(( + UploadOutcome { + duration: started.elapsed(), + pinned_quotes, + total_quotes, + stored_acks, + }, + address, + )) +} + +/// Rebuild every node's storage commitment (simulates the hourly rotation +/// that re-arms pin-level dedup in production). +async fn rebuild_all_commitments(network: &TestNetwork) { + for node in network.nodes() { + if let Some(engine) = node.replication_engine.as_ref() { + if let Err(e) = engine.rebuild_commitment_now().await { + eprintln!( + "AB-WARN commitment rebuild failed on node {}: {e}", + node.index + ); + } + } + } +} + +fn percentile_ms(sorted: &[Duration], pct: f64) -> u128 { + if sorted.is_empty() { + return 0; + } + let last = sorted.len() - 1; + #[allow(clippy::cast_possible_truncation, clippy::cast_sign_loss)] + let idx = ((sorted.len() as f64 * pct) as usize).min(last); + sorted.get(idx).map_or(0, |d| d.as_millis()) +} + +#[tokio::test(flavor = "multi_thread")] +#[serial] +#[allow(clippy::too_many_lines)] +async fn first_audit_ab_workload() { + if std::env::var("FIRST_AUDIT_AB").is_err() { + eprintln!("first_audit_ab_workload skipped: set FIRST_AUDIT_AB=1 to run"); + return; + } + // Route node logs to stderr so the runner script can count audit events. + let _ = tracing_subscriber::fmt() + .with_env_filter( + tracing_subscriber::EnvFilter::try_from_default_env() + .unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("info")), + ) + .with_writer(std::io::stderr) + .try_init(); + + let nodes = env_usize("AB_NODES", 20); + let seed_uploads = env_usize("AB_SEED_UPLOADS", 25); + let uploads = env_usize("AB_UPLOADS", 40); + let gap_ms = env_u64("AB_GAP_MS", 2000); + let rebuild_every = env_usize("AB_REBUILD_EVERY", 15); + let downloads = env_usize("AB_DOWNLOADS", 15); + let tail_secs = env_u64("AB_TAIL_SECS", 90); + let chunk_kib = env_usize("AB_CHUNK_KIB", 64); + + eprintln!( + "AB-CONFIG nodes={nodes} seed_uploads={seed_uploads} uploads={uploads} gap_ms={gap_ms} \ + rebuild_every={rebuild_every} downloads={downloads} tail_secs={tail_secs} chunk_kib={chunk_kib}" + ); + + // Anvil FIRST so nodes verify payments against it for real. + let anvil = TestAnvil::new().await.expect("start anvil"); + let wallet = anvil.create_funded_wallet().expect("funded wallet"); + + let mut config = TestNetworkConfig { + node_count: nodes, + payment_enforcement: true, + ..TestNetworkConfig::default() + }; + config.evm_network = Some(anvil.to_network()); + let mut network = TestNetwork::new(config).await.expect("build network"); + network.start().await.expect("start network"); + network.warmup_dht().await.expect("warmup dht"); + + // Wire each node exactly like production `node.rs`: commitment source into + // the quote generator, gossip cache into the verifier, and the + // monetized-pin sender into the verifier (the trio the e2e testnet does + // not wire by default). + for node in network.nodes() { + if let (Some(protocol), Some(engine)) = + (node.ant_protocol.as_ref(), node.replication_engine.as_ref()) + { + let concrete = Arc::clone(engine.commitment_state()); + let source: Arc = concrete; + protocol.attach_commitment_source(source); + protocol + .payment_verifier_arc() + .attach_commitment_cache(Arc::clone(engine.last_commitment_by_peer())); + protocol + .payment_verifier_arc() + .attach_monetized_pin_sender(engine.monetized_pin_sender()); + } + } + + // The last node acts as the uploading/downloading client. + let client_index = nodes.saturating_sub(1); + let client = network + .nodes() + .get(client_index) + .and_then(|n| n.p2p_node.as_ref()) + .map(Arc::clone) + .expect("client node p2p"); + let client_peer = *client.peer_id(); + + let chunk_bytes = chunk_kib * 1024; + let mut rng = rand::thread_rng(); + let mut make_chunk = move || { + let mut data = vec![0u8; chunk_bytes]; + rng.fill(&mut data[..]); + data + }; + + // ---- Seed phase: give nodes stored chunks, then build commitments so + // subsequent quotes carry pins (production steady state). + let mut seeded = 0usize; + for _ in 0..seed_uploads { + let data = make_chunk(); + match paid_upload(&client, &client_peer, &wallet, &data).await { + Ok(_) => seeded += 1, + Err(e) => eprintln!("AB-WARN seed upload failed: {e}"), + } + } + eprintln!("AB-METRIC phase=seed uploads_ok={seeded}/{seed_uploads}"); + rebuild_all_commitments(&network).await; + tokio::time::sleep(Duration::from_secs(5)).await; + + // ---- Measurement phase: steady paid uploads, with periodic commitment + // rotation (the production re-arming mechanism). + let mut upload_durations: Vec = Vec::with_capacity(uploads); + let mut uploaded_addresses: Vec<[u8; 32]> = Vec::with_capacity(uploads); + let mut pinned_total = 0usize; + let mut quotes_total = 0usize; + let mut acks_total = 0usize; + let mut failures = 0usize; + let measure_started = Instant::now(); + for i in 0..uploads { + if i > 0 && rebuild_every > 0 && i % rebuild_every == 0 { + rebuild_all_commitments(&network).await; + eprintln!("AB-METRIC phase=measure rotation_at_upload={i}"); + } + let data = make_chunk(); + match paid_upload(&client, &client_peer, &wallet, &data).await { + Ok((outcome, address)) => { + eprintln!( + "AB-METRIC phase=measure upload={i} ms={} pinned={}/{} acks={}", + outcome.duration.as_millis(), + outcome.pinned_quotes, + outcome.total_quotes, + outcome.stored_acks + ); + pinned_total += outcome.pinned_quotes; + quotes_total += outcome.total_quotes; + acks_total += outcome.stored_acks; + upload_durations.push(outcome.duration); + uploaded_addresses.push(address); + } + Err(e) => { + failures += 1; + eprintln!("AB-WARN measure upload {i} failed: {e}"); + } + } + tokio::time::sleep(Duration::from_millis(gap_ms)).await; + } + + // ---- Download phase. + let mut download_durations: Vec = Vec::with_capacity(downloads); + let mut download_failures = 0usize; + for (i, address) in uploaded_addresses.iter().rev().take(downloads).enumerate() { + let started = Instant::now(); + let closest = client + .dht() + .find_closest_nodes(address, CLOSE_GROUP_SIZE) + .await + .unwrap_or_default(); + let mut found = false; + for peer in closest.iter().map(|n| n.peer_id) { + if peer == client_peer { + continue; + } + if get_chunk(&client, &peer, *address).await == Ok(true) { + found = true; + break; + } + } + if found { + let elapsed = started.elapsed(); + eprintln!( + "AB-METRIC phase=download get={i} ms={}", + elapsed.as_millis() + ); + download_durations.push(elapsed); + } else { + download_failures += 1; + eprintln!("AB-WARN download {i} failed"); + } + } + + // ---- Tail: let in-flight and pending audits play out before teardown so + // their outcomes land in the captured logs. + eprintln!("AB-METRIC phase=tail sleeping_secs={tail_secs}"); + tokio::time::sleep(Duration::from_secs(tail_secs)).await; + + upload_durations.sort_unstable(); + download_durations.sort_unstable(); + eprintln!( + "AB-SUMMARY uploads_ok={} uploads_failed={failures} upload_p50_ms={} upload_p90_ms={} \ + downloads_ok={} downloads_failed={download_failures} download_p50_ms={} download_p90_ms={} \ + pinned_quote_ratio={pinned_total}/{quotes_total} storer_acks={acks_total} \ + measure_wall_secs={}", + upload_durations.len(), + percentile_ms(&upload_durations, 0.50), + percentile_ms(&upload_durations, 0.90), + download_durations.len(), + percentile_ms(&download_durations, 0.50), + percentile_ms(&download_durations, 0.90), + measure_started.elapsed().as_secs(), + ); + + network.shutdown().await.expect("shutdown network"); +} diff --git a/tests/e2e/mod.rs b/tests/e2e/mod.rs index 994dc31f..d5e9ae5c 100644 --- a/tests/e2e/mod.rs +++ b/tests/e2e/mod.rs @@ -66,6 +66,9 @@ mod security_attacks; #[cfg(test)] mod subtree_audit_testnet; +#[cfg(test)] +mod first_audit_ab; + pub use anvil::TestAnvil; pub use harness::TestHarness; pub use testnet::{NetworkState, NodeState, TestNetwork, TestNetworkConfig, TestNode}; From 9f11f18427d542cb0d458809958e4c4aff2f140e Mon Sep 17 00:00:00 2001 From: grumbach Date: Tue, 14 Jul 2026 12:22:05 +0900 Subject: [PATCH 02/21] fix(payment): bind settlement to the quote's rewards address; harden limiter accounting Review-round fixes for the first-audit scheduler change: - Settlement-redirect rejection: completedPayments stores whatever (rewardsAddress, amount) the payer supplied for a quote hash, so the amount-only check accepted a payment the client redirected to its own wallet while the median issuer was treated (and first-audited) as paid. The verifier now compares the vault's bytes16 payee prefix against the quote's rewards address; validated against the real contract via the Anvil workload (honest pay_for_quotes settlements still verify). - Drop-guarded in-flight slot: a panicking or cancelled first-audit task can no longer leak an in-flight slot and wedge the cap shut. - Correct the launch-rate wording (sustained 12/h plus at most the burst of 2 in the first hour) in config docs and ADR-0004 Amendment 2. - A/B workload driver only runs when FIRST_AUDIT_AB=1 exactly. --- ...ADR-0004-commitment-bound-quote-pricing.md | 3 +- src/payment/verifier.rs | 122 ++++++++++++++++-- src/replication/config.rs | 9 +- src/replication/mod.rs | 33 ++++- tests/e2e/first_audit_ab.rs | 2 +- 5 files changed, 150 insertions(+), 19 deletions(-) diff --git a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md index 138d8fc4..1f8f1b9a 100644 --- a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md +++ b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md @@ -365,7 +365,8 @@ runs on every quote and candidate. **Launches are budgeted, not deterministic.** Each node's first-audit drainer runs a launch limiter: a token bucket (burst 2, one token per 5 minutes — a -hard per-node ceiling of 12 launches/hour), an in-flight cap (2), and a uniform +sustained per-node rate of 12 launches/hour, plus at most the burst of 2 in +the first hour), an in-flight cap (2), and a uniform 0-30s launch jitter so the storers of one chunk, which all verify the same payment at the same instant, do not challenge the paid peer simultaneously. Fleet-wide first-audit pressure is therefore `nodes x refill-rate`, independent diff --git a/src/payment/verifier.rs b/src/payment/verifier.rs index 7e116b0e..599f08ce 100644 --- a/src/payment/verifier.rs +++ b/src/payment/verifier.rs @@ -351,6 +351,11 @@ type CommitmentCache = Arc< /// handle without borrowing the verifier. type PinFetchNegativeCache = Arc>>; +/// Test-override shape for one on-chain settlement: the settled amount and +/// the recorded `bytes16` rewards-address prefix (`None` = matches the quote). +#[cfg(any(test, feature = "test-utils"))] +type TestSettlementOverride = (Amount, Option<[u8; 16]>); + /// Main payment verifier for ant-node. /// /// Uses: @@ -396,9 +401,13 @@ pub struct PaymentVerifier { /// Test-only override for `completedPayments(quote_hash)`. /// /// Production always queries the payment vault; unit tests use this to - /// exercise the full verifier path without starting an EVM chain. + /// exercise the full verifier path without starting an EVM chain. Maps a + /// quote hash to the settled amount plus the recorded rewards-address + /// prefix (`None` means "matches the quote", the honest-settlement + /// default). The prefix is 16 bytes because the vault packs the address + /// as `bytes16` in the `completedPayments` slot. #[cfg(any(test, feature = "test-utils"))] - test_completed_payments_override: RwLock>, + test_completed_payments_override: RwLock>, // NOTE: the test-only own-peer-id override was removed with the ADR-retired // quote-freshness/staleness gate (ADR-0004 binds price to the committed // count instead), so it no longer has any reader. @@ -677,12 +686,40 @@ impl PaymentVerifier { self.set_paid_quote_k_closest_for_tests(peer_ids); } - /// Test-only setter for an on-chain completed payment amount. + /// Test-only setter for an on-chain completed payment amount. The recorded + /// rewards address is treated as matching the quote (honest settlement). #[cfg(any(test, feature = "test-utils"))] pub fn set_completed_payment_for_tests(&self, quote_hash: QuoteHash, amount: Amount) { self.test_completed_payments_override .write() - .insert(quote_hash, amount); + .insert(quote_hash, (amount, None)); + } + + /// Test-only setter for an on-chain completed payment with an explicit + /// recorded rewards address, to exercise the settlement-redirect rejection. + #[cfg(any(test, feature = "test-utils"))] + pub fn set_completed_payment_with_address_for_tests( + &self, + quote_hash: QuoteHash, + amount: Amount, + rewards_address: evmlib::RewardsAddress, + ) { + self.test_completed_payments_override.write().insert( + quote_hash, + (amount, Some(Self::rewards_address_prefix(&rewards_address))), + ); + } + + /// The vault's `completedPayments` slot packs the payee as + /// `bytes16(bytes20(rewardsAddress))` — the leading 16 bytes of the + /// address (Solidity `bytesN` conversions truncate on the right). This is + /// the form the on-chain record is compared in. + fn rewards_address_prefix(rewards_address: &evmlib::RewardsAddress) -> [u8; 16] { + let mut prefix = [0u8; 16]; + if let Some(head) = rewards_address.as_slice().get(..16) { + prefix.copy_from_slice(head); + } + prefix } /// Check if payment is required for the given `XorName`. @@ -1103,9 +1140,29 @@ impl PaymentVerifier { Self::validate_paid_quote_signature(candidate).await?; - let on_chain_amount = self - .completed_payment_amount(candidate.quote.hash()) + let (on_chain_amount, recorded_rewards_prefix) = self + .completed_payment_settlement(candidate.quote.hash()) .await?; + // ADR-0004 Amendment 2: the settlement must be RECORDED FOR THE QUOTE'S + // rewards address, not merely under its quote hash. The vault stores + // whatever `(rewardsAddress, amount)` the payer supplied for the hash, + // so an amount-only check would accept a payment the client redirected + // to its own wallet — the issuer would be treated (and first-audited) + // as "paid" without ever being compensated. Honest clients build the + // payment from the quote itself, so this never rejects a legit upload. + // The vault packs the payee as `bytes16`, so the comparison is on the + // leading 16 bytes (128 bits — far beyond grinding range). + if let Some(recorded) = recorded_rewards_prefix { + let expected = Self::rewards_address_prefix(&candidate.quote.rewards_address); + if recorded != expected { + return Err(Error::Payment(format!( + "Median-priced quote settlement for peer {:?} was redirected: recorded rewards address prefix {} does not match the quote's {}", + candidate.encoded_peer_id, + hex::encode(recorded), + hex::encode(expected) + ))); + } + } if on_chain_amount >= candidate.expected_amount { return Ok(on_chain_amount); } @@ -1220,7 +1277,15 @@ impl PaymentVerifier { .map_err(|e| Error::Payment(format!("Signature verification task failed: {e}")))? } - async fn completed_payment_amount(&self, quote_hash: QuoteHash) -> Result { + /// Look up the on-chain settlement recorded for `quote_hash`: the settled + /// amount and the `bytes16` rewards-address prefix it was recorded for. + /// The prefix is `None` only on the test-override path (meaning "matches + /// the quote"); the production contract read always returns the recorded + /// prefix so the caller's redirect check always applies. + async fn completed_payment_settlement( + &self, + quote_hash: QuoteHash, + ) -> Result<(Amount, Option<[u8; 16]>)> { #[cfg(any(test, feature = "test-utils"))] { let completed_payment_override = { @@ -1229,8 +1294,8 @@ impl PaymentVerifier { .get("e_hash) .copied() }; - if let Some(amount) = completed_payment_override { - return Ok(amount); + if let Some((amount, recorded)) = completed_payment_override { + return Ok((amount, recorded)); } } @@ -1244,7 +1309,7 @@ impl PaymentVerifier { .await .map_err(|e| Error::Payment(format!("completedPayments lookup failed: {e}")))?; - Ok(Amount::from(result.amount)) + Ok((Amount::from(result.amount), Some(result.rewardsAddress.0))) } fn validate_paid_quote_peer_binding( @@ -3283,6 +3348,43 @@ mod tests { ); } + /// ADR-0004 Amendment 2: a settlement recorded under the median quote's + /// hash but redirected to a DIFFERENT rewards address must be rejected — + /// otherwise a client could "pay" its own wallet while the issuer is + /// treated (and first-audited) as paid without ever being compensated. + #[tokio::test] + async fn test_legacy_paid_median_redirected_settlement_rejected() { + let verifier = create_test_verifier(); + let xorname = [0xA2u8; 32]; + let peer_quotes = make_signed_legacy_bundle(xorname, unique_test_prices()); + mark_k_closest_paid_candidates(&verifier, &peer_quotes); + let expected_amount = expected_median_payment(&peer_quotes); + let paid_quote = median_test_candidates(&peer_quotes) + .first() + .expect("median candidate") + .1 + .clone(); + // Full amount, but recorded for an address that is not the quote's. + let attacker_address = RewardsAddress::new([0xEEu8; 20]); + assert_ne!(attacker_address, paid_quote.rewards_address); + verifier.set_completed_payment_with_address_for_tests( + paid_quote.hash(), + expected_amount, + attacker_address, + ); + + let proof_bytes = serialize_proof(peer_quotes); + let result = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await; + + let err = result.expect_err("redirected settlement must be rejected"); + assert!( + format!("{err}").contains("redirected"), + "Error should mention the settlement redirect: {err}" + ); + } + #[tokio::test] async fn test_legacy_single_quote_proof_accepted() { let verifier = create_test_verifier(); diff --git a/src/replication/config.rs b/src/replication/config.rs index 815964c6..e670cc62 100644 --- a/src/replication/config.rs +++ b/src/replication/config.rs @@ -412,10 +412,11 @@ pub const FIRST_AUDIT_DRAIN_BATCH: usize = 64; /// scheduler scaled with `uploads x pinned-quotes-per-proof x verifying-storers` /// (measured at ~110 storage-commitment audit events/hour/service in the /// DEV-01 staging run, with 77.7% of launches timing out in the DEV-03 -/// per-service-concurrency attempt). At 5 minutes the hard ceiling is -/// 12 launches/hour/node; steady-state demand sits far below it because -/// nomination is paid-pin-only and re-nominations are suppressed by -/// [`FIRST_AUDIT_PEER_REAUDIT_INTERVAL`]. +/// per-service-concurrency attempt). At 5 minutes the sustained rate is +/// 12 launches/hour/node (a drained burst bucket adds at most +/// [`FIRST_AUDIT_BUDGET_BURST`] more in the first hour); steady-state demand +/// sits far below it because nomination is paid-pin-only and re-nominations +/// are suppressed by [`FIRST_AUDIT_PEER_REAUDIT_INTERVAL`]. pub const FIRST_AUDIT_LAUNCH_INTERVAL: Duration = Duration::from_secs(5 * 60); /// ADR-0004 Amendment 2: token-bucket capacity for monetized first-audit diff --git a/src/replication/mod.rs b/src/replication/mod.rs index fad0f62a..15d7f514 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -176,6 +176,24 @@ struct RecentFirstAudit { key_count: u32, } +/// Holds one first-audit in-flight slot; decrements the gauge on drop so a +/// panicking or cancelled audit task can never leak a slot and wedge the +/// [`config::FIRST_AUDIT_MAX_INFLIGHT`] cap shut. +struct FirstAuditInflightSlot(Arc); + +impl FirstAuditInflightSlot { + fn acquire(observability: &Arc) -> Self { + observability.inflight.fetch_add(1, Ordering::Relaxed); + Self(Arc::clone(observability)) + } +} + +impl Drop for FirstAuditInflightSlot { + fn drop(&mut self) { + self.0.inflight.fetch_sub(1, Ordering::Relaxed); + } +} + /// ADR-0004 Amendment 2: whether `new_count` exceeds `audited_count` by more /// than the [`config::FIRST_AUDIT_COUNT_JUMP_NUM`]/ /// [`config::FIRST_AUDIT_COUNT_JUMP_DEN`] ratio (`new > old * NUM / DEN`, @@ -1322,11 +1340,19 @@ impl ReplicationEngine { } } // Audit is launching: consume budget, stamp the per-peer - // window, and mark the pin first-audited. + // window, and mark the pin first-audited. All three commit + // at LAUNCH (not at completion), matching the pre-existing + // `first_audited` semantics: an unproductive launch consumes + // the peer's slot, and re-coverage comes from the count-jump + // override or the gossip lottery — never from re-launching, + // which an unresponsive peer could otherwise farm for load. limiter.commit_launch(peer, event.key_count, assess_now); first_audited.put(event.pin, ()); observability.launched.fetch_add(1, Ordering::Relaxed); - observability.inflight.fetch_add(1, Ordering::Relaxed); + // Drop-guarded slot: released when the audit task finishes, + // panics, or is cancelled — the in-flight cap can never + // wedge shut on a leaked slot. + let inflight_slot = FirstAuditInflightSlot::acquire(&observability); debug!( "First-audit scheduler: audit_trigger=first_monetized outcome=launched peer={peer} pin={} key_count={} pending={} inflight={}", hex::encode(event.pin), event.key_count, pending.len(), @@ -1335,6 +1361,7 @@ impl ReplicationEngine { let trigger = gossip_audit.clone(); let audit_observability = Arc::clone(&observability); tokio::spawn(async move { + let inflight_slot = inflight_slot; // ADR-0004 Amendment 2: jitter the send. Every storer of // a chunk verifies the same payment at the same instant; // unjittered, the whole close group would challenge the @@ -1388,7 +1415,7 @@ impl ReplicationEngine { .fetch_add(1, Ordering::Relaxed); } } - audit_observability.inflight.fetch_sub(1, Ordering::Relaxed); + drop(inflight_slot); debug!( "First-audit scheduler: audit_trigger=first_monetized outcome={} peer={} pin={} key_count={} elapsed_ms={} inflight={}", outcome.as_str(), diff --git a/tests/e2e/first_audit_ab.rs b/tests/e2e/first_audit_ab.rs index 62fe141e..2425a363 100644 --- a/tests/e2e/first_audit_ab.rs +++ b/tests/e2e/first_audit_ab.rs @@ -348,7 +348,7 @@ fn percentile_ms(sorted: &[Duration], pct: f64) -> u128 { #[serial] #[allow(clippy::too_many_lines)] async fn first_audit_ab_workload() { - if std::env::var("FIRST_AUDIT_AB").is_err() { + if std::env::var("FIRST_AUDIT_AB").ok().as_deref() != Some("1") { eprintln!("first_audit_ab_workload skipped: set FIRST_AUDIT_AB=1 to run"); return; } From 46818c4caa75789839c24a9e65793c0178b156c3 Mon Sep 17 00:00:00 2001 From: grumbach Date: Tue, 14 Jul 2026 13:04:03 +0900 Subject: [PATCH 03/21] fix(replication): bound nomination ingress; re-screen answerability after jitter Final review-round hardenings against the production-incident context: - The verifier-to-drainer monetized-pin channel is now bounded (FIRST_AUDIT_INGRESS_CAPACITY = 1024); producers try_send and drop on full. Every stage of the first-audit pipeline is now capacity-limited: ingress queue, pending LRU, and launch-rate token bucket. A dropped nomination is penalty-free and the peer stays covered by the lottery. - The A1 answerability screen runs again after the 0-30s launch jitter, so a pin can never be challenged outside its window regardless of how deferral time, jitter, and the skew margin compose. No false-conviction window remains by construction. - ADR-0004 Amendment 2 documents the accepted residuals: the budget-exhaustion starvation economics (bounded by real settled payment costs, lottery backstop) and the PRE-EXISTING vault-level completedPayments overwrite race (already defeats the amount check with a 1-wei payment on all node versions; needs a contract-side fix, tracked as follow-up). --- ...ADR-0004-commitment-bound-quote-pricing.md | 25 ++++++++++++ src/payment/verifier.rs | 18 +++++---- src/replication/config.rs | 11 +++++ src/replication/mod.rs | 40 +++++++++++++++---- 4 files changed, 80 insertions(+), 14 deletions(-) diff --git a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md index 1f8f1b9a..39358c4a 100644 --- a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md +++ b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md @@ -396,3 +396,28 @@ an explicit load budget, not a per-payment guarantee. The scheduler's funnel cooldown_deferred / launched / terminal outcomes, plus tokens and in-flight gauges) is exported in the periodic scheduler summary so this coverage is measurable in production. + +**Every pipeline stage is bounded, and the invariants hold by construction.** +The verifier-to-drainer nomination channel is bounded (producers `try_send` +and drop on full — penalty-free, lottery-covered), the pending set is a +bounded newest-per-peer LRU, and launches are token-bucketed. The A1 +answerability screen runs both before launch selection and again after the +jitter sleep, so a pin can never be challenged outside its window regardless +of how deferral time, jitter, and the skew margin compose. + +**Accepted residuals.** (1) *Budget-exhaustion starvation:* an uploader can +keep an observer's launch budget saturated with newer settled payments so an +older pending pin expires un-first-audited. Each such nomination requires a +distinct real settled payment (window dedup suppresses repeats per issuer), +so starving all ~5-9 storers of one chunk for the whole 2.5h eligible window +costs dozens of on-chain payments plus gas per observer to shield a single +overcharged payment — economically irrational, and the cheater's gossiped +commitments remain under the ADR-0002 lottery throughout. (2) *Settlement +overwrite (pre-existing, contract-level):* the vault's `payForQuotes` +unconditionally overwrites `completedPayments[quoteHash]`, so a third party +who learns a quote hash can overwrite the record (already breaking the +long-standing amount check with a 1-wei payment, on all node versions) and +now equivalently the rewards-address binding. This griefing race predates +this amendment and needs a vault-side fix (reject or accumulate on existing +entries); it is tracked as follow-up work, not a property this ADR can +enforce from the node side. diff --git a/src/payment/verifier.rs b/src/payment/verifier.rs index 599f08ce..d7e29420 100644 --- a/src/payment/verifier.rs +++ b/src/payment/verifier.rs @@ -432,7 +432,7 @@ pub struct PaymentVerifier { /// `None` until [`Self::attach_monetized_pin_sender`] (unit tests, or /// pre-replication startup), in which case no first audit is scheduled. monetized_pin_tx: - RwLock>>, + RwLock>>, /// Price-floor input: the SAME live commitment source the local /// `QuoteGenerator` prices from, read via the non-mutating snapshot so the /// floor never extends commitment answerability. `None` until @@ -584,7 +584,7 @@ impl PaymentVerifier { /// absent (unit tests / pre-replication) no first audit is scheduled. pub fn attach_monetized_pin_sender( &self, - tx: tokio::sync::mpsc::UnboundedSender, + tx: tokio::sync::mpsc::Sender, ) { *self.monetized_pin_tx.write() = Some(tx); debug!("PaymentVerifier: ADR-0004 monetized-pin sender attached"); @@ -1749,7 +1749,9 @@ impl PaymentVerifier { let is_paid = paid_peer.is_some_and(|paid| paid == *encoded_peer_id.as_bytes()); if is_paid { if let Some(ref tx) = monetized_pin_tx { - let _ = tx.send(crate::replication::MonetizedPinEvent { + // Bounded queue: drop on full (best-effort, penalty-free; + // the gossip lottery still covers the peer). + let _ = tx.try_send(crate::replication::MonetizedPinEvent { peer: peer_id, pin, key_count: quote.committed_key_count, @@ -2871,7 +2873,9 @@ impl PaymentVerifier { // pool candidate (16) was nominated per verified proof. if paid_indices.contains(&idx) { if let Some(ref tx) = monetized_pin_tx { - let _ = tx.send(crate::replication::MonetizedPinEvent { + // Bounded queue: drop on full (best-effort, penalty-free; + // the gossip lottery still covers the peer). + let _ = tx.try_send(crate::replication::MonetizedPinEvent { peer: peer_id, pin, key_count: candidate.committed_key_count, @@ -4861,7 +4865,7 @@ mod tests { use evmlib::{EncodedPeerId, RewardsAddress}; let verifier = create_test_verifier(); - let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel(); + let (tx, mut rx) = tokio::sync::mpsc::channel(8); verifier.attach_monetized_pin_sender(tx); let ids: Vec<[u8; 32]> = (1..=3u8).map(|b| [b; 32]).collect(); @@ -4904,7 +4908,7 @@ mod tests { use evmlib::{EncodedPeerId, RewardsAddress}; let verifier = create_test_verifier(); - let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel(); + let (tx, mut rx) = tokio::sync::mpsc::channel(8); verifier.attach_monetized_pin_sender(tx); let mut quote = @@ -4926,7 +4930,7 @@ mod tests { use evmlib::merkle_payments::{MerklePaymentCandidatePool, MerkleTree}; let verifier = create_test_verifier(); - let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel(); + let (tx, mut rx) = tokio::sync::mpsc::channel(8); verifier.attach_monetized_pin_sender(tx); let timestamp = std::time::SystemTime::now() diff --git a/src/replication/config.rs b/src/replication/config.rs index e670cc62..a0acfe40 100644 --- a/src/replication/config.rs +++ b/src/replication/config.rs @@ -472,6 +472,17 @@ pub const FIRST_AUDIT_COUNT_JUMP_NUM: u64 = 3; /// [`FIRST_AUDIT_COUNT_JUMP_NUM`]. pub const FIRST_AUDIT_COUNT_JUMP_DEN: u64 = 2; +/// ADR-0004 Amendment 2: capacity of the bounded verifier-to-drainer +/// monetized-pin channel. +/// +/// Nominations are gated behind SETTLED on-chain payments, so legitimate +/// ingress is tiny; the drainer drains every wake (batched at +/// [`FIRST_AUDIT_DRAIN_BATCH`]) and coalesces newest-per-peer, so a backlog +/// this deep implies the drainer is starved, not that work is arriving fast. +/// Producers `try_send` and drop on full: a dropped nomination is +/// penalty-free, and the peer remains covered by the gossip-lottery path. +pub const FIRST_AUDIT_INGRESS_CAPACITY: usize = 1024; + /// Number of subtree leaves spot-checked against real chunk bytes per audit /// (ADR-0002 real-bytes layer). /// diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 15d7f514..146e53e8 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -610,11 +610,14 @@ pub struct ReplicationEngine { possession_check_rx: Option>, /// ADR-0004: sender the payment verifier clones to surface monetized pins /// for a deterministic first audit. The matching receiver is drained by - /// `start_first_audit_drainer`. - monetized_pin_tx: mpsc::UnboundedSender, + /// `start_first_audit_drainer`. BOUNDED (Amendment 2): the producer + /// `try_send`s and drops on a full queue, so ingress memory is capped just + /// like launches; a dropped nomination is penalty-free and the peer stays + /// covered by the gossip lottery. + monetized_pin_tx: mpsc::Sender, /// ADR-0004: receiver half of the monetized-pin channel, taken by /// `start_first_audit_drainer`. - monetized_pin_rx: Option>, + monetized_pin_rx: Option>, /// Shutdown token. shutdown: CancellationToken, /// Background task handles. @@ -652,7 +655,11 @@ impl ReplicationEngine { let (possession_check_tx, possession_check_rx) = mpsc::unbounded_channel(); // ADR-0004: monetized-pin channel (verifier -> first-audit drainer). - let (monetized_pin_tx, monetized_pin_rx) = mpsc::unbounded_channel(); + // Bounded (Amendment 2): every stage of the first-audit pipeline is + // now capacity-limited — ingress queue here, pending set (LRU), and + // launch rate (token bucket). + let (monetized_pin_tx, monetized_pin_rx) = + mpsc::channel(config::FIRST_AUDIT_INGRESS_CAPACITY); let engine = Self { config: Arc::clone(&config), @@ -697,10 +704,11 @@ impl ReplicationEngine { } /// ADR-0004: a sender the payment verifier uses to surface monetized pins - /// (commitments that backed a payment) for a deterministic first audit. - /// Cloneable; the engine drains the matching receiver. + /// (commitments that backed a payment) for a first audit. Cloneable; the + /// engine drains the matching receiver. Bounded: senders must `try_send` + /// and treat a full queue as a benign drop (Amendment 2 best-effort). #[must_use] - pub fn monetized_pin_sender(&self) -> mpsc::UnboundedSender { + pub fn monetized_pin_sender(&self) -> mpsc::Sender { self.monetized_pin_tx.clone() } @@ -1375,6 +1383,24 @@ impl ReplicationEngine { .unwrap_or(u64::MAX), ); tokio::time::sleep(Duration::from_millis(jitter_ms)).await; + // Re-screen answerability AFTER the jitter sleep so the + // no-false-conviction invariant (ADR-0004 A1) holds by + // construction, independent of how the skew margin and + // the jitter are sized relative to each other. A pin + // that aged out during the sleep is skipped with no + // consequence for the peer (slot released by the drop + // guard; pin stays first_audited — lottery covers it). + if !quote_within_audit_window(event.quote_ts, SystemTime::now()) { + audit_observability + .outside_answerability_window + .fetch_add(1, Ordering::Relaxed); + debug!( + "First-audit scheduler: audit_trigger=first_monetized outcome=outside_answerability_window post_jitter=true peer={} pin={} key_count={}", + event.peer, hex::encode(event.pin), event.key_count + ); + drop(inflight_slot); + return; + } let started = Instant::now(); let credit = storage_commitment_audit::AuditCredit { recent_provers: &trigger.recent_provers, From c50e75b2fc317c5b6b684dbfab715ede49e77cee Mon Sep 17 00:00:00 2001 From: grumbach Date: Tue, 14 Jul 2026 13:16:02 +0900 Subject: [PATCH 04/21] fix(replication): alternate first-audit launch passes newest/oldest Anti-starvation lane for the launch budget: passes alternate between newest-first (freshest answerability windows) and oldest-first, so a stream of fresh settled decoy nominations cannot keep an aging pin from launching before its eligibility window closes. Suppressing a pin now additionally requires pre-aged pending decoys at every observer, which the oldest lane itself drains and the per-peer re-audit window blocks from refreshing. Also restate the ADR-0004 Amendment 2 starvation residual with accurate economics (decoy settlements hit a whole storer cohort at once, merkle settlements nominate every paid index, principal recycles through sybil reward addresses, and sidecar-only pins have no lottery backstop) and correct over-claimed lottery-coverage comments: a dropped nomination is re-covered by the peer's next settled payment, the lottery covers only gossiped commitments. --- ...ADR-0004-commitment-bound-quote-pricing.md | 42 ++++++++++------- src/payment/verifier.rs | 4 +- src/replication/config.rs | 3 +- src/replication/mod.rs | 47 ++++++++++++++----- 4 files changed, 64 insertions(+), 32 deletions(-) diff --git a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md index 39358c4a..617e31c2 100644 --- a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md +++ b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md @@ -405,19 +405,29 @@ answerability screen runs both before launch selection and again after the jitter sleep, so a pin can never be challenged outside its window regardless of how deferral time, jitter, and the skew margin compose. -**Accepted residuals.** (1) *Budget-exhaustion starvation:* an uploader can -keep an observer's launch budget saturated with newer settled payments so an -older pending pin expires un-first-audited. Each such nomination requires a -distinct real settled payment (window dedup suppresses repeats per issuer), -so starving all ~5-9 storers of one chunk for the whole 2.5h eligible window -costs dozens of on-chain payments plus gas per observer to shield a single -overcharged payment — economically irrational, and the cheater's gossiped -commitments remain under the ADR-0002 lottery throughout. (2) *Settlement -overwrite (pre-existing, contract-level):* the vault's `payForQuotes` -unconditionally overwrites `completedPayments[quoteHash]`, so a third party -who learns a quote hash can overwrite the record (already breaking the -long-standing amount check with a 1-wei payment, on all node versions) and -now equivalently the rewards-address binding. This griefing race predates -this amendment and needs a vault-side fix (reject or accumulate on existing -entries); it is tracked as follow-up work, not a property this ADR can -enforce from the node side. +**Accepted residuals.** (1) *Budget-exhaustion starvation, mitigated by +alternating lanes:* an attacker can try to keep observers' launch budgets +saturated with newer settled decoy nominations so an older pending pin +expires un-first-audited. Stated precisely: suppressing one pin requires +outcompeting roughly 31 launch opportunities per observer over the 150-minute +eligibility window; a single decoy settlement consumes budget at every storer +of its chunk simultaneously (a cohort, not per-observer, cost), a merkle +settlement nominates every paid index, principal recycles through +sybil-controlled reward addresses (the real costs are gas, liquidity, and +close-group positioning, not the transferred amount), and a sidecar-only +target pin has NO lottery backstop. The scheduler therefore does not rely on +economics alone: launch passes alternate between newest-first and +oldest-first, so fresh decoys cannot displace an aging pin from the oldest +lane — suppressing it would additionally require pre-aged pending decoys at +every observer, which that lane itself drains and the per-peer re-audit +window blocks from refreshing. Residual exposure is the reduced coverage +latency any bounded scheduler has under sustained hostile load, not +indefinite suppression. (2) *Settlement overwrite (pre-existing, +contract-level):* the vault's `payForQuotes` unconditionally overwrites +`completedPayments[quoteHash]`, so a third party who learns a quote hash can +overwrite the record (already breaking the long-standing amount check with a +1-wei payment, on all node versions) and now equivalently the +rewards-address binding. This griefing race predates this amendment and +needs a vault-side fix (reject or accumulate on existing entries); it is +tracked as follow-up work, not a property this ADR can enforce from the node +side. diff --git a/src/payment/verifier.rs b/src/payment/verifier.rs index d7e29420..1fcdd9bc 100644 --- a/src/payment/verifier.rs +++ b/src/payment/verifier.rs @@ -1750,7 +1750,7 @@ impl PaymentVerifier { if is_paid { if let Some(ref tx) = monetized_pin_tx { // Bounded queue: drop on full (best-effort, penalty-free; - // the gossip lottery still covers the peer). + // the peer's next settled payment re-nominates it). let _ = tx.try_send(crate::replication::MonetizedPinEvent { peer: peer_id, pin, @@ -2874,7 +2874,7 @@ impl PaymentVerifier { if paid_indices.contains(&idx) { if let Some(ref tx) = monetized_pin_tx { // Bounded queue: drop on full (best-effort, penalty-free; - // the gossip lottery still covers the peer). + // the peer's next settled payment re-nominates it). let _ = tx.try_send(crate::replication::MonetizedPinEvent { peer: peer_id, pin, diff --git a/src/replication/config.rs b/src/replication/config.rs index a0acfe40..2ce8f8c3 100644 --- a/src/replication/config.rs +++ b/src/replication/config.rs @@ -480,7 +480,8 @@ pub const FIRST_AUDIT_COUNT_JUMP_DEN: u64 = 2; /// [`FIRST_AUDIT_DRAIN_BATCH`]) and coalesces newest-per-peer, so a backlog /// this deep implies the drainer is starved, not that work is arriving fast. /// Producers `try_send` and drop on full: a dropped nomination is -/// penalty-free, and the peer remains covered by the gossip-lottery path. +/// penalty-free; the peer's gossiped commitments stay lottery-covered and +/// its next settled payment re-nominates the paid pin. pub const FIRST_AUDIT_INGRESS_CAPACITY: usize = 1024; /// Number of subtree leaves spot-checked against real chunk bytes per audit diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 146e53e8..64ca4a04 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -612,8 +612,9 @@ pub struct ReplicationEngine { /// for a deterministic first audit. The matching receiver is drained by /// `start_first_audit_drainer`. BOUNDED (Amendment 2): the producer /// `try_send`s and drops on a full queue, so ingress memory is capped just - /// like launches; a dropped nomination is penalty-free and the peer stays - /// covered by the gossip lottery. + /// like launches; a dropped nomination is penalty-free — the peer's + /// gossiped commitments stay lottery-covered and its next settled payment + /// re-nominates the paid pin. monetized_pin_tx: mpsc::Sender, /// ADR-0004: receiver half of the monetized-pin channel, taken by /// `start_first_audit_drainer`. @@ -1108,6 +1109,14 @@ impl ReplicationEngine { // first-audit load independent of upload volume: payments only // NOMINATE pins, the clock launches them. let mut limiter = FirstAuditLimiter::new(Instant::now()); + // Launch-pass direction flip-flop (Amendment 2 anti-starvation): + // passes alternate between newest-first (freshest answerability + // windows get budget) and oldest-first (an aging pin cannot be + // starved out of its eligibility window by a stream of newer + // nominations — the attacker would need PRE-aged decoys at every + // observer, which the oldest lane itself drains and the per-peer + // re-audit window blocks from refreshing). + let mut oldest_first_pass = false; // Periodic retry tick for pending (cooldown/budget-blocked) pins. // Created once; `Skip` so a backlog of missed ticks collapses to one. let mut tick = tokio::time::interval(config::FIRST_AUDIT_RETRY_INTERVAL); @@ -1257,16 +1266,18 @@ impl ReplicationEngine { // Try to launch an audit for each pending peer; keep the ones // blocked by cooldown or budget for the next tick. Drain into a // vec first (LruCache has no drain). `iter()` yields most- to - // least-recently-used, and with the Amendment 2 launch budget - // that order is load-bearing: the NEWEST-monetized peers must - // consume tokens first (freshest answerability windows, per the - // ADR's newest-first coverage rule), with older pins deferred. - // Deferred entries are collected and re-inserted afterwards in - // reverse order so relative recency is restored (oldest re-put + // least-recently-used; the pass direction alternates (see + // `oldest_first_pass`) so budget is shared between the newest + // pins (freshest answerability) and the oldest (starvation + // resistance). Deferred entries are collected and re-inserted + // afterwards with relative recency restored (oldest re-put // first → stays the eviction victim). - let snapshot: Vec<(PeerId, MonetizedPinEvent)> = + let mut snapshot: Vec<(PeerId, MonetizedPinEvent)> = pending.iter().map(|(p, e)| (*p, *e)).collect(); pending.clear(); + if oldest_first_pass { + snapshot.reverse(); + } let mut deferred: Vec<(PeerId, MonetizedPinEvent)> = Vec::new(); for (peer, event) in snapshot { // Dedup: a pin already first-audited is dropped (done). @@ -1389,7 +1400,8 @@ impl ReplicationEngine { // the jitter are sized relative to each other. A pin // that aged out during the sleep is skipped with no // consequence for the peer (slot released by the drop - // guard; pin stays first_audited — lottery covers it). + // guard; pin stays first_audited; the peer's next + // settled payment re-nominates it). if !quote_within_audit_window(event.quote_ts, SystemTime::now()) { audit_observability .outside_answerability_window @@ -1461,10 +1473,19 @@ impl ReplicationEngine { } // Re-insert deferred entries oldest-first so the LRU's // relative recency is restored (newest stays most-recently- - // used and is the last to be capacity-evicted). - for (peer, event) in deferred.into_iter().rev() { - pending.put(peer, event); + // used and is the last to be capacity-evicted). The collection + // order follows the pass direction, so a newest-first pass + // reverses and an oldest-first pass inserts in order. + if oldest_first_pass { + for (peer, event) in deferred { + pending.put(peer, event); + } + } else { + for (peer, event) in deferred.into_iter().rev() { + pending.put(peer, event); + } } + oldest_first_pass = !oldest_first_pass; } debug!("First-audit drainer shut down"); }); From d141613c084fad1aa747bea830623ea388ce3171 Mon Sep 17 00:00:00 2001 From: grumbach Date: Tue, 14 Jul 2026 13:26:06 +0900 Subject: [PATCH 05/21] fix(replication): advance the first-audit launch lane per launch, not per pass MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The alternating newest/oldest lane flipped on every drainer pass, but most passes spend no token (the bucket is empty between refills) and a pass is triggered by incoming nominations as well as by the retry tick. An attacker could therefore inject a nomination to force a barren scan, flip lane parity, and keep every token-bearing pass on the newest lane — starving an aging pin with nothing but fresh decoy settlements, no pre-aged decoys required. The lane now advances only when a launch token is actually spent, so consecutive LAUNCHES strictly alternate whatever the scan pattern in between, and the oldest lane cannot be skipped. Adds a regression test modelling the barren-scan attack, and restates the Amendment 2 residual: the remaining suppression routes (pre-aged decoys at every observer, or evicting the target from the 4096-entry pending LRU) both cost real settled payments at scale. --- ...ADR-0004-commitment-bound-quote-pricing.md | 18 +++-- src/replication/mod.rs | 78 ++++++++++++++++++- 2 files changed, 88 insertions(+), 8 deletions(-) diff --git a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md index 617e31c2..206e106d 100644 --- a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md +++ b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md @@ -416,13 +416,17 @@ settlement nominates every paid index, principal recycles through sybil-controlled reward addresses (the real costs are gas, liquidity, and close-group positioning, not the transferred amount), and a sidecar-only target pin has NO lottery backstop. The scheduler therefore does not rely on -economics alone: launch passes alternate between newest-first and -oldest-first, so fresh decoys cannot displace an aging pin from the oldest -lane — suppressing it would additionally require pre-aged pending decoys at -every observer, which that lane itself drains and the per-peer re-audit -window blocks from refreshing. Residual exposure is the reduced coverage -latency any bounded scheduler has under sustained hostile load, not -indefinite suppression. (2) *Settlement overwrite (pre-existing, +economics alone: consecutive LAUNCHES strictly alternate between a +newest-first and an oldest-first lane (the lane advances only when a token is +actually spent, never on a barren scan, so nomination timing cannot steer +lane parity). Fresh decoys therefore cannot capture the oldest lane, and an +aging pin is served there. The remaining suppression routes both cost real +settled payments at scale: pre-aged pending decoys at every observer (which +the oldest lane itself drains, and which the per-peer re-audit window blocks +from refreshing), or evicting the target from a 4096-entry pending LRU with +that many fresher distinct-peer nominations. Residual exposure is therefore +increased coverage latency under sustained, paid, well-positioned hostile +load — not free or indefinite suppression. (2) *Settlement overwrite (pre-existing, contract-level):* the vault's `payForQuotes` unconditionally overwrites `completedPayments[quoteHash]`, so a third party who learns a quote hash can overwrite the record (already breaking the long-standing amount check with a diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 64ca4a04..49e06930 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -1278,6 +1278,16 @@ impl ReplicationEngine { if oldest_first_pass { snapshot.reverse(); } + // Whether this pass actually spent a launch token. The lane + // alternates on LAUNCHES, never on passes: most passes launch + // nothing (the bucket is empty between refills), and a pass is + // triggered by ingress as well as by the tick — so flipping per + // pass would let an attacker steer lane parity with the timing + // of its own nominations, and always hand the token-bearing pass + // to the newest lane. Flipping per launch makes CONSECUTIVE + // LAUNCHES strictly alternate, whatever the scan pattern in + // between, so the oldest lane cannot be skipped. + let mut launched_this_pass = false; let mut deferred: Vec<(PeerId, MonetizedPinEvent)> = Vec::new(); for (peer, event) in snapshot { // Dedup: a pin already first-audited is dropped (done). @@ -1367,6 +1377,7 @@ impl ReplicationEngine { // which an unresponsive peer could otherwise farm for load. limiter.commit_launch(peer, event.key_count, assess_now); first_audited.put(event.pin, ()); + launched_this_pass = true; observability.launched.fetch_add(1, Ordering::Relaxed); // Drop-guarded slot: released when the audit task finishes, // panics, or is cancelled — the in-flight cap can never @@ -1485,7 +1496,12 @@ impl ReplicationEngine { pending.put(peer, event); } } - oldest_first_pass = !oldest_first_pass; + // Flip ONLY when a token was actually spent (see + // `launched_this_pass`): a barren scan — however it was + // triggered — must not advance the lane. + if launched_this_pass { + oldest_first_pass = !oldest_first_pass; + } } debug!("First-audit drainer shut down"); }); @@ -6051,6 +6067,66 @@ mod tests { ); } + /// ADR-0004 Amendment 2 anti-starvation invariant: the launch lane must + /// advance per LAUNCH, never per pass. Passes are triggered by ingress as + /// well as by the retry tick and most of them spend no token (the bucket + /// is empty between refills), so a per-pass flip would let an attacker + /// steer lane parity purely with the timing of its own nominations and + /// keep every token-bearing pass on the newest lane — starving an aging + /// pin with nothing but fresh decoys. This models the drainer's lane rule + /// against that exact attack: barren scans (however many, however timed) + /// must not advance the lane, so consecutive launches strictly alternate. + #[test] + fn first_audit_lane_alternates_per_launch_not_per_pass() { + // Mirrors the drainer: `oldest_first_pass` flips iff a token was spent. + fn run_pass(lane_oldest: &mut bool, tokens: &mut u32) -> Option { + let lane = *lane_oldest; + if *tokens == 0 { + return None; // barren scan: no launch, no flip + } + *tokens -= 1; + *lane_oldest = !*lane_oldest; + Some(lane) + } + + let mut lane_oldest = false; + let mut tokens = 0u32; + let mut launch_lanes = Vec::new(); + + // The attacker injects nominations to force barren scans, trying to + // land every refill on the newest lane. + for _ in 0..8 { + for _ in 0..5 { + // Barren scans between refills: attacker-controlled in number. + assert_eq!( + run_pass(&mut lane_oldest, &mut tokens), + None, + "a scan with no token must not launch" + ); + } + tokens += 1; // refill + let lane = + run_pass(&mut lane_oldest, &mut tokens).expect("a refilled bucket must launch"); + launch_lanes.push(lane); + } + + // Consecutive launches alternate, so the oldest lane gets served every + // other launch no matter how the attacker times its ingress. + for (i, lane) in launch_lanes.iter().enumerate() { + assert_eq!( + *lane, + i % 2 == 1, + "launch {i} landed on the wrong lane: barren scans must not \ + advance lane parity" + ); + } + assert_eq!( + launch_lanes.iter().filter(|oldest| **oldest).count(), + 4, + "half of all launches must serve the oldest lane" + ); + } + #[test] fn first_audit_count_jump_boundaries() { // Exactly 1.5x is NOT a jump; strictly above is. From 0c7733a434c1876607cc24681b1de418e58c02b1 Mon Sep 17 00:00:00 2001 From: grumbach Date: Tue, 14 Jul 2026 13:59:37 +0900 Subject: [PATCH 06/21] fix(replication): flip the first-audit lane on every launch, not once per pass MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The lane still advanced once per launching pass, so a full burst spent both tokens in the same lane: the launch sequence could be newest, newest, oldest instead of a strict alternation, and the anti-starvation argument did not hold for burst passes. The launch loop now pops candidates from a deque — front for the newest lane, back for the oldest — and flips the lane at every committed launch, so consecutive launches strictly alternate including within one burst pass. Deferred entries carry their snapshot index and are re-inserted oldest-first, restoring LRU recency regardless of which end they were popped from. The regression test now models the real loop (deque + multi-token passes) and rejects both wrong flip rules: per pass (attacker steers parity with barren scans) and once per launching pass (a burst keeps one lane). ADR-0004 Amendment 2 restated accordingly, and the residual no longer claims suppression cannot be indefinite: evicting a target from an observer's 4096-entry pending LRU permanently drops that nomination, after which coverage depends on the peer's next settled payment (or the ADR-0002 lottery for a gossiped commitment). --- ...ADR-0004-commitment-bound-quote-pricing.md | 28 ++- src/replication/mod.rs | 209 ++++++++++-------- 2 files changed, 135 insertions(+), 102 deletions(-) diff --git a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md index 206e106d..197b7e92 100644 --- a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md +++ b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md @@ -417,16 +417,24 @@ sybil-controlled reward addresses (the real costs are gas, liquidity, and close-group positioning, not the transferred amount), and a sidecar-only target pin has NO lottery backstop. The scheduler therefore does not rely on economics alone: consecutive LAUNCHES strictly alternate between a -newest-first and an oldest-first lane (the lane advances only when a token is -actually spent, never on a barren scan, so nomination timing cannot steer -lane parity). Fresh decoys therefore cannot capture the oldest lane, and an -aging pin is served there. The remaining suppression routes both cost real -settled payments at scale: pre-aged pending decoys at every observer (which -the oldest lane itself drains, and which the per-peer re-audit window blocks -from refreshing), or evicting the target from a 4096-entry pending LRU with -that many fresher distinct-peer nominations. Residual exposure is therefore -increased coverage latency under sustained, paid, well-positioned hostile -load — not free or indefinite suppression. (2) *Settlement overwrite (pre-existing, +newest-first and an oldest-first lane. The lane advances on every committed +launch — not per scheduling pass (most pass over an empty bucket, and a pass +is triggered by ingress, so per-pass parity would be attacker-steerable) and +not once per launching pass (a full burst spends two tokens in one pass, and +those two must alternate with each other). Fresh decoy nominations therefore +cannot capture the oldest lane, and an aging pin is served there. + +Two suppression routes remain, both requiring real settled payments at scale. +An attacker holding pre-aged pending decoys at every observer can occupy the +oldest lane too — though that lane drains them, and the per-peer re-audit +window blocks the same issuers from refreshing them. And 4096 fresher +distinct-peer nominations at an observer evict the target from that +observer's pending LRU, which is permanent for that nomination: coverage then +depends on the peer's NEXT settled payment (or, for a gossiped commitment, on +the ADR-0002 lottery). We accept this: both routes cost per-observer sybil +positioning plus a stream of genuine on-chain settlements, and neither is +free or silent — the scheduler funnel exports capacity evictions and lane +outcomes. (2) *Settlement overwrite (pre-existing, contract-level):* the vault's `payForQuotes` unconditionally overwrites `completedPayments[quoteHash]`, so a third party who learns a quote hash can overwrite the record (already breaking the long-standing amount check with a diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 49e06930..37700df1 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -33,7 +33,7 @@ pub mod storage_commitment_audit; pub mod subtree; pub mod types; -use std::collections::{HashMap, HashSet}; +use std::collections::{HashMap, HashSet, VecDeque}; use std::fmt; use std::num::NonZeroUsize; use std::path::{Path, PathBuf}; @@ -1109,14 +1109,13 @@ impl ReplicationEngine { // first-audit load independent of upload volume: payments only // NOMINATE pins, the clock launches them. let mut limiter = FirstAuditLimiter::new(Instant::now()); - // Launch-pass direction flip-flop (Amendment 2 anti-starvation): - // passes alternate between newest-first (freshest answerability - // windows get budget) and oldest-first (an aging pin cannot be - // starved out of its eligibility window by a stream of newer - // nominations — the attacker would need PRE-aged decoys at every - // observer, which the oldest lane itself drains and the per-peer - // re-audit window blocks from refreshing). - let mut oldest_first_pass = false; + // Launch lane (Amendment 2 anti-starvation), flipped on every + // COMMITTED LAUNCH — see the launch loop below. Consecutive + // launches therefore strictly alternate between the newest pending + // pin (freshest answerability window) and the oldest (which can + // then never be starved out of its eligibility window by a stream + // of fresher nominations, however the attacker times them). + let mut oldest_first_lane = false; // Periodic retry tick for pending (cooldown/budget-blocked) pins. // Created once; `Skip` so a backlog of missed ticks collapses to one. let mut tick = tokio::time::interval(config::FIRST_AUDIT_RETRY_INTERVAL); @@ -1265,31 +1264,39 @@ impl ReplicationEngine { // Try to launch an audit for each pending peer; keep the ones // blocked by cooldown or budget for the next tick. Drain into a - // vec first (LruCache has no drain). `iter()` yields most- to - // least-recently-used; the pass direction alternates (see - // `oldest_first_pass`) so budget is shared between the newest - // pins (freshest answerability) and the oldest (starvation - // resistance). Deferred entries are collected and re-inserted - // afterwards with relative recency restored (oldest re-put - // first → stays the eviction victim). - let mut snapshot: Vec<(PeerId, MonetizedPinEvent)> = - pending.iter().map(|(p, e)| (*p, *e)).collect(); + // deque first (LruCache has no drain). `iter()` yields most- to + // least-recently-used, so the FRONT is the newest-monetized pin + // and the BACK is the oldest. + // + // ADR-0004 Amendment 2 anti-starvation lane: each launch takes + // its candidate from the front (newest: freshest answerability + // window) or the back (oldest: cannot be starved), and the lane + // flips ON EVERY COMMITTED LAUNCH — never per pass. Passes are + // triggered by ingress as well as by the retry tick, and most + // spend no token (the bucket is empty between refills), so a + // per-pass flip would let an attacker steer lane parity with the + // timing of its own nominations (barren scans) and keep every + // token-bearing launch on the newest lane. Flipping per launch + // makes CONSECUTIVE LAUNCHES strictly alternate — including the + // two launches of a full burst inside ONE pass — so an aging pin + // is always served by the next oldest-lane launch. + // + // Each entry keeps its snapshot index (0 = newest) so deferred + // entries can be re-inserted oldest-first afterwards, restoring + // the LRU's relative recency (oldest re-put first → stays the + // eviction victim) no matter which ends they were popped from. + let mut queue: VecDeque<(usize, PeerId, MonetizedPinEvent)> = pending + .iter() + .enumerate() + .map(|(i, (p, e))| (i, *p, *e)) + .collect(); pending.clear(); - if oldest_first_pass { - snapshot.reverse(); - } - // Whether this pass actually spent a launch token. The lane - // alternates on LAUNCHES, never on passes: most passes launch - // nothing (the bucket is empty between refills), and a pass is - // triggered by ingress as well as by the tick — so flipping per - // pass would let an attacker steer lane parity with the timing - // of its own nominations, and always hand the token-bearing pass - // to the newest lane. Flipping per launch makes CONSECUTIVE - // LAUNCHES strictly alternate, whatever the scan pattern in - // between, so the oldest lane cannot be skipped. - let mut launched_this_pass = false; - let mut deferred: Vec<(PeerId, MonetizedPinEvent)> = Vec::new(); - for (peer, event) in snapshot { + let mut deferred: Vec<(usize, PeerId, MonetizedPinEvent)> = Vec::new(); + while let Some((idx, peer, event)) = if oldest_first_lane { + queue.pop_back() + } else { + queue.pop_front() + } { // Dedup: a pin already first-audited is dropped (done). if first_audited.contains(&event.pin) { observability.duplicates.fetch_add(1, Ordering::Relaxed); @@ -1334,7 +1341,7 @@ impl ReplicationEngine { continue; } LimiterVerdict::RateDeferred => { - deferred.push((peer, event)); + deferred.push((idx, peer, event)); observability .rate_deferred_attempts .fetch_add(1, Ordering::Relaxed); @@ -1357,7 +1364,7 @@ impl ReplicationEngine { let now = Instant::now(); let mut map = gossip_audit.cooldown.write().await; if !cooldown_allows_audit(&mut map, &peer, now) { - deferred.push((peer, event)); + deferred.push((idx, peer, event)); observability .cooldown_deferred_attempts .fetch_add(1, Ordering::Relaxed); @@ -1377,7 +1384,10 @@ impl ReplicationEngine { // which an unresponsive peer could otherwise farm for load. limiter.commit_launch(peer, event.key_count, assess_now); first_audited.put(event.pin, ()); - launched_this_pass = true; + // Flip the lane on EVERY committed launch (see the lane + // comment above): consecutive launches strictly alternate, + // including the two launches of a full burst in one pass. + oldest_first_lane = !oldest_first_lane; observability.launched.fetch_add(1, Ordering::Relaxed); // Drop-guarded slot: released when the audit task finishes, // panics, or is cancelled — the in-flight cap can never @@ -1482,25 +1492,15 @@ impl ReplicationEngine { .await; }); } - // Re-insert deferred entries oldest-first so the LRU's - // relative recency is restored (newest stays most-recently- - // used and is the last to be capacity-evicted). The collection - // order follows the pass direction, so a newest-first pass - // reverses and an oldest-first pass inserts in order. - if oldest_first_pass { - for (peer, event) in deferred { - pending.put(peer, event); - } - } else { - for (peer, event) in deferred.into_iter().rev() { - pending.put(peer, event); - } - } - // Flip ONLY when a token was actually spent (see - // `launched_this_pass`): a barren scan — however it was - // triggered — must not advance the lane. - if launched_this_pass { - oldest_first_pass = !oldest_first_pass; + // Re-insert deferred entries oldest-first so the LRU's relative + // recency is restored (oldest re-put first → stays the eviction + // victim; newest stays most-recently-used). Entries were popped + // from BOTH ends as the lane alternated, so sort by the snapshot + // index (0 = newest) descending rather than relying on the + // collection order. + deferred.sort_unstable_by_key(|(idx, _, _)| std::cmp::Reverse(*idx)); + for (_, peer, event) in deferred { + pending.put(peer, event); } } debug!("First-audit drainer shut down"); @@ -6068,61 +6068,86 @@ mod tests { } /// ADR-0004 Amendment 2 anti-starvation invariant: the launch lane must - /// advance per LAUNCH, never per pass. Passes are triggered by ingress as - /// well as by the retry tick and most of them spend no token (the bucket - /// is empty between refills), so a per-pass flip would let an attacker - /// steer lane parity purely with the timing of its own nominations and - /// keep every token-bearing pass on the newest lane — starving an aging - /// pin with nothing but fresh decoys. This models the drainer's lane rule - /// against that exact attack: barren scans (however many, however timed) - /// must not advance the lane, so consecutive launches strictly alternate. + /// advance on every COMMITTED LAUNCH — never per pass, and never once per + /// pass regardless of how many tokens that pass spends. + /// + /// Two ways to get this wrong, both of which let fresh decoys crowd out an + /// aging pin, and both of which this test rejects: + /// + /// 1. *Flip per pass.* Passes are triggered by ingress as well as by the + /// retry tick, and most spend no token (the bucket is empty between + /// refills), so an attacker could inject nominations to force barren + /// scans, steer lane parity, and keep every token-bearing launch on the + /// newest lane. + /// 2. *Flip once per pass that launched.* A full burst spends two tokens + /// inside ONE pass; a single flip afterwards yields `newest, newest, + /// oldest` instead of a strict alternation. + /// + /// This models the drainer's real loop — a deque popped from the front + /// (newest) or the back (oldest) with the lane flipped at each launch — + /// and asserts strict alternation across barren scans AND multi-token + /// bursts. #[test] - fn first_audit_lane_alternates_per_launch_not_per_pass() { - // Mirrors the drainer: `oldest_first_pass` flips iff a token was spent. - fn run_pass(lane_oldest: &mut bool, tokens: &mut u32) -> Option { - let lane = *lane_oldest; - if *tokens == 0 { - return None; // barren scan: no launch, no flip + fn first_audit_lane_alternates_per_launch_across_barren_scans_and_bursts() { + /// One drainer pass over `queue_len` pending pins with `tokens` + /// available. Returns the lane each launch used, in order. + fn run_pass(lane_oldest: &mut bool, tokens: &mut u32, queue_len: usize) -> Vec { + let mut launched = Vec::new(); + let mut remaining = queue_len; + while remaining > 0 && *tokens > 0 { + // The lane picks the end of the deque; the launch flips it. + launched.push(*lane_oldest); + *lane_oldest = !*lane_oldest; + *tokens -= 1; + remaining -= 1; } - *tokens -= 1; - *lane_oldest = !*lane_oldest; - Some(lane) + launched } let mut lane_oldest = false; let mut tokens = 0u32; - let mut launch_lanes = Vec::new(); + let mut lanes = Vec::new(); - // The attacker injects nominations to force barren scans, trying to - // land every refill on the newest lane. - for _ in 0..8 { + // Attacker forces barren scans between refills, trying to land every + // token on the newest lane. A barren pass launches nothing, so it + // cannot advance the lane. + for _ in 0..4 { for _ in 0..5 { - // Barren scans between refills: attacker-controlled in number. - assert_eq!( - run_pass(&mut lane_oldest, &mut tokens), - None, - "a scan with no token must not launch" + assert!( + run_pass(&mut lane_oldest, &mut tokens, 8).is_empty(), + "a pass with no tokens must not launch and must not flip" ); } - tokens += 1; // refill - let lane = - run_pass(&mut lane_oldest, &mut tokens).expect("a refilled bucket must launch"); - launch_lanes.push(lane); + tokens += 1; + lanes.extend(run_pass(&mut lane_oldest, &mut tokens, 8)); } - // Consecutive launches alternate, so the oldest lane gets served every - // other launch no matter how the attacker times its ingress. - for (i, lane) in launch_lanes.iter().enumerate() { + // A full burst spends BOTH tokens in a single pass: those two launches + // must still alternate with each other. + tokens += config::FIRST_AUDIT_BUDGET_BURST; + let burst = run_pass(&mut lane_oldest, &mut tokens, 8); + assert_eq!( + burst.len(), + config::FIRST_AUDIT_BUDGET_BURST as usize, + "a full burst must launch every token it holds" + ); + lanes.extend(burst); + + // ...and a later single-token pass continues the same alternation. + tokens += 1; + lanes.extend(run_pass(&mut lane_oldest, &mut tokens, 8)); + + for (i, lane) in lanes.iter().enumerate() { assert_eq!( *lane, i % 2 == 1, - "launch {i} landed on the wrong lane: barren scans must not \ - advance lane parity" + "launch {i} landed on the wrong lane: consecutive launches must \ + strictly alternate across barren scans AND multi-token bursts" ); } assert_eq!( - launch_lanes.iter().filter(|oldest| **oldest).count(), - 4, + lanes.iter().filter(|oldest| **oldest).count(), + lanes.len() / 2, "half of all launches must serve the oldest lane" ); } From 22bb41cd06ed2b2191d9f02faf896e15c658cdb4 Mon Sep 17 00:00:00 2001 From: grumbach Date: Tue, 14 Jul 2026 15:18:48 +0900 Subject: [PATCH 07/21] feat(replication): observe first-audit ingress drops; clarify launched telemetry MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prod-readiness review follow-ups (both minor/nit, no behavior change): - Count nominations dropped at the bounded ingress channel when it is full (Full only, not Closed) in a process-global counter, surfaced as ingress_dropped in the 5-minute scheduler summary. A non-zero value is the rollout signal that nomination ingress is saturating — benign but the thing to watch when widening the rollout. - Document that launched counts SCHEDULED audits (token spent); a pin that ages out at the post-jitter answerability re-screen is counted under outside_answerability_window and sends nothing, so actual wire challenges are launched minus post-jitter aborts, reconciled by the terminal counters. --- src/payment/verifier.rs | 48 +++++++++++++++++++++++++---------------- src/replication/mod.rs | 24 ++++++++++++++++++++- 2 files changed, 53 insertions(+), 19 deletions(-) diff --git a/src/payment/verifier.rs b/src/payment/verifier.rs index 1fcdd9bc..4810ccd0 100644 --- a/src/payment/verifier.rs +++ b/src/payment/verifier.rs @@ -1750,13 +1750,19 @@ impl PaymentVerifier { if is_paid { if let Some(ref tx) = monetized_pin_tx { // Bounded queue: drop on full (best-effort, penalty-free; - // the peer's next settled payment re-nominates it). - let _ = tx.try_send(crate::replication::MonetizedPinEvent { - peer: peer_id, - pin, - key_count: quote.committed_key_count, - quote_ts: quote.timestamp, - }); + // the peer's next settled payment re-nominates it). Count a + // Full drop so ingress saturation is observable; a Closed + // channel just means the engine is shutting down. + if let Err(tokio::sync::mpsc::error::TrySendError::Full(_)) = + tx.try_send(crate::replication::MonetizedPinEvent { + peer: peer_id, + pin, + key_count: quote.committed_key_count, + quote_ts: quote.timestamp, + }) + { + crate::replication::note_monetized_ingress_drop(); + } } } // Resolution order: sidecar (synchronous, no state) -> gossip cache @@ -2874,17 +2880,23 @@ impl PaymentVerifier { if paid_indices.contains(&idx) { if let Some(ref tx) = monetized_pin_tx { // Bounded queue: drop on full (best-effort, penalty-free; - // the peer's next settled payment re-nominates it). - let _ = tx.try_send(crate::replication::MonetizedPinEvent { - peer: peer_id, - pin, - key_count: candidate.committed_key_count, - quote_ts: std::time::UNIX_EPOCH - .checked_add(std::time::Duration::from_secs( - candidate.merkle_payment_timestamp, - )) - .unwrap_or(std::time::UNIX_EPOCH), - }); + // the peer's next settled payment re-nominates it). Count a + // Full drop so ingress saturation is observable; a Closed + // channel just means the engine is shutting down. + if let Err(tokio::sync::mpsc::error::TrySendError::Full(_)) = + tx.try_send(crate::replication::MonetizedPinEvent { + peer: peer_id, + pin, + key_count: candidate.committed_key_count, + quote_ts: std::time::UNIX_EPOCH + .checked_add(std::time::Duration::from_secs( + candidate.merkle_payment_timestamp, + )) + .unwrap_or(std::time::UNIX_EPOCH), + }) + { + crate::replication::note_monetized_ingress_drop(); + } } } diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 37700df1..39374981 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -81,6 +81,21 @@ use saorsa_core::identity::{NodeIdentity, PeerId}; use saorsa_core::{DhtNetworkEvent, P2PEvent, P2PNode, TrustEvent}; use saorsa_pqc::api::sig::{MlDsaSecretKey, MlDsaVariant}; +/// Count of monetized-pin nominations DROPPED at the bounded ingress channel +/// because it was full (Amendment 2). Process-global because the producer is +/// the payment verifier (a different module) and the drop happens before the +/// per-drainer `received` counter. A non-zero value is the rollout signal that +/// nomination ingress is saturating — benign (penalty-free, lottery/next- +/// payment covered) but worth watching. `Closed` (engine shut down) is not +/// counted: it is not a saturation signal. +static FIRST_AUDIT_INGRESS_DROPPED: AtomicU64 = AtomicU64::new(0); + +/// Record one ingress-full drop. Called by the payment verifier's `try_send` +/// sites; read by the drainer's periodic summary. +pub(crate) fn note_monetized_ingress_drop() { + FIRST_AUDIT_INGRESS_DROPPED.fetch_add(1, Ordering::Relaxed); +} + #[derive(Default)] struct FirstAuditObservability { received: AtomicU64, @@ -1234,7 +1249,8 @@ impl ReplicationEngine { if last_summary.elapsed() >= config::FIRST_AUDIT_SUMMARY_INTERVAL { info!( - "First-audit scheduler summary: audit_trigger=first_monetized received={} queued={} coalesced={} duplicates={} capacity_evicted={} cooldown_deferred_attempts={} rate_deferred_attempts={} window_deduped={} launched={} passed={} timeout={} failed={} bootstrap_claims={} idle={} insufficient_keys={} outside_answerability_window={} pending={} inflight={} tokens={}", + "First-audit scheduler summary: audit_trigger=first_monetized ingress_dropped={} received={} queued={} coalesced={} duplicates={} capacity_evicted={} cooldown_deferred_attempts={} rate_deferred_attempts={} window_deduped={} launched={} passed={} timeout={} failed={} bootstrap_claims={} idle={} insufficient_keys={} outside_answerability_window={} pending={} inflight={} tokens={}", + FIRST_AUDIT_INGRESS_DROPPED.load(Ordering::Relaxed), observability.received.load(Ordering::Relaxed), observability.queued.load(Ordering::Relaxed), observability.coalesced.load(Ordering::Relaxed), @@ -1388,6 +1404,12 @@ impl ReplicationEngine { // comment above): consecutive launches strictly alternate, // including the two launches of a full burst in one pass. oldest_first_lane = !oldest_first_lane; + // `launched` counts SCHEDULED audits (a token was spent). + // A pin that ages out during the post-jitter re-screen is + // counted separately under `outside_answerability_window` + // and sends nothing, so actual wire challenges are + // `launched - (post-jitter aborts)`; the terminal counters + // (passed/timeout/…) reconcile it exactly. observability.launched.fetch_add(1, Ordering::Relaxed); // Drop-guarded slot: released when the audit task finishes, // panics, or is cancelled — the in-flight cap can never From 4bca0fa3003f9939c2b805158a43cf93490e6f2d Mon Sep 17 00:00:00 2001 From: grumbach Date: Wed, 15 Jul 2026 13:26:51 +0900 Subject: [PATCH 08/21] fix(replication): reserve/promote first audits so cancels leave no suppression MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Blocking review finding (dirvine): the drainer committed durable suppression (first_audited + the 2h per-peer window + token + launched) at LAUNCH time, then the child slept for jitter and re-checked answerability; a quote near the answerability boundary could obtain full 'already audited' suppression and drop a same-count successor for two hours despite no audit ever sending. Reworked to a drainer-owned reserve/promote/cancel scheduler (consensus design with codex xhigh over four rounds): - Payments NOMINATE; the drainer holds at most ONE reservation at a time (preserving per-launch lane alternation), consuming a token and an in-flight slot but stamping NO suppression. - A jitter timer (select arm) fires at the reservation's ready_at; the AUTHORITATIVE answerability + shared-cooldown check-and-stamp runs then, under the cooldown write lock, immediately before the send. Only on that pass are first_audited/recent stamped, the lane flipped, and launched counted. - A cancelled reservation (answerability lapsed during jitter, or a gossip audit won the cooldown) refunds the token, releases the slot, and leaves NO durable state — nothing to roll back. A cooldown-race requeues the event. - A same-peer successor arriving during a reservation is retained (window bypass for the reserved peer), never dropped, and becomes the next reservation if the first cancels. - B horizon prefilter: a nomination is only admitted if it stays answerable through now + max_jitter + slack, so a committed reservation can still be challenged at send; the authoritative check remains at promotion so A1 holds regardless of jitter-vs-window sizing. All scheduler state mutation stays single-threaded in the drainer; the spawned child only runs the audit I/O and holds the moved-in in-flight slot. Tests: horizon-prefilter boundary; answerability-cancel is state-neutral and retains a same-count successor (the reviewer's exact hole); per-launch lane alternation driven through reserve/resolve. 699 lib tests + cfd green; e2e smoke on a live anvil testnet exercises reserve->promote end to end. --- src/replication/mod.rs | 1197 ++++++++++++++++++++++++---------------- 1 file changed, 734 insertions(+), 463 deletions(-) diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 39374981..42520546 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -33,7 +33,7 @@ pub mod storage_commitment_audit; pub mod subtree; pub mod types; -use std::collections::{HashMap, HashSet, VecDeque}; +use std::collections::{HashMap, HashSet}; use std::fmt; use std::num::NonZeroUsize; use std::path::{Path, PathBuf}; @@ -183,9 +183,16 @@ fn queue_first_audit_event( } } +/// ADR-0004 Amendment 2 (E′): slack added to the max launch jitter when +/// prefiltering a nomination's answerability at schedule time, covering the +/// spawn/dispatch latency between the timer firing and the wire challenge so a +/// jitter==MAX pin is not admitted only to fail the authoritative check by a +/// few milliseconds. Tiny against the multi-hour answerability window. +const FIRST_AUDIT_SEND_LATENCY_SLACK: Duration = Duration::from_secs(1); + /// A first audit the limiter recently launched at a peer: when, and the /// committed key count that was audited (for the count-jump override). -#[derive(Debug, Clone, Copy)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] struct RecentFirstAudit { launched_at: Instant, key_count: u32, @@ -223,8 +230,8 @@ const fn first_audit_count_jump(audited_count: u32, new_count: u32) -> bool { /// The launch limiter's verdict for one pending monetized pin. #[derive(Debug, Clone, Copy, PartialEq, Eq)] enum LimiterVerdict { - /// Within budget and window: the caller may launch (and MUST call - /// [`FirstAuditLimiter::commit_launch`] if it actually does). + /// Within budget and window: the caller may reserve a launch (and consumes + /// a token via [`FirstAuditLimiter::reserve_token`] only if it does). Admit, /// Launch-rate budget or in-flight cap exhausted. Penalty-free: keep the /// pin pending and retry on a later tick. @@ -329,10 +336,27 @@ impl FirstAuditLimiter { LimiterVerdict::Admit } - /// Consume one token and stamp the per-peer window for a launch that is - /// actually happening. - fn commit_launch(&mut self, peer: PeerId, key_count: u32, now: Instant) { + /// Consume one launch token for a RESERVATION (ADR-0004 Amendment 2 E′). + /// Does NOT stamp the per-peer window: the durable `recent` stamp happens + /// only at [`Self::promote`], after the authoritative post-jitter + /// answerability check, so a reservation that is later cancelled leaves no + /// suppression behind. + fn reserve_token(&mut self) { self.tokens = self.tokens.saturating_sub(1); + } + + /// Return a token consumed by a reservation that was cancelled before it + /// launched (answerability lapsed or a concurrent gossip audit won the + /// cooldown). Capped at the burst so a spurious double-refund cannot exceed + /// the bucket. + fn refund_token(&mut self) { + self.tokens = (self.tokens + 1).min(config::FIRST_AUDIT_BUDGET_BURST); + } + + /// Stamp the per-peer re-audit window for a launch that is ACTUALLY firing + /// (promotion). Separated from token consumption so suppression is only + /// ever recorded for a real send. + fn promote(&mut self, peer: PeerId, key_count: u32, now: Instant) { self.recent.put( peer, RecentFirstAudit { @@ -341,6 +365,303 @@ impl FirstAuditLimiter { }, ); } + + /// Test convenience: the pre-E′ atomic "a launch happened" — consume a token + /// and stamp the window in one call. Production splits these across + /// reservation and promotion; the limiter's own budget/window unit tests do + /// not model the jitter reservation and use this shorthand. + #[cfg(test)] + fn commit_launch(&mut self, peer: PeerId, key_count: u32, now: Instant) { + self.reserve_token(); + self.promote(peer, key_count, now); + } +} + +/// ADR-0004 Amendment 2 (E′ B-prefilter): whether a monetized pin is answerable +/// across the ENTIRE launch-jitter window ending at +/// `now + FIRST_AUDIT_LAUNCH_JITTER_MAX + slack`, so committing scheduling state +/// for it cannot later require aborting an out-of-window challenge. The +/// too-future bound is enforced at `now` (a jitter delay only ages a pin +/// forward, never toward the future); the too-old bound is enforced at the +/// latest possible send time. This is a conservative admission prefilter; the +/// authoritative answerability check still runs at promotion against the real +/// send-time wall clock, so A1 (no false conviction) holds regardless of how +/// jitter and the answerability margin are sized. `checked_add` overflow fails +/// closed (skip the pin). +fn quote_answerable_through_nominal_jitter(quote_ts: SystemTime, now: SystemTime) -> bool { + let Some(latest_send) = now + .checked_add(config::FIRST_AUDIT_LAUNCH_JITTER_MAX) + .and_then(|t| t.checked_add(FIRST_AUDIT_SEND_LATENCY_SLACK)) + else { + return false; + }; + quote_within_audit_window(quote_ts, now) && quote_within_audit_window(quote_ts, latest_send) +} + +/// A far-future `Instant` used to effectively DISABLE the promotion-timer +/// select arm when no reservation is outstanding. The drainer still wakes at +/// least every [`config::FIRST_AUDIT_RETRY_INTERVAL`] via its tick, so this only +/// needs to be comfortably past the next tick. +fn first_audit_far_future() -> Instant { + Instant::now() + .checked_add(Duration::from_secs(3600)) + .unwrap_or_else(Instant::now) +} + +/// ADR-0004 Amendment 2 (E′): one outstanding first-audit reservation. Holds +/// its in-flight slot and launch token from schedule time until the jitter +/// timer fires; the durable suppression (`recent` + `first_audited`) is stamped +/// only if the authoritative post-jitter answerability + cooldown checks pass at +/// promotion, so a cancelled reservation leaves NO suppression behind. +struct FirstAuditReservation { + event: MonetizedPinEvent, + ready_at: Instant, + inflight: FirstAuditInflightSlot, +} + +/// ADR-0004 Amendment 2 (E′): the drainer-owned first-audit scheduler. Owns the +/// pending queue, the dedup set, the launch limiter, the alternating lane, and +/// the single outstanding reservation. All mutation is single-threaded in the +/// drainer task; the only asynchrony is the spawned audit I/O (which just holds +/// the moved-in in-flight slot). Kept as a struct so the reserve/promote/cancel +/// state machine is unit-testable with injected clocks. +struct FirstAuditScheduler { + /// Pins already given a first audit (dedup). A pin enters only at PROMOTION + /// (a real send), never at reservation, so a cancelled reservation can be + /// re-nominated. + first_audited: LruCache<[u8; 32], ()>, + /// Newest-per-peer pending nominations not yet launched. + pending: LruCache, + /// Token bucket + per-peer re-audit window. + limiter: FirstAuditLimiter, + /// The single outstanding reservation (E′ serializes reservations so the + /// per-launch lane alternation is preserved and at most one jitter timer is + /// live). + reserved: Option, + /// Alternating launch lane, flipped on every PROMOTION (real launch). + oldest_first_lane: bool, +} + +impl FirstAuditScheduler { + fn new(now: Instant) -> Self { + let cap = NonZeroUsize::new(MAX_LAST_COMMITMENT_BY_PEER).unwrap_or(NonZeroUsize::MIN); + Self { + first_audited: LruCache::new(cap), + pending: LruCache::new(cap), + limiter: FirstAuditLimiter::new(now), + reserved: None, + oldest_first_lane: false, + } + } + + fn pending_len(&self) -> usize { + self.pending.len() + } + + fn tokens(&self) -> u32 { + self.limiter.tokens + } + + fn has_reservation(&self) -> bool { + self.reserved.is_some() + } + + /// When the outstanding reservation becomes eligible for promotion. + fn reserved_ready_at(&self) -> Option { + self.reserved.as_ref().map(|r| r.ready_at) + } + + /// The peer of the outstanding reservation, if any. + fn reserved_peer(&self) -> Option { + self.reserved.as_ref().map(|r| r.event.peer) + } + + /// Admit a monetized nomination into `pending`. Dropped as a duplicate if + /// already first-audited; queued unconditionally if it is for the currently + /// reserved peer (so a successor is retained across the reservation, never + /// window-dropped); otherwise window-screened. Newest-per-peer coalescing. + fn enqueue(&mut self, event: MonetizedPinEvent, obs: &Arc) { + if self.first_audited.contains(&event.pin) { + obs.duplicates.fetch_add(1, Ordering::Relaxed); + return; + } + let reserved_peer = self.reserved_peer(); + let is_reserved_peer = reserved_peer == Some(event.peer); + if !is_reserved_peer + && !self + .limiter + .window_allows(&event.peer, event.key_count, Instant::now()) + { + obs.window_deduped.fetch_add(1, Ordering::Relaxed); + return; + } + match queue_first_audit_event(&mut self.pending, event) { + FirstAuditQueueOutcome::Queued => { + obs.queued.fetch_add(1, Ordering::Relaxed); + } + FirstAuditQueueOutcome::Coalesced => { + obs.coalesced.fetch_add(1, Ordering::Relaxed); + } + FirstAuditQueueOutcome::CapacityEvicted { .. } => { + obs.queued.fetch_add(1, Ordering::Relaxed); + obs.capacity_evicted.fetch_add(1, Ordering::Relaxed); + } + } + } + + /// Attempt to create the single reservation from `pending` (E′ reserve). + /// Scans in the current lane order and reserves the FIRST eligible pin: + /// consumes one token, acquires one in-flight slot, and sets `ready_at = + /// mono_now + jitter`. Does NOT stamp `recent`/`first_audited` and does NOT + /// flip the lane — those happen only at promotion. Returns whether a + /// reservation was made. `cooldown` is a read-only snapshot (the + /// authoritative check-and-stamp is at promotion). + fn try_reserve( + &mut self, + mono_now: Instant, + inflight: u64, + jitter: Duration, + cooldown: &HashMap, + obs: &Arc, + ) -> bool { + if self.reserved.is_some() || self.pending.is_empty() { + return false; + } + self.limiter.refill(mono_now); + if inflight >= config::FIRST_AUDIT_MAX_INFLIGHT || self.limiter.tokens == 0 { + obs.rate_deferred_attempts.fetch_add(1, Ordering::Relaxed); + return false; + } + // MRU->LRU order; reverse for the oldest-first lane so the preferred + // end is at the front. + let mut ordered: Vec<(usize, PeerId, MonetizedPinEvent)> = self + .pending + .iter() + .enumerate() + .map(|(i, (p, e))| (i, *p, *e)) + .collect(); + self.pending.clear(); + if self.oldest_first_lane { + ordered.reverse(); + } + let mut chosen: Option = None; + let mut kept: Vec<(usize, PeerId, MonetizedPinEvent)> = Vec::new(); + for (idx, peer, event) in ordered { + if chosen.is_some() { + kept.push((idx, peer, event)); + continue; + } + if self.first_audited.contains(&event.pin) { + obs.duplicates.fetch_add(1, Ordering::Relaxed); + continue; // drop: already audited + } + if !quote_answerable_through_nominal_jitter(event.quote_ts, SystemTime::now()) { + obs.outside_answerability_window + .fetch_add(1, Ordering::Relaxed); + continue; // drop: cannot stay answerable through the jitter + } + // Window + budget + inflight. Tokens do not decrease during the + // scan (reserve happens after the loop) and `inflight` is fixed, so + // after the upfront budget gate `assess` never returns RateDeferred + // here; a defensive RateDeferred keeps the pin. + match self + .limiter + .assess(&peer, event.key_count, mono_now, inflight) + { + LimiterVerdict::WindowDeduped => { + obs.window_deduped.fetch_add(1, Ordering::Relaxed); + continue; // drop: recently first-audited, no count jump + } + LimiterVerdict::RateDeferred => { + kept.push((idx, peer, event)); + continue; // keep (defensive; unreachable after upfront gate) + } + LimiterVerdict::Admit => {} + } + if !cooldown_would_allow(cooldown, &peer, mono_now) { + obs.cooldown_deferred_attempts + .fetch_add(1, Ordering::Relaxed); + kept.push((idx, peer, event)); + continue; // keep: on shared cooldown, retry later + } + chosen = Some(event); + } + // Restore relative recency (oldest re-put first -> newest stays MRU). + kept.sort_unstable_by_key(|(idx, _, _)| std::cmp::Reverse(*idx)); + for (_, peer, event) in kept { + self.pending.put(peer, event); + } + let Some(event) = chosen else { + return false; + }; + self.limiter.reserve_token(); + let inflight_slot = FirstAuditInflightSlot::acquire(obs); + let ready_at = mono_now.checked_add(jitter).unwrap_or(mono_now); + self.reserved = Some(FirstAuditReservation { + event, + ready_at, + inflight: inflight_slot, + }); + true + } + + /// Take the outstanding reservation if its jitter has elapsed at `mono_now`. + fn take_due_reservation(&mut self, mono_now: Instant) -> Option { + if self + .reserved + .as_ref() + .is_some_and(|r| mono_now >= r.ready_at) + { + self.reserved.take() + } else { + None + } + } + + /// Authoritative promotion of a due reservation (E′). The caller holds the + /// shared cooldown write lock and passes the real send-time `wall_now` and + /// `mono_now`. Returns `Some((event, slot))` to spawn the audit on a real + /// launch, or `None` when the launch was cancelled (answerability lapsed + /// during jitter) or requeued (a concurrent gossip audit won the cooldown). + /// On both `None` paths the token is refunded, the in-flight slot released, + /// and NO suppression is recorded. + fn resolve( + &mut self, + reservation: FirstAuditReservation, + wall_now: SystemTime, + mono_now: Instant, + cooldown: &mut HashMap, + obs: &Arc, + ) -> Option<(MonetizedPinEvent, FirstAuditInflightSlot)> { + let FirstAuditReservation { + event, inflight, .. + } = reservation; + // Authoritative answerability at the REAL send time. + if !quote_within_audit_window(event.quote_ts, wall_now) { + self.limiter.refund_token(); + obs.outside_answerability_window + .fetch_add(1, Ordering::Relaxed); + drop(inflight); + return None; // cancelled; nothing stamped, nothing to roll back + } + // Authoritative shared-cooldown check-and-stamp. Losing this race to a + // concurrent gossip audit requeues the event (newest-per-peer coalesce) + // rather than dropping it. + if !cooldown_allows_audit(cooldown, &event.peer, mono_now) { + self.limiter.refund_token(); + obs.cooldown_deferred_attempts + .fetch_add(1, Ordering::Relaxed); + drop(inflight); + let _ = queue_first_audit_event(&mut self.pending, event); + return None; + } + // Promote: stamp durable suppression, flip the lane, count the launch. + self.limiter.promote(event.peer, event.key_count, mono_now); + self.first_audited.put(event.pin, ()); + self.oldest_first_lane = !self.oldest_first_lane; + obs.launched.fetch_add(1, Ordering::Relaxed); + Some((event, inflight)) + } } // --------------------------------------------------------------------------- @@ -1099,153 +1420,173 @@ impl ReplicationEngine { let observability = Arc::new(FirstAuditObservability::default()); let handle = tokio::spawn(async move { - // Bounded dedup of pins that have ALREADY been given their - // deterministic first audit. A pin is inserted ONLY when an audit is - // actually launched (never on a cooldown skip), so a pin skipped now - // can still be first-audited later. - let mut first_audited: LruCache<[u8; 32], ()> = LruCache::new( - NonZeroUsize::new(MAX_LAST_COMMITMENT_BY_PEER).unwrap_or(NonZeroUsize::MIN), - ); - // PERSISTENT pending queue: the most-recently-monetized pin per peer - // that has NOT yet been first-audited. A pin stays here until it is - // ACTUALLY first-audited (enters `first_audited`) — never removed for - // any weaker reason (e.g. a cooldown stamp), so an unaudited monetized - // pin is never silently forgotten. Newest-per-peer: a fresher pin for - // the same peer replaces the older one. Memory is bounded by an LRU: - // each entry needs a SETTLED on-chain payment, so the realistic count - // is tiny; the LRU is a pure DoS backstop that, only under an absurd - // flood, evicts the LEAST-RECENTLY-MONETIZED peer (the one most likely - // already superseded) — never the newest. - let mut pending: LruCache = LruCache::new( - NonZeroUsize::new(MAX_LAST_COMMITMENT_BY_PEER).unwrap_or(NonZeroUsize::MIN), - ); - // ADR-0004 Amendment 2: the launch limiter (token bucket + - // in-flight cap + per-peer re-audit window). This is what makes - // first-audit load independent of upload volume: payments only - // NOMINATE pins, the clock launches them. - let mut limiter = FirstAuditLimiter::new(Instant::now()); - // Launch lane (Amendment 2 anti-starvation), flipped on every - // COMMITTED LAUNCH — see the launch loop below. Consecutive - // launches therefore strictly alternate between the newest pending - // pin (freshest answerability window) and the oldest (which can - // then never be starved out of its eligibility window by a stream - // of fresher nominations, however the attacker times them). - let mut oldest_first_lane = false; - // Periodic retry tick for pending (cooldown/budget-blocked) pins. - // Created once; `Skip` so a backlog of missed ticks collapses to one. + // ADR-0004 Amendment 2 (E'): the drainer-owned first-audit + // scheduler. Payments only NOMINATE pins; the token bucket launches + // them at a fixed per-node rate, and durable suppression is stamped + // only at PROMOTION (after an authoritative post-jitter answerability + // + cooldown check), so a cancelled reservation leaves nothing behind. + let mut scheduler = FirstAuditScheduler::new(Instant::now()); + // Periodic retry tick so budget/cooldown-deferred pins get retried + // even when no new nomination arrives. `Skip` collapses a backlog. let mut tick = tokio::time::interval(config::FIRST_AUDIT_RETRY_INTERVAL); tick.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); let mut last_summary = Instant::now(); loop { - // Wake on: shutdown, a new monetized pin, OR a periodic tick so - // pending (cooldown-blocked) pins get retried once their window - // reopens even if no new pin arrives. - let drained_new = tokio::select! { + // The reservation's jitter deadline is a wake source: if a + // reservation is outstanding, sleep until it is due; otherwise + // a far-future deadline effectively disables that arm and only + // shutdown/rx/tick wake the loop. Recreated each iteration so a + // newly reserved (earlier) deadline is honoured next turn. + let promotion_due = scheduler + .reserved_ready_at() + .unwrap_or_else(first_audit_far_future); + tokio::select! { () = shutdown.cancelled() => break, event = rx.recv() => match event { Some(e) => { observability.received.fetch_add(1, Ordering::Relaxed); - // Newest-per-peer: a fresher pin replaces the older one, - // BUT only if it is not already first-audited — an - // already-audited duplicate must never overwrite an - // unaudited pending pin for the same peer (it would then - // be dropped as "done" and the unaudited pin lost). Cap - // the per-wake batch drain (FIRST_AUDIT_DRAIN_BATCH) so a - // sustained flood can't starve the audit-launch phase. - if first_audited.contains(&e.pin) { - observability.duplicates.fetch_add(1, Ordering::Relaxed); - debug!( - "First-audit scheduler: audit_trigger=first_monetized outcome=duplicate peer={} pin={} key_count={} pending={}", - e.peer, hex::encode(e.pin), e.key_count, pending.len() - ); - } else if !limiter.window_allows(&e.peer, e.key_count, Instant::now()) { - // Recently first-audited peer, no count jump: - // drop at the door so rotated pins never - // occupy pending slots or re-arm launches. - observability.window_deduped.fetch_add(1, Ordering::Relaxed); - debug!( - "First-audit scheduler: audit_trigger=first_monetized outcome=window_deduped peer={} pin={} key_count={} pending={}", - e.peer, hex::encode(e.pin), e.key_count, pending.len() - ); - } else { - match queue_first_audit_event(&mut pending, e) { - FirstAuditQueueOutcome::Queued => { - observability.queued.fetch_add(1, Ordering::Relaxed); - debug!( - "First-audit scheduler: audit_trigger=first_monetized outcome=queued peer={} pin={} key_count={} pending={}", - e.peer, hex::encode(e.pin), e.key_count, pending.len() - ); - } - FirstAuditQueueOutcome::Coalesced => { - observability.coalesced.fetch_add(1, Ordering::Relaxed); - debug!( - "First-audit scheduler: audit_trigger=first_monetized outcome=coalesced peer={} pin={} key_count={} pending={}", - e.peer, hex::encode(e.pin), e.key_count, pending.len() - ); - } - FirstAuditQueueOutcome::CapacityEvicted { peer, pin } => { - observability.queued.fetch_add(1, Ordering::Relaxed); - observability - .capacity_evicted - .fetch_add(1, Ordering::Relaxed); - debug!( - "First-audit scheduler: audit_trigger=first_monetized outcome=queued capacity_evicted=true evicted_peer={peer} evicted_pin={} replacement_peer={} replacement_pin={} pending={}", - hex::encode(pin), e.peer, hex::encode(e.pin), pending.len() - ); - } - } - } + scheduler.enqueue(e, &observability); + // Drain a bounded burst so a flood cannot starve the + // launch phase. let mut drained = 1usize; while drained < config::FIRST_AUDIT_DRAIN_BATCH { match rx.try_recv() { Ok(e) => { observability.received.fetch_add(1, Ordering::Relaxed); - if first_audited.contains(&e.pin) { - observability - .duplicates - .fetch_add(1, Ordering::Relaxed); - } else if !limiter.window_allows( - &e.peer, - e.key_count, - Instant::now(), - ) { - observability - .window_deduped - .fetch_add(1, Ordering::Relaxed); - } else { - match queue_first_audit_event(&mut pending, e) { - FirstAuditQueueOutcome::Queued => { - observability - .queued - .fetch_add(1, Ordering::Relaxed); - } - FirstAuditQueueOutcome::Coalesced => { - observability - .coalesced - .fetch_add(1, Ordering::Relaxed); - } - FirstAuditQueueOutcome::CapacityEvicted { .. } => { - observability - .queued - .fetch_add(1, Ordering::Relaxed); - observability - .capacity_evicted - .fetch_add(1, Ordering::Relaxed); - } - } - } + scheduler.enqueue(e, &observability); drained += 1; } Err(_) => break, } } - true } None => break, }, - _ = tick.tick() => false, - }; - let _ = drained_new; + _ = tick.tick() => {} + () = tokio::time::sleep_until(promotion_due.into()) => {} + } + + // 1) Promote a due reservation. Resolved after EVERY wake (not + // only when the timer arm wins) so a continuously-ready `rx` + // cannot indefinitely delay a due promotion. The authoritative + // answerability + cooldown check-and-stamp happens here under + // the shared cooldown write lock, immediately before the send. + if let Some(reservation) = scheduler.take_due_reservation(Instant::now()) { + let promoted = { + let mut cooldown = gossip_audit.cooldown.write().await; + scheduler.resolve( + reservation, + SystemTime::now(), + Instant::now(), + &mut cooldown, + &observability, + ) + }; + if let Some((event, inflight_slot)) = promoted { + debug!( + "First-audit scheduler: audit_trigger=first_monetized outcome=launched peer={} pin={} key_count={} inflight={}", + event.peer, hex::encode(event.pin), event.key_count, + observability.inflight.load(Ordering::Relaxed) + ); + let trigger = gossip_audit.clone(); + let audit_observability = Arc::clone(&observability); + tokio::spawn(async move { + // The jitter already elapsed as the reservation's + // timer; the slot is held for the audit's duration + // and released on drop (panic-safe). + let inflight_slot = inflight_slot; + let started = Instant::now(); + let credit = storage_commitment_audit::AuditCredit { + recent_provers: &trigger.recent_provers, + }; + let result = storage_commitment_audit::run_subtree_audit( + &trigger.p2p_node, + &trigger.config, + &event.peer, + event.pin, + event.key_count, + Some(&credit), + ) + .await; + let outcome = first_audit_terminal_outcome(&result); + match outcome { + FirstAuditTerminalOutcome::Passed => { + audit_observability.passed.fetch_add(1, Ordering::Relaxed); + } + FirstAuditTerminalOutcome::Timeout => { + audit_observability + .timed_out + .fetch_add(1, Ordering::Relaxed); + } + FirstAuditTerminalOutcome::Failed => { + audit_observability.failed.fetch_add(1, Ordering::Relaxed); + } + FirstAuditTerminalOutcome::BootstrapClaim => { + audit_observability + .bootstrap_claims + .fetch_add(1, Ordering::Relaxed); + } + FirstAuditTerminalOutcome::Idle => { + audit_observability.idle.fetch_add(1, Ordering::Relaxed); + } + FirstAuditTerminalOutcome::InsufficientKeys => { + audit_observability + .insufficient_keys + .fetch_add(1, Ordering::Relaxed); + } + } + drop(inflight_slot); + debug!( + "First-audit scheduler: audit_trigger=first_monetized outcome={} peer={} pin={} key_count={} elapsed_ms={} inflight={}", + outcome.as_str(), + event.peer, hex::encode(event.pin), event.key_count, + started.elapsed().as_millis(), + audit_observability.inflight.load(Ordering::Relaxed) + ); + handle_subtree_audit_result( + &result, + &trigger.p2p_node, + &trigger.sync_state, + &trigger.recent_provers, + &trigger.config, + ) + .await; + }); + } + } + + // 2) Open the single reservation from the pending queue if none + // is outstanding (read-only cooldown snapshot; the + // authoritative stamp is at promotion). Serializing + // reservations preserves per-launch lane alternation and keeps + // at most one jitter timer live. + if !scheduler.has_reservation() { + let jitter = Duration::from_millis( + rand::thread_rng().gen_range( + 0..=u64::try_from(config::FIRST_AUDIT_LAUNCH_JITTER_MAX.as_millis()) + .unwrap_or(u64::MAX), + ), + ); + let inflight = observability.inflight.load(Ordering::Relaxed); + let reserved = { + let cooldown = gossip_audit.cooldown.read().await; + scheduler.try_reserve( + Instant::now(), + inflight, + jitter, + &cooldown, + &observability, + ) + }; + if reserved { + if let Some(peer) = scheduler.reserved_peer() { + debug!( + "First-audit scheduler: audit_trigger=first_monetized outcome=reserved peer={peer} pending={}", + scheduler.pending_len() + ); + } + } + } if last_summary.elapsed() >= config::FIRST_AUDIT_SUMMARY_INTERVAL { info!( @@ -1267,263 +1608,12 @@ impl ReplicationEngine { observability.idle.load(Ordering::Relaxed), observability.insufficient_keys.load(Ordering::Relaxed), observability.outside_answerability_window.load(Ordering::Relaxed), - pending.len(), + scheduler.pending_len(), observability.inflight.load(Ordering::Relaxed), - limiter.tokens, + scheduler.tokens(), ); last_summary = Instant::now(); } - - if pending.is_empty() { - continue; - } - - // Try to launch an audit for each pending peer; keep the ones - // blocked by cooldown or budget for the next tick. Drain into a - // deque first (LruCache has no drain). `iter()` yields most- to - // least-recently-used, so the FRONT is the newest-monetized pin - // and the BACK is the oldest. - // - // ADR-0004 Amendment 2 anti-starvation lane: each launch takes - // its candidate from the front (newest: freshest answerability - // window) or the back (oldest: cannot be starved), and the lane - // flips ON EVERY COMMITTED LAUNCH — never per pass. Passes are - // triggered by ingress as well as by the retry tick, and most - // spend no token (the bucket is empty between refills), so a - // per-pass flip would let an attacker steer lane parity with the - // timing of its own nominations (barren scans) and keep every - // token-bearing launch on the newest lane. Flipping per launch - // makes CONSECUTIVE LAUNCHES strictly alternate — including the - // two launches of a full burst inside ONE pass — so an aging pin - // is always served by the next oldest-lane launch. - // - // Each entry keeps its snapshot index (0 = newest) so deferred - // entries can be re-inserted oldest-first afterwards, restoring - // the LRU's relative recency (oldest re-put first → stays the - // eviction victim) no matter which ends they were popped from. - let mut queue: VecDeque<(usize, PeerId, MonetizedPinEvent)> = pending - .iter() - .enumerate() - .map(|(i, (p, e))| (i, *p, *e)) - .collect(); - pending.clear(); - let mut deferred: Vec<(usize, PeerId, MonetizedPinEvent)> = Vec::new(); - while let Some((idx, peer, event)) = if oldest_first_lane { - queue.pop_back() - } else { - queue.pop_front() - } { - // Dedup: a pin already first-audited is dropped (done). - if first_audited.contains(&event.pin) { - observability.duplicates.fetch_add(1, Ordering::Relaxed); - continue; - } - // ADR-0004 A1 (guardrail A): only first-audit a pin whose SIGNED - // quote_ts lands inside the answerability window. With grace - // removed, auditing an out-of-window pin the responder may have - // aged out would false-convict, so a stale OR far-future/skewed - // quote is skipped. Legit first-audits fire moments after - // payment (quote_ts ≈ now); a skipped pin can still be - // gossip-lottery audited. - if !quote_within_audit_window(event.quote_ts, SystemTime::now()) { - observability - .outside_answerability_window - .fetch_add(1, Ordering::Relaxed); - debug!( - "First-audit scheduler: audit_trigger=first_monetized outcome=outside_answerability_window peer={peer} pin={} key_count={} pending={}", - hex::encode(event.pin), event.key_count, pending.len() - ); - continue; - } - // ADR-0004 Amendment 2 launch limiter: per-peer re-audit - // window (drop), then launch-rate budget + in-flight cap - // (penalty-free deferral: the pin stays pending). Assessed - // BEFORE the cooldown gate so a budget deferral never burns - // the peer's 30-min cooldown stamp; nothing is consumed - // until `commit_launch` below. - let assess_now = Instant::now(); - match limiter.assess( - &peer, - event.key_count, - assess_now, - observability.inflight.load(Ordering::Relaxed), - ) { - LimiterVerdict::WindowDeduped => { - observability.window_deduped.fetch_add(1, Ordering::Relaxed); - debug!( - "First-audit scheduler: audit_trigger=first_monetized outcome=window_deduped peer={peer} pin={} key_count={} pending={}", - hex::encode(event.pin), event.key_count, pending.len() - ); - continue; - } - LimiterVerdict::RateDeferred => { - deferred.push((idx, peer, event)); - observability - .rate_deferred_attempts - .fetch_add(1, Ordering::Relaxed); - debug!( - "First-audit scheduler: audit_trigger=first_monetized outcome=rate_deferred peer={peer} pin={} key_count={} deferred={} tokens={}", - hex::encode(event.pin), event.key_count, deferred.len(), limiter.tokens - ); - continue; - } - LimiterVerdict::Admit => {} - } - // Cooldown: if the peer's per-peer audit window is closed, keep - // this pin pending and retry on a later tick once it reopens. - // We do NOT treat "cooldown closed" as "already audited" (a - // losing gossip lottery can stamp the window without auditing), - // so the pin stays pending until it gets a REAL first audit; it - // is only ever evicted by the LRU memory backstop above, which - // drops the least-recently-monetized peer, not this newest one. - { - let now = Instant::now(); - let mut map = gossip_audit.cooldown.write().await; - if !cooldown_allows_audit(&mut map, &peer, now) { - deferred.push((idx, peer, event)); - observability - .cooldown_deferred_attempts - .fetch_add(1, Ordering::Relaxed); - debug!( - "First-audit scheduler: audit_trigger=first_monetized outcome=cooldown_deferred peer={peer} pin={} key_count={} deferred={}", - hex::encode(event.pin), event.key_count, deferred.len() - ); - continue; - } - } - // Audit is launching: consume budget, stamp the per-peer - // window, and mark the pin first-audited. All three commit - // at LAUNCH (not at completion), matching the pre-existing - // `first_audited` semantics: an unproductive launch consumes - // the peer's slot, and re-coverage comes from the count-jump - // override or the gossip lottery — never from re-launching, - // which an unresponsive peer could otherwise farm for load. - limiter.commit_launch(peer, event.key_count, assess_now); - first_audited.put(event.pin, ()); - // Flip the lane on EVERY committed launch (see the lane - // comment above): consecutive launches strictly alternate, - // including the two launches of a full burst in one pass. - oldest_first_lane = !oldest_first_lane; - // `launched` counts SCHEDULED audits (a token was spent). - // A pin that ages out during the post-jitter re-screen is - // counted separately under `outside_answerability_window` - // and sends nothing, so actual wire challenges are - // `launched - (post-jitter aborts)`; the terminal counters - // (passed/timeout/…) reconcile it exactly. - observability.launched.fetch_add(1, Ordering::Relaxed); - // Drop-guarded slot: released when the audit task finishes, - // panics, or is cancelled — the in-flight cap can never - // wedge shut on a leaked slot. - let inflight_slot = FirstAuditInflightSlot::acquire(&observability); - debug!( - "First-audit scheduler: audit_trigger=first_monetized outcome=launched peer={peer} pin={} key_count={} pending={} inflight={}", - hex::encode(event.pin), event.key_count, pending.len(), - observability.inflight.load(Ordering::Relaxed) - ); - let trigger = gossip_audit.clone(); - let audit_observability = Arc::clone(&observability); - tokio::spawn(async move { - let inflight_slot = inflight_slot; - // ADR-0004 Amendment 2: jitter the send. Every storer of - // a chunk verifies the same payment at the same instant; - // unjittered, the whole close group would challenge the - // paid peer simultaneously and trip its per-peer - // responder admission cap (drops that auditors then - // record as Timeout). The in-flight slot is already - // held, so jittered launches cannot pile up beyond the - // cap either. - let jitter_ms = rand::thread_rng().gen_range( - 0..=u64::try_from(config::FIRST_AUDIT_LAUNCH_JITTER_MAX.as_millis()) - .unwrap_or(u64::MAX), - ); - tokio::time::sleep(Duration::from_millis(jitter_ms)).await; - // Re-screen answerability AFTER the jitter sleep so the - // no-false-conviction invariant (ADR-0004 A1) holds by - // construction, independent of how the skew margin and - // the jitter are sized relative to each other. A pin - // that aged out during the sleep is skipped with no - // consequence for the peer (slot released by the drop - // guard; pin stays first_audited; the peer's next - // settled payment re-nominates it). - if !quote_within_audit_window(event.quote_ts, SystemTime::now()) { - audit_observability - .outside_answerability_window - .fetch_add(1, Ordering::Relaxed); - debug!( - "First-audit scheduler: audit_trigger=first_monetized outcome=outside_answerability_window post_jitter=true peer={} pin={} key_count={}", - event.peer, hex::encode(event.pin), event.key_count - ); - drop(inflight_slot); - return; - } - let started = Instant::now(); - let credit = storage_commitment_audit::AuditCredit { - recent_provers: &trigger.recent_provers, - }; - let result = storage_commitment_audit::run_subtree_audit( - &trigger.p2p_node, - &trigger.config, - &event.peer, - event.pin, - event.key_count, - Some(&credit), - ) - .await; - let outcome = first_audit_terminal_outcome(&result); - match outcome { - FirstAuditTerminalOutcome::Passed => { - audit_observability.passed.fetch_add(1, Ordering::Relaxed); - } - FirstAuditTerminalOutcome::Timeout => { - audit_observability - .timed_out - .fetch_add(1, Ordering::Relaxed); - } - FirstAuditTerminalOutcome::Failed => { - audit_observability.failed.fetch_add(1, Ordering::Relaxed); - } - FirstAuditTerminalOutcome::BootstrapClaim => { - audit_observability - .bootstrap_claims - .fetch_add(1, Ordering::Relaxed); - } - FirstAuditTerminalOutcome::Idle => { - audit_observability.idle.fetch_add(1, Ordering::Relaxed); - } - FirstAuditTerminalOutcome::InsufficientKeys => { - audit_observability - .insufficient_keys - .fetch_add(1, Ordering::Relaxed); - } - } - drop(inflight_slot); - debug!( - "First-audit scheduler: audit_trigger=first_monetized outcome={} peer={} pin={} key_count={} elapsed_ms={} inflight={}", - outcome.as_str(), - event.peer, hex::encode(event.pin), event.key_count, - started.elapsed().as_millis(), - audit_observability.inflight.load(Ordering::Relaxed) - ); - handle_subtree_audit_result( - &result, - &trigger.p2p_node, - &trigger.sync_state, - &trigger.recent_provers, - &trigger.config, - ) - .await; - }); - } - // Re-insert deferred entries oldest-first so the LRU's relative - // recency is restored (oldest re-put first → stays the eviction - // victim; newest stays most-recently-used). Entries were popped - // from BOTH ends as the lane alternated, so sort by the snapshot - // index (0 = newest) descending rather than relying on the - // collection order. - deferred.sort_unstable_by_key(|(idx, _, _)| std::cmp::Reverse(*idx)); - for (_, peer, event) in deferred { - pending.put(peer, event); - } } debug!("First-audit drainer shut down"); }); @@ -5174,6 +5264,18 @@ fn cooldown_allows_audit(map: &mut HashMap, peer: &PeerId, now: true } +/// Read-only companion to [`cooldown_allows_audit`]: whether `peer` is OUTSIDE +/// its cooldown at `now`, WITHOUT stamping. Used by the first-audit reserve gate +/// (ADR-0004 Amendment 2 E′) to avoid reserving a peer that a recent audit +/// already covered; the authoritative check-and-stamp still runs at promotion, +/// so this is only an optimization and never the security boundary. +fn cooldown_would_allow(map: &HashMap, peer: &PeerId, now: Instant) -> bool { + let cooldown = Duration::from_secs(config::AUDIT_ON_GOSSIP_COOLDOWN_SECS); + map.get(peer).map_or(true, |&last| { + now.saturating_duration_since(last) >= cooldown + }) +} + /// The gossip-audit launch decision in ONE place so the ordering is shared /// between production and its test (ADR-0002 "occasional surprise exams"). /// @@ -5787,17 +5889,22 @@ mod tests { audit_failure_revokes_holder_credit, audit_launch_decision, config, cooldown_allows_audit, first_audit_count_jump, first_audit_terminal_outcome, first_failed_key_label, fresh_offer_payment_context, paid_notify_payment_context, queue_first_audit_event, - quote_within_audit_window, FirstAuditLimiter, FirstAuditQueueOutcome, - FirstAuditTerminalOutcome, LimiterVerdict, MonetizedPinEvent, MONETIZED_AUDIT_SKEW_MARGIN, + quote_answerable_through_nominal_jitter, quote_within_audit_window, FirstAuditLimiter, + FirstAuditObservability, FirstAuditQueueOutcome, FirstAuditScheduler, + FirstAuditTerminalOutcome, LimiterVerdict, MonetizedPinEvent, + FIRST_AUDIT_SEND_LATENCY_SLACK, MONETIZED_AUDIT_SKEW_MARGIN, }; use crate::payment::VerificationContext; use crate::replication::audit::AuditTickResult; + use crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL; use crate::replication::recent_provers::RecentProvers; use crate::replication::types::{AuditFailureReason, AuditFailureSummary, FailureEvidence}; use lru::LruCache; use saorsa_core::identity::PeerId; use std::collections::HashMap; use std::num::NonZeroUsize; + use std::sync::atomic::Ordering; + use std::sync::Arc; use std::time::Duration; use std::time::Instant; use std::time::SystemTime; @@ -6089,88 +6196,252 @@ mod tests { ); } - /// ADR-0004 Amendment 2 anti-starvation invariant: the launch lane must - /// advance on every COMMITTED LAUNCH — never per pass, and never once per - /// pass regardless of how many tokens that pass spends. - /// - /// Two ways to get this wrong, both of which let fresh decoys crowd out an - /// aging pin, and both of which this test rejects: - /// - /// 1. *Flip per pass.* Passes are triggered by ingress as well as by the - /// retry tick, and most spend no token (the bucket is empty between - /// refills), so an attacker could inject nominations to force barren - /// scans, steer lane parity, and keep every token-bearing launch on the - /// newest lane. - /// 2. *Flip once per pass that launched.* A full burst spends two tokens - /// inside ONE pass; a single flip afterwards yields `newest, newest, - /// oldest` instead of a strict alternation. - /// - /// This models the drainer's real loop — a deque popped from the front - /// (newest) or the back (oldest) with the lane flipped at each launch — - /// and asserts strict alternation across barren scans AND multi-token - /// bursts. + /// ADR-0004 Amendment 2 (E'): the B horizon prefilter rejects a quote that + /// is answerable now but would age out of the answerability window during + /// the launch jitter, so scheduling state is only ever committed for a pin + /// that can still be challenged when it actually sends. #[test] - fn first_audit_lane_alternates_per_launch_across_barren_scans_and_bursts() { - /// One drainer pass over `queue_len` pending pins with `tokens` - /// available. Returns the lane each launch used, in order. - fn run_pass(lane_oldest: &mut bool, tokens: &mut u32, queue_len: usize) -> Vec { - let mut launched = Vec::new(); - let mut remaining = queue_len; - while remaining > 0 && *tokens > 0 { - // The lane picks the end of the deque; the launch flips it. - launched.push(*lane_oldest); - *lane_oldest = !*lane_oldest; - *tokens -= 1; - remaining -= 1; - } - launched - } + fn first_audit_horizon_prefilter_boundary() { + let now = SystemTime::now(); + // C = the too-old cutoff; H = the worst-case send horizon. + let c = GOSSIP_ANSWERABILITY_TTL.saturating_sub(MONETIZED_AUDIT_SKEW_MARGIN); + let h = config::FIRST_AUDIT_LAUNCH_JITTER_MAX + FIRST_AUDIT_SEND_LATENCY_SLACK; + // A quote whose age is exactly C at `now + H` (in the past, since C > H). + let boundary = now + .checked_add(h) + .and_then(|t| t.checked_sub(c)) + .expect("boundary time"); + + // In window at `now` (age = C - H < C)... + assert!(quote_within_audit_window(boundary, now)); + // ...but the horizon prefilter rejects it (age == C at now + H). + assert!(!quote_answerable_through_nominal_jitter(boundary, now)); + + // One nanosecond newer stays answerable through the horizon; one older + // does not. + let newer = boundary + .checked_add(Duration::from_nanos(1)) + .expect("newer"); + let older = boundary + .checked_sub(Duration::from_nanos(1)) + .expect("older"); + assert!(quote_answerable_through_nominal_jitter(newer, now)); + assert!(!quote_answerable_through_nominal_jitter(older, now)); + + // A quote too far in the FUTURE is rejected at `now`, independent of the + // horizon. + let future = now + .checked_add(MONETIZED_AUDIT_SKEW_MARGIN) + .and_then(|t| t.checked_add(Duration::from_secs(60))) + .expect("future"); + assert!(!quote_answerable_through_nominal_jitter(future, now)); + } + + /// ADR-0004 Amendment 2 (E'): a reservation whose AUTHORITATIVE post-jitter + /// answerability check fails at promotion is fully state-neutral — it stamps + /// no `first_audited`, no per-peer window, refunds its token, releases its + /// in-flight slot, does not flip the lane, and does not count a launch — and + /// a same-peer, same-count successor enqueued DURING the reservation is + /// retained and becomes the next reservation after the cancel. This is the + /// exact hole the reviewer flagged: suppression must never outlive a launch + /// that did not send. + #[test] + #[allow(clippy::too_many_lines)] + fn first_audit_answerability_cancel_is_state_neutral_and_retains_successor() { + let obs = Arc::new(FirstAuditObservability::default()); + let mono = Instant::now(); + let mut scheduler = FirstAuditScheduler::new(mono); + + // A pre-existing window sentinel for an UNRELATED peer must survive the + // cancel byte-for-byte (cancel never touches `recent`). + let sentinel_peer = test_peer(0xAA); + scheduler.limiter.promote(sentinel_peer, 500, mono); + let sentinel_before = scheduler + .limiter + .recent + .peek(&sentinel_peer) + .copied() + .expect("sentinel present"); - let mut lane_oldest = false; - let mut tokens = 0u32; - let mut lanes = Vec::new(); - - // Attacker forces barren scans between refills, trying to land every - // token on the newest lane. A barren pass launches nothing, so it - // cannot advance the lane. - for _ in 0..4 { - for _ in 0..5 { - assert!( - run_pass(&mut lane_oldest, &mut tokens, 8).is_empty(), - "a pass with no tokens must not launch and must not flip" - ); - } - tokens += 1; - lanes.extend(run_pass(&mut lane_oldest, &mut tokens, 8)); - } + let peer = test_peer(1); + let a = MonetizedPinEvent { + peer, + pin: [1; 32], + key_count: 100, + quote_ts: SystemTime::now(), // fresh: passes the horizon prefilter now + }; + scheduler.enqueue(a, &obs); - // A full burst spends BOTH tokens in a single pass: those two launches - // must still alternate with each other. - tokens += config::FIRST_AUDIT_BUDGET_BURST; - let burst = run_pass(&mut lane_oldest, &mut tokens, 8); + let tokens_before = scheduler.tokens(); + let lane_before = scheduler.oldest_first_lane; + let launched_before = obs.launched.load(Ordering::Relaxed); + let mut cooldown: HashMap = HashMap::new(); + + // Reserve A (jitter 0 so it is immediately due). + let inflight0 = obs.inflight.load(Ordering::Relaxed); + assert!(scheduler.try_reserve(mono, inflight0, Duration::ZERO, &cooldown, &obs)); + assert_eq!(scheduler.reserved_peer(), Some(peer)); assert_eq!( - burst.len(), - config::FIRST_AUDIT_BUDGET_BURST as usize, - "a full burst must launch every token it holds" + scheduler.tokens(), + tokens_before - 1, + "reserve consumes a token" ); - lanes.extend(burst); - - // ...and a later single-token pass continues the same alternation. - tokens += 1; - lanes.extend(run_pass(&mut lane_oldest, &mut tokens, 8)); - - for (i, lane) in lanes.iter().enumerate() { - assert_eq!( - *lane, - i % 2 == 1, - "launch {i} landed on the wrong lane: consecutive launches must \ - strictly alternate across barren scans AND multi-token bursts" - ); + assert_eq!( + obs.inflight.load(Ordering::Relaxed), + 1, + "reserve holds a slot" + ); + + // A same-peer, same-count successor arrives DURING the reservation. It + // must be retained (bypasses the window for the reserved peer) and must + // NOT create a second reservation. + let b = MonetizedPinEvent { + peer, + pin: [2; 32], + key_count: 100, + quote_ts: SystemTime::now(), + }; + scheduler.enqueue(b, &obs); + assert_eq!(scheduler.pending_len(), 1, "successor retained in pending"); + assert!( + !scheduler.try_reserve( + mono, + obs.inflight.load(Ordering::Relaxed), + Duration::ZERO, + &cooldown, + &obs + ), + "no second reservation while one is outstanding" + ); + + // Resolve A with an injected wall time PAST A's answerability cutoff. + let reservation = scheduler + .take_due_reservation(mono) + .expect("A is due at jitter 0"); + let wall_fail = a + .quote_ts + .checked_add(GOSSIP_ANSWERABILITY_TTL) + .and_then(|t| t.checked_add(Duration::from_secs(1))) + .expect("past-cutoff wall time"); + let promoted = scheduler.resolve(reservation, wall_fail, mono, &mut cooldown, &obs); + assert!( + promoted.is_none(), + "answerability lapsed -> cancelled, not promoted" + ); + + // State-neutral cancel. + assert!(scheduler.first_audited.is_empty(), "no pin marked audited"); + assert!( + scheduler.limiter.recent.peek(&peer).is_none(), + "cancel stamps no per-peer window" + ); + assert_eq!( + scheduler.limiter.recent.peek(&sentinel_peer).copied(), + Some(sentinel_before), + "unrelated window sentinel untouched" + ); + assert!(!cooldown.contains_key(&peer), "cancel stamps no cooldown"); + assert_eq!( + scheduler.tokens(), + tokens_before, + "token refunded on cancel" + ); + assert_eq!( + obs.inflight.load(Ordering::Relaxed), + 0, + "in-flight slot released" + ); + assert_eq!(scheduler.oldest_first_lane, lane_before, "lane not flipped"); + assert_eq!( + obs.launched.load(Ordering::Relaxed), + launched_before, + "no launch counted" + ); + assert_eq!( + obs.outside_answerability_window.load(Ordering::Relaxed), + 1, + "the cancel reason is recorded" + ); + + // The successor is still pending and is now fully schedulable: a fresh + // reserve makes B the next reservation (proving eligibility, not merely + // that the limiter would admit it). + assert_eq!( + scheduler.pending_len(), + 1, + "successor still pending after cancel" + ); + assert!(scheduler.reserved.is_none()); + assert!(scheduler.try_reserve( + mono, + obs.inflight.load(Ordering::Relaxed), + Duration::ZERO, + &cooldown, + &obs + )); + assert_eq!( + scheduler.reserved_peer(), + Some(peer), + "the retained successor becomes the next reservation" + ); + } + + /// ADR-0004 Amendment 2 (E'): consecutive PROMOTIONS strictly alternate the + /// launch lane, driven through the real scheduler (reserve -> resolve -> + /// promote), so a stream of fresh nominations cannot keep every launch on + /// the newest lane and starve the oldest. + #[test] + fn first_audit_lane_alternates_across_promotions() { + let obs = Arc::new(FirstAuditObservability::default()); + let mono = Instant::now(); + let mut scheduler = FirstAuditScheduler::new(mono); + let mut cooldown: HashMap = HashMap::new(); + + // Two distinct peers; the newest-inserted is the MRU (newest lane end). + let oldest_peer = test_peer(1); + let newest_peer = test_peer(2); + scheduler.enqueue( + MonetizedPinEvent { + peer: oldest_peer, + pin: [1; 32], + key_count: 100, + quote_ts: SystemTime::now(), + }, + &obs, + ); + scheduler.enqueue( + MonetizedPinEvent { + peer: newest_peer, + pin: [2; 32], + key_count: 100, + quote_ts: SystemTime::now(), + }, + &obs, + ); + + let mut launched_peers = Vec::new(); + for _ in 0..2 { + assert!(scheduler.try_reserve( + mono, + obs.inflight.load(Ordering::Relaxed), + Duration::ZERO, + &cooldown, + &obs + )); + let reservation = scheduler.take_due_reservation(mono).expect("due"); + // Answerable wall time == the quote's own time (age 0). + let (event, _slot) = scheduler + .resolve(reservation, SystemTime::now(), mono, &mut cooldown, &obs) + .expect("fresh in-window quote promotes"); + launched_peers.push(event.peer); } + + // First launch takes the newest lane (lane starts false = newest), the + // second takes the oldest lane: strict alternation. assert_eq!( - lanes.iter().filter(|oldest| **oldest).count(), - lanes.len() / 2, - "half of all launches must serve the oldest lane" + launched_peers, + vec![newest_peer, oldest_peer], + "consecutive promotions alternate newest-then-oldest lane" ); } From 535f25dcc5e9e022ac2990f41e178231cb9a1108 Mon Sep 17 00:00:00 2001 From: grumbach Date: Wed, 15 Jul 2026 13:35:14 +0900 Subject: [PATCH 09/21] fix(replication): cooldown-race requeue keeps the newer same-peer nomination Codex review: resolving a reservation that loses the shared-cooldown race requeued the reserved event unconditionally, which could overwrite a newer same-peer successor (a count jump) already pending, losing coverage. Requeue only when no same-peer entry is pending; the successor arrived later so it is the newer nomination. Adds a regression test. --- src/replication/mod.rs | 65 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 62 insertions(+), 3 deletions(-) diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 42520546..40e35a60 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -645,14 +645,20 @@ impl FirstAuditScheduler { return None; // cancelled; nothing stamped, nothing to roll back } // Authoritative shared-cooldown check-and-stamp. Losing this race to a - // concurrent gossip audit requeues the event (newest-per-peer coalesce) - // rather than dropping it. + // concurrent gossip audit requeues the reserved event rather than + // dropping it — BUT only if no same-peer successor is already pending: a + // successor necessarily arrived AFTER this reservation, so it is the + // newer nomination and must not be overwritten by the older reserved + // event (`queue_first_audit_event` would replace it). `contains` does + // not disturb LRU recency. if !cooldown_allows_audit(cooldown, &event.peer, mono_now) { self.limiter.refund_token(); obs.cooldown_deferred_attempts .fetch_add(1, Ordering::Relaxed); drop(inflight); - let _ = queue_first_audit_event(&mut self.pending, event); + if !self.pending.contains(&event.peer) { + let _ = queue_first_audit_event(&mut self.pending, event); + } return None; } // Promote: stamp durable suppression, flip the lane, count the launch. @@ -6386,6 +6392,59 @@ mod tests { ); } + /// ADR-0004 Amendment 2 (E'): when a promotion loses the shared-cooldown + /// race, the reserved event is requeued ONLY if no same-peer successor is + /// already pending. A successor arrived after the reservation, so it is the + /// newer nomination (e.g. a count jump) and must not be overwritten by the + /// older reserved event. + #[test] + fn first_audit_cooldown_race_requeue_preserves_newer_successor() { + let obs = Arc::new(FirstAuditObservability::default()); + let mono = Instant::now(); + let mut scheduler = FirstAuditScheduler::new(mono); + let peer = test_peer(1); + + // Reserve A. + let a = MonetizedPinEvent { + peer, + pin: [1; 32], + key_count: 100, + quote_ts: SystemTime::now(), + }; + scheduler.enqueue(a, &obs); + let cooldown_reserve: HashMap = HashMap::new(); + assert!(scheduler.try_reserve(mono, 0, Duration::ZERO, &cooldown_reserve, &obs)); + + // A newer same-peer successor B (a count jump) arrives during the + // reservation and is retained. + let b = MonetizedPinEvent { + peer, + pin: [2; 32], + key_count: 400, + quote_ts: SystemTime::now(), + }; + scheduler.enqueue(b, &obs); + assert_eq!(scheduler.pending_len(), 1); + + // Resolve A: answerability PASSES (fresh quote) but the shared cooldown + // is already stamped for the peer, so promotion loses the race. + let reservation = scheduler.take_due_reservation(mono).expect("due"); + let mut cooldown: HashMap = HashMap::new(); + cooldown.insert(peer, mono); // freshly on cooldown + let promoted = scheduler.resolve(reservation, SystemTime::now(), mono, &mut cooldown, &obs); + assert!(promoted.is_none(), "cooldown race -> not promoted"); + + // B (newer) is preserved; A did NOT overwrite it. + assert_eq!(scheduler.pending_len(), 1, "still exactly one pending"); + assert_eq!( + scheduler.pending.peek(&peer).map(|e| e.pin), + Some([2; 32]), + "the newer successor B is retained, not the older reserved A" + ); + assert!(scheduler.first_audited.is_empty()); + assert!(scheduler.limiter.recent.peek(&peer).is_none()); + } + /// ADR-0004 Amendment 2 (E'): consecutive PROMOTIONS strictly alternate the /// launch lane, driven through the real scheduler (reserve -> resolve -> /// promote), so a stream of fresh nominations cannot keep every launch on From 8247399fad5dd961fa6aea0f65dde38f2b8d7308 Mon Sep 17 00:00:00 2001 From: grumbach Date: Wed, 15 Jul 2026 13:50:12 +0900 Subject: [PATCH 10/21] fix(ci): no-logging dead_code on observability getters; windows time granularity in test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - pending_len/tokens are used only by the scheduler summary log and unit tests, both absent from cargo build --release --no-default-features, so they tripped dead_code under RUSTFLAGS=-D warnings. Gate the lint the same way the crate already gates logging-only unused items. - first_audit_horizon_prefilter_boundary used a 1ns epsilon; Windows SystemTime has 100ns (FILETIME) granularity, so the nanosecond step rounded to the same instant and the boundary assertion failed there. Use 1µs, exact on every platform. --- src/replication/mod.rs | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 40e35a60..19955db0 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -454,10 +454,16 @@ impl FirstAuditScheduler { } } + /// Observability getter: used by the scheduler summary log and unit tests, + /// both absent from a release `--no-default-features` build, so it is dead + /// only in that configuration. + #[cfg_attr(not(feature = "logging"), allow(dead_code))] fn pending_len(&self) -> usize { self.pending.len() } + /// Observability getter (see [`Self::pending_len`]). + #[cfg_attr(not(feature = "logging"), allow(dead_code))] fn tokens(&self) -> u32 { self.limiter.tokens } @@ -6223,13 +6229,14 @@ mod tests { // ...but the horizon prefilter rejects it (age == C at now + H). assert!(!quote_answerable_through_nominal_jitter(boundary, now)); - // One nanosecond newer stays answerable through the horizon; one older - // does not. + // A hair newer stays answerable through the horizon; a hair older does + // not. Use 1µs (not 1ns): Windows `SystemTime` has 100ns granularity, so + // a nanosecond step would round to the same instant there. let newer = boundary - .checked_add(Duration::from_nanos(1)) + .checked_add(Duration::from_micros(1)) .expect("newer"); let older = boundary - .checked_sub(Duration::from_nanos(1)) + .checked_sub(Duration::from_micros(1)) .expect("older"); assert!(quote_answerable_through_nominal_jitter(newer, now)); assert!(!quote_answerable_through_nominal_jitter(older, now)); From 81ec24698f1d3d55bd3f47431ab67c20b423dd95 Mon Sep 17 00:00:00 2001 From: grumbach Date: Wed, 15 Jul 2026 15:26:35 +0900 Subject: [PATCH 11/21] fix(replication): highest-count-per-peer coalescing so a cheaper pin cannot erase an inflated one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Re-review blocker (dirvine): per-peer pending coalescing replaced same-peer work by arrival order only, so a peer could suppress an inflated (audit-worthy) sidecar-only commitment for the cost of one cheaper same-peer settlement — the count-jump override only compares against the last AUDITED count and sidecar- only pins have no gossip-lottery backstop. Coalescing is now highest-count-per-peer (newest on an equal-count tie): a strictly-lower-count nomination never displaces a higher-count pending pin, and a suppressed lower nomination leaves the retained pin's LRU recency untouched (so low-count floods cannot steer the lane). The same rule governs the cooldown-race requeue of a reserved event (incoming_is_newer=false, so a higher-count reserved event beats a lower successor), which now also counts a capacity eviction in the funnel (previously unobserved). ADR-0004 Amendment 2 restated (highest-count coalescing + accepted residual: a single per-peer slot may drop the lower fallback if the retained higher pin later ages out). Non-blocking notes documented: single-node tied-median nominates only the first settled candidate (sidecar-only extras are best- effort residual; gossiped extras keep the lottery); the re-audit window is stamped at launch not outcome (Transient/silence is not auto-relaunched) — both to be quantified in the matched staging run. Tests: coalescing keeps highest count + exposes eviction; pending high vs later lower; reserved high vs cooldown-race lower successor. 702 lib tests + cfd green. --- ...ADR-0004-commitment-bound-quote-pricing.md | 46 ++- src/payment/verifier.rs | 6 + src/replication/mod.rs | 301 +++++++++++++++--- 3 files changed, 303 insertions(+), 50 deletions(-) diff --git a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md index 197b7e92..fc485ea1 100644 --- a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md +++ b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md @@ -371,9 +371,24 @@ the first hour), an in-flight cap (2), and a uniform payment at the same instant, do not challenge the paid peer simultaneously. Fleet-wide first-audit pressure is therefore `nodes x refill-rate`, independent of upload volume. Budget deferral is penalty-free: a deferred pin stays pending -(newest-per-peer, bounded) and launches when tokens allow; only audits that +(bounded, one entry per peer) and launches when tokens allow; only audits that actually launch have consequences. +**Pending coalescing keeps the highest count per peer, not merely the newest.** +The per-peer pending slot retains the pin that most needs auditing — the +HIGHEST committed key count (newest on an equal-count tie). A strictly-lower- +count nomination for a peer never displaces a higher-count pending pin, and a +suppressed lower nomination does not disturb the retained pin's queue position. +Without this rule a peer could erase an inflated (audit-worthy) commitment for +the cost of one cheaper same-peer settlement, since the count-jump override +only compares against the last AUDITED count and a sidecar-only inflated pin +has no gossip-lottery backstop. The same rule governs the requeue of a +reservation that loses the shared-cooldown race. Accepted residual: if the +retained higher pin later ages out of the answerability window (un-auditable +once aged, grace removed), the single per-peer slot has already discarded the +lower fallback — a small coverage reduction versus keeping both, not a revival +of the lower-count-displacement bypass. + **Per-peer re-audit window survives pin rotation.** After a first audit launches at a peer, further nominations for that peer are dropped for 2 hours (inside the 3h answerability TTL) — unless the new pin's committed key count @@ -399,11 +414,30 @@ measurable in production. **Every pipeline stage is bounded, and the invariants hold by construction.** The verifier-to-drainer nomination channel is bounded (producers `try_send` -and drop on full — penalty-free, lottery-covered), the pending set is a -bounded newest-per-peer LRU, and launches are token-bucketed. The A1 -answerability screen runs both before launch selection and again after the -jitter sleep, so a pin can never be challenged outside its window regardless -of how deferral time, jitter, and the skew margin compose. +and drop on full — penalty-free), the pending set is a bounded +highest-count-per-peer LRU, and launches are token-bucketed. Durable +suppression (the first-audited pin dedup, the per-peer re-audit window, the +shared cooldown stamp, the lane flip, and the launch count) is committed ONLY +at promotion, after an authoritative answerability + cooldown check-and-stamp +taken immediately before the wire challenge; a reservation cancelled during +its jitter refunds its token, releases its in-flight slot, and leaves no +suppression behind. The answerability screen runs both as a schedule-time +prefilter (over the whole jitter horizon) and authoritatively at promotion, so +a pin can never be challenged outside its window regardless of how deferral +time, jitter, and the skew margin compose. + +**Nomination scope and known coverage limits (measured in staging).** (a) On +the single-node path only the FIRST settlement-verified median candidate is +nominated; if a client settles several tied-median candidates, gossiped extras +retain the ADR-0002 lottery but sidecar-only extra settled pins do not and are +an accepted best-effort residual. The merkle path nominates every +contract-paid index. (b) The per-peer re-audit window is stamped at LAUNCH, not +at a passing outcome: a peer that answers with a `Transient`/silence reaches +the ADR-0002 timeout lane and is not automatically re-launched by this +scheduler (automatic retries would change the load bound and need their own +policy). Both effects reduce coverage without weakening the price ceiling and +are to be quantified against the terminal-outcome counters in the matched +staging run. **Accepted residuals.** (1) *Budget-exhaustion starvation, mitigated by alternating lanes:* an attacker can try to keep observers' launch budgets diff --git a/src/payment/verifier.rs b/src/payment/verifier.rs index 4810ccd0..143366bc 100644 --- a/src/payment/verifier.rs +++ b/src/payment/verifier.rs @@ -1028,6 +1028,12 @@ impl PaymentVerifier { { Ok(settled_amount) => { verified_paid_quote = Some((paid_price, settled_amount)); + // First settlement-verified median candidate wins the paid + // slot and the sole first-audit nomination. If a client + // settled several TIED-median candidates, only this one is + // first-audited; gossiped extras keep the ADR-0002 lottery, + // sidecar-only extras are an accepted best-effort residual + // (ADR-0004 Amendment 2). The honest client pays one. paid_peer = Some(candidate_peer); break; } diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 19955db0..f7c89b47 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -160,22 +160,54 @@ fn first_audit_terminal_outcome(result: &AuditTickResult) -> FirstAuditTerminalO #[derive(Debug, Clone, Copy, PartialEq, Eq)] enum FirstAuditQueueOutcome { + /// New peer inserted with free capacity. Queued, + /// Collapsed with an existing same-peer entry; the incoming won (higher + /// count, or equal count and newer) and replaced it. Coalesced, + /// Collapsed with an existing same-peer entry; the incoming LOST (strictly + /// lower count) and was dropped, leaving the higher-count pin in place. + SuppressedLower, + /// A DIFFERENT peer's entry was evicted by the bounded LRU to make room. CapacityEvicted { peer: PeerId, pin: [u8; 32] }, } -/// Insert newest-per-peer work while exposing the bounded LRU's otherwise -/// silent capacity eviction. This preserves `LruCache::put` semantics exactly. -fn queue_first_audit_event( +/// Coalesce a monetized nomination into the per-peer pending queue with a +/// SECURITY-AWARE rule (ADR-0004 Amendment 2): keep the pin that most needs +/// auditing — the HIGHEST committed key count for that peer, newest on a tie. +/// +/// A strictly-lower-count incoming must NOT displace a higher-count pending pin, +/// otherwise a peer can erase an inflated (audit-worthy) commitment by simply +/// monetizing a cheaper one right after — and a sidecar-only inflated pin has no +/// gossip-lottery backstop. When the incoming loses, the retained entry's LRU +/// recency is left UNTOUCHED (via `peek`), so a flood of low-count nominations +/// cannot promote the retained pin's lane position. +/// +/// `incoming_is_newer` distinguishes ordinary enqueue (the incoming arrived +/// last, so it wins an equal-count tie for freshness) from a cooldown-race +/// requeue of an older reserved event (the pending successor is newer, so it +/// wins the tie). +fn coalesce_first_audit_event( pending: &mut LruCache, - event: MonetizedPinEvent, + incoming: MonetizedPinEvent, + incoming_is_newer: bool, ) -> FirstAuditQueueOutcome { - match pending.push(event.peer, event) { - None => FirstAuditQueueOutcome::Queued, - Some((replaced_peer, _)) if replaced_peer == event.peer => { - FirstAuditQueueOutcome::Coalesced + if let Some(existing) = pending.peek(&incoming.peer) { + let incoming_wins = incoming.key_count > existing.key_count + || (incoming.key_count == existing.key_count && incoming_is_newer); + if !incoming_wins { + // Retain the higher/equal-newer existing pin; do NOT touch its + // recency. + return FirstAuditQueueOutcome::SuppressedLower; } + // Same-peer replace: `push` updates the value and bumps MRU. Replacing + // an existing key never evicts a different peer. + let _ = pending.push(incoming.peer, incoming); + return FirstAuditQueueOutcome::Coalesced; + } + // No same-peer entry: a `push` at capacity evicts the LRU (a DIFFERENT peer). + match pending.push(incoming.peer, incoming) { + None => FirstAuditQueueOutcome::Queued, Some((evicted_peer, evicted)) => FirstAuditQueueOutcome::CapacityEvicted { peer: evicted_peer, pin: evicted.pin, @@ -483,9 +515,11 @@ impl FirstAuditScheduler { } /// Admit a monetized nomination into `pending`. Dropped as a duplicate if - /// already first-audited; queued unconditionally if it is for the currently + /// already first-audited; the window screen is bypassed for the currently /// reserved peer (so a successor is retained across the reservation, never - /// window-dropped); otherwise window-screened. Newest-per-peer coalescing. + /// window-dropped); otherwise window-screened. Coalescing is + /// highest-count-per-peer (newest on a tie) — a lower-count successor never + /// displaces a higher-count pending pin. fn enqueue(&mut self, event: MonetizedPinEvent, obs: &Arc) { if self.first_audited.contains(&event.pin) { obs.duplicates.fetch_add(1, Ordering::Relaxed); @@ -501,11 +535,13 @@ impl FirstAuditScheduler { obs.window_deduped.fetch_add(1, Ordering::Relaxed); return; } - match queue_first_audit_event(&mut self.pending, event) { + // Ordinary enqueue: the incoming arrived last, so it wins an equal-count + // tie. + match coalesce_first_audit_event(&mut self.pending, event, true) { FirstAuditQueueOutcome::Queued => { obs.queued.fetch_add(1, Ordering::Relaxed); } - FirstAuditQueueOutcome::Coalesced => { + FirstAuditQueueOutcome::Coalesced | FirstAuditQueueOutcome::SuppressedLower => { obs.coalesced.fetch_add(1, Ordering::Relaxed); } FirstAuditQueueOutcome::CapacityEvicted { .. } => { @@ -651,19 +687,23 @@ impl FirstAuditScheduler { return None; // cancelled; nothing stamped, nothing to roll back } // Authoritative shared-cooldown check-and-stamp. Losing this race to a - // concurrent gossip audit requeues the reserved event rather than - // dropping it — BUT only if no same-peer successor is already pending: a - // successor necessarily arrived AFTER this reservation, so it is the - // newer nomination and must not be overwritten by the older reserved - // event (`queue_first_audit_event` would replace it). `contains` does - // not disturb LRU recency. + // concurrent gossip audit requeues the reserved event through the SAME + // security-aware coalescing: the reserved event is OLDER than any + // same-peer successor (`incoming_is_newer = false`), so a higher-count + // reserved event still wins over a lower-count successor (the inflated + // pin must be audited), while an equal/higher successor is preserved. if !cooldown_allows_audit(cooldown, &event.peer, mono_now) { self.limiter.refund_token(); obs.cooldown_deferred_attempts .fetch_add(1, Ordering::Relaxed); drop(inflight); - if !self.pending.contains(&event.peer) { - let _ = queue_first_audit_event(&mut self.pending, event); + // Count a capacity eviction if the requeue pushed a different peer + // out of the full LRU (the ADR promises capacity loss is + // observable); the nomination itself was already counted at ingress. + if let FirstAuditQueueOutcome::CapacityEvicted { .. } = + coalesce_first_audit_event(&mut self.pending, event, false) + { + obs.capacity_evicted.fetch_add(1, Ordering::Relaxed); } return None; } @@ -5898,9 +5938,9 @@ mod tests { use super::*; use super::{ apply_audit_failure_credit_revocation, audit_failure_clears_bootstrap_claim, - audit_failure_revokes_holder_credit, audit_launch_decision, config, cooldown_allows_audit, - first_audit_count_jump, first_audit_terminal_outcome, first_failed_key_label, - fresh_offer_payment_context, paid_notify_payment_context, queue_first_audit_event, + audit_failure_revokes_holder_credit, audit_launch_decision, coalesce_first_audit_event, + config, cooldown_allows_audit, first_audit_count_jump, first_audit_terminal_outcome, + first_failed_key_label, fresh_offer_payment_context, paid_notify_payment_context, quote_answerable_through_nominal_jitter, quote_within_audit_window, FirstAuditLimiter, FirstAuditObservability, FirstAuditQueueOutcome, FirstAuditScheduler, FirstAuditTerminalOutcome, LimiterVerdict, MonetizedPinEvent, @@ -6029,44 +6069,89 @@ mod tests { } #[test] - fn first_audit_queue_exposes_coalescing_and_capacity_eviction() { + fn first_audit_coalescing_keeps_highest_count_and_exposes_eviction() { let mut pending = LruCache::new(NonZeroUsize::new(1).unwrap()); - let first = MonetizedPinEvent { - peer: test_peer(1), + let peer = test_peer(1); + let base = MonetizedPinEvent { + peer, pin: [1; 32], - key_count: 1, + key_count: 100, quote_ts: SystemTime::now(), }; - let replacement = MonetizedPinEvent { + + // First insert into an empty slot: Queued. + assert_eq!( + coalesce_first_audit_event(&mut pending, base, true), + FirstAuditQueueOutcome::Queued + ); + + // A strictly LOWER-count same-peer nomination must NOT displace it. + let lower = MonetizedPinEvent { pin: [2; 32], - ..first + key_count: 50, + ..base }; - let other_peer = MonetizedPinEvent { - peer: test_peer(2), + assert_eq!( + coalesce_first_audit_event(&mut pending, lower, true), + FirstAuditQueueOutcome::SuppressedLower + ); + assert_eq!( + pending.peek(&peer).map(|e| (e.pin, e.key_count)), + Some(([1; 32], 100)), + "the higher-count pin is retained" + ); + + // A HIGHER-count same-peer nomination wins (the inflated pin to audit). + let higher = MonetizedPinEvent { pin: [3; 32], - ..first + key_count: 400, + ..base }; - assert_eq!( - queue_first_audit_event(&mut pending, first), - FirstAuditQueueOutcome::Queued + coalesce_first_audit_event(&mut pending, higher, true), + FirstAuditQueueOutcome::Coalesced ); assert_eq!( - queue_first_audit_event(&mut pending, replacement), + pending.peek(&peer).map(|e| (e.pin, e.key_count)), + Some(([3; 32], 400)) + ); + + // EQUAL count: an ordinary (newer) enqueue replaces for freshness... + let equal_newer = MonetizedPinEvent { + pin: [4; 32], + key_count: 400, + ..base + }; + assert_eq!( + coalesce_first_audit_event(&mut pending, equal_newer, true), FirstAuditQueueOutcome::Coalesced ); + assert_eq!(pending.peek(&peer).map(|e| e.pin), Some([4; 32])); + // ...but an equal-count OLDER requeue (incoming_is_newer=false) does not. + let equal_older = MonetizedPinEvent { + pin: [5; 32], + key_count: 400, + ..base + }; assert_eq!( - queue_first_audit_event(&mut pending, other_peer), - FirstAuditQueueOutcome::CapacityEvicted { - peer: first.peer, - pin: replacement.pin, - } + coalesce_first_audit_event(&mut pending, equal_older, false), + FirstAuditQueueOutcome::SuppressedLower ); - assert_eq!(pending.len(), 1); + assert_eq!(pending.peek(&peer).map(|e| e.pin), Some([4; 32])); + + // A different peer at capacity 1 evicts the LRU (a DIFFERENT peer). + let other_peer = MonetizedPinEvent { + peer: test_peer(2), + pin: [6; 32], + key_count: 100, + ..base + }; assert_eq!( - pending.peek(&other_peer.peer).map(|event| event.pin), - Some([3; 32]) + coalesce_first_audit_event(&mut pending, other_peer, true), + FirstAuditQueueOutcome::CapacityEvicted { peer, pin: [4; 32] } ); + assert_eq!(pending.len(), 1); + assert_eq!(pending.peek(&other_peer.peer).map(|e| e.pin), Some([6; 32])); } // -- ADR-0004 Amendment 2: first-audit launch limiter -------------------- @@ -6452,6 +6537,134 @@ mod tests { assert!(scheduler.limiter.recent.peek(&peer).is_none()); } + /// ADR-0004 Amendment 2 (reviewer blocker): a strictly-lower-count same-peer + /// nomination arriving while an inflated pin is PENDING must not displace it. + /// The inflated pin stays and is the one launched. + #[test] + fn first_audit_pending_lower_count_does_not_replace_higher() { + let obs = Arc::new(FirstAuditObservability::default()); + let mono = Instant::now(); + let mut scheduler = FirstAuditScheduler::new(mono); + let peer = test_peer(1); + + // Inflated (high-count) sidecar pin lands in pending. + scheduler.enqueue( + MonetizedPinEvent { + peer, + pin: [1; 32], + key_count: 400, + quote_ts: SystemTime::now(), + }, + &obs, + ); + // A cheaper same-peer settlement arrives right after. + scheduler.enqueue( + MonetizedPinEvent { + peer, + pin: [2; 32], + key_count: 100, + quote_ts: SystemTime::now(), + }, + &obs, + ); + + assert_eq!(scheduler.pending_len(), 1); + assert_eq!( + scheduler.pending.peek(&peer).map(|e| (e.pin, e.key_count)), + Some(([1; 32], 400)), + "the inflated pin must not be erased by the cheaper successor" + ); + + // It reserves and promotes as the inflated pin/count. + let cooldown_read: HashMap = HashMap::new(); + assert!(scheduler.try_reserve(mono, 0, Duration::ZERO, &cooldown_read, &obs)); + let reservation = scheduler.take_due_reservation(mono).expect("due"); + let mut cooldown: HashMap = HashMap::new(); + let (event, _slot) = scheduler + .resolve(reservation, SystemTime::now(), mono, &mut cooldown, &obs) + .expect("promotes"); + assert_eq!((event.pin, event.key_count), ([1; 32], 400)); + } + + /// ADR-0004 Amendment 2 (reviewer blocker): a RESERVED inflated pin that + /// loses the cooldown race must be requeued OVER a lower-count same-peer + /// successor that arrived during its jitter, and must remain launchable once + /// the shared cooldown expires — the cheaper successor cannot suppress it. + #[test] + fn first_audit_cooldown_race_requeues_reserved_higher_over_lower_successor() { + let obs = Arc::new(FirstAuditObservability::default()); + let mono = Instant::now(); + let mut scheduler = FirstAuditScheduler::new(mono); + let peer = test_peer(1); + + // Reserve the inflated pin. + scheduler.enqueue( + MonetizedPinEvent { + peer, + pin: [1; 32], + key_count: 400, + quote_ts: SystemTime::now(), + }, + &obs, + ); + let cooldown_read: HashMap = HashMap::new(); + assert!(scheduler.try_reserve(mono, 0, Duration::ZERO, &cooldown_read, &obs)); + + // A cheaper successor arrives during the reservation (bypasses the + // window as the reserved peer) and sits in pending. + scheduler.enqueue( + MonetizedPinEvent { + peer, + pin: [2; 32], + key_count: 100, + quote_ts: SystemTime::now(), + }, + &obs, + ); + assert_eq!(scheduler.pending_len(), 1); + + // Resolve: answerability passes but the shared cooldown is already + // stamped, so the reservation loses the race and requeues. + let reservation = scheduler.take_due_reservation(mono).expect("due"); + let mut cooldown: HashMap = HashMap::new(); + cooldown.insert(peer, mono); + assert!(scheduler + .resolve(reservation, SystemTime::now(), mono, &mut cooldown, &obs) + .is_none()); + + // The inflated pin (400) replaced the cheaper successor (100). + assert_eq!(scheduler.pending_len(), 1); + assert_eq!( + scheduler.pending.peek(&peer).map(|e| (e.pin, e.key_count)), + Some(([1; 32], 400)), + "the inflated reserved pin must survive the requeue over the cheaper successor" + ); + assert!(scheduler.first_audited.is_empty()); + assert!(scheduler.limiter.recent.peek(&peer).is_none()); + + // Once the shared cooldown expires, the inflated pin reserves and + // promotes with its intended pin/count. + let later = mono + .checked_add(Duration::from_secs( + config::AUDIT_ON_GOSSIP_COOLDOWN_SECS + 1, + )) + .expect("later"); + let cooldown_read_later: HashMap = HashMap::new(); + assert!(scheduler.try_reserve(later, 0, Duration::ZERO, &cooldown_read_later, &obs)); + let reservation = scheduler.take_due_reservation(later).expect("due"); + let mut cooldown_later: HashMap = HashMap::new(); + let (event, _slot) = scheduler + .resolve( + reservation, + SystemTime::now(), + later, + &mut cooldown_later, + &obs, + ) + .expect("promotes after cooldown"); + assert_eq!((event.pin, event.key_count), ([1; 32], 400)); + } + /// ADR-0004 Amendment 2 (E'): consecutive PROMOTIONS strictly alternate the /// launch lane, driven through the real scheduler (reserve -> resolve -> /// promote), so a stream of fresh nominations cannot keep every launch on From e1a140bd4db8679c2854ff9eb9f33c1c2a705cdb Mon Sep 17 00:00:00 2001 From: grumbach Date: Wed, 15 Jul 2026 15:40:54 +0900 Subject: [PATCH 12/21] feat(replication): observe suppressed-lower self-erasure attempts; add edge tests Review nits (both non-blocking): add a dedicated suppressed_lower funnel counter so a burst of attempted cheaper-pin self-erasure (a lower-count same-peer nomination dropped so a higher-count pin survives) is visible in the scheduler summary, on both the enqueue and cooldown-race requeue paths; add edge tests for (a) a suppressed-lower flood leaving the retained pin's LRU recency/lane position unchanged (peek, no push) and (b) the cooldown-race requeue counting a different-peer capacity eviction. ADR-0004 funnel list updated. 704 lib tests + cfd green. --- ...ADR-0004-commitment-bound-quote-pricing.md | 9 +- src/replication/mod.rs | 165 +++++++++++++++++- 2 files changed, 161 insertions(+), 13 deletions(-) diff --git a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md index fc485ea1..682ac5c0 100644 --- a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md +++ b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md @@ -407,10 +407,11 @@ soon (minutes, from multiple independent storers, each within its own budget); commitments rotated without material count change rely on the ADR-0002 lottery for re-audit. First-audit coverage is best-effort supplementary sampling under an explicit load budget, not a per-payment guarantee. The scheduler's funnel -(received / queued / coalesced / duplicates / window_deduped / rate_deferred / -cooldown_deferred / launched / terminal outcomes, plus tokens and in-flight -gauges) is exported in the periodic scheduler summary so this coverage is -measurable in production. +(received / queued / coalesced / suppressed_lower / duplicates / +capacity_evicted / window_deduped / rate_deferred / cooldown_deferred / +launched / terminal outcomes, plus tokens and in-flight gauges) is exported in +the periodic scheduler summary so this coverage is measurable in production; +`suppressed_lower` in particular surfaces attempted cheaper-pin self-erasure. **Every pipeline stage is bounded, and the invariants hold by construction.** The verifier-to-drainer nomination channel is bounded (producers `try_send` diff --git a/src/replication/mod.rs b/src/replication/mod.rs index f7c89b47..09b9deb7 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -103,6 +103,10 @@ struct FirstAuditObservability { coalesced: AtomicU64, duplicates: AtomicU64, capacity_evicted: AtomicU64, + /// A strictly-lower-count same-peer nomination that was dropped so a + /// higher-count pending pin survived. A sustained rise is the signal of an + /// attempted "erase the inflated pin with a cheaper one" self-suppression. + suppressed_lower: AtomicU64, cooldown_deferred_attempts: AtomicU64, rate_deferred_attempts: AtomicU64, window_deduped: AtomicU64, @@ -541,9 +545,12 @@ impl FirstAuditScheduler { FirstAuditQueueOutcome::Queued => { obs.queued.fetch_add(1, Ordering::Relaxed); } - FirstAuditQueueOutcome::Coalesced | FirstAuditQueueOutcome::SuppressedLower => { + FirstAuditQueueOutcome::Coalesced => { obs.coalesced.fetch_add(1, Ordering::Relaxed); } + FirstAuditQueueOutcome::SuppressedLower => { + obs.suppressed_lower.fetch_add(1, Ordering::Relaxed); + } FirstAuditQueueOutcome::CapacityEvicted { .. } => { obs.queued.fetch_add(1, Ordering::Relaxed); obs.capacity_evicted.fetch_add(1, Ordering::Relaxed); @@ -697,13 +704,18 @@ impl FirstAuditScheduler { obs.cooldown_deferred_attempts .fetch_add(1, Ordering::Relaxed); drop(inflight); - // Count a capacity eviction if the requeue pushed a different peer - // out of the full LRU (the ADR promises capacity loss is - // observable); the nomination itself was already counted at ingress. - if let FirstAuditQueueOutcome::CapacityEvicted { .. } = - coalesce_first_audit_event(&mut self.pending, event, false) - { - obs.capacity_evicted.fetch_add(1, Ordering::Relaxed); + // Account for the requeue outcome (the nomination itself was already + // counted at ingress, so `queued` is not re-incremented): a capacity + // eviction of a DIFFERENT peer and a suppressed reserved event are + // both observable per the ADR funnel. + match coalesce_first_audit_event(&mut self.pending, event, false) { + FirstAuditQueueOutcome::CapacityEvicted { .. } => { + obs.capacity_evicted.fetch_add(1, Ordering::Relaxed); + } + FirstAuditQueueOutcome::SuppressedLower => { + obs.suppressed_lower.fetch_add(1, Ordering::Relaxed); + } + FirstAuditQueueOutcome::Queued | FirstAuditQueueOutcome::Coalesced => {} } return None; } @@ -1642,11 +1654,12 @@ impl ReplicationEngine { if last_summary.elapsed() >= config::FIRST_AUDIT_SUMMARY_INTERVAL { info!( - "First-audit scheduler summary: audit_trigger=first_monetized ingress_dropped={} received={} queued={} coalesced={} duplicates={} capacity_evicted={} cooldown_deferred_attempts={} rate_deferred_attempts={} window_deduped={} launched={} passed={} timeout={} failed={} bootstrap_claims={} idle={} insufficient_keys={} outside_answerability_window={} pending={} inflight={} tokens={}", + "First-audit scheduler summary: audit_trigger=first_monetized ingress_dropped={} received={} queued={} coalesced={} suppressed_lower={} duplicates={} capacity_evicted={} cooldown_deferred_attempts={} rate_deferred_attempts={} window_deduped={} launched={} passed={} timeout={} failed={} bootstrap_claims={} idle={} insufficient_keys={} outside_answerability_window={} pending={} inflight={} tokens={}", FIRST_AUDIT_INGRESS_DROPPED.load(Ordering::Relaxed), observability.received.load(Ordering::Relaxed), observability.queued.load(Ordering::Relaxed), observability.coalesced.load(Ordering::Relaxed), + observability.suppressed_lower.load(Ordering::Relaxed), observability.duplicates.load(Ordering::Relaxed), observability.capacity_evicted.load(Ordering::Relaxed), observability.cooldown_deferred_attempts.load(Ordering::Relaxed), @@ -6537,6 +6550,140 @@ mod tests { assert!(scheduler.limiter.recent.peek(&peer).is_none()); } + /// A flood of strictly-lower-count same-peer nominations must neither + /// displace the retained higher pin NOR disturb its LRU position (each is + /// suppressed via `peek`, no `push`), and each must be counted as + /// `suppressed_lower` — the attempted cheaper-pin self-erasure signal. + #[test] + fn first_audit_suppressed_lower_flood_leaves_recency_and_counts() { + let mut pending: LruCache = + LruCache::new(NonZeroUsize::new(2).unwrap()); + let victim = test_peer(1); + let other = test_peer(2); + // Victim (high count) inserted first (older), then `other` (newer/MRU). + let _ = coalesce_first_audit_event( + &mut pending, + MonetizedPinEvent { + peer: victim, + pin: [1; 32], + key_count: 400, + quote_ts: SystemTime::now(), + }, + true, + ); + let _ = coalesce_first_audit_event( + &mut pending, + MonetizedPinEvent { + peer: other, + pin: [9; 32], + key_count: 100, + quote_ts: SystemTime::now(), + }, + true, + ); + + // Flood the victim with cheaper nominations. + let mut suppressed = 0u64; + for i in 0..8u8 { + let out = coalesce_first_audit_event( + &mut pending, + MonetizedPinEvent { + peer: victim, + pin: [i; 32], + key_count: 50, + quote_ts: SystemTime::now(), + }, + true, + ); + assert_eq!(out, FirstAuditQueueOutcome::SuppressedLower); + suppressed += 1; + } + assert_eq!(suppressed, 8); + // Victim pin/count unchanged. + assert_eq!( + pending.peek(&victim).map(|e| (e.pin, e.key_count)), + Some(([1; 32], 400)) + ); + // Recency unchanged: `other` is still MRU (a new distinct peer at cap 2 + // would evict the LRU; the victim must be the LRU, so `third` evicts the + // victim, not `other`). + let third = test_peer(3); + let out = coalesce_first_audit_event( + &mut pending, + MonetizedPinEvent { + peer: third, + pin: [7; 32], + key_count: 100, + quote_ts: SystemTime::now(), + }, + true, + ); + assert_eq!( + out, + FirstAuditQueueOutcome::CapacityEvicted { + peer: victim, + pin: [1; 32] + }, + "the suppressed-lower flood must not have promoted the victim above `other`" + ); + } + + /// The cooldown-race requeue counts a genuine different-peer capacity + /// eviction (the ADR promises capacity loss is observable). + #[test] + fn first_audit_cooldown_race_requeue_counts_capacity_eviction() { + let obs = Arc::new(FirstAuditObservability::default()); + let mono = Instant::now(); + let mut scheduler = FirstAuditScheduler::new(mono); + // Pending capacity 1 so a requeue of a different peer must evict. + scheduler.pending = LruCache::new(NonZeroUsize::new(1).unwrap()); + let reserved_peer = test_peer(1); + let other_peer = test_peer(2); + + // Reserve peer 1. + scheduler.enqueue( + MonetizedPinEvent { + peer: reserved_peer, + pin: [1; 32], + key_count: 100, + quote_ts: SystemTime::now(), + }, + &obs, + ); + let cooldown_read: HashMap = HashMap::new(); + assert!(scheduler.try_reserve(mono, 0, Duration::ZERO, &cooldown_read, &obs)); + + // A DIFFERENT peer fills the single pending slot during the reservation. + scheduler.enqueue( + MonetizedPinEvent { + peer: other_peer, + pin: [2; 32], + key_count: 100, + quote_ts: SystemTime::now(), + }, + &obs, + ); + assert_eq!(scheduler.pending_len(), 1); + + // Resolve peer 1: cooldown race -> requeue peer 1, evicting peer 2. + let reservation = scheduler.take_due_reservation(mono).expect("due"); + let mut cooldown: HashMap = HashMap::new(); + cooldown.insert(reserved_peer, mono); + let cap_before = obs.capacity_evicted.load(Ordering::Relaxed); + assert!(scheduler + .resolve(reservation, SystemTime::now(), mono, &mut cooldown, &obs) + .is_none()); + assert_eq!( + obs.capacity_evicted.load(Ordering::Relaxed), + cap_before + 1, + "the requeue eviction of a different peer is counted" + ); + assert_eq!( + scheduler.pending.peek(&reserved_peer).map(|e| e.pin), + Some([1; 32]) + ); + } + /// ADR-0004 Amendment 2 (reviewer blocker): a strictly-lower-count same-peer /// nomination arriving while an inflated pin is PENDING must not displace it. /// The inflated pin stays and is the one launched. From dcd7cf8c35a9d16acbe83df2cedf1248196075ee Mon Sep 17 00:00:00 2001 From: grumbach Date: Wed, 15 Jul 2026 15:48:12 +0900 Subject: [PATCH 13/21] fix(replication): separate equal-count tie from suppressed-lower in the funnel Codex review: SuppressedLower was returned for both a strictly-lower incoming (the attack signal) and an equal-count freshness tie, so the requeue counted a benign tie as an attempted suppression. Add a distinct RetainedOnTie outcome; only a strictly-lower drop increments suppressed_lower. Assert the counter through the enqueue path in the pending-lower test. --- src/replication/mod.rs | 40 ++++++++++++++++++++++++++++------------ 1 file changed, 28 insertions(+), 12 deletions(-) diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 09b9deb7..ac8dbd92 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -169,9 +169,14 @@ enum FirstAuditQueueOutcome { /// Collapsed with an existing same-peer entry; the incoming won (higher /// count, or equal count and newer) and replaced it. Coalesced, - /// Collapsed with an existing same-peer entry; the incoming LOST (strictly - /// lower count) and was dropped, leaving the higher-count pin in place. + /// Collapsed with an existing same-peer entry; the incoming LOST because it + /// had a STRICTLY LOWER count and was dropped, leaving the higher-count pin + /// in place. This is the attempted cheaper-pin self-erasure signal. SuppressedLower, + /// Collapsed with an existing same-peer entry of EQUAL count; the incoming + /// lost the freshness tie (it was not the newer of the two) and the + /// existing entry was retained. Benign, not an attack signal. + RetainedOnTie, /// A DIFFERENT peer's entry was evicted by the bounded LRU to make room. CapacityEvicted { peer: PeerId, pin: [u8; 32] }, } @@ -197,15 +202,18 @@ fn coalesce_first_audit_event( incoming_is_newer: bool, ) -> FirstAuditQueueOutcome { if let Some(existing) = pending.peek(&incoming.peer) { - let incoming_wins = incoming.key_count > existing.key_count - || (incoming.key_count == existing.key_count && incoming_is_newer); - if !incoming_wins { - // Retain the higher/equal-newer existing pin; do NOT touch its - // recency. + // Strictly lower -> the incoming loses and is dropped WITHOUT touching + // the retained pin's recency (the security-relevant self-erasure case). + if incoming.key_count < existing.key_count { return FirstAuditQueueOutcome::SuppressedLower; } - // Same-peer replace: `push` updates the value and bumps MRU. Replacing - // an existing key never evicts a different peer. + // Equal count -> keep the fresher; an older incoming loses a benign tie. + if incoming.key_count == existing.key_count && !incoming_is_newer { + return FirstAuditQueueOutcome::RetainedOnTie; + } + // Strictly higher, or equal-and-newer: the incoming wins. `push` updates + // the value and bumps MRU; replacing an existing key never evicts a + // different peer. let _ = pending.push(incoming.peer, incoming); return FirstAuditQueueOutcome::Coalesced; } @@ -545,7 +553,7 @@ impl FirstAuditScheduler { FirstAuditQueueOutcome::Queued => { obs.queued.fetch_add(1, Ordering::Relaxed); } - FirstAuditQueueOutcome::Coalesced => { + FirstAuditQueueOutcome::Coalesced | FirstAuditQueueOutcome::RetainedOnTie => { obs.coalesced.fetch_add(1, Ordering::Relaxed); } FirstAuditQueueOutcome::SuppressedLower => { @@ -715,7 +723,9 @@ impl FirstAuditScheduler { FirstAuditQueueOutcome::SuppressedLower => { obs.suppressed_lower.fetch_add(1, Ordering::Relaxed); } - FirstAuditQueueOutcome::Queued | FirstAuditQueueOutcome::Coalesced => {} + FirstAuditQueueOutcome::Queued + | FirstAuditQueueOutcome::Coalesced + | FirstAuditQueueOutcome::RetainedOnTie => {} } return None; } @@ -6148,7 +6158,7 @@ mod tests { }; assert_eq!( coalesce_first_audit_event(&mut pending, equal_older, false), - FirstAuditQueueOutcome::SuppressedLower + FirstAuditQueueOutcome::RetainedOnTie ); assert_eq!(pending.peek(&peer).map(|e| e.pin), Some([4; 32])); @@ -6721,6 +6731,12 @@ mod tests { Some(([1; 32], 400)), "the inflated pin must not be erased by the cheaper successor" ); + // The dropped cheaper nomination is counted through the enqueue path. + assert_eq!( + obs.suppressed_lower.load(Ordering::Relaxed), + 1, + "the attempted cheaper-pin self-erasure is observable" + ); // It reserves and promotes as the inflated pin/count. let cooldown_read: HashMap = HashMap::new(); From a118d7201d3fba5be1fffcd594df410632ab8bfb Mon Sep 17 00:00:00 2001 From: grumbach Date: Fri, 17 Jul 2026 12:51:07 +0900 Subject: [PATCH 14/21] feat(replication): log oldest-pending-quote age in first-audit summary Adds oldest_pending_quote_age_ms to the first-audit scheduler summary so the pending-queue depth can be interpreted: a small, steady value means the queue is draining promptly; a value climbing toward the answerability window means pending work is expiring unaudited instead of launching. Gives the queue-age evidence the last testnet report asked for. Observability only: no scheduling or protocol behaviour change. --- src/replication/mod.rs | 72 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 71 insertions(+), 1 deletion(-) diff --git a/src/replication/mod.rs b/src/replication/mod.rs index ac8dbd92..391db429 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -512,6 +512,23 @@ impl FirstAuditScheduler { self.limiter.tokens } + /// Observability getter (see [`Self::pending_len`]). Milliseconds since the + /// signed quote timestamp of the oldest pin still awaiting a first audit: + /// how close the longest-waiting pending pin is to aging out of the + /// answerability window. A value climbing toward the window means pending + /// work is expiring unaudited instead of launching; a small, steady value + /// means the queue is draining promptly. Returns `0` when `pending` is empty + /// and saturates to `0` for a future-dated quote (clock skew), never panics. + #[cfg_attr(not(feature = "logging"), allow(dead_code))] + fn oldest_pending_quote_age_ms(&self, now: SystemTime) -> u64 { + self.pending + .iter() + .map(|(_, e)| e.quote_ts) + .min() + .and_then(|oldest| now.duration_since(oldest).ok()) + .map_or(0, |age| u64::try_from(age.as_millis()).unwrap_or(u64::MAX)) + } + fn has_reservation(&self) -> bool { self.reserved.is_some() } @@ -1664,7 +1681,7 @@ impl ReplicationEngine { if last_summary.elapsed() >= config::FIRST_AUDIT_SUMMARY_INTERVAL { info!( - "First-audit scheduler summary: audit_trigger=first_monetized ingress_dropped={} received={} queued={} coalesced={} suppressed_lower={} duplicates={} capacity_evicted={} cooldown_deferred_attempts={} rate_deferred_attempts={} window_deduped={} launched={} passed={} timeout={} failed={} bootstrap_claims={} idle={} insufficient_keys={} outside_answerability_window={} pending={} inflight={} tokens={}", + "First-audit scheduler summary: audit_trigger=first_monetized ingress_dropped={} received={} queued={} coalesced={} suppressed_lower={} duplicates={} capacity_evicted={} cooldown_deferred_attempts={} rate_deferred_attempts={} window_deduped={} launched={} passed={} timeout={} failed={} bootstrap_claims={} idle={} insufficient_keys={} outside_answerability_window={} pending={} oldest_pending_quote_age_ms={} inflight={} tokens={}", FIRST_AUDIT_INGRESS_DROPPED.load(Ordering::Relaxed), observability.received.load(Ordering::Relaxed), observability.queued.load(Ordering::Relaxed), @@ -1684,6 +1701,7 @@ impl ReplicationEngine { observability.insufficient_keys.load(Ordering::Relaxed), observability.outside_answerability_window.load(Ordering::Relaxed), scheduler.pending_len(), + scheduler.oldest_pending_quote_age_ms(SystemTime::now()), observability.inflight.load(Ordering::Relaxed), scheduler.tokens(), ); @@ -6560,6 +6578,58 @@ mod tests { assert!(scheduler.limiter.recent.peek(&peer).is_none()); } + /// The oldest-pending-quote-age gauge reports the age of the OLDEST quote + /// still awaiting a first audit (not the newest), is `0` on an empty queue, + /// and saturates to `0` for a future-dated quote (clock skew) without + /// panicking. A climbing value is the pending-work-aging-out signal. + #[test] + fn first_audit_oldest_pending_quote_age_tracks_the_oldest() { + let now = SystemTime::now(); + let mut scheduler = FirstAuditScheduler::new(Instant::now()); + assert_eq!( + scheduler.oldest_pending_quote_age_ms(now), + 0, + "empty pending -> zero" + ); + + // Two peers: quotes 10s and 2s old. The gauge must track the older. + for (peer_id, pin, secs) in [(1u8, [1; 32], 10u64), (2, [2; 32], 2)] { + let _ = coalesce_first_audit_event( + &mut scheduler.pending, + MonetizedPinEvent { + peer: test_peer(peer_id), + pin, + key_count: 100, + quote_ts: now - Duration::from_secs(secs), + }, + true, + ); + } + assert_eq!( + scheduler.oldest_pending_quote_age_ms(now), + 10_000, + "tracks the oldest (10s), not the newest (2s)" + ); + + // A future-dated quote (clock skew) saturates to zero, never panics. + let mut skewed = FirstAuditScheduler::new(Instant::now()); + let _ = coalesce_first_audit_event( + &mut skewed.pending, + MonetizedPinEvent { + peer: test_peer(3), + pin: [3; 32], + key_count: 100, + quote_ts: now + Duration::from_secs(5), + }, + true, + ); + assert_eq!( + skewed.oldest_pending_quote_age_ms(now), + 0, + "future quote_ts saturates to zero" + ); + } + /// A flood of strictly-lower-count same-peer nominations must neither /// displace the retained higher pin NOR disturb its LRU position (each is /// suppressed via `peek`, no `push`), and each must be counted as From c02aeab09bd759d965f4a86628528e5f2b8c196e Mon Sep 17 00:00:00 2001 From: grumbach Date: Fri, 17 Jul 2026 16:17:20 +0900 Subject: [PATCH 15/21] feat(replication): tally audit outcomes and admission drops in the periodic summary The per-variant traffic counters give audit volume but not results, so release comparisons cannot attribute CPU changes to audit work. Add cumulative pass/fail-per-reason counters for responsible and subtree audits plus responder-side admission-drop counters, emitted as a fourth summary line that telegraf lifts into per-field ES/Influx data. --- src/replication/mod.rs | 8 ++ src/replication/protocol.rs | 150 ++++++++++++++++++++++++++++++++++++ 2 files changed, 158 insertions(+) diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 391db429..a974acda 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -2196,6 +2196,7 @@ impl ReplicationEngine { )) => { protocol::log_traffic_summary(); protocol::log_served_peers_summary(); + protocol::log_audit_outcome_summary(); } } } @@ -2940,6 +2941,7 @@ async fn handle_replication_message( { Ok(guard) => guard, Err(failure) => { + protocol::record_audit_drop(protocol::AuditDropKind::Responsible); warn!( "Audit challenge reply not sent: kind=responsible response=dropped \ source={source} {failure}" @@ -2997,6 +2999,7 @@ async fn handle_replication_message( { Ok(guard) => guard, Err(failure) => { + protocol::record_audit_drop(protocol::AuditDropKind::Subtree); warn!( "Audit challenge reply not sent: kind=subtree response=dropped \ source={source} {failure}" @@ -3065,6 +3068,7 @@ async fn handle_replication_message( { Ok(guard) => guard, Err(failure) => { + protocol::record_audit_drop(protocol::AuditDropKind::Byte); warn!( "Audit challenge reply not sent: kind=byte response=dropped \ source={source} {failure}" @@ -5042,6 +5046,7 @@ async fn handle_subtree_audit_result( challenged_peer, keys_checked, } => { + protocol::record_audit_pass(protocol::AuditOutcomeKind::Subtree); debug!("Audit passed for {challenged_peer} ({keys_checked} keys)"); // Peer responded normally — clear the active bootstrap claim while // retaining history so a later claim is treated as repeated abuse. @@ -5065,6 +5070,7 @@ async fn handle_subtree_audit_result( .. } = evidence { + protocol::record_audit_fail(protocol::AuditOutcomeKind::Subtree, reason); // Rich diagnostics (from main's audit-failure logging) + the // first-failed-key correlation handle. let first_failed_key = first_failed_key_label(confirmed_failed_keys); @@ -5163,6 +5169,7 @@ async fn handle_audit_result( challenged_peer, keys_checked, } => { + protocol::record_audit_pass(protocol::AuditOutcomeKind::Responsible); debug!("Audit passed for {challenged_peer} ({keys_checked} keys)"); { let mut state = sync_state.write().await; @@ -5184,6 +5191,7 @@ async fn handle_audit_result( .. } = evidence { + protocol::record_audit_fail(protocol::AuditOutcomeKind::Responsible, reason); let first_failed_key = first_failed_key_label(confirmed_failed_keys); error!( "Audit failure for {challenged_peer}: reason={reason:?}, confirmed_failed_keys={}, challenged_keys={}, absent_keys={}, digest_mismatch_keys={}, first_failed_key={first_failed_key}", diff --git a/src/replication/protocol.rs b/src/replication/protocol.rs index 19c397ba..efc2366f 100644 --- a/src/replication/protocol.rs +++ b/src/replication/protocol.rs @@ -11,6 +11,8 @@ use serde::{Deserialize, Serialize}; use crate::ant_protocol::XorName; +use super::types::AuditFailureReason; + pub use super::config::MAX_REPLICATION_MESSAGE_SIZE; /// Sentinel digest value indicating the challenged key is absent from storage. @@ -209,6 +211,106 @@ impl ReplicationMessageBody { } } +// --------------------------------------------------------------------------- +// Cumulative audit outcome accounting (extends V2-623) +// --------------------------------------------------------------------------- +// +// The per-variant traffic counters above already give audit *volume* (challenge +// tx = launched, response tx = served). These tally the *outcomes*, which the +// traffic counters cannot express: auditor-side pass/fail per failure reason, +// and responder-side admission drops (challenges refused at the flood-fair +// caps). Same conventions: process-global relaxed atomics, monotonic since +// process start, emitted next to the traffic summary and diffed at query time. + +/// Auditor-side audit kinds with pass/fail outcome tallies. +#[derive(Clone, Copy)] +pub(crate) enum AuditOutcomeKind { + /// Responsible-chunk audit (audit #2, `AuditChallenge`). + Responsible = 0, + /// Storage-bound subtree audit (ADR-0002). + Subtree = 1, +} + +/// Responder-side challenge kinds dropped at the admission caps. +#[derive(Clone, Copy)] +pub(crate) enum AuditDropKind { + /// Responsible-chunk / prune-confirmation `AuditChallenge`. + Responsible = 0, + /// Subtree audit round-1 challenge. + Subtree = 1, + /// Subtree audit round-2 byte challenge. + Byte = 2, +} + +/// One slot per [`AuditFailureReason`] variant, per [`AuditOutcomeKind`]. +const N_FAIL_REASONS: usize = 5; +const N_OUTCOME_KINDS: usize = 2; +const N_DROP_KINDS: usize = 3; + +static AUDIT_PASS: [AtomicU64; N_OUTCOME_KINDS] = [const { AtomicU64::new(0) }; N_OUTCOME_KINDS]; +static AUDIT_FAIL: [AtomicU64; N_OUTCOME_KINDS * N_FAIL_REASONS] = + [const { AtomicU64::new(0) }; N_OUTCOME_KINDS * N_FAIL_REASONS]; +static AUDIT_DROPPED: [AtomicU64; N_DROP_KINDS] = [const { AtomicU64::new(0) }; N_DROP_KINDS]; + +/// Stable counter index for a failure reason. Matches declaration order of +/// [`AuditFailureReason`]; exhaustive so a new variant is a compile error here. +fn fail_reason_index(reason: &AuditFailureReason) -> usize { + match reason { + AuditFailureReason::Timeout => 0, + AuditFailureReason::MalformedResponse => 1, + AuditFailureReason::DigestMismatch => 2, + AuditFailureReason::KeyAbsent => 3, + AuditFailureReason::Rejected => 4, + } +} + +/// Record an auditor-side audit pass. +pub(crate) fn record_audit_pass(kind: AuditOutcomeKind) { + AUDIT_PASS[kind as usize].fetch_add(1, Ordering::Relaxed); +} + +/// Record an auditor-side audit failure with its reason. +pub(crate) fn record_audit_fail(kind: AuditOutcomeKind, reason: &AuditFailureReason) { + AUDIT_FAIL[kind as usize * N_FAIL_REASONS + fail_reason_index(reason)] + .fetch_add(1, Ordering::Relaxed); +} + +/// Record a responder-side challenge dropped at the admission caps. +pub(crate) fn record_audit_drop(kind: AuditDropKind) { + AUDIT_DROPPED[kind as usize].fetch_add(1, Ordering::Relaxed); +} + +/// Emit the cumulative audit outcome tallies as one INFO summary line, same +/// target and cadence as [`log_traffic_summary`] (`group = 4`), so the +/// telegraf→Elasticsearch pipeline lifts each key into a `tail.*` field and +/// pass/fail/drop rates per hour fall out of max-per-hour deltas. +pub(crate) fn log_audit_outcome_summary() { + let pass = |k: usize| AUDIT_PASS[k].load(Ordering::Relaxed); + let fail = |k: usize, r: usize| AUDIT_FAIL[k * N_FAIL_REASONS + r].load(Ordering::Relaxed); + let drop = |k: usize| AUDIT_DROPPED[k].load(Ordering::Relaxed); + + crate::logging::info!( + target: "ant_node::replication::traffic", + group = 4, + responsible_audit_pass = pass(0), + responsible_audit_fail_timeout = fail(0, 0), + responsible_audit_fail_malformed = fail(0, 1), + responsible_audit_fail_digest_mismatch = fail(0, 2), + responsible_audit_fail_key_absent = fail(0, 3), + responsible_audit_fail_rejected = fail(0, 4), + subtree_audit_pass = pass(1), + subtree_audit_fail_timeout = fail(1, 0), + subtree_audit_fail_malformed = fail(1, 1), + subtree_audit_fail_digest_mismatch = fail(1, 2), + subtree_audit_fail_key_absent = fail(1, 3), + subtree_audit_fail_rejected = fail(1, 4), + audit_dropped_responsible = drop(0), + audit_dropped_subtree = drop(1), + audit_dropped_byte = drop(2), + "audit outcome summary (cumulative)" + ); +} + /// Record an encoded (tx) replication message against its variant. fn record_tx(body: &ReplicationMessageBody, bytes: usize) { let i = body.variant_index(); @@ -991,6 +1093,54 @@ impl std::error::Error for ReplicationProtocolError {} mod tests { use super::*; + // === Audit outcome counters === + + /// No other test mutates the audit outcome counters, so per-slot deltas are + /// stable even under parallel test execution. + #[test] + fn audit_outcome_counters_tally_by_kind_and_reason() { + let pass_slot = AuditOutcomeKind::Subtree as usize; + let before = AUDIT_PASS[pass_slot].load(Ordering::Relaxed); + record_audit_pass(AuditOutcomeKind::Subtree); + assert_eq!(AUDIT_PASS[pass_slot].load(Ordering::Relaxed), before + 1); + + let reasons = [ + AuditFailureReason::Timeout, + AuditFailureReason::MalformedResponse, + AuditFailureReason::DigestMismatch, + AuditFailureReason::KeyAbsent, + AuditFailureReason::Rejected, + ]; + for (kind, kind_index) in [ + (AuditOutcomeKind::Responsible, 0usize), + (AuditOutcomeKind::Subtree, 1usize), + ] { + for (reason_index, reason) in reasons.iter().enumerate() { + let slot = kind_index * N_FAIL_REASONS + reason_index; + let before = AUDIT_FAIL[slot].load(Ordering::Relaxed); + record_audit_fail(kind, reason); + assert_eq!( + AUDIT_FAIL[slot].load(Ordering::Relaxed), + before + 1, + "kind {kind_index} reason {reason:?} must land in its own slot", + ); + } + } + + for (kind, kind_index) in [ + (AuditDropKind::Responsible, 0usize), + (AuditDropKind::Subtree, 1usize), + (AuditDropKind::Byte, 2usize), + ] { + let before = AUDIT_DROPPED[kind_index].load(Ordering::Relaxed); + record_audit_drop(kind); + assert_eq!( + AUDIT_DROPPED[kind_index].load(Ordering::Relaxed), + before + 1 + ); + } + } + // === Round-2 byte response sizing === #[test] From 598f5c38a754556b5eaf7c2b06f0dcb27c27ec69 Mon Sep 17 00:00:00 2001 From: grumbach Date: Tue, 21 Jul 2026 16:27:46 +0900 Subject: [PATCH 16/21] fix(replication): drop expired first-audit pending entries eagerly Expired pending nominations were only collected during reserve scans, which the token-budget gate skips entirely under sustained load, so dead entries squatted the bounded per-peer LRU and a dead higher-count incumbent could permanently veto live lower-count nominations for its peer. Enqueue now drops a dead incumbent before coalescing and the summary tick sweeps expired entries regardless of tokens; both removals are accounted as outside_answerability_window, so the pending count and oldest-pending-age gauge report only launchable work. --- src/replication/mod.rs | 200 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 199 insertions(+), 1 deletion(-) diff --git a/src/replication/mod.rs b/src/replication/mod.rs index a974acda..569f5c6b 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -529,6 +529,33 @@ impl FirstAuditScheduler { .map_or(0, |age| u64::try_from(age.as_millis()).unwrap_or(u64::MAX)) } + /// Drop every pending nomination that has aged past the answerability + /// horizon. `try_reserve` collects expired entries only while it can scan — + /// under token starvation it returns at the budget gate first — so without + /// a periodic sweep dead entries squat the bounded LRU (evicting live peers + /// at capacity) and inflate the `pending`/`oldest_pending_quote_age_ms` + /// telemetry. Each removal is accounted as `outside_answerability_window`, + /// exactly like a scan-time expiry. Returns how many entries were dropped; + /// survivor recency is untouched. + fn sweep_expired(&mut self, wall_now: SystemTime, obs: &Arc) -> usize { + let expired: Vec = self + .pending + .iter() + .filter(|(_, event)| !quote_answerable_through_nominal_jitter(event.quote_ts, wall_now)) + .map(|(peer, _)| *peer) + .collect(); + for peer in &expired { + self.pending.pop(peer); + } + if !expired.is_empty() { + obs.outside_answerability_window.fetch_add( + u64::try_from(expired.len()).unwrap_or(u64::MAX), + Ordering::Relaxed, + ); + } + expired.len() + } + fn has_reservation(&self) -> bool { self.reserved.is_some() } @@ -548,7 +575,9 @@ impl FirstAuditScheduler { /// reserved peer (so a successor is retained across the reservation, never /// window-dropped); otherwise window-screened. Coalescing is /// highest-count-per-peer (newest on a tie) — a lower-count successor never - /// displaces a higher-count pending pin. + /// displaces a higher-count pending pin. An incumbent that has aged past + /// the answerability horizon is dropped (and accounted as an expiry) before + /// coalescing, so a dead pin never vetoes a live nomination. fn enqueue(&mut self, event: MonetizedPinEvent, obs: &Arc) { if self.first_audited.contains(&event.pin) { obs.duplicates.fetch_add(1, Ordering::Relaxed); @@ -564,6 +593,19 @@ impl FirstAuditScheduler { obs.window_deduped.fetch_add(1, Ordering::Relaxed); return; } + // A pending incumbent past the answerability horizon can never launch, + // yet its (possibly higher) key count would still win the coalesce + // against a live incoming — e.g. a fresh post-prune lower-count pin — + // and under token starvation no reserve scan runs to collect it. Drop + // it first so the incoming is judged on its own merits. + let stale_incumbent = self.pending.peek(&event.peer).is_some_and(|existing| { + !quote_answerable_through_nominal_jitter(existing.quote_ts, SystemTime::now()) + }); + if stale_incumbent { + self.pending.pop(&event.peer); + obs.outside_answerability_window + .fetch_add(1, Ordering::Relaxed); + } // Ordinary enqueue: the incoming arrived last, so it wins an equal-count // tie. match coalesce_first_audit_event(&mut self.pending, event, true) { @@ -1680,6 +1722,17 @@ impl ReplicationEngine { } if last_summary.elapsed() >= config::FIRST_AUDIT_SUMMARY_INTERVAL { + // Token-independent hygiene: collect entries that aged past + // the answerability horizon while the budget gate kept the + // reserve scan from running, so the summary below reports + // only live work. + let swept = scheduler.sweep_expired(SystemTime::now(), &observability); + if swept > 0 { + debug!( + "First-audit scheduler: audit_trigger=first_monetized outcome=expired_swept count={swept} pending={}", + scheduler.pending_len() + ); + } info!( "First-audit scheduler summary: audit_trigger=first_monetized ingress_dropped={} received={} queued={} coalesced={} suppressed_lower={} duplicates={} capacity_evicted={} cooldown_deferred_attempts={} rate_deferred_attempts={} window_deduped={} launched={} passed={} timeout={} failed={} bootstrap_claims={} idle={} insufficient_keys={} outside_answerability_window={} pending={} oldest_pending_quote_age_ms={} inflight={} tokens={}", FIRST_AUDIT_INGRESS_DROPPED.load(Ordering::Relaxed), @@ -6392,6 +6445,151 @@ mod tests { /// retained and becomes the next reservation after the cancel. This is the /// exact hole the reviewer flagged: suppression must never outlive a launch /// that did not send. + /// A pending pin that has aged past the answerability horizon must not + /// suppress a live lower-count nomination for the same peer: under token + /// starvation no reserve scan ever collects the dead entry, so without the + /// enqueue-time check the peer would stay unauditable through this path + /// indefinitely (e.g. after a prune legitimately lowered its key count). + #[test] + fn first_audit_stale_incumbent_does_not_suppress_live_lower_count() { + let obs = Arc::new(FirstAuditObservability::default()); + let mut scheduler = FirstAuditScheduler::new(Instant::now()); + let peer = test_peer(1); + + let dead_quote = SystemTime::now() + .checked_sub(GOSSIP_ANSWERABILITY_TTL) + .and_then(|t| t.checked_sub(Duration::from_secs(60))) + .expect("past wall time"); + let stale_high = MonetizedPinEvent { + peer, + pin: [1; 32], + key_count: 400, + quote_ts: dead_quote, + }; + scheduler.enqueue(stale_high, &obs); + assert_eq!(scheduler.pending_len(), 1); + + // A fresh, lower-count nomination (a post-prune commitment). + let fresh_lower = MonetizedPinEvent { + peer, + pin: [2; 32], + key_count: 100, + quote_ts: SystemTime::now(), + }; + scheduler.enqueue(fresh_lower, &obs); + + assert_eq!(scheduler.pending_len(), 1); + assert_eq!( + scheduler.pending.peek(&peer).map(|e| e.pin), + Some([2; 32]), + "the live nomination replaced the dead incumbent" + ); + assert_eq!( + obs.suppressed_lower.load(Ordering::Relaxed), + 0, + "no self-erasure signal for displacing a dead pin" + ); + assert_eq!(obs.queued.load(Ordering::Relaxed), 2); + assert_eq!( + obs.outside_answerability_window.load(Ordering::Relaxed), + 1, + "the dead incumbent is accounted as an expiry" + ); + } + + /// The self-erasure defence is untouched for LIVE incumbents: a lower-count + /// nomination still loses to an answerable higher-count pending pin. + #[test] + fn first_audit_live_incumbent_still_suppresses_lower_count() { + let obs = Arc::new(FirstAuditObservability::default()); + let mut scheduler = FirstAuditScheduler::new(Instant::now()); + let peer = test_peer(1); + + let live_high = MonetizedPinEvent { + peer, + pin: [1; 32], + key_count: 400, + quote_ts: SystemTime::now(), + }; + scheduler.enqueue(live_high, &obs); + let cheaper = MonetizedPinEvent { + peer, + pin: [2; 32], + key_count: 100, + quote_ts: SystemTime::now(), + }; + scheduler.enqueue(cheaper, &obs); + + assert_eq!(scheduler.pending_len(), 1); + assert_eq!( + scheduler.pending.peek(&peer).map(|e| e.pin), + Some([1; 32]), + "the higher-count live pin is retained" + ); + assert_eq!(obs.suppressed_lower.load(Ordering::Relaxed), 1); + assert_eq!(obs.outside_answerability_window.load(Ordering::Relaxed), 0); + } + + /// The periodic sweep collects expired pending entries even when the token + /// bucket is empty — the reserve path returns at the budget gate and never + /// scans — keeping the bounded LRU and the pending/oldest-age telemetry + /// honest under fleet-wide starvation. + #[test] + fn first_audit_sweep_expired_collects_dead_entries_without_tokens() { + let obs = Arc::new(FirstAuditObservability::default()); + let mono = Instant::now(); + let mut scheduler = FirstAuditScheduler::new(mono); + + let dead_quote = SystemTime::now() + .checked_sub(GOSSIP_ANSWERABILITY_TTL) + .and_then(|t| t.checked_sub(Duration::from_secs(60))) + .expect("past wall time"); + scheduler.enqueue( + MonetizedPinEvent { + peer: test_peer(1), + pin: [1; 32], + key_count: 100, + quote_ts: dead_quote, + }, + &obs, + ); + scheduler.enqueue( + MonetizedPinEvent { + peer: test_peer(2), + pin: [2; 32], + key_count: 100, + quote_ts: SystemTime::now(), + }, + &obs, + ); + assert_eq!(scheduler.pending_len(), 2); + + // Empty the bucket so the reserve path is budget-gated (fleet-wide + // starvation) and cannot collect the dead entry itself. + scheduler.limiter.tokens = 0; + let cooldown: HashMap = HashMap::new(); + assert!(!scheduler.try_reserve(mono, 0, Duration::ZERO, &cooldown, &obs)); + assert_eq!( + scheduler.pending_len(), + 2, + "budget-gated reserve scans nothing" + ); + + let wall_now = SystemTime::now(); + assert_eq!(scheduler.sweep_expired(wall_now, &obs), 1); + assert_eq!(scheduler.pending_len(), 1); + assert!( + scheduler.pending.peek(&test_peer(2)).is_some(), + "the live entry survives the sweep" + ); + assert_eq!(obs.outside_answerability_window.load(Ordering::Relaxed), 1); + let age_ms = scheduler.oldest_pending_quote_age_ms(wall_now); + assert!( + Duration::from_millis(age_ms) < GOSSIP_ANSWERABILITY_TTL, + "the age gauge reflects only live work after the sweep" + ); + } + #[test] #[allow(clippy::too_many_lines)] fn first_audit_answerability_cancel_is_state_neutral_and_retains_successor() { From c98d3b18dc69885a2f3980f42ba54fe96773992c Mon Sep 17 00:00:00 2001 From: grumbach Date: Tue, 21 Jul 2026 16:27:46 +0900 Subject: [PATCH 17/21] docs(adr): ADR-0004 Amendment 3 - eager expiry and detection model --- ...ADR-0004-commitment-bound-quote-pricing.md | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md index 682ac5c0..04f7a954 100644 --- a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md +++ b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md @@ -478,3 +478,26 @@ rewards-address binding. This griefing race predates this amendment and needs a vault-side fix (reject or accumulate on existing entries); it is tracked as follow-up work, not a property this ADR can enforce from the node side. + +## Amendment 3 (2026-07-21): expired pending entries are dropped eagerly + +A pending entry past the answerability horizon can never launch, but the +reserve scan that collected such entries only runs when launch tokens are +available — under sustained load, effectively never (2026-07-17 staging: +median oldest-pending age 270 minutes against the 150-minute eligibility +window). Dead entries squatted the per-peer LRU, skewed the +`pending`/`oldest_pending_quote_age_ms` telemetry, and — because coalescing +compares key counts only — a dead higher-count pin permanently vetoed every +live lower-count nomination for its peer (realistic after a legitimate +prune). Fixed: enqueue drops a dead incumbent before coalescing, and the +summary tick sweeps expired entries regardless of tokens; both removals are +accounted as `outside_answerability_window`. + +Coverage remains as Amendment 2 states — a budgeted probabilistic exam, not +a per-payment guarantee. Quantitatively: round 1 is structural, so only the +3-5 fresh-random round-2 byte openings prove possession; per-audit detection +of a peer missing fraction `x` of committed bytes is ~`1-(1-x)^k` dispersed +(`k` = opened leaves), degrading toward the subtree-selection rate when +clustered, compounding across the fleet's per-peer audit rate. Proposed +target, pending an adversarial staging run with hash-retaining deleters: +at least 99% detection within 24 hours for at least 1% missing bytes. From c723e8af525c13f9cfae4de728192942a0a6dd8a Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 23 Jul 2026 13:40:06 +0900 Subject: [PATCH 18/21] chore(replication): drop test import orphaned by rebase onto main The pre-rebase merge of main had resolved the tests-module import list to use the qualified path for AuditFailureSummary; replaying the linear commits resurrected the unused import. --- src/replication/mod.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 569f5c6b..5e7ef54a 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -6052,7 +6052,7 @@ mod tests { use crate::replication::audit::AuditTickResult; use crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL; use crate::replication::recent_provers::RecentProvers; - use crate::replication::types::{AuditFailureReason, AuditFailureSummary, FailureEvidence}; + use crate::replication::types::{AuditFailureReason, FailureEvidence}; use lru::LruCache; use saorsa_core::identity::PeerId; use std::collections::HashMap; From b5b90dd6cad22de27c6aae68c38128a820ffee11 Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 23 Jul 2026 15:17:06 +0900 Subject: [PATCH 19/21] fix(replication): losing gossip lottery no longer stamps the shared audit cooldown audit_launch_decision stamped the shared audit_on_gossip_cooldown before consulting the lottery, so a losing ticket consumed the 30-minute window despite sending no challenge. The monetized first-audit scheduler reads the same map as recent-audit coverage, so a peer whose changed commitment kept losing the lottery near each reopening could hold a paid first audit pending until its answerability window expired, and sidecar-only pins have no gossip-lottery backstop. The lottery attempt window is now a gossip-private map (stamped on every roll, win or lose, preserving one-roll-per-window flood resistance) and the shared cooldown is stamped only when a real audit launches. Also documents the promotion-before-send stamped-but-unsent residual on resolve(). --- src/replication/mod.rs | 191 +++++++++++++++++++++++++++++++---------- 1 file changed, 144 insertions(+), 47 deletions(-) diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 5e7ef54a..c913b3a0 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -741,6 +741,15 @@ impl FirstAuditScheduler { /// during jitter) or requeued (a concurrent gossip audit won the cooldown). /// On both `None` paths the token is refunded, the in-flight slot released, /// and NO suppression is recorded. + /// + /// Known operational residual: suppression is stamped here, at promotion, + /// while the wire challenge is sent by the detached task the caller spawns + /// with the returned event. A task-start or encoding failure between the + /// two therefore leaves a stamped-but-unsent window. No remote input can + /// force that failure (it requires local task-spawn/alloc failure), so it + /// is accepted rather than closed; closing it would require stamping from + /// inside the spawned task and re-introduce the cancel-leaves-suppression + /// race this design exists to prevent. fn resolve( &mut self, reservation: FirstAuditReservation, @@ -987,6 +996,11 @@ pub struct ReplicationEngine { /// commitment changes cannot spawn back-to-back audits of the same peer. /// Bounded by routing-table membership and cleaned on `PeerRemoved`. audit_on_gossip_cooldown: Arc>>, + /// Gossip-private lottery attempt window (one roll per peer per window, + /// win or lose). Kept separate from `audit_on_gossip_cooldown` so a losing + /// ticket never stamps the shared map and thus never defers a monetized + /// first audit. Bounded like its sibling and cleaned on `PeerRemoved`. + gossip_lottery_attempts: Arc>>, /// Completed local neighbor-sync cycle epoch for proof maturity. sync_cycle_epoch: Arc>, /// Per-key repair proof tracking for audit eligibility. @@ -1141,6 +1155,7 @@ impl ReplicationEngine { sync_state: Arc::new(RwLock::new(initial_neighbors)), sync_history: Arc::new(RwLock::new(HashMap::new())), audit_on_gossip_cooldown: Arc::new(RwLock::new(HashMap::new())), + gossip_lottery_attempts: Arc::new(RwLock::new(HashMap::new())), sync_cycle_epoch: Arc::new(RwLock::new(0)), repair_proofs: Arc::new(RwLock::new(RepairProofs::new())), bootstrap_state: Arc::new(RwLock::new(BootstrapState::new())), @@ -1548,6 +1563,7 @@ impl ReplicationEngine { recent_provers: Arc::clone(&self.recent_provers), sync_state: Arc::clone(&self.sync_state), cooldown: Arc::clone(&self.audit_on_gossip_cooldown), + lottery_attempts: Arc::clone(&self.gossip_lottery_attempts), }; let shutdown = self.shutdown.clone(); let observability = Arc::new(FirstAuditObservability::default()); @@ -1789,6 +1805,7 @@ impl ReplicationEngine { let recent_provers = Arc::clone(&self.recent_provers); let sig_verify_attempts = Arc::clone(&self.sig_verify_attempts); let audit_on_gossip_cooldown = Arc::clone(&self.audit_on_gossip_cooldown); + let gossip_lottery_attempts = Arc::clone(&self.gossip_lottery_attempts); let sync_state = Arc::clone(&self.sync_state); let audit_responder_semaphore = Arc::clone(&self.audit_responder_semaphore); let audit_responder_inflight = Arc::clone(&self.audit_responder_inflight); @@ -1801,6 +1818,7 @@ impl ReplicationEngine { recent_provers: Arc::clone(&recent_provers), sync_state: Arc::clone(&sync_state), cooldown: Arc::clone(&audit_on_gossip_cooldown), + lottery_attempts: Arc::clone(&gossip_lottery_attempts), }; let handle = tokio::spawn(async move { @@ -1921,8 +1939,10 @@ impl ReplicationEngine { last_commitment_by_peer.write().await.remove(&peer_id); recent_provers.write().await.forget_peer(&peer_id); sig_verify_attempts.write().await.remove(&peer_id); - // Same for the gossip-audit cooldown (ADR-0002). + // Same for the gossip-audit cooldown (ADR-0002) + // and the lottery-attempt window. audit_on_gossip_cooldown.write().await.remove(&peer_id); + gossip_lottery_attempts.write().await.remove(&peer_id); // The sticky `commitment_capable` flag is // preserved orthogonally via // `ever_capable_peers` — even after this @@ -1968,6 +1988,7 @@ impl ReplicationEngine { recent_provers: Arc::clone(&self.recent_provers), sync_state: Arc::clone(&sync_state), cooldown: Arc::clone(&self.audit_on_gossip_cooldown), + lottery_attempts: Arc::clone(&self.gossip_lottery_attempts), }; let handle = tokio::spawn(async move { @@ -5352,7 +5373,15 @@ struct GossipAuditTrigger { config: Arc, recent_provers: Arc>, sync_state: Arc>, + /// Shared "an audit actually launched" cooldown, consulted by BOTH the + /// gossip-lottery path and the monetized first-audit scheduler. Stamped + /// only when a real audit is about to be sent — never by a losing lottery + /// ticket — so gossip traffic alone can never suppress a paid first audit. cooldown: Arc>>, + /// Gossip-private lottery attempt window: stamped on every roll (win or + /// lose) so a gossip flood cannot re-roll the lottery within the window. + /// The first-audit scheduler never reads this map. + lottery_attempts: Arc>>, } /// What a gossip ingest yields for the audit trigger: the commitment hash to @@ -5433,27 +5462,39 @@ fn cooldown_would_allow(map: &HashMap, peer: &PeerId, now: Inst /// The gossip-audit launch decision in ONE place so the ordering is shared /// between production and its test (ADR-0002 "occasional surprise exams"). /// -/// Order matters and is the security-relevant property: the per-peer cooldown is -/// checked-and-stamped FIRST, THEN the probability lottery (`lottery_wins`) is -/// applied. If the lottery were sampled first, a gossip flood would re-roll it on -/// every message until one won, multiplying audits. Because the cooldown is -/// stamped before the lottery is consulted, a LOSING ticket still consumes the -/// window — so each peer gets at most one audit lottery per cooldown window -/// regardless of how often it gossips. Production calls this with -/// `lottery_wins = gen_bool(AUDIT_ON_GOSSIP_PROBABILITY)`; the test calls it with -/// a deterministic `lottery_wins`, so a reorder regression here fails the test. +/// Order matters and is the security-relevant property. Gate 1 checks-and-stamps +/// the gossip-PRIVATE `attempts` window, win or lose: if the lottery were +/// sampled first, a gossip flood would re-roll it on every message until one +/// won, multiplying audits, so each peer gets at most one lottery roll per +/// window regardless of how often it gossips. Gate 3 checks-and-stamps the +/// SHARED `launched` cooldown — the map the monetized first-audit scheduler +/// also consults — only after a WIN, i.e. only when a real audit is about to be +/// sent. A losing ticket must never stamp `launched`: no challenge went on the +/// wire, so it must not defer a paid first audit (repeatable losses could +/// otherwise hold a monetized pin past its answerability window). Production +/// calls this with `lottery_wins = gen_bool(AUDIT_ON_GOSSIP_PROBABILITY)`; the +/// test calls it with a deterministic `lottery_wins`, so a reorder regression +/// here fails the test. fn audit_launch_decision( - map: &mut HashMap, + attempts: &mut HashMap, + launched: &mut HashMap, peer: &PeerId, now: Instant, lottery_wins: bool, ) -> bool { - // Gate 1: cooldown check-and-stamp (consumes the window even on a loss). - if !cooldown_allows_audit(map, peer, now) { + // Gate 1: lottery-attempt window check-and-stamp (consumes the attempt + // window even on a loss; private to the gossip path). + if !cooldown_allows_audit(attempts, peer, now) { + return false; + } + // Gate 2: the probability lottery. A loss stops here and stamps nothing + // shared. + if !lottery_wins { return false; } - // Gate 2: the probability lottery. - lottery_wins + // Gate 3: the shared actual-audit cooldown (a recent real audit from + // either path still suppresses this launch), stamped only on launch. + cooldown_allows_audit(launched, peer, now) } /// On a peer's *changed* gossiped commitment, maybe launch a subtree audit @@ -5465,14 +5506,17 @@ async fn maybe_trigger_gossip_audit( peer: &PeerId, target: AuditTarget, ) { - // The launch decision (cooldown-then-lottery ordering) lives in the pure - // `audit_launch_decision` so the ordering is shared with its test. Sample - // the lottery here, then let the helper apply it AFTER the cooldown stamp. + // The launch decision (attempt-window, lottery, shared-cooldown ordering) + // lives in the pure `audit_launch_decision` so the ordering is shared with + // its test. Sample the lottery here, then let the helper apply the gates. let now = Instant::now(); let lottery_wins = rand::thread_rng().gen_bool(config::AUDIT_ON_GOSSIP_PROBABILITY); { - let mut map = trigger.cooldown.write().await; - if !audit_launch_decision(&mut map, peer, now, lottery_wins) { + // Lock order: attempts before the shared cooldown; this is the only + // place both are held together. + let mut attempts = trigger.lottery_attempts.write().await; + let mut launched = trigger.cooldown.write().await; + if !audit_launch_decision(&mut attempts, &mut launched, peer, now, lottery_wins) { return; } } @@ -7265,58 +7309,111 @@ mod tests { // would still pass them while breaking flood-resistance: a flood would then // re-roll the lottery on EVERY message until one won, multiplying audits. // - // We model the exact production gate order (cooldown-then-lottery) with a - // lottery driven by a fixed outcome instead of `gen_bool(..)`. The first - // message LOSES the lottery; the remaining flood messages all WIN. With the - // production order, the losing first ticket burns the window and every later - // winner in the same window is blocked, so there are 0 audits this window. If - // the gates were flipped, the second message's winning ticket would slip - // through. The window only reopens after the cooldown elapses. + // We model the exact production gate order (attempt-window, lottery, + // shared cooldown) with a lottery driven by a fixed outcome instead of + // `gen_bool(..)`. The first message LOSES the lottery; the remaining flood + // messages all WIN. With the production order, the losing first ticket + // burns the ATTEMPT window and every later winner in the same window is + // blocked, so there are 0 audits this window. If the gates were flipped, + // the second message's winning ticket would slip through. The window only + // reopens after it elapses. // - // FLIPS IF: the lottery is sampled before `cooldown_allows_audit` (a losing - // ticket no longer consumes the window), re-enabling a flood-amplified audit - // storm. + // FLIPS IF: the lottery is sampled before the attempt-window + // check-and-stamp (a losing ticket no longer consumes the window), + // re-enabling a flood-amplified audit storm. #[test] - fn losing_lottery_still_consumes_cooldown_window() { - // Faithful re-implementation of the two gates in - // `maybe_trigger_gossip_audit`, with the lottery outcome made - // deterministic instead of `rand::thread_rng().gen_bool(..)`. + fn losing_lottery_still_consumes_attempt_window() { // Calls the SHIPPED `audit_launch_decision` (the same function - // `maybe_trigger_gossip_audit` uses), so a reorder of the two gates in + // `maybe_trigger_gossip_audit` uses), so a reorder of the gates in // production fails this test — not a local reimplementation. let peer = strike_peer(3); - let mut map: HashMap = HashMap::new(); + let mut attempts: HashMap = HashMap::new(); + let mut launched: HashMap = HashMap::new(); let t0 = Instant::now(); - // First flooded message at t0 LOSES the lottery, but the cooldown is - // stamped BEFORE the lottery is consulted, so the window is now consumed. + // First flooded message at t0 LOSES the lottery, but the attempt window + // is stamped BEFORE the lottery is consulted, so the window is consumed. assert!( - !audit_launch_decision(&mut map, &peer, t0, false), + !audit_launch_decision(&mut attempts, &mut launched, &peer, t0, false), "a losing ticket launches no audit" ); // 99 more flooded messages at the same instant would all WIN the lottery, - // yet every one must be blocked by the cooldown the loser already stamped. + // yet every one must be blocked by the attempt window the loser stamped. // (If production sampled the lottery FIRST, these would each get a fresh // roll and audits would multiply — this assertion catches that reorder.) let mut audits = 0; for _ in 0..99 { - if audit_launch_decision(&mut map, &peer, t0, true) { + if audit_launch_decision(&mut attempts, &mut launched, &peer, t0, true) { audits += 1; } } assert_eq!( audits, 0, - "a losing first ticket must consume the window so no later flooded \ - message in the same window can audit" + "a losing first ticket must consume the attempt window so no later \ + flooded message in the same window can audit" ); - // The window only reopens after the cooldown elapses; the next winning - // ticket then launches exactly one audit. + // The window only reopens after it elapses; the next winning ticket + // then launches exactly one audit and stamps the SHARED cooldown. let after = t0 + Duration::from_secs(config::AUDIT_ON_GOSSIP_COOLDOWN_SECS + 1); assert!( - audit_launch_decision(&mut map, &peer, after, true), - "after the cooldown a winning ticket audits again" + audit_launch_decision(&mut attempts, &mut launched, &peer, after, true), + "after the window a winning ticket audits again" + ); + assert!( + launched.contains_key(&peer), + "a real launch stamps the shared cooldown" + ); + } + + /// The reviewer-flagged suppression route: a LOSING gossip lottery must not + /// stamp the SHARED audit cooldown, otherwise repeated losses (one per + /// 30-minute window, no challenge ever sent) keep a paid monetized pin's + /// first audit deferred until its answerability window expires. + /// + /// FLIPS IF: `audit_launch_decision` stamps the shared `launched` map on a + /// loss (the pre-split behavior, where both paths shared one map). + #[test] + fn losing_lottery_does_not_suppress_monetized_first_audit() { + let peer = strike_peer(4); + let mut attempts: HashMap = HashMap::new(); + let mut launched: HashMap = HashMap::new(); + let t0 = Instant::now(); + + // A losing ticket consumes the gossip attempt window... + assert!(!audit_launch_decision( + &mut attempts, + &mut launched, + &peer, + t0, + false + )); + // ...but leaves the shared map untouched, so the first-audit reserve + // gate (read-only) and the authoritative promotion check-and-stamp both + // still allow the paid audit to launch immediately. + assert!( + cooldown_would_allow(&launched, &peer, t0), + "reserve gate must not see a losing ticket as audit coverage" + ); + assert!( + cooldown_allows_audit(&mut launched, &peer, t0), + "promotion must not be deferred by a losing ticket" + ); + + // Conversely a WINNING ticket (real audit sent) does suppress the + // first audit for the window, which is the intended shared semantics. + let peer_won = strike_peer(5); + assert!(audit_launch_decision( + &mut attempts, + &mut launched, + &peer_won, + t0, + true + )); + assert!( + !cooldown_would_allow(&launched, &peer_won, t0), + "a real gossip audit still covers the peer for the window" ); } From 4309dac836812691b7dd03a41c27ef71f345012e Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 23 Jul 2026 13:18:27 +0900 Subject: [PATCH 20/21] fix(replication): cap first-audit pending admission at the launch budget The pending queue shared the 4096-entry commitment-cache cap while the token bucket can launch at most ~32 audits inside one effective answerability window, so under production-scale arrival rates (staging 2026-07-17: ~113 admitted nominations per node-hour against ~12 launches) over 99% of admitted work could never launch. The queue became an aging backlog whose entries mostly expired, and the pending/oldest-age telemetry measured the backlog instead of schedulable work. pending is now capped at FIRST_AUDIT_PENDING_CAP, derived from the budget and window constants with compile-time guardrails. At capacity the LRU displaces the least-recently-refreshed entry, counted as capacity_evicted; keep-newest displacement preserves the newest-first lane's prompt-audit deterrence under flood, and displacement stamps no suppression state. The summary line reports pending_cap so occupancy is a dashboard quantity. ADR-0004 Amendment 4 documents the sampling coverage semantics. --- ...ADR-0004-commitment-bound-quote-pricing.md | 39 ++++ src/replication/mod.rs | 169 +++++++++++++++++- 2 files changed, 202 insertions(+), 6 deletions(-) diff --git a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md index 04f7a954..b85425c3 100644 --- a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md +++ b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md @@ -501,3 +501,42 @@ of a peer missing fraction `x` of committed bytes is ~`1-(1-x)^k` dispersed clustered, compounding across the fleet's per-peer audit rate. Proposed target, pending an adversarial staging run with hash-retaining deleters: at least 99% detection within 24 hours for at least 1% missing bytes. + +## Amendment 4 (2026-07-23): pending admission is sized to the launch budget + +Amendment 3 stopped dead entries from squatting the queue, but admission +itself was still unbounded relative to the budget: `pending` shared the +4096-entry commitment-cache cap while the token bucket can launch at most +~32 audits inside one effective answerability window (150 min of +eligibility at one token per 5 minutes, plus a burst of 2). At the +2026-07-17 staging arrival rate (~113 admitted nominations per node-hour +against ~12 launches) over 99% of admitted work could never launch: the +queue was an aging backlog whose entries mostly expired, and the +`pending` / `oldest_pending_quote_age_ms` telemetry measured the backlog's +age rather than schedulable work. + +`pending` is now capped at `FIRST_AUDIT_PENDING_CAP`, derived from the same +constants that define the budget and the window +(`(GOSSIP_ANSWERABILITY_TTL − MONETIZED_AUDIT_SKEW_MARGIN) / +FIRST_AUDIT_LAUNCH_INTERVAL + FIRST_AUDIT_BUDGET_BURST`, currently 32), so +retuning either side re-sizes admission automatically. At capacity the LRU +displaces the least-recently-refreshed entry, accounted as +`capacity_evicted` — now the steady overload signal — and the summary line +reports `pending_cap` so occupancy is a first-class dashboard quantity. + +Keep-newest displacement is deliberate. Under overload the queue degrades +to a budget-sized rolling sample biased toward fresh nominations, so the +newest-first lane keeps its prompt-audit deterrence: a cheater's fresh pin +always enters the sample and retains an unpredictable chance of +near-immediate audit. Refusing new admissions at capacity would invert +that — a sustained payment flood would keep the queue full and guarantee +every later pin (including the flooder's next target) is never admitted. +Displacement stamps no suppression state, so a displaced peer's next +nomination is judged like any newcomer. Under light load (arrivals below +the drain rate) the cap is not reached and coverage remains exhaustive. + +Coverage semantics are otherwise unchanged from Amendments 2-3: a budgeted +probabilistic exam backstopped by the gossip-lottery path. What changes is +the honesty of the funnel — admitted now approximates launchable, and +overflow is explicit at admission time (`capacity_evicted`) instead of +surfacing hours later as mass expiry (`outside_answerability_window`). diff --git a/src/replication/mod.rs b/src/replication/mod.rs index c913b3a0..2f169570 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -474,7 +474,10 @@ struct FirstAuditScheduler { /// (a real send), never at reservation, so a cancelled reservation can be /// re-nominated. first_audited: LruCache<[u8; 32], ()>, - /// Newest-per-peer pending nominations not yet launched. + /// Newest-per-peer pending nominations not yet launched. Capped at + /// [`FIRST_AUDIT_PENDING_CAP`] (what the token budget can launch within + /// one effective answerability window); at capacity the least-recently + /// refreshed entry is displaced and counted as `capacity_evicted`. pending: LruCache, /// Token bucket + per-peer re-audit window. limiter: FirstAuditLimiter, @@ -488,10 +491,11 @@ struct FirstAuditScheduler { impl FirstAuditScheduler { fn new(now: Instant) -> Self { - let cap = NonZeroUsize::new(MAX_LAST_COMMITMENT_BY_PEER).unwrap_or(NonZeroUsize::MIN); + let dedup_cap = NonZeroUsize::new(MAX_LAST_COMMITMENT_BY_PEER).unwrap_or(NonZeroUsize::MIN); + let pending_cap = NonZeroUsize::new(FIRST_AUDIT_PENDING_CAP).unwrap_or(NonZeroUsize::MIN); Self { - first_audited: LruCache::new(cap), - pending: LruCache::new(cap), + first_audited: LruCache::new(dedup_cap), + pending: LruCache::new(pending_cap), limiter: FirstAuditLimiter::new(now), reserved: None, oldest_first_lane: false, @@ -577,7 +581,11 @@ impl FirstAuditScheduler { /// highest-count-per-peer (newest on a tie) — a lower-count successor never /// displaces a higher-count pending pin. An incumbent that has aged past /// the answerability horizon is dropped (and accounted as an expiry) before - /// coalescing, so a dead pin never vetoes a live nomination. + /// coalescing, so a dead pin never vetoes a live nomination. Admission for + /// a NEW peer at capacity displaces the least-recently-refreshed entry + /// (see [`FIRST_AUDIT_PENDING_CAP`]): displacement is accounted as + /// `capacity_evicted` and stamps no suppression, so a displaced peer's + /// next nomination is judged like any newcomer. fn enqueue(&mut self, event: MonetizedPinEvent, obs: &Arc) { if self.first_audited.contains(&event.pin) { obs.duplicates.fetch_add(1, Ordering::Relaxed); @@ -954,6 +962,43 @@ const COMMITMENT_SIG_VERIFY_MIN_INTERVAL: Duration = Duration::from_secs(60); /// the cap is the third line of defence against sybil/churn flooding. const MAX_LAST_COMMITMENT_BY_PEER: usize = 4096; +/// ADR-0004 Amendment 4: admission cap for the first-audit pending queue, +/// sized to the launch budget instead of the commitment cache. +/// +/// The token bucket can launch at most one audit per +/// [`config::FIRST_AUDIT_LAUNCH_INTERVAL`] (plus the +/// [`config::FIRST_AUDIT_BUDGET_BURST`] allowance), and a pending pin stays +/// launchable for at most the effective answerability window +/// ([`GOSSIP_ANSWERABILITY_TTL`] − [`MONETIZED_AUDIT_SKEW_MARGIN`]). Any +/// occupancy beyond `window / interval + burst` is work that cannot launch +/// before it expires, so admitting it only builds an aging backlog whose +/// telemetry measures the backlog instead of schedulable work. +/// +/// At capacity `LruCache::push` displaces the least-recently-refreshed entry +/// (counted as `capacity_evicted`). Keep-newest is deliberate: under overload +/// `pending` degrades to a budget-sized rolling sample biased toward fresh +/// nominations, so the newest-first lane keeps its prompt-audit deterrence — +/// a fresh pin always enters the sample with an unpredictable chance of +/// near-immediate audit — whereas refusing at the door would let a sustained +/// payment flood guarantee that every later pin is never admitted. +#[allow(clippy::cast_possible_truncation)] // bounded by TTL-secs/interval-secs + burst (~32) +const FIRST_AUDIT_PENDING_CAP: usize = { + let cutoff_secs = GOSSIP_ANSWERABILITY_TTL + .as_secs() + .saturating_sub(MONETIZED_AUDIT_SKEW_MARGIN.as_secs()); + // Compile-time division-by-zero if the launch interval is ever zeroed: + // a zero interval makes the budget (and this cap) meaningless. + let launchable = cutoff_secs / config::FIRST_AUDIT_LAUNCH_INTERVAL.as_secs(); + (launchable + config::FIRST_AUDIT_BUDGET_BURST as u64) as usize +}; + +// Compile-time guardrails: the cap must be a usable `LruCache` capacity and +// strictly tighter than the commitment-cache bound it replaced. +const _: () = { + assert!(FIRST_AUDIT_PENDING_CAP >= 1); + assert!(FIRST_AUDIT_PENDING_CAP < MAX_LAST_COMMITMENT_BY_PEER); +}; + /// Cap on the sticky `ever_capable_peers` set. Bounds memory so a /// long-running bootstrap node cannot have the set grow without limit /// from peer-id churn. Sized at 4x `MAX_LAST_COMMITMENT_BY_PEER` so @@ -1750,7 +1795,7 @@ impl ReplicationEngine { ); } info!( - "First-audit scheduler summary: audit_trigger=first_monetized ingress_dropped={} received={} queued={} coalesced={} suppressed_lower={} duplicates={} capacity_evicted={} cooldown_deferred_attempts={} rate_deferred_attempts={} window_deduped={} launched={} passed={} timeout={} failed={} bootstrap_claims={} idle={} insufficient_keys={} outside_answerability_window={} pending={} oldest_pending_quote_age_ms={} inflight={} tokens={}", + "First-audit scheduler summary: audit_trigger=first_monetized ingress_dropped={} received={} queued={} coalesced={} suppressed_lower={} duplicates={} capacity_evicted={} cooldown_deferred_attempts={} rate_deferred_attempts={} window_deduped={} launched={} passed={} timeout={} failed={} bootstrap_claims={} idle={} insufficient_keys={} outside_answerability_window={} pending={} pending_cap={} oldest_pending_quote_age_ms={} inflight={} tokens={}", FIRST_AUDIT_INGRESS_DROPPED.load(Ordering::Relaxed), observability.received.load(Ordering::Relaxed), observability.queued.load(Ordering::Relaxed), @@ -1770,6 +1815,7 @@ impl ReplicationEngine { observability.insufficient_keys.load(Ordering::Relaxed), observability.outside_answerability_window.load(Ordering::Relaxed), scheduler.pending_len(), + FIRST_AUDIT_PENDING_CAP, scheduler.oldest_pending_quote_age_ms(SystemTime::now()), observability.inflight.load(Ordering::Relaxed), scheduler.tokens(), @@ -6574,6 +6620,117 @@ mod tests { assert_eq!(obs.outside_answerability_window.load(Ordering::Relaxed), 0); } + /// A fresh nomination for peer `b` (distinct pin per peer). + fn live_nomination(b: u8) -> MonetizedPinEvent { + MonetizedPinEvent { + peer: test_peer(b), + pin: [b; 32], + key_count: 100, + quote_ts: SystemTime::now(), + } + } + + /// The pending admission cap must equal what the token budget can launch + /// inside one effective answerability window (plus burst): any larger and + /// admitted work is guaranteed to expire unlaunched; any smaller and the + /// budget idles while nominations are refused. + #[test] + fn first_audit_pending_cap_matches_window_budget() { + let cutoff_secs = GOSSIP_ANSWERABILITY_TTL + .saturating_sub(MONETIZED_AUDIT_SKEW_MARGIN) + .as_secs(); + let expected = usize::try_from( + cutoff_secs / config::FIRST_AUDIT_LAUNCH_INTERVAL.as_secs() + + u64::from(config::FIRST_AUDIT_BUDGET_BURST), + ) + .expect("cap fits usize"); + assert_eq!(FIRST_AUDIT_PENDING_CAP, expected); + } + + /// The pending queue is sized to the launch budget while the dedup set + /// keeps the commitment-cache bound: shrinking `first_audited` would + /// forget audited pins and re-admit duplicates. + #[test] + fn first_audit_pending_cap_independent_of_dedup_cap() { + let scheduler = FirstAuditScheduler::new(Instant::now()); + assert_eq!(scheduler.pending.cap().get(), FIRST_AUDIT_PENDING_CAP); + assert_eq!( + scheduler.first_audited.cap().get(), + MAX_LAST_COMMITMENT_BY_PEER + ); + } + + /// Admission beyond the cap displaces the least-recently-refreshed entry + /// (keep-newest sampling) and is accounted as `capacity_evicted` — the + /// steady overload signal — never silent loss. + #[test] + fn first_audit_admission_beyond_cap_displaces_oldest_and_counts() { + let obs = Arc::new(FirstAuditObservability::default()); + let mut scheduler = FirstAuditScheduler::new(Instant::now()); + let overflow: usize = 5; + let total = FIRST_AUDIT_PENDING_CAP + overflow; + + for i in 0..total { + let b = u8::try_from(i + 1).expect("test peer count fits u8"); + scheduler.enqueue(live_nomination(b), &obs); + } + + assert_eq!(scheduler.pending_len(), FIRST_AUDIT_PENDING_CAP); + assert_eq!( + obs.capacity_evicted.load(Ordering::Relaxed), + u64::try_from(overflow).expect("overflow fits u64"), + "each admission past the cap displaces exactly one entry" + ); + assert_eq!( + obs.queued.load(Ordering::Relaxed), + u64::try_from(total).expect("total fits u64"), + "displacement is not an admission refusal" + ); + for i in 0..overflow { + let b = u8::try_from(i + 1).expect("test peer count fits u8"); + assert!( + scheduler.pending.peek(&test_peer(b)).is_none(), + "oldest-admitted entries are the displaced ones" + ); + } + for i in overflow..total { + let b = u8::try_from(i + 1).expect("test peer count fits u8"); + assert!( + scheduler.pending.peek(&test_peer(b)).is_some(), + "newest entries survive as the budget-sized sample" + ); + } + } + + /// Displacement stamps no suppression state: a displaced peer's next + /// nomination re-enters like any newcomer (it is NOT window-deduped or + /// treated as a duplicate), so under overload every peer keeps an + /// unpredictable chance of prompt first audit. + #[test] + fn first_audit_displaced_peer_may_be_renominated() { + let obs = Arc::new(FirstAuditObservability::default()); + let mut scheduler = FirstAuditScheduler::new(Instant::now()); + let total = FIRST_AUDIT_PENDING_CAP + 1; + + for i in 0..total { + let b = u8::try_from(i + 1).expect("test peer count fits u8"); + scheduler.enqueue(live_nomination(b), &obs); + } + let displaced = test_peer(1); + assert!(scheduler.pending.peek(&displaced).is_none()); + + scheduler.enqueue(live_nomination(1), &obs); + + assert!( + scheduler.pending.peek(&displaced).is_some(), + "the displaced peer re-enters the sample" + ); + assert_eq!(scheduler.pending_len(), FIRST_AUDIT_PENDING_CAP); + assert_eq!(obs.capacity_evicted.load(Ordering::Relaxed), 2); + assert_eq!(obs.duplicates.load(Ordering::Relaxed), 0); + assert_eq!(obs.window_deduped.load(Ordering::Relaxed), 0); + } + /// The periodic sweep collects expired pending entries even when the token /// bucket is empty — the reserve path returns at the budget gate and never /// scans — keeping the bounded LRU and the pending/oldest-age telemetry From 3462edf3c4f183f47c7dc83f68a1fb46fb3f2c3a Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 23 Jul 2026 15:24:22 +0900 Subject: [PATCH 21/21] fix(replication): random-victim retention and pre-overflow reservation for the pending cap Review findings on the admission cap: - Deterministic keep-newest LRU displacement let an ordered batch of cap distinct-peer nominations flush a chosen target inside one 64-event ingress drain, before either launch lane could see it, cutting the documented suppression cost from 4096 nominations to 32. Overflow now displaces a uniformly random incumbent (per-nomination eviction probability 1/cap, never certain), and the drainer grants pending work a reservation opportunity before any destructive overflow, so a burst can never flush eligible work past a ready token. - The cap derivation over-counted by one: the strict reserve predicate (answerable through max jitter plus send slack) permits 31 usable launches per window, not 32. The unit test now simulates the shipped predicate instant-by-instant instead of repeating the formula. - The reservation is held outside pending, so the summary now reports reserved alongside pending/pending_cap; schedulable occupancy is their sum. Regressions: ordered-flood non-determinism (100-trial statistical bounds with miss odds below 1e-20), reserve-before-eviction at capacity, and displaced-peer re-nomination under random retention. --- ...ADR-0004-commitment-bound-quote-pricing.md | 59 ++- src/replication/mod.rs | 354 ++++++++++++++---- 2 files changed, 311 insertions(+), 102 deletions(-) diff --git a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md index b85425c3..aa16795d 100644 --- a/docs/adr/ADR-0004-commitment-bound-quote-pricing.md +++ b/docs/adr/ADR-0004-commitment-bound-quote-pricing.md @@ -507,7 +507,7 @@ at least 99% detection within 24 hours for at least 1% missing bytes. Amendment 3 stopped dead entries from squatting the queue, but admission itself was still unbounded relative to the budget: `pending` shared the 4096-entry commitment-cache cap while the token bucket can launch at most -~32 audits inside one effective answerability window (150 min of +~31 audits inside one effective answerability window (150 min of eligibility at one token per 5 minutes, plus a burst of 2). At the 2026-07-17 staging arrival rate (~113 admitted nominations per node-hour against ~12 launches) over 99% of admitted work could never launch: the @@ -515,25 +515,44 @@ queue was an aging backlog whose entries mostly expired, and the `pending` / `oldest_pending_quote_age_ms` telemetry measured the backlog's age rather than schedulable work. -`pending` is now capped at `FIRST_AUDIT_PENDING_CAP`, derived from the same -constants that define the budget and the window -(`(GOSSIP_ANSWERABILITY_TTL − MONETIZED_AUDIT_SKEW_MARGIN) / -FIRST_AUDIT_LAUNCH_INTERVAL + FIRST_AUDIT_BUDGET_BURST`, currently 32), so -retuning either side re-sizes admission automatically. At capacity the LRU -displaces the least-recently-refreshed entry, accounted as -`capacity_evicted` — now the steady overload signal — and the summary line -reports `pending_cap` so occupancy is a first-class dashboard quantity. - -Keep-newest displacement is deliberate. Under overload the queue degrades -to a budget-sized rolling sample biased toward fresh nominations, so the -newest-first lane keeps its prompt-audit deterrence: a cheater's fresh pin -always enters the sample and retains an unpredictable chance of -near-immediate audit. Refusing new admissions at capacity would invert -that — a sustained payment flood would keep the queue full and guarantee -every later pin (including the flooder's next target) is never admitted. -Displacement stamps no suppression state, so a displaced peer's next -nomination is judged like any newcomer. Under light load (arrivals below -the drain rate) the cap is not reached and coverage remains exhaustive. +`pending` is now capped at `FIRST_AUDIT_PENDING_CAP`, the number of usable +launches inside one window under the STRICT reserve-time predicate: burst +tokens at age zero plus every refill that still leaves the quote answerable +through the maximum launch jitter and send slack (currently 31). The unit +test simulates the shipped predicate instant-by-instant and must agree with +the derivation, so retuning the budget, window, or jitter re-sizes +admission automatically. The summary line reports `pending_cap` and +`reserved` (the one reservation held outside the queue), so schedulable +occupancy is `pending + reserved`, a first-class dashboard quantity. + +Overflow follows two rules, in order. First, an arrival that would displace +a different peer grants the queue a reservation opportunity before anything +is evicted: if a token and in-flight slot are free, one pending entry moves +into the reservation and the arrival is admitted with no eviction, so an +ingress burst can never flush eligible work past a ready token. Second, +when no launch is possible, admission displaces a uniformly RANDOM +incumbent, accounted as `capacity_evicted` — the steady overload signal. +Random selection is the security property: with deterministic retention +(oldest-out or newest-kept), an ordered batch of `cap` distinct-peer paid +nominations flushes a chosen target with certainty; with a random victim, +each overflow evicts a specific target with probability `1/cap`, so +suppressing it with confidence `1−e` costs about `cap · ln(1/e)` +distinct-peer nominations (~95 paid pins for 95% at cap 31, ~140 for 99%) +and is never certain, while the target may launch in the meantime. A fresh +pin still always enters the sample with an unpredictable chance of prompt +audit; refusing at the door would instead let a sustained flood guarantee +every later pin is never admitted. Displacement stamps no suppression +state, so a displaced peer's next nomination is judged like any newcomer. +Under light load (arrivals below the drain rate) the cap is not reached +and coverage remains exhaustive. + +For comparison, the pre-cap queue's 4,096-entry bound made targeted +eviction expensive but protected work that mostly could not launch anyway: +at observed staging rates, over 99% of those entries expired unaudited, so +the practical suppression of any given pin was already the budget share, +not the eviction threshold. The honest quantity to compare across designs +is P(audited before answerability expiry), which the cap leaves at the +budget share while making the skipped remainder visible at admission time. Coverage semantics are otherwise unchanged from Amendments 2-3: a budgeted probabilistic exam backstopped by the gossip-lottery path. What changes is diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 2f169570..71a73933 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -217,9 +217,33 @@ fn coalesce_first_audit_event( let _ = pending.push(incoming.peer, incoming); return FirstAuditQueueOutcome::Coalesced; } - // No same-peer entry: a `push` at capacity evicts the LRU (a DIFFERENT peer). + // No same-peer entry. At capacity, displace a UNIFORMLY RANDOM incumbent, + // never the LRU: deterministic keep-newest would let an ordered batch of + // `cap` distinct-peer nominations flush a chosen target before any launch + // lane sees it, cutting the eviction cost from probabilistic to exact. A + // random victim caps an attacker's per-nomination eviction probability at + // `1/cap` regardless of arrival order or timing, so suppressing a specific + // target with confidence `1-e` costs ~`cap*ln(1/e)` distinct-peer paid + // nominations and is never certain (ADR-0004 Amendment 4). + if pending.len() >= pending.cap().get() { + let victim = { + let idx = rand::thread_rng().gen_range(0..pending.len()); + pending.iter().nth(idx).map(|(p, _)| *p) + }; + if let Some(victim_peer) = victim { + if let Some(evicted) = pending.pop(&victim_peer) { + let _ = pending.push(incoming.peer, incoming); + return FirstAuditQueueOutcome::CapacityEvicted { + peer: victim_peer, + pin: evicted.pin, + }; + } + } + } match pending.push(incoming.peer, incoming) { None => FirstAuditQueueOutcome::Queued, + // Defensive: unreachable after the explicit random displacement above, + // but a racing cap change must still be accounted, never silent. Some((evicted_peer, evicted)) => FirstAuditQueueOutcome::CapacityEvicted { peer: evicted_peer, pin: evicted.pin, @@ -442,6 +466,38 @@ fn quote_answerable_through_nominal_jitter(quote_ts: SystemTime, now: SystemTime quote_within_audit_window(quote_ts, now) && quote_within_audit_window(quote_ts, latest_send) } +/// Open the single first-audit reservation from `pending` if none is +/// outstanding: samples the launch jitter, snapshots the shared cooldown +/// read-only, and delegates to [`FirstAuditScheduler::try_reserve`]. One +/// shared implementation for both drainer call sites — the per-wake launch +/// phase and the pre-overflow opportunity inside the ingress batch — so the +/// two cannot drift. +async fn open_first_audit_reservation( + scheduler: &mut FirstAuditScheduler, + cooldown: &RwLock>, + observability: &Arc, +) { + if scheduler.has_reservation() { + return; + } + let jitter = Duration::from_millis(rand::thread_rng().gen_range( + 0..=u64::try_from(config::FIRST_AUDIT_LAUNCH_JITTER_MAX.as_millis()).unwrap_or(u64::MAX), + )); + let inflight = observability.inflight.load(Ordering::Relaxed); + let reserved = { + let cooldown = cooldown.read().await; + scheduler.try_reserve(Instant::now(), inflight, jitter, &cooldown, observability) + }; + if reserved { + if let Some(peer) = scheduler.reserved_peer() { + debug!( + "First-audit scheduler: audit_trigger=first_monetized outcome=reserved peer={peer} pending={}", + scheduler.pending_len() + ); + } + } +} + /// A far-future `Instant` used to effectively DISABLE the promotion-timer /// select arm when no reservation is outstanding. The drainer still wakes at /// least every [`config::FIRST_AUDIT_RETRY_INTERVAL`] via its tick, so this only @@ -474,10 +530,14 @@ struct FirstAuditScheduler { /// (a real send), never at reservation, so a cancelled reservation can be /// re-nominated. first_audited: LruCache<[u8; 32], ()>, - /// Newest-per-peer pending nominations not yet launched. Capped at + /// Highest-count-per-peer pending nominations not yet launched. Capped at /// [`FIRST_AUDIT_PENDING_CAP`] (what the token budget can launch within - /// one effective answerability window); at capacity the least-recently - /// refreshed entry is displaced and counted as `capacity_evicted`. + /// one effective answerability window); at capacity a uniformly RANDOM + /// incumbent is displaced and counted as `capacity_evicted`, so eviction + /// of a specific entry can never be forced deterministically. One further + /// entry may be held in `reserved` outside this queue; the summary line + /// reports both (`pending`/`reserved`) so schedulable occupancy is + /// `pending + reserved`. pending: LruCache, /// Token bucket + per-peer re-audit window. limiter: FirstAuditLimiter, @@ -564,6 +624,16 @@ impl FirstAuditScheduler { self.reserved.is_some() } + /// Whether admitting `event` would displace a DIFFERENT peer's pending + /// entry: the queue is at capacity and `event.peer` has no incumbent to + /// coalesce into. The drainer checks this before enqueueing so pending + /// work gets a reservation opportunity BEFORE destructive overflow + /// (ADR-0004 Amendment 4) — a successful reservation moves one entry out + /// of `pending`, freeing the slot without any eviction. + fn would_displace(&self, event: &MonetizedPinEvent) -> bool { + self.pending.len() >= self.pending.cap().get() && self.pending.peek(&event.peer).is_none() + } + /// When the outstanding reservation becomes eligible for promotion. fn reserved_ready_at(&self) -> Option { self.reserved.as_ref().map(|r| r.ready_at) @@ -582,10 +652,13 @@ impl FirstAuditScheduler { /// displaces a higher-count pending pin. An incumbent that has aged past /// the answerability horizon is dropped (and accounted as an expiry) before /// coalescing, so a dead pin never vetoes a live nomination. Admission for - /// a NEW peer at capacity displaces the least-recently-refreshed entry - /// (see [`FIRST_AUDIT_PENDING_CAP`]): displacement is accounted as - /// `capacity_evicted` and stamps no suppression, so a displaced peer's - /// next nomination is judged like any newcomer. + /// a NEW peer at capacity displaces a uniformly RANDOM incumbent (see + /// [`FIRST_AUDIT_PENDING_CAP`] and [`coalesce_first_audit_event`]): + /// displacement is accounted as `capacity_evicted` and stamps no + /// suppression, so a displaced peer's next nomination is judged like any + /// newcomer. The drainer additionally offers a reservation opportunity + /// via [`Self::would_displace`] before calling this on an overflowing + /// arrival. fn enqueue(&mut self, event: MonetizedPinEvent, obs: &Arc) { if self.first_audited.contains(&event.pin) { obs.duplicates.fetch_add(1, Ordering::Relaxed); @@ -974,22 +1047,34 @@ const MAX_LAST_COMMITMENT_BY_PEER: usize = 4096; /// before it expires, so admitting it only builds an aging backlog whose /// telemetry measures the backlog instead of schedulable work. /// -/// At capacity `LruCache::push` displaces the least-recently-refreshed entry -/// (counted as `capacity_evicted`). Keep-newest is deliberate: under overload -/// `pending` degrades to a budget-sized rolling sample biased toward fresh -/// nominations, so the newest-first lane keeps its prompt-audit deterrence — -/// a fresh pin always enters the sample with an unpredictable chance of -/// near-immediate audit — whereas refusing at the door would let a sustained -/// payment flood guarantee that every later pin is never admitted. -#[allow(clippy::cast_possible_truncation)] // bounded by TTL-secs/interval-secs + burst (~32) +/// At capacity, admission displaces a uniformly RANDOM incumbent (counted as +/// `capacity_evicted`), so under overload `pending` degrades to a budget-sized +/// rolling sample in which no arrival order lets an attacker deterministically +/// flush a chosen target; a fresh pin always enters the sample with an +/// unpredictable chance of near-immediate audit, whereas refusing at the door +/// would let a sustained payment flood guarantee that every later pin is never +/// admitted. See [`coalesce_first_audit_event`] for the displacement rule and +/// ADR-0004 Amendment 4 for the eviction-cost math. +/// +/// Derivation uses the STRICT reserve-time horizon: a launch at quote age `a` +/// is usable only if the quote stays answerable through `a` plus the maximum +/// launch jitter and send slack, so the last usable refill is the largest +/// `k` with `k * interval < cutoff - jitter_max - slack`. The unit test +/// simulates the shipped predicate instant-by-instant and must agree. +#[allow(clippy::cast_possible_truncation)] // bounded by TTL-secs/interval-secs + burst (~31) const FIRST_AUDIT_PENDING_CAP: usize = { - let cutoff_secs = GOSSIP_ANSWERABILITY_TTL + let strict_horizon_secs = GOSSIP_ANSWERABILITY_TTL .as_secs() - .saturating_sub(MONETIZED_AUDIT_SKEW_MARGIN.as_secs()); - // Compile-time division-by-zero if the launch interval is ever zeroed: - // a zero interval makes the budget (and this cap) meaningless. - let launchable = cutoff_secs / config::FIRST_AUDIT_LAUNCH_INTERVAL.as_secs(); - (launchable + config::FIRST_AUDIT_BUDGET_BURST as u64) as usize + .saturating_sub(MONETIZED_AUDIT_SKEW_MARGIN.as_secs()) + .saturating_sub(config::FIRST_AUDIT_LAUNCH_JITTER_MAX.as_secs()) + .saturating_sub(FIRST_AUDIT_SEND_LATENCY_SLACK.as_secs()); + // Strictly-inside count: a refill landing exactly ON the horizon is dead + // (`age >= cutoff` rejects), hence the `- 1`. Compile-time division-by-zero + // if the launch interval is ever zeroed: a zero interval makes the budget + // (and this cap) meaningless. + let refills = + strict_horizon_secs.saturating_sub(1) / config::FIRST_AUDIT_LAUNCH_INTERVAL.as_secs(); + (refills + config::FIRST_AUDIT_BUDGET_BURST as u64) as usize }; // Compile-time guardrails: the cap must be a usable `LruCache` capacity and @@ -1639,6 +1724,20 @@ impl ReplicationEngine { event = rx.recv() => match event { Some(e) => { observability.received.fetch_add(1, Ordering::Relaxed); + // Pre-overflow reservation opportunity: before an + // arrival may displace a DIFFERENT peer, let the + // queue launch — a successful reservation frees the + // slot without eviction, so a single ingress batch + // can never flush eligible work past a ready token + // (ADR-0004 Amendment 4). + if scheduler.would_displace(&e) { + open_first_audit_reservation( + &mut scheduler, + &gossip_audit.cooldown, + &observability, + ) + .await; + } scheduler.enqueue(e, &observability); // Drain a bounded burst so a flood cannot starve the // launch phase. @@ -1647,6 +1746,14 @@ impl ReplicationEngine { match rx.try_recv() { Ok(e) => { observability.received.fetch_add(1, Ordering::Relaxed); + if scheduler.would_displace(&e) { + open_first_audit_reservation( + &mut scheduler, + &gossip_audit.cooldown, + &observability, + ) + .await; + } scheduler.enqueue(e, &observability); drained += 1; } @@ -1754,33 +1861,12 @@ impl ReplicationEngine { // authoritative stamp is at promotion). Serializing // reservations preserves per-launch lane alternation and keeps // at most one jitter timer live. - if !scheduler.has_reservation() { - let jitter = Duration::from_millis( - rand::thread_rng().gen_range( - 0..=u64::try_from(config::FIRST_AUDIT_LAUNCH_JITTER_MAX.as_millis()) - .unwrap_or(u64::MAX), - ), - ); - let inflight = observability.inflight.load(Ordering::Relaxed); - let reserved = { - let cooldown = gossip_audit.cooldown.read().await; - scheduler.try_reserve( - Instant::now(), - inflight, - jitter, - &cooldown, - &observability, - ) - }; - if reserved { - if let Some(peer) = scheduler.reserved_peer() { - debug!( - "First-audit scheduler: audit_trigger=first_monetized outcome=reserved peer={peer} pending={}", - scheduler.pending_len() - ); - } - } - } + open_first_audit_reservation( + &mut scheduler, + &gossip_audit.cooldown, + &observability, + ) + .await; if last_summary.elapsed() >= config::FIRST_AUDIT_SUMMARY_INTERVAL { // Token-independent hygiene: collect entries that aged past @@ -1795,7 +1881,7 @@ impl ReplicationEngine { ); } info!( - "First-audit scheduler summary: audit_trigger=first_monetized ingress_dropped={} received={} queued={} coalesced={} suppressed_lower={} duplicates={} capacity_evicted={} cooldown_deferred_attempts={} rate_deferred_attempts={} window_deduped={} launched={} passed={} timeout={} failed={} bootstrap_claims={} idle={} insufficient_keys={} outside_answerability_window={} pending={} pending_cap={} oldest_pending_quote_age_ms={} inflight={} tokens={}", + "First-audit scheduler summary: audit_trigger=first_monetized ingress_dropped={} received={} queued={} coalesced={} suppressed_lower={} duplicates={} capacity_evicted={} cooldown_deferred_attempts={} rate_deferred_attempts={} window_deduped={} launched={} passed={} timeout={} failed={} bootstrap_claims={} idle={} insufficient_keys={} outside_answerability_window={} pending={} pending_cap={} reserved={} oldest_pending_quote_age_ms={} inflight={} tokens={}", FIRST_AUDIT_INGRESS_DROPPED.load(Ordering::Relaxed), observability.received.load(Ordering::Relaxed), observability.queued.load(Ordering::Relaxed), @@ -1816,6 +1902,7 @@ impl ReplicationEngine { observability.outside_answerability_window.load(Ordering::Relaxed), scheduler.pending_len(), FIRST_AUDIT_PENDING_CAP, + u8::from(scheduler.has_reservation()), scheduler.oldest_pending_quote_age_ms(SystemTime::now()), observability.inflight.load(Ordering::Relaxed), scheduler.tokens(), @@ -6630,21 +6717,34 @@ mod tests { } } - /// The pending admission cap must equal what the token budget can launch - /// inside one effective answerability window (plus burst): any larger and - /// admitted work is guaranteed to expire unlaunched; any smaller and the - /// budget idles while nominations are refused. + /// The pending admission cap must equal the number of usable launches the + /// token budget can perform inside one answerability window, counted by + /// SIMULATING the shipped reserve-time predicate instant-by-instant (burst + /// tokens at age zero, one refill per launch interval), not by repeating + /// the derivation formula. Any larger and admitted work is guaranteed to + /// expire unlaunched; any smaller and the budget idles while nominations + /// are displaced. #[test] - fn first_audit_pending_cap_matches_window_budget() { - let cutoff_secs = GOSSIP_ANSWERABILITY_TTL - .saturating_sub(MONETIZED_AUDIT_SKEW_MARGIN) - .as_secs(); - let expected = usize::try_from( - cutoff_secs / config::FIRST_AUDIT_LAUNCH_INTERVAL.as_secs() - + u64::from(config::FIRST_AUDIT_BUDGET_BURST), - ) - .expect("cap fits usize"); - assert_eq!(FIRST_AUDIT_PENDING_CAP, expected); + fn first_audit_pending_cap_matches_strict_launch_horizon() { + let quote_ts = SystemTime::now(); + // Burst launches fire at age zero and must pass the reserve predicate. + assert!(quote_answerable_through_nominal_jitter(quote_ts, quote_ts)); + let mut usable = usize::try_from(config::FIRST_AUDIT_BUDGET_BURST).expect("burst fits"); + let mut refill = 1u32; + loop { + let age = config::FIRST_AUDIT_LAUNCH_INTERVAL * refill; + let at = quote_ts + age; + if !quote_answerable_through_nominal_jitter(quote_ts, at) { + break; + } + usable += 1; + refill += 1; + assert!(refill < 10_000, "runaway horizon simulation"); + } + assert_eq!( + FIRST_AUDIT_PENDING_CAP, usable, + "cap must match the simulated strict launch horizon" + ); } /// The pending queue is sized to the launch budget while the dedup set @@ -6660,11 +6760,11 @@ mod tests { ); } - /// Admission beyond the cap displaces the least-recently-refreshed entry - /// (keep-newest sampling) and is accounted as `capacity_evicted` — the - /// steady overload signal — never silent loss. + /// Admission beyond the cap displaces exactly one incumbent per overflow + /// and every displacement is accounted as `capacity_evicted` — the steady + /// overload signal — never silent loss, and never an admission refusal. #[test] - fn first_audit_admission_beyond_cap_displaces_oldest_and_counts() { + fn first_audit_admission_beyond_cap_displaces_and_counts() { let obs = Arc::new(FirstAuditObservability::default()); let mut scheduler = FirstAuditScheduler::new(Instant::now()); let overflow: usize = 5; @@ -6686,20 +6786,106 @@ mod tests { u64::try_from(total).expect("total fits u64"), "displacement is not an admission refusal" ); - for i in 0..overflow { - let b = u8::try_from(i + 1).expect("test peer count fits u8"); - assert!( - scheduler.pending.peek(&test_peer(b)).is_none(), - "oldest-admitted entries are the displaced ones" + // Every arrival was admitted (the newest always enters the sample). + let last = u8::try_from(total).expect("test peer count fits u8"); + assert!( + scheduler.pending.peek(&test_peer(last)).is_some(), + "the newest arrival always enters the sample" + ); + } + + /// The reviewer-demonstrated suppression route: a target followed by an + /// ordered batch of distinct-peer nominations, all inside one ingress + /// drain batch (`pending_cap < batch <= FIRST_AUDIT_DRAIN_BATCH`), with + /// the token bucket EMPTY so no reservation can intervene. Under the old + /// keep-newest LRU the target was evicted with certainty; under random + /// displacement its per-overflow eviction probability is `1/cap`, so + /// across repeated trials the target must survive some runs and be + /// displaced in others — never deterministically flushed, and every + /// displacement accounted. + #[test] + fn first_audit_ordered_flood_cannot_deterministically_evict_target() { + let flood: usize = 60; // pending_cap < 60 <= FIRST_AUDIT_DRAIN_BATCH + assert!(FIRST_AUDIT_PENDING_CAP < flood); + assert!(flood <= config::FIRST_AUDIT_DRAIN_BATCH); + + let trials = 100u32; + let mut survived = 0u32; + let mut evicted = 0u32; + for _ in 0..trials { + let obs = Arc::new(FirstAuditObservability::default()); + let mut scheduler = FirstAuditScheduler::new(Instant::now()); + // Drain the burst tokens so the pre-overflow reservation + // opportunity cannot fire: this isolates pure retention. + scheduler.limiter.reserve_token(); + scheduler.limiter.reserve_token(); + + let target = test_peer(200); + scheduler.enqueue(live_nomination(200), &obs); + for i in 0..flood { + let b = u8::try_from(i + 1).expect("flood peer fits u8"); + scheduler.enqueue(live_nomination(b), &obs); + } + + assert_eq!(scheduler.pending_len(), FIRST_AUDIT_PENDING_CAP); + let overflows = u64::try_from(1 + flood - FIRST_AUDIT_PENDING_CAP) + .expect("overflow count fits u64"); + assert_eq!( + obs.capacity_evicted.load(Ordering::Relaxed), + overflows, + "every overflow past the cap is accounted, none silent" ); + if scheduler.pending.peek(&target).is_some() { + survived += 1; + } else { + evicted += 1; + } } - for i in overflow..total { + // With eviction probability 1/cap per overflow, P(target survives one + // trial) ~= (1 - 1/31)^30 ~= 0.37: both outcomes are overwhelmingly + // certain to appear across 100 trials (miss odds < 1e-20 each side). + assert!( + survived > 0, + "target must survive some ordered floods — deterministic eviction \ + would mean keep-newest retention regressed" + ); + assert!( + evicted > 0, + "target must also be displaceable — otherwise overflow is refusing \ + admissions instead of sampling" + ); + } + + /// With a token AVAILABLE, an overflowing arrival must first give pending + /// work a launch opportunity: the drainer's pre-overflow reservation pulls + /// one entry out of the queue, so admission proceeds without any eviction. + #[test] + fn first_audit_overflow_reserves_before_destructive_eviction() { + let obs = Arc::new(FirstAuditObservability::default()); + let mut scheduler = FirstAuditScheduler::new(Instant::now()); + let cooldown: HashMap = HashMap::new(); + + for i in 0..FIRST_AUDIT_PENDING_CAP { let b = u8::try_from(i + 1).expect("test peer count fits u8"); - assert!( - scheduler.pending.peek(&test_peer(b)).is_some(), - "newest entries survive as the budget-sized sample" - ); + scheduler.enqueue(live_nomination(b), &obs); } + let overflowing = live_nomination(250); + assert!(scheduler.would_displace(&overflowing)); + + // The drainer's pre-overflow sequence: reserve, then enqueue. + assert!(scheduler.try_reserve(Instant::now(), 0, Duration::ZERO, &cooldown, &obs)); + assert!( + !scheduler.would_displace(&overflowing), + "a successful reservation frees the slot" + ); + scheduler.enqueue(overflowing, &obs); + + assert_eq!(scheduler.pending_len(), FIRST_AUDIT_PENDING_CAP); + assert_eq!( + obs.capacity_evicted.load(Ordering::Relaxed), + 0, + "no destructive eviction when a launch opportunity existed" + ); } /// Displacement stamps no suppression state: a displaced peer's next @@ -6716,13 +6902,17 @@ mod tests { let b = u8::try_from(i + 1).expect("test peer count fits u8"); scheduler.enqueue(live_nomination(b), &obs); } - let displaced = test_peer(1); - assert!(scheduler.pending.peek(&displaced).is_none()); + assert_eq!(obs.capacity_evicted.load(Ordering::Relaxed), 1); + // Find whichever peer the random displacement removed. + let displaced = (1..=total) + .map(|i| u8::try_from(i).expect("test peer count fits u8")) + .find(|b| scheduler.pending.peek(&test_peer(*b)).is_none()) + .expect("exactly one admitted peer was displaced"); - scheduler.enqueue(live_nomination(1), &obs); + scheduler.enqueue(live_nomination(displaced), &obs); assert!( - scheduler.pending.peek(&displaced).is_some(), + scheduler.pending.peek(&test_peer(displaced)).is_some(), "the displaced peer re-enters the sample" ); assert_eq!(scheduler.pending_len(), FIRST_AUDIT_PENDING_CAP);