fix(replication): cap first-audit pending admission at the launch budget

[grumbach]

Summary

Follow-up to #177 (stacked on its head; see merge-order note below). Sizes first-audit pending admission to what the launch budget can actually audit.

#177 bounded launches (token bucket) and expired dead entries eagerly, but admission was still unbounded relative to the budget: pending shared the 4096-entry commitment-cache cap while the token bucket can launch at most ~31 audits inside one effective answerability window. On the 2026-07-17 DEV-02 staging run this meant ~113 admitted nominations per node-hour against ~12 launches: over 99% of admitted work could never launch, pending sat at ~331-431k fleet-wide, and the oldest-pending age tracked the age of the deployment itself — an aging backlog whose steady state is mass expiry.

Changes

• pending is capped at FIRST_AUDIT_PENDING_CAP, derived from the strict reserve-time launch horizon (burst at age zero plus every refill that stays answerable through max jitter + send slack; currently 31). The unit test simulates the shipped predicate instant-by-instant and must agree with the derivation. The dedup set keeps the 4096 bound.
• Overflow follows two rules, in order (review feedback):
  1. Reservation before destructive overflow — an arrival that would displace a different peer first gives the queue a launch opportunity; with a token free, one entry moves into the reservation and admission proceeds with no eviction, so an ingress burst cannot flush eligible work past a ready token.
  2. Random-victim displacement — when no launch is possible, a uniformly random incumbent is displaced, counted as capacity_evicted. Per-nomination eviction probability of a specific target is 1/cap regardless of arrival order: suppressing it with 95% confidence costs ~95 distinct-peer paid nominations and is never certain (deterministic keep-newest LRU, flagged in review, allowed a certain flush with 32).
• The summary line gains pending_cap and reserved (the single reservation held outside the queue), so schedulable occupancy pending + reserved is directly computable in Elasticsearch.
• ADR-0004 Amendment 4 documents the admission sizing, the two overflow rules, and the eviction-cost math.

Evidence

• 745 lib tests pass. Regressions include: ordered-flood non-determinism at the drain/reserve boundary (pending_cap < 60 <= FIRST_AUDIT_DRAIN_BATCH, 100-trial statistical bounds with miss odds < 1e-20), reserve-before-eviction at capacity, displaced-peer re-nomination, cap-vs-predicate simulation.
• Expected staging signals: pending ≤ 31 with oldest_pending_quote_age_ms far below the answerability window (queue drains instead of aging), overflow visible as capacity_evicted growth at admission time instead of outside_answerability_window mass expiry hours later.

Semver

Patch: no protocol or config-surface change; two additive fields in the summary log line.

Tracking: V2-738

[grumbach]

Merge order: this PR is stacked on #177 and must be merged AFTER it. Until #177 lands, this diff shows #177's commits too; the admission-cap change itself is the top two commits (4309dac + review fixes in 3462edf).

[Copilot]

Pull request overview

This PR tightens ADR-0004’s monetized first-audit pipeline so the pending admission queue is sized to what the launch budget can realistically drain within the answerability window, preventing an ever-aging backlog under sustained nomination load. It also expands observability (audit outcome/drops counters, scheduler summary fields) and updates payment-verifier nomination behavior and ADR documentation.

Changes:

• Cap first-audit pending admission at FIRST_AUDIT_PENDING_CAP derived from the launch budget + effective answerability window, with LRU displacement counted as capacity_evicted.
• Add cumulative audit outcome/drop accounting and emit it alongside the existing replication traffic summary.
• Update payment verification to nominate only the actually-paid candidate(s), add settlement payee-prefix checking, and add an opt-in A/B e2e workload driver plus ADR amendments.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/e2e/mod.rs	Registers the new opt-in A/B workload module.
tests/e2e/first_audit_ab.rs	Adds an environment-gated A/B workload driver for production-shaped paid uploads + first-audit behavior comparison.
src/replication/protocol.rs	Adds process-global audit pass/fail-by-reason and responder-drop counters + summary log.
src/replication/mod.rs	Implements the capped pending scheduler + reservation/promotion flow, adds new scheduler observability, and wires audit outcome/drop recording.
src/replication/config.rs	Documents and centralizes first-audit limiter constants (launch interval, burst, inflight, jitter, ingress capacity, etc.).
src/payment/verifier.rs	Narrows deterministic first-audit nomination to paid candidates, adds settlement redirect rejection, and updates test hooks accordingly.
docs/adr/ADR-0004-commitment-bound-quote-pricing.md	Adds ADR-0004 Amendments 2–4, including the pending admission sizing rationale and semantics.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 7 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (3)

src/replication/protocol.rs:1105

• These assertions use exact deltas on process-global counters, which can fail if another test increments the same counter between the load and the assert. Use a >= check so the test remains correct under parallel execution.

        let before = AUDIT_PASS[pass_slot].load(Ordering::Relaxed);
        record_audit_pass(AuditOutcomeKind::Subtree);
        assert_eq!(AUDIT_PASS[pass_slot].load(Ordering::Relaxed), before + 1);

src/replication/protocol.rs:1126

• Same flakiness risk as the pass assertion: these are global counters and other tests may increment them. Assert that the intended slot increased by at least 1 instead of exactly 1.

                assert_eq!(
                    AUDIT_FAIL[slot].load(Ordering::Relaxed),
                    before + 1,
                    "kind {kind_index} reason {reason:?} must land in its own slot",
                );

src/replication/protocol.rs:1140

• Same flakiness risk as above: AUDIT_DROPPED is process-global and may be incremented by other tests. Use a >= assertion to avoid nondeterministic failures in parallel runs.

            assert_eq!(
                AUDIT_DROPPED[kind_index].load(Ordering::Relaxed),
                before + 1
            );

[dirvine]

Reviewed exact head d1fc1ddf0a4f710e16debd471cd3ec4c3916e92f with a six-seat panel (four substantive completions; two tool-failed), then independently checked the scheduler ordering and focused tests.

Verdict: changes requested. The queue-sizing diagnosis is sound, but the current keep-newest cap introduces a deterministic suppression route before the scheduler can reserve work.

Blocking — drain-before-reserve can evict a target deterministically

FIRST_AUDIT_PENDING_CAP is 32 (src/replication/mod.rs:956-984), while FIRST_AUDIT_DRAIN_BATCH is 64 (src/replication/config.rs:399-405). On an ingress wake the event loop enqueues/drains up to 64 nominations first (src/replication/mod.rs:1621-1640) and only then attempts reservation (:1736-1758). At capacity, every new distinct peer deterministically evicts the LRU peer (:199-227, :619-632).

Therefore an ordered batch of target + 32 later distinct-peer nominations removes the target before either newest/oldest lane can inspect it, even when a token and in-flight slot are available. Lane alternation cannot help because the target is already gone.

This cuts the stacked ADR's documented capacity-eviction threshold from 4,096 nominations (docs/adr/ADR-0004-commitment-bound-quote-pricing.md:462-470) to 32 — a 128x reduction — while Amendment 4 says every fresh pin retains an “unpredictable chance” of prompt audit (:527-533). The current replacement is deterministic under controlled arrival order. That matters because the ADR already assumes recyclable payment principal and sidecar-only pins without a lottery backstop (:443-453).

Please either give eligible work a reservation opportunity before destructive overflow, or use genuine adversary-resistant/random/weighted retention rather than deterministic keep-newest LRU. Add a production-order regression with pending_cap < ordered_batch <= drain_batch; the new enqueue-only test currently proves that the oldest target is evicted but does not exercise the drain/reserve boundary.

Also fix in this head

• Cap maths/wording: the strict predicate rejects age >= 150m (src/replication/mod.rs:920-928). For a fresh quote, burst 2 plus refills at 5…145 minutes gives 31 usable launches, not 32; the max-jitter/slack prefilter is stricter again. The test at :6590-6604 repeats the formula rather than simulating the predicate. Derive the cap from the same strict launch predicate/horizon.
• Occupancy telemetry: a reservation is held outside pending (:477-487, :721-728), so the scheduler may hold one reserved + 32 pending while reporting pending=32, pending_cap=32. Report reserved/total schedulable occupancy or narrow the claim.
• Stack dependency: #183 must remain after #177. The merge-order comment names stale cap SHA cbc2f5e; the current cap commit is this head, d1fc1dd.

Verification

• cargo fmt --all -- --check — pass
• cargo test first_audit_ --lib --no-fail-fast — 28/28 pass
• GitHub CI — all current checks green across Linux/macOS/Windows, clippy, format, docs, no-logging, audit and ADR validation
• Head revalidated immediately before review

The causal note on DEV-02/DEV-03 is fair: that comparison cannot attribute the upload p90 change. I also agree another attribution run is not necessary for this queue finding; the scheduler counters already show the admission/budget mismatch. This review is about the proposed retention policy, not the diagnosis.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 7 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

src/payment/verifier.rs:1165

• The settlement-redirect check runs before verifying the on-chain amount is sufficient. For an unpaid quote hash, the vault mapping will usually return a zeroed rewards-address prefix, which will be treated as a “redirected settlement” and yields a misleading error.

Consider applying the redirect check only after on_chain_amount >= candidate.expected_amount so “not paid enough” remains the failure mode when no sufficient settlement exists.

        if let Some(recorded) = recorded_rewards_prefix {
            let expected = Self::rewards_address_prefix(&candidate.quote.rewards_address);
            if recorded != expected {
                return Err(Error::Payment(format!(
                    "Median-priced quote settlement for peer {:?} was redirected: recorded rewards address prefix {} does not match the quote's {}",