Skip to content

Reduce payment receipt contention - #53

Merged
j0sh merged 1 commit into
mainfrom
ja-codex/payment-receipt-perf
Nov 6, 2026
Merged

j0sh merged 1 commit into
mainfrom
ja-codex/payment-receipt-perf

Conversation

@j0sh

@j0sh j0sh commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Concurrent payment receipts queue behind the global payment-engine mutex. Verify request-local tickets and encode challenges before that lock, reuse prepared admission SELECTs, combine watermark and exposure into one read, and record accepted counters after unlocking.

Stacked directly on #43 (15c6057), without the main rebase. Session identity, freshness, expiry, capacity, replay, collateral, liability, randomness exposure, durable winning-ticket storage, and credit publication retain their checks and transaction boundary. No admission results are cached. Counters finish before Receive returns and only emit for accepted batches.

The 32-caller replay regression requires exactly one accepted two-ticket batch, durable winners, exact credit, and matching counters. The existing SQL rollback test now also requires zero accepted counters on failure.

The local follow-up at 9cff0b6 targets the remaining contention: CPU/mutex/block profiles show receipts holding the engine mutex across SQLite admission and WAL work, which also queues challenge and charge operations. The follow-up combines only the watermark/exposure read in the same transaction and moves immutable challenge encoding before the mutex; the liability query is unchanged.

Eight counterbalanced local 32-client trials (8,192 paid HTTP jobs each, real signatures and SQLite, constant chain) improved the median of trial p50s from 2.010 to 1.882 ms (6.4%), with throughput 13,406 to 14,105 jobs/s (5.2%). Every paired p50 improved. Serial latency was effectively unchanged (0.297/0.298 ms, five trials). Direct receipts with 16 and 256 outstanding winners showed no regression. A larger one-statement prototype regressed with populated liability tables and was discarded. These follow-up results are local; the cloud table below applies to 2c11ea6.

The previous frozen-source e2e comparison measured 15c6057 against this change (2c11ea6) and clean upstream go-livepeer 2a4f7c4. Three repetitions rotated implementation order on one GCP n2-standard-8, using Go 1.27.3, native Anvil, protocol 3c01f3a, and 0/110 ms added latency per JSON-RPC HTTP exchange. Each repetition used 128 serial or 1,024 concurrent warm jobs after eight warmups.

Added RPC delay Clients #43 p50, ms Measured 2c11ea6 p50, ms go-livepeer p50, ms
0 ms 1 1.541 1.411 1.626
0 ms 32 13.341 7.960 7.316
110 ms 1 1.547 1.412 1.651
110 ms 32 13.178 7.865 7.407

On that cloud run, at 32 clients median dispatch fell about 40% and observed goodput rose about 52% versus #43. Median dispatch improved in every paired repetition and condition. At 110 ms / 32 clients, p95 fell from 20.194 to 14.261 ms. Measured 2c11ea6 medians were 6–9% higher than go-livepeer at 32 clients; 2c11ea6 was faster in the serial conditions.

Both Node revisions succeeded on 6,912/6,912 warm requests; go-livepeer succeeded on 6,910/6,912, with two paid-request HTTP 409 session already exists responses. All three succeeded on 12/12 separate cold probes in this run. Failed requests remain in attempted counts and goodput; latency quantiles use successes. A separate unchanged go-livepeer manifest-ID diagnostic reproduced a data race and 246 duplicate IDs in 32,768 generations.

Cloud validation at 15c6057 and 2c11ea6: 38/38 real-chain protocol cases, 6/6 recovery cases, 13/13 controlled-cache scenarios, and full repository go test -race ./... passed. The candidate also passed the local orchestrator race suite and repeated targeted race checks. The requested Sol/Astra, Opus, Gemini, Muse, Grok, Terra documentation, and final minimization review cycle is complete with no accepted findings remaining.

Dispatch measures challenge → remote signing → paid forwarding to the echo runner's recorded arrival. These finite closed-loop bursts exclude GPU inference and Internet gateway latency. The go-livepeer reference matches the PR #43 benchmark pin; it is separate from the modified local checkout. Raw requests/RPC traces, manifests, binary/source hashes, and review reports are retained. The temporary VM and boot disk were deleted.

GitHub CI at 9cff0b6 passed the race/SDK process tests, vet and staticcheck. Its vulnerability scan fails on GO-2026-6865 in klauspost/compress v1.20.1 (fixed in v1.20.2). The same govulncheck v1.8.0 scan reproduces the identical finding on unchanged parent 15c6057; this PR leaves go.mod and go.sum unchanged. This inherited CI blocker remains at the measured revision.

The 9cff0b6 follow-up passed the full orchestrator race suite in 67.670 seconds and the same full review workflow. No benchmark fixtures, dependency changes, cached admission results or weakened checks are included.

@j0sh
j0sh added this pull request to stack #60 November 6, 2026 22:50
@j0sh
j0sh force-pushed the ja-codex/payment-receipt-perf branch from 9cff0b6 to 8d8bb48 Compare November 6, 2026 23:09
@j0sh
j0sh force-pushed the ja-codex/payment-receipt-perf branch from 8d8bb48 to 0c7ad1e Compare November 6, 2026 23:14
Copilot AI balanced review requested due to automatic review settings November 6, 2026 23:14

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

There are no active review issues with the changes aimed at reducing payment receipt contention.

No open issues 🎉

What changed in this PR

This PR reduces payment receipt contention by moving request-local work outside the payment-engine mutex and reducing database admission overhead, while retaining transactional checks before publishing credit.

Changes:

  • Move ticket verification and challenge encoding before locking, and record accepted counters after unlocking.
  • Reuse prepared admission queries and combine watermark and randomness-exposure checks into one transactional read.
  • Add concurrent replay coverage and verify storage failures emit no accepted counters.
File Description
orchestrator/​redeemer_db.go Prepare reusable admission queries and combine watermark/exposure reads.
orchestrator/​payment_recipient_test.go Test concurrent replay handling and counter behavior on storage failure.
orchestrator/​payment_engine.go Shorten mutex-held work while preserving serialized admission and credit publication.

Review effort: Balanced

@j0sh
j0sh temporarily deployed to docker-publish November 6, 2026 23:16 — with GitHub Actions Inactive
@j0sh
j0sh temporarily deployed to docker-publish November 6, 2026 23:17 — with GitHub Actions Inactive
@j0sh
j0sh temporarily deployed to docker-publish November 6, 2026 23:17 — with GitHub Actions Inactive
@j0sh
j0sh temporarily deployed to docker-publish November 6, 2026 23:17 — with GitHub Actions Inactive
@j0sh
j0sh force-pushed the ja-codex/payment-receipt-perf branch from 0c7ad1e to 725d988 Compare November 6, 2026 23:35
@j0sh
j0sh temporarily deployed to docker-publish November 6, 2026 23:38 — with GitHub Actions Inactive
Base automatically changed from ja-codex/payment-36-performance-metrics to main November 6, 2026 23:38
Verify request-local tickets and encode challenges before acquiring the receipt mutex. Reuse prepared admission queries, combine watermark and exposure reads, and record accepted receipt counters after unlocking.

Cover concurrent replay acceptance, durable winners, exact credit, and rollback counters with regression tests.
@j0sh
j0sh force-pushed the ja-codex/payment-receipt-perf branch from 725d988 to ce93e64 Compare November 6, 2026 23:38
@j0sh
j0sh merged commit 22e6b55 into main Nov 6, 2026
5 checks passed
@j0sh
j0sh deleted the ja-codex/payment-receipt-perf branch November 6, 2026 23:41
@j0sh
j0sh deployed to docker-publish November 6, 2026 23:45 — with GitHub Actions Active
@j0sh
j0sh deployed to docker-publish November 6, 2026 23:46 — with GitHub Actions Active
@j0sh
j0sh deployed to docker-publish November 6, 2026 23:46 — with GitHub Actions Active
@j0sh
j0sh deployed to docker-publish November 6, 2026 23:46 — with GitHub Actions Active

This branch was successfully deployed

1 active deployment
docker-publish — ce93e647 Deployed Nov 6, 2026 by j0sh via docker-pr / promote #124
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants