Repository navigation
Reduce payment receipt contention - #53
Merged
Merged
Conversation
j0sh
force-pushed
the
ja-codex/payment-receipt-perf
branch
from
November 6, 2026 23:09
9cff0b6 to
8d8bb48
Compare
j0sh
force-pushed
the
ja-codex/payment-receipt-perf
branch
from
November 6, 2026 23:14
8d8bb48 to
0c7ad1e
Compare
There was a problem hiding this comment.
🟢 Approval recommended
There are no active review issues with the changes aimed at reducing payment receipt contention.
No open issues 🎉
What changed in this PR
This PR reduces payment receipt contention by moving request-local work outside the payment-engine mutex and reducing database admission overhead, while retaining transactional checks before publishing credit.
Changes:
- Move ticket verification and challenge encoding before locking, and record accepted counters after unlocking.
- Reuse prepared admission queries and combine watermark and randomness-exposure checks into one transactional read.
- Add concurrent replay coverage and verify storage failures emit no accepted counters.
| File | Description |
|---|---|
orchestrator/redeemer_db.go |
Prepare reusable admission queries and combine watermark/exposure reads. |
orchestrator/payment_recipient_test.go |
Test concurrent replay handling and counter behavior on storage failure. |
orchestrator/payment_engine.go |
Shorten mutex-held work while preserving serialized admission and credit publication. |
Review effort: Balanced
j0sh
force-pushed
the
ja-codex/payment-receipt-perf
branch
from
November 6, 2026 23:35
0c7ad1e to
725d988
Compare
Base automatically changed from
ja-codex/payment-36-performance-metrics
to
main
November 6, 2026 23:38
Verify request-local tickets and encode challenges before acquiring the receipt mutex. Reuse prepared admission queries, combine watermark and exposure reads, and record accepted receipt counters after unlocking. Cover concurrent replay acceptance, durable winners, exact credit, and rollback counters with regression tests.
j0sh
force-pushed
the
ja-codex/payment-receipt-perf
branch
from
November 6, 2026 23:38
725d988 to
ce93e64
Compare
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Concurrent payment receipts queue behind the global payment-engine mutex. Verify request-local tickets and encode challenges before that lock, reuse prepared admission SELECTs, combine watermark and exposure into one read, and record accepted counters after unlocking.
Stacked directly on #43 (
15c6057), without the main rebase. Session identity, freshness, expiry, capacity, replay, collateral, liability, randomness exposure, durable winning-ticket storage, and credit publication retain their checks and transaction boundary. No admission results are cached. Counters finish beforeReceivereturns and only emit for accepted batches.The 32-caller replay regression requires exactly one accepted two-ticket batch, durable winners, exact credit, and matching counters. The existing SQL rollback test now also requires zero accepted counters on failure.
The local follow-up at
9cff0b6targets the remaining contention: CPU/mutex/block profiles show receipts holding the engine mutex across SQLite admission and WAL work, which also queues challenge and charge operations. The follow-up combines only the watermark/exposure read in the same transaction and moves immutable challenge encoding before the mutex; the liability query is unchanged.Eight counterbalanced local 32-client trials (8,192 paid HTTP jobs each, real signatures and SQLite, constant chain) improved the median of trial p50s from 2.010 to 1.882 ms (6.4%), with throughput 13,406 to 14,105 jobs/s (5.2%). Every paired p50 improved. Serial latency was effectively unchanged (0.297/0.298 ms, five trials). Direct receipts with 16 and 256 outstanding winners showed no regression. A larger one-statement prototype regressed with populated liability tables and was discarded. These follow-up results are local; the cloud table below applies to
2c11ea6.The previous frozen-source e2e comparison measured
15c6057against this change (2c11ea6) and clean upstream go-livepeer2a4f7c4. Three repetitions rotated implementation order on one GCPn2-standard-8, using Go 1.27.3, native Anvil, protocol3c01f3a, and 0/110 ms added latency per JSON-RPC HTTP exchange. Each repetition used 128 serial or 1,024 concurrent warm jobs after eight warmups.2c11ea6p50, msOn that cloud run, at 32 clients median dispatch fell about 40% and observed goodput rose about 52% versus #43. Median dispatch improved in every paired repetition and condition. At 110 ms / 32 clients, p95 fell from 20.194 to 14.261 ms. Measured
2c11ea6medians were 6–9% higher than go-livepeer at 32 clients;2c11ea6was faster in the serial conditions.Both Node revisions succeeded on 6,912/6,912 warm requests; go-livepeer succeeded on 6,910/6,912, with two paid-request HTTP 409
session already existsresponses. All three succeeded on 12/12 separate cold probes in this run. Failed requests remain in attempted counts and goodput; latency quantiles use successes. A separate unchanged go-livepeer manifest-ID diagnostic reproduced a data race and 246 duplicate IDs in 32,768 generations.Cloud validation at
15c6057and2c11ea6: 38/38 real-chain protocol cases, 6/6 recovery cases, 13/13 controlled-cache scenarios, and full repositorygo test -race ./...passed. The candidate also passed the local orchestrator race suite and repeated targeted race checks. The requested Sol/Astra, Opus, Gemini, Muse, Grok, Terra documentation, and final minimization review cycle is complete with no accepted findings remaining.Dispatch measures challenge → remote signing → paid forwarding to the echo runner's recorded arrival. These finite closed-loop bursts exclude GPU inference and Internet gateway latency. The go-livepeer reference matches the PR #43 benchmark pin; it is separate from the modified local checkout. Raw requests/RPC traces, manifests, binary/source hashes, and review reports are retained. The temporary VM and boot disk were deleted.
GitHub CI at
9cff0b6passed the race/SDK process tests, vet and staticcheck. Its vulnerability scan fails on GO-2026-6865 inklauspost/compress v1.20.1(fixed inv1.20.2). The samegovulncheck v1.8.0scan reproduces the identical finding on unchanged parent15c6057; this PR leavesgo.modandgo.sumunchanged. This inherited CI blocker remains at the measured revision.The
9cff0b6follow-up passed the full orchestrator race suite in 67.670 seconds and the same full review workflow. No benchmark fixtures, dependency changes, cached admission results or weakened checks are included.