feat(prover): revive cancelled proving jobs from a persisted aborted state#24578
Merged
PhilWindle merged 3 commits intoJul 8, 2026
Merged
Conversation
…tate Cancelling a proving job records a persisted 'aborted' status instead of a dropped/rejected one; re-enqueuing the same job id revives it (clearing the persisted result via deleteProvingJobResult and re-running). An abort never permanently blocks the proof - in the same process or after a restart.
AztecBot
added
ci-draft
alexghr
reviewed
Jul 8, 2026
| }); | ||
| this.cleanUpProvingJobState([job.id]); | ||
| await this.database.deleteProvingJobResult(job.id); | ||
| jobStatus = this.#getProvingJobStatus(job.id); |
Contributor
There was a problem hiding this comment.
looking at the code below this if, the broker adds the job back to the database, is this operation idempotent or is it going to fail with a duplicate error?
Collaborator
There was a problem hiding this comment.
It is idempotent. But upon closer inspection, there were a couple of other issues which have now been addressed.
Two robustness fixes to the revive path added for cancelled proving jobs: - The revive now re-sets jobsCache synchronously before awaiting the DB delete, so the enqueue lock is held across the whole revive. A concurrent re-request of the same aborted job takes the cached no-op path instead of falling through and enqueuing a duplicate. - The persisted DB routes the result-delete through the batch queue as an [id, undefined] tombstone rather than writing straight to the store, so it stays FIFO-ordered behind the queued abort. Otherwise a still-queued abort could flush after the direct delete and resurrect the cancelled state on disk. Revive now reports 'in-queue' rather than 'not-found' (the job stays cached through the revive); this is inert for the broker facade, which treats all non-terminal statuses identically.
alexghr
approved these changes
Jul 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v5 version of #24558 (same change, based on
merge-train/spartan-v5).Problem
When the prover broker cancelled a proving job it settled the job as a cached
{ status: 'rejected', reason: 'Aborted' }result — in memory and in the database. Because a job's id is a deterministic hash of its inputs, any later enqueue of the same job returned that cached abort (Cached proving job … Not enqueuing again) and the facade failed the job withAborted. So once a proof was aborted it stayed aborted, and a retry — a fresh epoch attempt after a reorg/failure, or a re-orchestration after a restart — got the cached abort instead of re-proving. A planned mid-epoch redeploy hit this and turned into a testnet epoch prune.Fix
Cancellation is now its own first-class, revivable state:
abortedproving-job status (ProvingJobStatus/ProvingJobSettledResult).cancelProvingJobrecords it, notifies the current waiter, and persists it (ProvingBrokerDatabase.setProvingJobAborted).enqueueProvingJobrevives an aborted job: when the producer re-requests it, the broker clears the aborted state — in memory and, via the newdeleteProvingJobResult, in the database — and re-enqueues it as if new. So a restart mid-revival keeps the job pending rather than resurrecting the abort. Genuine agent errors and timeouts still settle as terminalrejectedresults.Net effect: cancelling frees the job cleanly and records why, but re-requesting the same proof always brings it back to life — in the same process or after a restart — so an abort can never permanently block an epoch from proving.
The stacked PR (prover-node clean-shutdown, v5) stops a clean prover-node shutdown from cancelling its in-flight jobs in the first place.
Tests
proving_broker.test.ts(both DB variants): cancel leaves a jobaborted; re-request revives it and it completes; the aborted state persists across a restart; and a revived job survives a restart mid-revival as pending (not aborted).Note: not run locally in this session — the prover-client suite needs a full noir/wasm bootstrap that wasn't available here — so relied on CI.
Created by claudebox · group:
slackbot