Skip to content

fix(prover-node): do not abort in-flight proving jobs on a clean shutdown#24579

Draft
AztecBot wants to merge 1 commit into
cb/prover-broker-abort-reenqueue-v5from
cb/prover-node-keep-jobs-on-shutdown-v5
Draft

fix(prover-node): do not abort in-flight proving jobs on a clean shutdown#24579
AztecBot wants to merge 1 commit into
cb/prover-broker-abort-reenqueue-v5from
cb/prover-node-keep-jobs-on-shutdown-v5

Conversation

@AztecBot

@AztecBot AztecBot commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

v5 version of #24562 (based on merge-train/spartan-v5). Stacked on #24578 — review/merge that first.

Note: this is a fresh implementation, not a port — the v5 prover-node architecture differs from next (it uses EpochSession / TopTreeJob rather than EpochProvingJob / ProvingOrchestrator), which is the structure actually running on testnet.

Problem

On a clean shutdown, SessionManager.stop() cancels every live session with reason 'prover-node stopping'. That flows EpochSession.cancelTopTreeJob.cancel()topTree.cancel({ abortJobs: true }) — the abortJobs: true was hardcoded, so a deploy/restart aborted the in-flight top-tree broker jobs. Aborting on a clean restart is wasteful: the proofs are still valid, agents are mid-flight, and a restarted node re-orchestrating the same epoch produces the same deterministically-hashed job ids, so it re-proves from scratch.

Fix

Thread an abortJobs decision from the cancel reason down to the broker:

  • SessionManager.stop() cancels with { abortJobs: false } — a clean shutdown preserves the jobs.
  • EpochSession.cancel(reason, { abortJobs }) forwards it to TopTreeJob.cancel(abortJobs)topTree.cancel({ abortJobs }).
  • All other cancels (reorg / supersede / deadline) keep the default abortJobs: true, since their inputs are stale.

Composes with #24578: a clean redeploy mid-epoch neither cancels the in-flight jobs nor (if one ever were aborted) poisons them, so the epoch keeps proving across the restart.

Tests

  • session-manager.test.ts: stop() cancels every session with abortJobs: false.
  • epoch-session.test.ts: a normal cancel forwards abortJobs: true to the top-tree orchestrator; a clean-shutdown cancel forwards abortJobs: false.

Note: not run locally in this session — the prover-node suite needs a full noir/wasm bootstrap that wasn't available here — so relied on CI.


Created by claudebox · group: slackbot

@AztecBot AztecBot added ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR. labels Jul 7, 2026
…down

On a clean shutdown SessionManager cancels sessions with abortJobs=false, so the
in-flight top-tree broker jobs are left in the broker for the restarted node to
reuse instead of being aborted and re-proven. Reorg/supersede/deadline cancels
still abort (their inputs are stale).
@AztecBot AztecBot force-pushed the cb/prover-node-keep-jobs-on-shutdown-v5 branch from 640550b to 41a8f66 Compare July 7, 2026 11:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant