Skip to content

Release the concurrency lock of a job whose class is missing - #825

Closed
andrewfader wants to merge 1 commit into
rails:mainfrom
andrewfader:release-concurrency-lock-for-missing-class
Closed

andrewfader wants to merge 1 commit into
rails:mainfrom
andrewfader:release-concurrency-lock-for-missing-class

Conversation

@andrewfader

@andrewfader andrewfader commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

A concurrency-limited job whose class exists where it was dispatched, but not where it runs, keeps its key's semaphore after it fails. That happens during a rolling deploy or a rollback, when a worker on the other release claims the job. ClaimedExecution#perform fails it with ClassMissingError (73602fe), but release_concurrency_lock returns early because concurrency_limited? needs job_class. The slot stays taken and the jobs blocked behind it wait until the semaphore expires (duration, 3 minutes by default).

This releases the lock by concurrency_key alone, and falls back to the default limit and default_concurrency_control_period when the class is missing.

Two trade-offs:

  • A job dispatched without its class never took the lock (the existing "dispatch job with missing class and concurrency key skips concurrency controls" test). If it fails while another job of its key holds the lock, this release frees that job's slot, so one extra job of the key can start early. Today, the leaked slot blocks the key for the whole duration instead.
  • For a key whose real limit is above 1, the fallback limit of 1 releases nothing unless the semaphore is at 0, so that slot still waits to expire, as it does today.

Happy to narrow it if you'd rather only release when the semaphore shows the lock was taken.

Test: a job dispatched with its class and then run without it releases the next blocked job of its key. It fails without the change.

A job dispatched where its class exists but run where it doesn't fails
with ClassMissingError and kept its key's semaphore until it expired,
because releasing the lock required the job class.
@rosa

rosa commented Nov 3, 2026

Copy link
Copy Markdown
Member

Thanks a lot for this, @andrewfader, and for spelling out the trade-offs so clearly! While reviewing it, we found that releasing by key alone can let jobs run over the limit: a job whose class was missing when it was dispatched never took a slot, so when it fails it frees a slot that belongs to another running job. That turned out to be the same problem as #827, since the semaphore counts slots but doesn't know who holds them. We're fixing both in #831 by having each job remember the semaphore it got its slot from. That also covers your case, so I'll close this one in favor of it.

@rosa rosa closed this Nov 3, 2026
rosa added a commit that referenced this pull request Nov 3, 2026
A semaphore counts free slots but doesn't know which jobs hold them, so
any job that gave a slot back by key counted as a holder:

- A job that ran past its duration gave its slot back to the semaphore
  created after its own expired, letting another job start while the
  new holder was still running (#827).
- A job dispatched while its class was missing skipped its limits, but
  gave a slot back if it later ran where its class loads.
- A job whose class was missing when it ran couldn't give back the slot
  it got when it was dispatched (#825).

Now a job records the id of the semaphore it got a slot on, and its
limit, in its serialized Active Job data, and gives the slot back only
to that semaphore. A job that skips its limits records that it holds
no slot. Jobs that got their slot before this change have nothing
recorded and keep giving it back by key, until 2.0.

The first two tests come from the reproduction in #827.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rosa added a commit that referenced this pull request Nov 3, 2026
A semaphore counts free slots but doesn't know which jobs hold them, so
any job that gave a slot back by key counted as a holder:

- A job that ran past its duration gave its slot back to the semaphore
  created after its own expired, letting another job start while the
  new holder was still running (#827).
- A job dispatched while its class was missing skipped its limits, but
  gave a slot back if it later ran where its class loads.
- A job whose class was missing when it ran couldn't give back the slot
  it got when it was dispatched (#825).

Now a job records the id of the semaphore it got a slot on, and its
limit, in its serialized Active Job data, and gives the slot back only
to that semaphore. A job that skips its limits records that it holds
no slot. Jobs that got their slot before this change have nothing
recorded and keep giving it back by key, until 2.0.

The first two tests come from the reproduction in #827.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rosa added a commit that referenced this pull request Nov 3, 2026
A semaphore counts free slots but doesn't know which jobs hold them, so
any job that gave a slot back by key counted as a holder:

- A job that ran past its duration gave its slot back to the semaphore
  created after its own expired, letting another job start while the
  new holder was still running (#827).
- A job dispatched while its class was missing skipped its limits, but
  gave a slot back if it later ran where its class loads.
- A job whose class was missing when it ran couldn't give back the slot
  it got when it was dispatched (#825).

Now a job records the id of the semaphore it got a slot on, and its
limit, in its serialized Active Job data, and gives the slot back only
to that semaphore. A job that skips its limits records that it holds
no slot. Jobs that got their slot before this change have nothing
recorded and keep giving it back by key, until 2.0.

The first two tests come from the reproduction in #827.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@andrewfader
andrewfader deleted the release-concurrency-lock-for-missing-class branch November 4, 2026 06:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants