Repository navigation
Release the concurrency lock of a job whose class is missing - #825
Closed
andrewfader wants to merge 1 commit into
Closed
andrewfader wants to merge 1 commit into
andrewfader wants to merge 1 commit into
Conversation
A job dispatched where its class exists but run where it doesn't fails with ClassMissingError and kept its key's semaphore until it expired, because releasing the lock required the job class.
Member
|
Thanks a lot for this, @andrewfader, and for spelling out the trade-offs so clearly! While reviewing it, we found that releasing by key alone can let jobs run over the limit: a job whose class was missing when it was dispatched never took a slot, so when it fails it frees a slot that belongs to another running job. That turned out to be the same problem as #827, since the semaphore counts slots but doesn't know who holds them. We're fixing both in #831 by having each job remember the semaphore it got its slot from. That also covers your case, so I'll close this one in favor of it. |
rosa
added a commit
that referenced
this pull request
Nov 3, 2026
A semaphore counts free slots but doesn't know which jobs hold them, so any job that gave a slot back by key counted as a holder: - A job that ran past its duration gave its slot back to the semaphore created after its own expired, letting another job start while the new holder was still running (#827). - A job dispatched while its class was missing skipped its limits, but gave a slot back if it later ran where its class loads. - A job whose class was missing when it ran couldn't give back the slot it got when it was dispatched (#825). Now a job records the id of the semaphore it got a slot on, and its limit, in its serialized Active Job data, and gives the slot back only to that semaphore. A job that skips its limits records that it holds no slot. Jobs that got their slot before this change have nothing recorded and keep giving it back by key, until 2.0. The first two tests come from the reproduction in #827. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rosa
added a commit
that referenced
this pull request
Nov 3, 2026
A semaphore counts free slots but doesn't know which jobs hold them, so any job that gave a slot back by key counted as a holder: - A job that ran past its duration gave its slot back to the semaphore created after its own expired, letting another job start while the new holder was still running (#827). - A job dispatched while its class was missing skipped its limits, but gave a slot back if it later ran where its class loads. - A job whose class was missing when it ran couldn't give back the slot it got when it was dispatched (#825). Now a job records the id of the semaphore it got a slot on, and its limit, in its serialized Active Job data, and gives the slot back only to that semaphore. A job that skips its limits records that it holds no slot. Jobs that got their slot before this change have nothing recorded and keep giving it back by key, until 2.0. The first two tests come from the reproduction in #827. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rosa
added a commit
that referenced
this pull request
Nov 3, 2026
A semaphore counts free slots but doesn't know which jobs hold them, so any job that gave a slot back by key counted as a holder: - A job that ran past its duration gave its slot back to the semaphore created after its own expired, letting another job start while the new holder was still running (#827). - A job dispatched while its class was missing skipped its limits, but gave a slot back if it later ran where its class loads. - A job whose class was missing when it ran couldn't give back the slot it got when it was dispatched (#825). Now a job records the id of the semaphore it got a slot on, and its limit, in its serialized Active Job data, and gives the slot back only to that semaphore. A job that skips its limits records that it holds no slot. Jobs that got their slot before this change have nothing recorded and keep giving it back by key, until 2.0. The first two tests come from the reproduction in #827. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A concurrency-limited job whose class exists where it was dispatched, but not where it runs, keeps its key's semaphore after it fails. That happens during a rolling deploy or a rollback, when a worker on the other release claims the job.
ClaimedExecution#performfails it withClassMissingError(73602fe), butrelease_concurrency_lockreturns early becauseconcurrency_limited?needsjob_class. The slot stays taken and the jobs blocked behind it wait until the semaphore expires (duration, 3 minutes by default).This releases the lock by
concurrency_keyalone, and falls back to the default limit anddefault_concurrency_control_periodwhen the class is missing.Two trade-offs:
Happy to narrow it if you'd rather only release when the semaphore shows the lock was taken.
Test: a job dispatched with its class and then run without it releases the next blocked job of its key. It fails without the change.