Repository navigation
Conversation
json 3 removed the quirks_mode option, which Active Support 7.1 and 8.0 still pass to JSON.generate. CI's `bundle update railties` picks json 3, so serializing job arguments raises ArgumentError, the processes started by integration tests die and those jobs hang until the 15-minute timeout.
ClaimedExecution.discard_all_in_batches and .discard_all_from_jobs raise UndiscardableError from inside `class << self`, where Ruby looks the constant up in the singleton class's lexical scope and ancestors. Neither has Execution::UndiscardableError, so both raised NameError (uninitialized constant #<Class:SolidQueue::ClaimedExecution>::UndiscardableError) instead of the intended error. Name the constant fully.
On MySQL, `error` is a TEXT column limited to 65,535 bytes. Only the backtrace was truncated to make room, and the budget was computed from the raw message size plus a fixed overhead. A message that didn't fit by itself, or that grew past the limit once JSON-escaped, made saving the failed execution raise ActiveRecord::ValueTooLong. The worker then stopped with a FinalizationError and released the job back to ready, so it ran and failed again on the next worker, over and over. Measure the error as it'll be stored, through the attribute's own type, drop backtrace lines first and, if that's not enough, truncate the message. Errors without a backtrace no longer crash the truncation. Fixes rails#821
The error column is serialized with Ruby's JSON, which escapes quotes but leaves <, > and & alone. Only Active Support's JSON encoder turns those into six-byte sequences. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eager loading lib/solid_queue/cli.rb loads Thor in every process of an application that eager loads, like its web servers, although only bin/jobs uses the CLI, and bin/jobs requires it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… job When the test ran before anything else in the process had looked up the jobs table, Active Record checked that the table exists with a query on sqlite_master that mentions solid_queue_jobs, and the test counted it as a reload. Active Record names those queries SCHEMA, so skip them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A worker prefix such as mail_* became LIKE 'mail_%', so _ and % matched any character. When that false match was the only row, the selector returned it without the literal prefix check and the worker claimed a queue it was not configured to run.
ESCAPE '\' is a valid string on SQLite and PostgreSQL, but MySQL treats the backslash as an escape and rejects the clause. MySQL needs the same one-character escape written as '\\'.
A backslash needs doubling inside MySQL string literals but not in SQLite or PostgreSQL, which is why the LIKE clause checked the adapter. Escaping with ! works the same everywhere, so the check goes away. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A worker listing queue_* and queue_b*, or queue_b* and queue_b1, polled queue_b1 twice on every pass and counted its jobs twice. Prefix matches were never removed from the remaining names, because `queues -= matches` only reassigned the block's local variable, and a queue matched both by name and by a prefix came back from both lookups. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a job outlives shutdown_timeout during graceful termination, the async supervisor ends with exit!, which skips the shutdown callbacks: neither the supervisor nor its threads deregister, and their claimed jobs stay claimed until a later supervisor prunes the stale registrations and fails them with ProcessPrunedError. On a TERM, which is what deploys send, the job is failed instead of going back to its queue as it does in fork mode. Worse, the supervisor and each worker wait for that same timeout, the supervisor starting slightly earlier, so it usually runs out first and calls exit! while the worker is still inside its own deregistration transaction, rolling back the release it had started. When the timeout runs out with threads still alive, deregister the supervisor before exit!, as the fork supervisor does after quitting its forks: the threads still running are deregistered with it and their claimed jobs go back to their queues. Deregister from a fresh copy of the registration, since the threads share theirs with the supervisor's in memory and a thread that already deregistered leaves a frozen record behind. QUIT is unchanged: it still exits without cleanup, as the async lifecycle test documents. The async lifecycle test for TERM past the timeout now expects the in-flight job back in its queue and a clean termination, like its fork counterpart, instead of accepting either outcome. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The signals section describes fork mode, where QUIT still returns in-flight jobs to the queue. In async mode QUIT exits without deregistering, so those jobs are failed once the processes are pruned, while a TERM that runs out of time now returns them (see rails#815). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bulk discards now free each job's concurrency slot and promote the next blocked job one job at a time, so this lives next to the single-job unblock_next_blocked_job as Job.unblock_next_blocked_jobs. Semaphore.signal_all had no callers left. It also returned only one slot when several discarded jobs shared a key, and never extended expires_at like signal does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ration A scheduler removes static tasks that are no longer configured when it starts, but removing the last one means no scheduler starts, so those rows stay around. Cleaning them up without a scheduler isn't safe when several machines run Solid Queue (see rails#803), so document the manual cleanup instead. See rails#623. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A task that's deleted and created again with the same key gets a new id, so keying the in-memory schedule by id lets the existing steps that schedule new tasks and unschedule deleted ones handle it, without a second map from keys to ids or a separate active flag. A run only schedules the next one if its task is still in the schedule, which also covers a replaced task and a stopped scheduler. The database query for dynamic tasks now runs before taking the lock, so it doesn't hold up a task that's due. Polling now only unschedules dynamic tasks. A static task whose row is deleted keeps running until the scheduler restarts, when it reads the configuration again. Also freeze time in the test for tasks updated in place, so a run at the top of the hour can't land between its two lookups. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatching a set of jobs with concurrency controls acquires each job's semaphore one by one, inside the transaction that dispatches the whole set, and in whatever order the jobs come in. Two dispatchers whose batches share concurrency keys can then acquire them in opposite order and deadlock, and so can bulk enqueues. The batch that's aborted stays scheduled, so no job is lost and the limits hold, but the dispatcher that hits the error is delayed. Go through the jobs by concurrency key instead, so everyone acquires the locks they share in the same order. Within a key, jobs keep their id order, so the same job as before gets the lock first. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Acq8BRNz9v7w9mKYLYRdCi
… tests It runs two dispatchers in threads against the same semaphores, like the lock tests in semaphore_test.rb, so it belongs there rather than in its own integration file. It still deadlocks on PostgreSQL and MySQL without dispatching by concurrency key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rosa
force-pushed
the
order-concurrency-locks-by-key
branch
from
November 3, 2026 17:17
dbab4f9 to
a6d5528
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
dispatch_next_batchdispatches a whole batch in one transaction and, for jobs with concurrency controls, acquires each semaphore in job ID order. Two dispatchers whose batches share keys can take them in opposite order and deadlock onsolid_queue_semaphores.perform_all_latergoes through the same path.However, no job is lost and limits hold, the batch rolls back and stays scheduled, and the dispatcher is restarted.
It only shows up under load (3 dispatchers, 200 limited jobs/s over 50 keys: ~20 deadlocks in 20 s on PostgreSQL 17).
This issue is different from rails#609 and rails#162.
There is a script to reproduce the issue with a rails single file https://gist.github.com/ceritium/82d13702de27d36831204bd126370568
Solution
The fix sorts by
[concurrency_key, id]so everyone takes shared locks in the same order. The batch commits atomically and workers claim by priority and id, so the order isn't visible outside.Let me know what you think.
Many thanks!