PersistedQueue hardening and optimization pass - #7547
Merged
Conversation
🦋 Changeset detectedLatest commit: b7eca51 The changes in this PR will be included in the next version bump. This PR includes changesets to release 30 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Bundle Size AnalysisGenerated from PR build output; treat the content below as untrusted.
|
tim-smart
commented
Aug 31, 2026
- Redis: park pending dedupe ids at +inf so timeToLive trims only completed entries, matching SQL/memory (with shared suite coverage) - Redis: trimFailed pops corrupt heads instead of blocking the whole list - Unify the unreachable configs maxAttempts fallback across stores - Fix takers counter leak when a waiting take is interrupted (onExit) - mssql: atomic MERGE WITH (HOLDLOCK) offer instead of racy IF NOT EXISTS
- Memory store: keep a pending subset so take stops scanning the completed backlog (was O(all entries ever offered) per take) - Merge the configs/nudges maps and fallback constant into one per-queue state record shared by offer/take/mailbox in both stores - One ack retry schedule and one SQL ack pipeline instead of six copies - Collapse secondsAgo/secondsFromNow into a single signed offset helper - Factory decode without the Exit round-trip; drop unused Exit/Arr imports - Redis: compute queue keys once per take, reuse the raw wire payload for requeue/retry, drain the failed-list trim, bounded-concurrent cleanup, poller failure logging parity with SQL - SQL: stop selecting the unused queue_name column, mutate polled rows in place instead of cloning (twice on mysql) - Tests: assertNotDelivered/advancePastTtl helpers, per-test timeout derived from the suite timeout instead of 12 magic literals
tim-smart
commented
Sep 1, 2026
The persisted attempt count is the schedule state: retryDelay replays a fresh schedule step up to the current attempt on every call, so delays keep progressing across takers and process restarts. Document that on make() and in the implementation, stop stepping a schedule that reported done, and add a memory-store test that fails if delays reset between retries.
tim-smart
force-pushed
the
agent/architect/f0e3f5086ee6
branch
from
September 1, 2026 01:05
3dd795a to
c5b2cee
Compare
tim-smart
commented
Sep 1, 2026
tim-smart
commented
Sep 1, 2026
- Shared suite: concurrency soak test — 24 elements, 3 workers, forced first-delivery failures, asserting exactly-once success and exact delivery counts on every backend - pg: exactly-once across two store instances with four workers; crashed worker recovery after lock expiration preserving the attempt count; reaper dead-lettering a row whose final attempt crashed - Redis: crashed worker recovery via the reset pass preserving the attempt count; crash on the final attempt landing in the failed list with the lock-expired reason - mysql: 200KB payload round-trip through the MEDIUMTEXT column Also fixes a shared-suite flake: advancePastTtl jumped the virtual clock by 2 minutes, which could fire the SQL client pool timers and time out in-flight connection acquisitions during cleanup. The jump only needs to outrun a 1 second virtual ttl, so it is now 2 seconds.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hardening and optimization pass over
PersistedQueue, covering the SQL, Redis, and memory stores plus the factory. SQL schema changes are applied by0002_upgrade_schema, which upgrades existing0001_create_tabledata in place.Closes EFF-980
Semantics
attempts, interrupt-release decrements it back, and the failure path no longer bumps. Handler metadata is now 1-based. A handler that crashes the process on its final attempt is flipped tofailedby a periodic pass over exhausted, lock-expired pending rows (mirrored in the Redis reset script).visible_atcolumn set from a retry schedule. The poll filtersvisible_at <= nowand orders byvisible_at, sequence. Redis uses a delayed zset swept by the take script; memory delays re-eligibility.completed BOOLEANbecomesstate(pending/completed/failed). Exhausted rows are marked failed instead of becoming invisible, matching the Redis:faileddestination. Inspect/requeue APIs are a later pass.last_failure, then takes the next item. Decode errors leavetake's error channel.make().maxAttemptsandretryScheduleare queue-definition options;take()loses its options. This also removesmaxAttemptsfrom the mailbox key.complete/retryretry with backoff up to the lock expiration window (the refresh fiber holds ownership meanwhile) instead of dying after ~1s.Retention and cleanup
Completed rows are kept for de-duplication (DurableQueue replays re-offer the same id) and pruned by a new
PersistedQueueStore.cleanupplusPersistedQueue.layerCleanup({ interval, timeToLive, failedTimeToLive }). Defaults: 30 days for completed; failed rows are the dead-letter record and exempt unlessfailedTimeToLiveis set. SQL batch-deletes with small LIMITs, Redis trims the ids zset and failed lists, memory prunes its entries.Bug fixes
id/queue_namewidened to 255 (DurableQueue ids like${queueName}/${idempotencyKey}overflowedVARCHAR(36)).sequenceis 64-bit on all dialects (was int32 on pg and mssql).element/last_failureareMEDIUMTEXT(64KB cap before).:idsdedup set no longer grows forever (timestamp-scored zset trimmed by cleanup).GETDATE()→SYSDATETIME()(1/300s rounding could put freshvisible_atvalues in the future, hiding them from the poll), and two mssql migration statements had invalid identifier interpolation.Optimizations
offernudges the local poll latch, so same-process work skips the up-to-pollIntervalwait. No LISTEN/NOTIFY this pass.Docs
At-least-once delivery (idempotent handlers) is now stated on the module and
take, and DurableQueue documents that a dead-lettered item parks its workflow until the future requeue API, plus the single-cleanup-instance recommendation.Testing
Shared queue behavior runs against memory, SQLite, PostgreSQL, MySQL, MSSQL, and Redis. Migration coverage exercises fresh installs and upgrades from the original
0001on every SQL backend, including preservation of pending and completed rows, de-duplication, widened identifiers, and payloads over 64 KB.pnpm checkandpnpm lintpass.