From a77d7d8418b2da0921f9e739f3e5f2edfab96562 Mon Sep 17 00:00:00 2001 From: "jeremy.barisch.rooney@channable.com" Date: Fri, 9 Oct 2026 16:36:24 +0200 Subject: [PATCH 1/6] Add state diagram for chunks --- README.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/README.md b/README.md index cc8a4760..1ef662df 100644 --- a/README.md +++ b/README.md @@ -341,6 +341,36 @@ Were a consumer to _raise an exception_ or _outright crash_ or _have network pro In the event of a consumer crash or (ephemeral) network problems, we do not want work to get lost. The opsqueue system takes the 'at least once' approach (rather than the 'at most once' approach). This means that your consumers **must be idempotent**. They have to handle the possibility of (part of a) chunk being re-executed multiple times. +## Chunks + +It may be helpful in understanding OpsQueue to see a state machine of a chunk: +diagram, definitions of each of the states follows below: + +```mermaid +stateDiagram + [*] --> Paused: Inserted paused + [*] --> Available: Inserted active + Paused --> Available: Submission unpaused + Available --> Reserved: Consumer reserves + Reserved --> Completed: Completion saved + Reserved --> Available: Failed attempt, retries remain + Reserved --> Available: Consumer disconnects + Reserved --> Available: Completion write fails (delayed release) + Reserved --> Failed: Retry limit reached + Paused --> Skipped: Submission cancelled + Available --> Skipped: Submission cancelled or failed + Reserved --> Skipped: Submission cancelled or failed +``` + +| State | Pseudo-SQL definition | +|---------------|---------------------------------------------------------------------------------------| +| **Paused** | `SELECT * FROM chunks_paused` | +| **Available** | `SELECT * FROM chunks WHERE opsqueue_is_reserved(submission_id, chunk_index) = FALSE` | +| **Reserved** | `SELECT * FROM chunks WHERE opsqueue_is_reserved(submission_id, chunk_index) = TRUE` | +| **Completed** | `SELECT * FROM chunks_completed` | +| **Failed** | `SELECT * FROM chunks_failed WHERE skipped = FALSE` | +| **Skipped** | `SELECT * FROM chunks_failed WHERE skipped = TRUE` | + ## API connections Under the hood, the producer and the queue talk with each other using a JSON-REST API over HTTP. Users of opsqueue don't need to think about this, as this is abstracted behind the client library. From 95cdb00a5c954211c0d7da7a7101145cf892efd5 Mon Sep 17 00:00:00 2001 From: "jeremy.barisch.rooney@channable.com" Date: Fri, 9 Oct 2026 16:37:56 +0200 Subject: [PATCH 2/6] fixup! Add state diagram for chunks --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 1ef662df..558e9c88 100644 --- a/README.md +++ b/README.md @@ -343,8 +343,8 @@ In the event of a consumer crash or (ephemeral) network problems, we do not want ## Chunks -It may be helpful in understanding OpsQueue to see a state machine of a chunk: -diagram, definitions of each of the states follows below: +It may be helpful in understanding OpsQueue to see a state machine of a chunk, +definitions of each of the states follows below: ```mermaid stateDiagram From 3091d37ecdaf700bfdb235906e8f345788a2881e Mon Sep 17 00:00:00 2001 From: "jeremy.barisch.rooney@channable.com" Date: Fri, 9 Oct 2026 16:50:42 +0200 Subject: [PATCH 3/6] fixup! Add state diagram for chunks --- README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 558e9c88..8c30898b 100644 --- a/README.md +++ b/README.md @@ -348,14 +348,14 @@ definitions of each of the states follows below: ```mermaid stateDiagram - [*] --> Paused: Inserted paused - [*] --> Available: Inserted active + [*] --> Available: Submission created + [*] --> Paused: Submission created paused Paused --> Available: Submission unpaused - Available --> Reserved: Consumer reserves - Reserved --> Completed: Completion saved - Reserved --> Available: Failed attempt, retries remain + Available --> Reserved: Consumer reserves chunk + Reserved --> Completed: Consumer reports completed chunk + Reserved --> Available: Consumer reports failed chunk, retries remain Reserved --> Available: Consumer disconnects - Reserved --> Available: Completion write fails (delayed release) + Reserved --> Available: Recording completion fails Reserved --> Failed: Retry limit reached Paused --> Skipped: Submission cancelled Available --> Skipped: Submission cancelled or failed From c858dbd0cf8cc3a4d54ad23a52ae354c9133eb97 Mon Sep 17 00:00:00 2001 From: "jeremy.barisch.rooney@channable.com" Date: Fri, 9 Oct 2026 16:55:26 +0200 Subject: [PATCH 4/6] fixup! Add state diagram for chunks --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 8c30898b..222b3979 100644 --- a/README.md +++ b/README.md @@ -343,8 +343,8 @@ In the event of a consumer crash or (ephemeral) network problems, we do not want ## Chunks -It may be helpful in understanding OpsQueue to see a state machine of a chunk, -definitions of each of the states follows below: +It may be helpful in understanding OpsQueue to see a state machine of a chunk. +Definitions of each of the states follows below: ```mermaid stateDiagram @@ -355,7 +355,7 @@ stateDiagram Reserved --> Completed: Consumer reports completed chunk Reserved --> Available: Consumer reports failed chunk, retries remain Reserved --> Available: Consumer disconnects - Reserved --> Available: Recording completion fails + Reserved --> Available: Recording completion/failures fails Reserved --> Failed: Retry limit reached Paused --> Skipped: Submission cancelled Available --> Skipped: Submission cancelled or failed From 2f6502774c54b092e1d7755cc9d149dbbb28b6ed Mon Sep 17 00:00:00 2001 From: "jeremy.barisch.rooney@channable.com" Date: Fri, 9 Oct 2026 17:11:25 +0200 Subject: [PATCH 5/6] fixup! Add state diagram for chunks --- README.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 222b3979..eca5aec9 100644 --- a/README.md +++ b/README.md @@ -352,16 +352,21 @@ stateDiagram [*] --> Paused: Submission created paused Paused --> Available: Submission unpaused Available --> Reserved: Consumer reserves chunk - Reserved --> Completed: Consumer reports completed chunk - Reserved --> Available: Consumer reports failed chunk, retries remain - Reserved --> Available: Consumer disconnects - Reserved --> Available: Recording completion/failures fails + Reserved --> Completed: Consumer reports success, completion recorded + Reserved --> Available: Consumer reports failure, retries remain + Reserved --> Available: Consumer disconnects, reservation released + Reserved --> Available: Reservation expires + Reserved --> Available: Completion/failure transaction fails, reservation released Reserved --> Failed: Retry limit reached Paused --> Skipped: Submission cancelled Available --> Skipped: Submission cancelled or failed Reserved --> Skipped: Submission cancelled or failed ``` +Note that due to late (after reservation expired) completion/failure of a chunk +is also possible, so we would have additional transitions `Available -> Completed` +and `Available -> Failed`, but kept out of the diagram for simplicity. + | State | Pseudo-SQL definition | |---------------|---------------------------------------------------------------------------------------| | **Paused** | `SELECT * FROM chunks_paused` | From 8f3a59b9c6d4ff390336e4d04367828443315f3f Mon Sep 17 00:00:00 2001 From: "jeremy.barisch.rooney@channable.com" Date: Fri, 9 Oct 2026 17:12:14 +0200 Subject: [PATCH 6/6] fixup! Add state diagram for chunks --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index eca5aec9..6cb9f44e 100644 --- a/README.md +++ b/README.md @@ -363,9 +363,9 @@ stateDiagram Reserved --> Skipped: Submission cancelled or failed ``` -Note that due to late (after reservation expired) completion/failure of a chunk -is also possible, so we would have additional transitions `Available -> Completed` -and `Available -> Failed`, but kept out of the diagram for simplicity. +Note that late (after reservation expired) completion/failure of a chunk is also +possible, so we would have additional transitions `Available -> Completed` and +`Available -> Failed`, but kept out of the diagram for simplicity. | State | Pseudo-SQL definition | |---------------|---------------------------------------------------------------------------------------|