Skip to content

fix(delivery): fire a hold that lost the race to its settle; skip held in start sweeps (RIG-4031) - #1351

Draft
rigel-mintaka wants to merge 3 commits into
compass-managed/rig-3107-delivery-cutoverfrom
compass-managed/rig-4031-held-gaps
Draft

rigel-mintaka wants to merge 3 commits into
compass-managed/rig-3107-delivery-cutoverfrom
compass-managed/rig-4031-held-gaps

Conversation

@rigel-mintaka

@rigel-mintaka rigel-mintaka commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

This PR is part of a stack containing 6 PRs:

  1. main
  2. feat(comms): publish message_posted on the event fabric (RIG-3107) #1333
  3. feat(fabric): propagate trace context, add OnReconnect, pin serial callbacks (RIG-3107) #1337
  4. feat(delivery): consume message_posted from the event fabric (RIG-3107) #1345
  5. "fix(delivery): fire a hold that lost the race to its settle; skip held in start sweeps (RIG-4031)" (this PR)
  6. test(delivery): prove single-claim across instances and slow-callback redelivery (RIG-3107) #1353
  7. feat(fabric): let a subscriber callback fail so the event is redelivered (RIG-4030) #1355

The fabric callback runs concurrently with the settle drain. A message
held after its author settled waited for the next settle, unbounded for
an author that stays READY. OnSessionSettled now records a per-session
settle time, and hold delivers at once from stored blocks when that
settle is at or after the message commit time. Entries are pruned by
the recovery pass and dropped on reap.

The session-start sweep also skips held messages, so a recipient that
starts mid-turn gets the settled blocks from fireHeld instead of a
partial deliver that dedup would keep.

Spec-impact: none. Refs RIG-4031, RIG-3107

Co-authored-by: Matt Wilkinson matt@rigel.build

…d in start sweeps (RIG-4031)

The fabric callback runs concurrently with the settle drain. A message
held after its author settled waited for the next settle, unbounded for
an author that stays READY. OnSessionSettled now records a per-session
settle time, and hold delivers at once from stored blocks when that
settle is at or after the message commit time. Entries are pruned by
the recovery pass and dropped on reap.

The session-start sweep also skips held messages, so a recipient that
starts mid-turn gets the settled blocks from fireHeld instead of a
partial deliver that dedup would keep.

Spec-impact: none. Refs RIG-4031, RIG-3107

Co-authored-by: Matt Wilkinson <matt@rigel.build>
@linear-code

linear-code Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

RIG-4031

RIG-3107

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

Compass engineering docs preview: https://compass-managed-rig-4031-hel.compass-eng-docs.pages.dev

Deployed from compass-managed/rig-4031-held-gaps at b4f5713.

rigel-mintaka and others added 2 commits September 26, 2026 05:43
…e by liveness (RIG-4031)

Review fixes:

- A hold that finds its settle already recorded now appends to c.held and
  queues a settle edge on the loop, instead of dispatching on the fabric
  goroutine. fireHeld sends the list in post order, so a late M2 no
  longer overtakes a held M1.
- Sweeps skip only messages held for a live author session, so an entry
  stranded under a dead session is still delivered by the cursor sweep.
- lastSettle is pruned by liveness, not age: settle >= at is correct at
  any age, and an age prune dropped the guard while a backlog replayed.

Spec-impact: none. Refs RIG-4031

Co-authored-by: Matt Wilkinson <matt@rigel.build>
…031)

The edge a late hold queues fired the whole held list, including a next-turn
message still streaming. Held entries now carry their commit time, and
the edge fires only entries at or before the settle it replays; a real
settle still fires all. heldIDs reads liveness after the held snapshot, so
a race leaves an entry skipped rather than sent partial.

Spec-impact: none. Refs RIG-4031

Co-authored-by: Matt Wilkinson <matt@rigel.build>
@rigelbuild-bot

Copy link
Copy Markdown
Contributor

Status at b4f5713:

  • Review loop: all-clear after 3 rounds (0 high/medium). The remaining low existed before this PR and is filed as the decision RIG-4033.
  • Local gate: delivery -race, targeted tests -count=20, golangci-lint 0 issues, and container pgtest for delivery and server are all green.
  • CI: the microvm job failed on TestInGuestEgressArmAutoloadsNetfilter. The test's asserts passed; the failure came from the cleanup-time Shutdown: waiting for virtiofsd: context canceled. This PR does not touch internal/runtime/microvm. Filed as RIG-4035 for its owning lane; not re-run.

This PR stays a draft with #1345; it is gated on RIG-4014 and RIG-4030. Enqueue at this PR, the stack tip, once promoted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants