Skip to content

feat(fabric): propagate trace context, add OnReconnect, pin serial callbacks (RIG-3107) - #1337

Draft
rigel-mintaka wants to merge 7 commits into
compass-managed/rig-3107-fabric-publishfrom
compass-managed/rig-3107-fabric-seam
Draft

rigel-mintaka wants to merge 7 commits into
compass-managed/rig-3107-fabric-publishfrom
compass-managed/rig-3107-fabric-seam

Conversation

@rigel-mintaka

@rigel-mintaka rigel-mintaka commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

This PR is part of a stack containing 6 PRs:

  1. main
  2. feat(comms): publish message_posted on the event fabric (RIG-3107) #1333
  3. "feat(fabric): propagate trace context, add OnReconnect, pin serial callbacks (RIG-3107)" (this PR)
  4. feat(delivery): consume message_posted from the event fabric (RIG-3107) #1345
  5. fix(delivery): fire a hold that lost the race to its settle; skip held in start sweeps (RIG-4031) #1351
  6. test(delivery): prove single-claim across instances and slow-callback redelivery (RIG-3107) #1353
  7. feat(fabric): let a subscriber callback fail so the event is redelivered (RIG-4030) #1355

Fabric-side prerequisites for moving delivery off the in-process bus. #1345 is the consumer.

  • Trace context. Publish writes a lowercase traceparent NATS header, and subscribe reads it in any case. nats-server 2.11 and 2.12 lowercase the key in place. Callbacks take func(ctx, EventRef) with the extracted span, and the subscriber's own span is stripped.
  • Callback ctx. It is WithoutCancel(subscribe ctx), bounded by AckWait. Events drained at teardown are no longer handled on a cancelled ctx and then acked.
  • OnReconnect(fn). Hooks run on a fabric goroutine, off the NATS callback goroutine. Each hook runs under a panic guard, and a burst of reconnects coalesces. A caller ReconnectHandler in Config.Options is chained after the fabric's handler, so it does not replace it. No hook runs once Close starts.
  • Serial callbacks. A per-subscription mutex makes serial dispatch a fabric guarantee. The no-overlap test guards that guarantee, not the mutex: nats.go already dispatches serially today.
  • Credential redaction. The reconnect log and the connect error redact every credential form in the NATS URL: a seed list, a token, and a scheme-less entry.

Hand-off hold

The callback signature change is RIG-4014 (Matt, unruled). Once it is ruled, the frozen record's T2 Interfaces line and DECISIONS.md get an amendment, and the Spec-impact line below changes to cite it. Enqueue at the stack tip, #1345.

Spec-impact: pending RIG-4014 (see above). Refs RIG-3107, RIG-4014

Co-authored-by: Matt Wilkinson matt@rigel.build

…llbacks (RIG-3107)

Fabric-side prerequisites for moving delivery off the in-process bus:

- Publish writes the W3C traceparent into a NATS header, and callbacks
  now take func(ctx, EventRef) with the extracted span context. Delivery
  keeps one trace per turn once its trigger moves to the fabric.
- OnReconnect(fn) registers a hook that runs after the fabric's own
  reconnect log line. The recovery sweep uses it to replace the bus-lag
  signal.
- A per-subscription mutex makes serial callback dispatch a fabric
  promise instead of a nats.go detail.

Tests cover each behaviour, and each fails when its mechanism is
removed. Nothing consumes the new seam yet; delivery still runs on the
bus.

The EventFabric callback signature change is pending Matt's ruling
in RIG-4014.

Spec-impact: none. Refs RIG-3107, RIG-4014

Co-authored-by: Matt Wilkinson <matt@rigel.build>
@linear-code

linear-code Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

RIG-4014

RIG-3107

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Compass engineering docs preview: https://compass-managed-rig-3107-fab-nrbp.compass-eng-docs.pages.dev

Deployed from compass-managed/rig-3107-fabric-seam at 15007c4.

rigel-mintaka and others added 6 commits September 26, 2026 02:57
…G-3107)

Review fixes on the fabric seam:

- nats-server 2.11/2.12 lowercase a Traceparent header in place, and
  NATS headers are case-sensitive. Publish writes lowercase
  traceparent; subscribe reads the header in any case.
- Reconnect hooks run on a fabric goroutine, off the NATS callback
  goroutine, each under a panic guard. A reconnect burst coalesces.
- A caller ReconnectHandler in Config.Options is chained after the
  fabric one instead of replacing it, so OnReconnect stays armed.
- Callbacks run on WithoutCancel(ctx) bounded by AckWait, with the
  subscriber span stripped. Events drained at teardown are no longer
  handled on a cancelled ctx and then acked.
- The reconnect log and the connect error redact URL credentials.

Spec-impact: none. Refs RIG-3107, RIG-4014

Co-authored-by: Matt Wilkinson <matt@rigel.build>
…(RIG-3107)

url.Redacted left a seed list past its first entry, a username-only
token, and a scheme-less entry unmasked. redactURL now splits the seed
list and replaces all userinfo; the reconnect log uses it too. The hook
loop also re-checks the fabric is open, because select picks randomly
once teardown and a pending signal are both ready.

Spec-impact: none. Refs RIG-3107

Co-authored-by: Matt Wilkinson <matt@rigel.build>
…G-3107)

url.Parse ends the authority at a raw /, ? or #, so a credential that
held one (common in base64 tokens) came out unredacted. NATS URLs have
no path, so redactURL now treats everything before the last @ as
credential and keeps only a letters-only scheme. Empty seed entries are
dropped rather than printed as nats:.

Spec-impact: none. Refs RIG-3107

Co-authored-by: Matt Wilkinson <matt@rigel.build>
nats.go quotes the raw URL in its *url.Error, so a password with an
unescaped %, ^ or # leaked through the wrapped error. New now returns a
fixed message for a URL parse error. redactURL cuts the whole string at
its last @, so a comma inside a credential can no longer split it into
a fake seed entry. Every credential form is tested through New and
through redactURL.

Spec-impact: none. Refs RIG-3107

Co-authored-by: Matt Wilkinson <matt@rigel.build>
…(RIG-3107)

nats.go splits the URL on commas, so an unencoded comma in a token made
a fake host whose DNS error named part of the credential. Whether it
leaked depended on the shuffled server order (12 of 30 runs). New now
passes only the fixed-text ErrNoServers through when the URL carries
credentials; any other cause is withheld. The URL doc says to
percent-encode a comma in a credential.

Spec-impact: none. Refs RIG-3107

Co-authored-by: Matt Wilkinson <matt@rigel.build>
…ed URLs (RIG-3107)

Withholding every cause hid a wrong password behind "cause withheld".
New now passes nats.ErrNoServers and nats.ErrAuthorization through,
wrapping the sentinel itself so neither server text nor URL text rides
along. Any other cause is still withheld. A test against a
password-protected nats-server pins the auth case.

Spec-impact: none. Refs RIG-3107

Co-authored-by: Matt Wilkinson <matt@rigel.build>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant