fix: prevent stream shutdown deadlock when Close races with event delivery#60
Merged
aaron-zeisler merged 1 commit intoJul 22, 2026
Merged
Conversation
…ivery When Close (or a restart/error) fired while the decoder goroutine had just decoded an event and was blocked sending it on the internal events channel, discardCurrentStream deadlocked: it drained the errs channel before events, but the decoder was stuck on the events send and never reached the code that closes errs. Neither side progressed, so stream.Events was never closed and a consumer draining it on shutdown would hang forever. Drain both internal channels concurrently. The decoder may be blocked sending on either channel, and once unblocked it always closes both, so draining them together lets it terminate regardless of which send it was parked on. Add a regression test that floods the stream and closes mid-delivery, which reliably reproduces the hang before this change.
keelerm84
approved these changes
Jul 22, 2026
aaron-zeisler
deleted the
aaronz/SDK-2549/fix-stream-close-drain-deadlock
branch
July 22, 2026 15:34
aaron-zeisler
pushed a commit
that referenced
this pull request
Jul 22, 2026
🤖 I have created a release *beep* *boop* --- ## [1.11.1](v1.11.0...v1.11.1) (2026-07-22) ### Bug Fixes * prevent stream shutdown deadlock when Close races with event delivery ([1067581](1067581)) * prevent stream shutdown deadlock when Close races with event delivery ([#60](#60)) ([c3ddae0](c3ddae0)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This was referenced Jul 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a deadlock in the stream goroutine's shutdown path that can leave
stream.Eventspermanently open. A consumer that drainsstream.Eventsafter callingClose(the documented shutdown pattern) then hangs forever.Background
When the stream is torn down — via
Close,Restart, or a read error —discardCurrentStreamshuts down the current connection's decoder goroutine and drains its two internal channels so the goroutine can exit:The decoder goroutine is a simple loop: decode, and either send the event on
events, or on error send onerrsand close both channels.The drain assumes that after
r.Close()the decoder is (or will shortly be) in the error path, so drainingerrsfirst lets it reach theclose(errs); close(events). But if the decoder had already decoded an event and was blocked onevents <- evwhen we closed,r.Close()doesn't unblock it — it's parked on a send, not insideDecode. The drain waits onerrs(nothing will ever arrive there), and the decoder waits for someone to receive itseventssend. Neither moves, soevents/errsare never closed,stream.streamnever reachesclose(stream.Events), and any consumer ranging overstream.Eventson shutdown blocks forever.It's timing-dependent — it needs
Closeto land while an event is mid-delivery — so it shows up as a rare, intermittent hang. It surfaced downstream inld-relay's auto-config stream (relay shutdown intermittently hanging until the process/test timed out).Fix
Drain both internal channels concurrently. The decoder can be blocked sending on either one, and once unblocked it always closes both — so draining them together lets it terminate regardless of which send it was parked on.
Verification
TestStreamCloseDuringEventDeliveryDoesNotDeadlockfloods the stream and closes mid-delivery, repeatedly. It hangs → fails on the old code (deadlock stack indiscardCurrentStream) and passes with the fix.-race:go test -race ./....ld-relay: with this fix pulled in via a localreplaceand no changes told-relay, its auto-config shutdown regression test passes; against stock v1.11.0 the same test hangs.