Skip to content

fix: let streams accepted before a GOAWAY finish - #961

Merged
benoitc merged 1 commit into
benoitc:masterfrom
lennartschoch:fix/h2-goaway-drain
Oct 2, 2026
Merged

benoitc merged 1 commit into
benoitc:masterfrom
lennartschoch:fix/h2-goaway-drain

Conversation

@lennartschoch

Copy link
Copy Markdown
Contributor

Ran into this after moving an Elixir app onto pooled HTTP/2. Every so often a request would come back as {error, {goaway, no_error}}, except the server had actually handled it. In our case that was a payment that went through and got recorded on our side as failed, which is about the worst way for a transport error to surface.

Turns out handle_h2_event/2 throws away the GOAWAY's last_stream_id, and h2_on_goaway/2 then fails every stream in flight. But a GOAWAY only refuses the streams above last_stream_id (RFC 9113 §6.8). The ones up to it may well still be processed, and the peer keeps the connection open to finish them. h2 already does its part here, it moves to goaway_received and keeps delivering frames for the existing streams. So the responses do arrive, hackney has just already told the callers they failed. Same story for the two-step shutdown §6.8 recommends, GOAWAY(2^31-1) followed by the real one: the first frame refuses nothing, and we were failing everything on it.

With this change only the streams above last_stream_id fail, still as {goaway, ErrorCode}, which for an ordinary request now actually means "the server didn't process this". If nothing accepted is in flight the connection closes right away like it does today. Otherwise it drains: it drops out of the pool and reports {ok, draining} from get_state, so checkout_h2/h2_conn_usable and request/5 skip it the way they'd skip a closed one and dial fresh. A request that was already checked out and lands on it anyway gets refused with {goaway, ErrorCode} before anything is sent, since h2_connection is in goaway_received by then and would only answer {error, unknown_request}, and the peer would ignore the stream regardless. Once the last accepted stream ends, the connection closes.

Two things I left alone on purpose. A connection with a streamed request or response body in flight still closes everything at once for now. That mode walks the connection through states of its own, so I'd rather do it as its own PR on top of this one than fold it in here. And the refused streams aren't RST'd: h2_connection doesn't take cancel_stream after a GOAWAY, so they just go with the connection when it closes.

rebar3 eunit is green, 1170 tests, xref and dialyzer clean. The four new ones in test/hackney_http2_goaway_drain_tests.erl all fail without the change with {error, {goaway, no_error}}: a GOAWAY that covers both in-flight streams (both complete, and a request made during the drain gets a fresh connection), one that covers only the first (the second fails fast, the first still completes), the two-step shutdown from §6.8, and a server that accepts a stream and then never answers it, where the stream's recv_timeout ends it and takes the connection down with it.

RFC 9113 6.8 only refuses the streams above last_stream_id; the peer may
still process the ones up to it and keeps the connection open to finish
them. h2_on_goaway failed every in-flight stream regardless, so a request
the server went on to complete came back as {error, {goaway, _}}. Fail only
the refused streams, leave the pool so no new request lands here, refuse one
that was already checked out, and close once the accepted streams end. A
streamed request or response body in flight still closes at once.
@benoitc

benoitc commented Oct 2, 2026

Copy link
Copy Markdown
Owner

Thanks @lennartschoch, clear writeup and solid tests.

@benoitc
benoitc merged commit 2038022 into benoitc:master Oct 2, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants