Repository navigation
fix: let streams accepted before a GOAWAY finish - #961
Merged
Merged
Conversation
RFC 9113 6.8 only refuses the streams above last_stream_id; the peer may
still process the ones up to it and keeps the connection open to finish
them. h2_on_goaway failed every in-flight stream regardless, so a request
the server went on to complete came back as {error, {goaway, _}}. Fail only
the refused streams, leave the pool so no new request lands here, refuse one
that was already checked out, and close once the accepted streams end. A
streamed request or response body in flight still closes at once.
Owner
|
Thanks @lennartschoch, clear writeup and solid tests. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ran into this after moving an Elixir app onto pooled HTTP/2. Every so often a request would come back as
{error, {goaway, no_error}}, except the server had actually handled it. In our case that was a payment that went through and got recorded on our side as failed, which is about the worst way for a transport error to surface.Turns out
handle_h2_event/2throws away the GOAWAY'slast_stream_id, andh2_on_goaway/2then fails every stream in flight. But a GOAWAY only refuses the streams abovelast_stream_id(RFC 9113 §6.8). The ones up to it may well still be processed, and the peer keeps the connection open to finish them.h2already does its part here, it moves togoaway_receivedand keeps delivering frames for the existing streams. So the responses do arrive, hackney has just already told the callers they failed. Same story for the two-step shutdown §6.8 recommends, GOAWAY(2^31-1) followed by the real one: the first frame refuses nothing, and we were failing everything on it.With this change only the streams above
last_stream_idfail, still as{goaway, ErrorCode}, which for an ordinary request now actually means "the server didn't process this". If nothing accepted is in flight the connection closes right away like it does today. Otherwise it drains: it drops out of the pool and reports{ok, draining}fromget_state, socheckout_h2/h2_conn_usableandrequest/5skip it the way they'd skip aclosedone and dial fresh. A request that was already checked out and lands on it anyway gets refused with{goaway, ErrorCode}before anything is sent, sinceh2_connectionis ingoaway_receivedby then and would only answer{error, unknown_request}, and the peer would ignore the stream regardless. Once the last accepted stream ends, the connection closes.Two things I left alone on purpose. A connection with a streamed request or response body in flight still closes everything at once for now. That mode walks the connection through states of its own, so I'd rather do it as its own PR on top of this one than fold it in here. And the refused streams aren't RST'd:
h2_connectiondoesn't takecancel_streamafter a GOAWAY, so they just go with the connection when it closes.rebar3 eunitis green, 1170 tests, xref and dialyzer clean. The four new ones intest/hackney_http2_goaway_drain_tests.erlall fail without the change with{error, {goaway, no_error}}: a GOAWAY that covers both in-flight streams (both complete, and a request made during the drain gets a fresh connection), one that covers only the first (the second fails fast, the first still completes), the two-step shutdown from §6.8, and a server that accepts a stream and then never answers it, where the stream'srecv_timeoutends it and takes the connection down with it.