diff --git a/lib/mint/http.ex b/lib/mint/http.ex index 45d65f78..277ac4a5 100644 --- a/lib/mint/http.ex +++ b/lib/mint/http.ex @@ -296,6 +296,42 @@ defmodule Mint.HTTP do this time, `connect/4` returns a `Mint.HTTPError` with reason `{:proxy, :tunnel_timeout}`. Defaults to `30_000` (30 seconds). + ### Proxying over HTTP/2 + + The `:proxy` option always speaks HTTP/1 to the proxy. The connection tunneled + *through* the proxy can still be HTTP/2 (negotiated via ALPN), but the `CONNECT` + exchange with the proxy itself is HTTP/1. Mint doesn't drive an HTTP/2 connection + to the proxy from `connect/4` on purpose: the benefit of HTTP/2 `CONNECT` is + multiplexing many tunnels over a single proxy connection, which means sharing that + connection across Mint connections. That is the job of a connection pool built on + top of Mint, not of a single connection. + + To tunnel through a proxy over HTTP/2, connect to the proxy with `Mint.HTTP2` and + open one `CONNECT` stream per tunnel (see the "CONNECT requests" section in + `Mint.HTTP2.request/5`): + + {:ok, conn} = Mint.HTTP2.connect(:https, proxy_host, proxy_port) + {:ok, conn, ref} = Mint.HTTP2.request(conn, "CONNECT", "example.com:443", [], :stream) + + Once the proxy replies with a `{:status, ref, 200}` response, the stream is a byte + tunnel: send bytes to the target with `Mint.HTTP2.stream_request_body/3` and receive + bytes from the target as `{:data, ref, data}` responses. + + To run TLS (or a whole Mint connection) *inside* such a tunnel, you need a relay: + `:ssl` can't run over an arbitrary byte stream, so a process has to own the proxy + connection and relay bytes between the `CONNECT` stream and a local TCP socket. The + inner connection then dials the relay while keeping the identity of the real host + through the `:hostname` option: + + Mint.HTTP.connect(:https, relay_address, relay_port, hostname: "example.com") + + so that SNI and certificate verification target the real host. Note that the default + request `Host` header (HTTP/1) and `:authority` pseudo-header (HTTP/2) contain the + port of the connection, which is the relay's port: either make the relay listen on + the target's port, or (on HTTP/1) pass an explicit `host` header with each request. + This is the natural shape for pooling libraries, which can multiplex many tunnels + over one proxy connection. + ## Transport options The options specified in `:transport_opts` are passed to the module that @@ -611,6 +647,12 @@ defmodule Mint.HTTP do `content-length` header yourself. If you're using HTTP/1, Mint will do chunked transfer-encoding when a content-length is not provided (see `Mint.HTTP1.request/5`). + ## CONNECT requests + + For requests with the `CONNECT` method, `path` is the target of the tunnel in + *authority form* (`"host:port"`), not a path. See `Mint.HTTP2.request/5` for HTTP/2 + tunnel semantics and for the extended CONNECT protocol, where `path` doesn't apply. + ## Examples Mint.HTTP.request(conn, "GET", "/", _headers = [], _body = nil) diff --git a/lib/mint/http2.ex b/lib/mint/http2.ex index 51788816..21527787 100644 --- a/lib/mint/http2.ex +++ b/lib/mint/http2.ex @@ -519,6 +519,40 @@ defmodule Mint.HTTP2 do body by initially passing `:stream` as the body and sending the body in chunks using `stream_request_body/3` and using `get_window_size/2` to get the window size of the request and connection. + + ## CONNECT requests + + You can open a tunnel to a target host through the server with the `CONNECT` + method ([RFC 9113, section 8.5](https://www.rfc-editor.org/rfc/rfc9113.html#section-8.5)). + For `CONNECT` requests (other than extended CONNECT, see below), `path` is not a + path: it's the host and port of the tunnel target, such as `"example.com:443"` + (IPv6 addresses must be enclosed in square brackets). Mint sends it as the + `:authority` pseudo-header and omits the `:scheme` and `:path` pseudo-headers. + *Available since v1.10.0*. + + Pass `:stream` as the body. Once the server replies with a 2xx status, the stream + becomes a tunnel: iodata you pass to `stream_request_body/3` is delivered to the + tunnel target, and bytes coming from the target are returned by `stream/2` as + `{:data, request_ref, data}` responses. The tunnel counts towards the limit of + concurrent requests for as long as it's open. + + Mint doesn't support the TCP-style half-close that RFC 9113 describes for tunnels: + when the server ends its side of the stream, Mint considers the whole tunnel closed, + returns `{:done, request_ref}`, and doesn't let you send more data for that request. + If instead you end your side first (by passing `:eof` to `stream_request_body/3`), + you keep receiving data until the server ends the stream. + + ### Extended CONNECT + + Mint also supports the extended CONNECT protocol + ([RFC 8441](https://www.rfc-editor.org/rfc/rfc8441.html)), which is used to bootstrap + protocols such as WebSocket over HTTP/2. If you pass a `:protocol` pseudo-header in + `headers`, the request is treated as an extended CONNECT request: you're expected to + also pass the `:scheme` and `:path` pseudo-headers explicitly (the `path` argument is + not used), and Mint sets `:authority` to the connection's authority, as in a normal + request. Note that servers only allow extended CONNECT after advertising the + `:enable_connect_protocol` setting (see `get_server_setting/2`); Mint doesn't + enforce this, so it's up to you to check the setting first. """ @impl true @spec request( @@ -1606,9 +1640,21 @@ defmodule Mint.HTTP2 do defp add_pseudo_headers(headers, conn, method, path) do if same_method?(method, "CONNECT") do + # In extended CONNECT (RFC 8441) the caller passes :scheme, :path, and + # :protocol explicitly and :authority is the origin, as in a normal + # request. In plain CONNECT (RFC 9113, section 8.5) the request target + # is the host and port of the tunnel destination, which request/5 + # receives as its "path" argument. + authority = + if List.keymember?(headers, ":protocol", 0) do + conn.authority + else + path + end + [ {":method", method}, - {":authority", conn.authority} + {":authority", authority} | headers ] else diff --git a/test/mint/http2/conn_test.exs b/test/mint/http2/conn_test.exs index e7f85054..b323be46 100644 --- a/test/mint/http2/conn_test.exs +++ b/test/mint/http2/conn_test.exs @@ -1151,18 +1151,18 @@ defmodule Mint.HTTP2Test do assert cookie == "a=b; c=d; e=f; g=h" end - test "a CONNECT request omits :scheme and :path pseudo-headers", %{conn: conn} do - assert {:ok, conn, _ref} = HTTP2.request(conn, "CONNECT", "/", [], nil) + test "a CONNECT request uses path as :authority and omits :scheme and :path", %{conn: conn} do + assert {:ok, conn, _ref} = HTTP2.request(conn, "CONNECT", "example.com:443", [], :stream) - assert_recv_frames [headers(hbf: hbf)] + assert_recv_frames [headers(hbf: hbf) = frame] - refute hbf - |> server_decode_headers() - |> List.keymember?(":scheme", 0) + assert [ + {":method", "CONNECT"}, + {":authority", "example.com:443"}, + {"user-agent", _} + ] = server_decode_headers(hbf) - refute hbf - |> server_decode_headers() - |> List.keymember?(":path", 0) + refute flag_set?(headers(frame, :flags), :headers, :end_stream) assert HTTP2.open?(conn) end @@ -1176,13 +1176,15 @@ defmodule Mint.HTTP2Test do {":protocol", "websocket"} ] - assert {:ok, conn, _ref} = HTTP2.request(conn, "CONNECT", "/", headers, :stream) + assert {:ok, conn, _ref} = HTTP2.request(conn, "CONNECT", "/ws", headers, :stream) assert_recv_frames [headers(hbf: hbf)] + connection_authority = conn.authority + assert [ {":method", "CONNECT"}, - {":authority", _}, + {":authority", ^connection_authority}, {":scheme", _}, {":path", "/ws"}, {":protocol", "websocket"}, @@ -2408,6 +2410,172 @@ defmodule Mint.HTTP2Test do end end + describe "CONNECT tunnels" do + test "tunnels bytes in both directions", %{conn: conn} do + assert {:ok, conn, ref} = HTTP2.request(conn, "CONNECT", "example.com:80", [], :stream) + + assert_recv_frames [headers(stream_id: stream_id)] + + hbf = server_encode_headers([{":status", "200"}]) + + assert {:ok, %HTTP2{} = conn, [{:status, ^ref, 200}, {:headers, ^ref, []}]} = + stream_frames(conn, [ + headers( + stream_id: stream_id, + hbf: hbf, + flags: set_flags(:headers, [:end_headers]) + ) + ]) + + assert {:ok, conn} = HTTP2.stream_request_body(conn, ref, "hello") + assert_recv_frames [data(stream_id: ^stream_id, data: "hello") = data_frame] + refute flag_set?(data(data_frame, :flags), :data, :end_stream) + + assert {:ok, %HTTP2{} = conn, [{:data, ^ref, "world"}]} = + stream_frames(conn, [ + data(stream_id: stream_id, data: "world", flags: set_flags(:data, [])) + ]) + + assert {:ok, conn} = HTTP2.stream_request_body(conn, ref, "again") + assert_recv_frames [data(stream_id: ^stream_id, data: "again")] + + assert HTTP2.open_request_count(conn) == 1 + assert HTTP2.open?(conn) + end + + test "content-length: 0 on the response does not end the tunnel", %{conn: conn} do + assert {:ok, conn, ref} = HTTP2.request(conn, "CONNECT", "example.com:443", [], :stream) + + assert_recv_frames [headers(stream_id: stream_id)] + + hbf = server_encode_headers([{":status", "200"}, {"content-length", "0"}]) + + assert {:ok, %HTTP2{} = conn, responses} = + stream_frames(conn, [ + headers( + stream_id: stream_id, + hbf: hbf, + flags: set_flags(:headers, [:end_headers]) + ), + data(stream_id: stream_id, data: "tunneled", flags: set_flags(:data, [])) + ]) + + assert responses == [ + {:status, ref, 200}, + {:headers, ref, [{"content-length", "0"}]}, + {:data, ref, "tunneled"} + ] + + assert {:ok, conn} = HTTP2.stream_request_body(conn, ref, "out") + assert_recv_frames [data(stream_id: ^stream_id, data: "out")] + + assert HTTP2.open_request_count(conn) == 1 + assert HTTP2.open?(conn) + end + + test "an END_STREAM from the server closes the whole tunnel", %{conn: conn} do + assert {:ok, conn, ref} = HTTP2.request(conn, "CONNECT", "example.com:443", [], :stream) + + assert_recv_frames [headers(stream_id: stream_id)] + + hbf = server_encode_headers([{":status", "200"}]) + + assert {:ok, %HTTP2{} = conn, responses} = + stream_frames(conn, [ + headers( + stream_id: stream_id, + hbf: hbf, + flags: set_flags(:headers, [:end_headers]) + ), + data(stream_id: stream_id, data: "bye", flags: set_flags(:data, [:end_stream])) + ]) + + assert responses == [ + {:status, ref, 200}, + {:headers, ref, []}, + {:data, ref, "bye"}, + {:done, ref} + ] + + assert_recv_frames [rst_stream(stream_id: ^stream_id, error_code: :no_error)] + + assert {:error, %HTTP2{} = conn, error} = HTTP2.stream_request_body(conn, ref, "x") + assert_http2_error error, :unknown_request_to_stream + + assert HTTP2.open_request_count(conn) == 0 + assert HTTP2.open?(conn) + end + + test "an :eof from the client half-closes the tunnel and data can still be received", + %{conn: conn} do + assert {:ok, conn, ref} = HTTP2.request(conn, "CONNECT", "example.com:443", [], :stream) + + assert_recv_frames [headers(stream_id: stream_id)] + + hbf = server_encode_headers([{":status", "200"}]) + + assert {:ok, %HTTP2{} = conn, [{:status, ^ref, 200}, {:headers, ^ref, []}]} = + stream_frames(conn, [ + headers( + stream_id: stream_id, + hbf: hbf, + flags: set_flags(:headers, [:end_headers]) + ) + ]) + + assert {:ok, conn} = HTTP2.stream_request_body(conn, ref, "last") + assert {:ok, conn} = HTTP2.stream_request_body(conn, ref, :eof) + + assert_recv_frames [ + data(stream_id: ^stream_id, data: "last") = data1, + data(stream_id: ^stream_id, data: "") = data2 + ] + + refute flag_set?(data(data1, :flags), :data, :end_stream) + assert flag_set?(data(data2, :flags), :data, :end_stream) + + assert {:ok, %HTTP2{} = conn, [{:data, ^ref, "still coming"}]} = + stream_frames(conn, [ + data(stream_id: stream_id, data: "still coming", flags: set_flags(:data, [])) + ]) + + assert {:ok, %HTTP2{} = conn, [{:data, ^ref, ""}, {:done, ^ref}]} = + stream_frames(conn, [ + data(stream_id: stream_id, data: "", flags: set_flags(:data, [:end_stream])) + ]) + + assert_recv_frames [] + + assert HTTP2.open_request_count(conn) == 0 + assert HTTP2.open?(conn) + end + + test "a CONNECT request with a nil body opens a receive-only tunnel", %{conn: conn} do + assert {:ok, conn, ref} = HTTP2.request(conn, "CONNECT", "example.com:443", [], nil) + + assert_recv_frames [headers(stream_id: stream_id) = headers_frame] + assert flag_set?(headers(headers_frame, :flags), :headers, :end_stream) + + hbf = server_encode_headers([{":status", "200"}]) + + assert {:ok, %HTTP2{} = conn, + [{:status, ^ref, 200}, {:headers, ^ref, []}, {:data, ^ref, "in"}]} = + stream_frames(conn, [ + headers( + stream_id: stream_id, + hbf: hbf, + flags: set_flags(:headers, [:end_headers]) + ), + data(stream_id: stream_id, data: "in", flags: set_flags(:data, [])) + ]) + + assert {:error, %HTTP2{} = conn, error} = HTTP2.stream_request_body(conn, ref, "x") + assert_http2_error error, :request_is_not_streaming + + assert HTTP2.open?(conn) + end + end + describe "stream_request_body/3" do test "streaming a request", %{conn: conn} do {conn, ref} = open_request(conn, :stream)