BUG: RuntimeError: deque mutated during iteration

The starting point for issues should usually be a discussion...

https://github.com/encode/httpx/discussions/

Possible bugs may be raised as a "Potential Issue" discussion, feature requests may be raised as an "Ideas" discussion. We can then determine if the discussion needs to be escalated into an "Issue" or not.

This will help us ensure that the "Issues" list properly reflects ongoing or needed work on the project.

Initially raised as discussion RuntimeError: deque mutated during iteration (hpack) #3279, but no response/action taken

Exception raised by this bug; it's already been discussed at python-hyper/hpack#275 and appears from time to time in a function that is decorated by backoff 2.2.1 (https://pypi.org/project/backoff/):

Backing off get_with_retry(...) for 0.1s (httpx.RemoteProtocolError: <ConnectionTerminated error_code:9, last_stream_id:15, additional_data:None>)
Backing off get_with_retry(...) for 0.4s (httpx.RemoteProtocolError: <ConnectionTerminated error_code:9, last_stream_id:15, additional_data:None>)
Traceback (most recent call last):
  File "/host/usr/local/bin/watch.py", line 567, in <module>
    main()
  File "/host/usr/local/bin/watch.py", line 396, in main
    for _ in executor.map(download, secs, repeat(c), repeat(ret_periods), repeat(display_price)):
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 619, in result_iterator
    yield _result_or_cancel(fs.pop())
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 317, in _result_or_cancel
    return fut.result(timeout)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/host/usr/local/bin/watch.py", line 145, in download
    resp = get_with_retry(url, c, timeout=5)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/backoff/_sync.py", line 48, in retry
    ret = target(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/host/usr/local/bin/mb_httpx.py", line 104, in get_with_retry
    return c.get(
           ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/httpx/_client.py", line 1054, in get
    return self.request(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/httpx/_client.py", line 827, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/httpx/_client.py", line 1015, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/httpx/_transports/default.py", line 233, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request
    raise exc from None
  File "/usr/local/lib/python3.12/dist-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
    response = connection.handle_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/httpcore/_sync/connection.py", line 101, in handle_request
    return self._connection.handle_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/httpcore/_sync/http2.py", line 185, in handle_request
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/httpcore/_sync/http2.py", line 142, in handle_request
    self._send_request_headers(request=request, stream_id=stream_id)
  File "/usr/local/lib/python3.12/dist-packages/httpcore/_sync/http2.py", line 247, in _send_request_headers
    self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
  File "/usr/local/lib/python3.12/dist-packages/h2/connection.py", line 770, in send_headers
    frames = stream.send_headers(
             ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/h2/stream.py", line 867, in send_headers
    frames = self._build_headers_frames(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/h2/stream.py", line 1254, in _build_headers_frames
    encoded_headers = encoder.encode(headers)
                      ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/hpack/hpack.py", line 255, in encode
    header_block.append(self.add(header, sensitive, huffman))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/hpack/hpack.py", line 280, in add
    match = self.header_table.search(name, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/hpack/table.py", line 184, in search
    for (i, (n, v)) in enumerate(self.dynamic_entries):
RuntimeError: deque mutated during iteration

Again, also discussed at python-hyper/hpack#281

    RuntimeError: deque mutated during iteration
    
    Traceback (most recent call last):
     File "/workspace/main.py", line 197, in get_status
        return get_with_retry_5s(url, c)
                                  ~~~~~~~~~~~~^^^^^^^^^
      File "/workspace/main.py", line 616, in uaflifo_main
        inb_status, _, _ = get_flifo_data(new_args, c=c)
                           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
      File "/workspace/main.py", line 223, in get_flifo_data
        fs_response = get_flight_status(args, c)
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/backoff/_sync.py", line 105, in retry
        ret = target(*args, **kwargs)
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/backoff/_sync.py", line 48, in retry
        ret = target(*args, **kwargs)
      File "/workspace/mb_httpx.py", line 213, in get_with_retry_5s
        return c.get(
               ~~~~~^
            url,
            ^^^^
        ...<6 lines>...
            extensions=extensions,
            ^^^^^^^^^^^^^^^^^^^^^^
        )
        ^
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpx/_client.py", line 1053, in get
        return self.request(
               ~~~~~~~~~~~~^
            "GET",
            ^^^^^^
        ...<7 lines>...
            extensions=extensions,
            ^^^^^^^^^^^^^^^^^^^^^^
        )
        ^
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpx/_client.py", line 825, in request
        return self.send(request, auth=auth, follow_redirects=follow_redirects)
               ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpx/_client.py", line 914, in send
        response = self._send_handling_auth(
            request,
        ...<2 lines>...
            history=[],
        )
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpx/_client.py", line 942, in _send_handling_auth
        response = self._send_handling_redirects(
            request,
            follow_redirects=follow_redirects,
            history=history,
        )
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
        response = self._send_single_request(request)
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpx/_client.py", line 1014, in _send_single_request
        response = transport.handle_request(request)
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpx/_transports/default.py", line 250, in handle_request
        resp = self._pool.handle_request(req)
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
        raise exc from None
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
        response = connection.handle_request(
            pool_request.request
        )
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpcore/_sync/connection.py", line 103, in handle_request
        return self._connection.handle_request(request)
               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpcore/_sync/http2.py", line 187, in handle_request
        raise exc
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpcore/_sync/http2.py", line 144, in handle_request
        self._send_request_headers(request=request, stream_id=stream_id)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpcore/_sync/http2.py", line 249, in _send_request_headers
        self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/h2/connection.py", line 806, in send_headers
        frames.extend(stream.send_headers(
                      ~~~~~~~~~~~~~~~~~~~^
            headers, self.encoder, end_stream,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        ))
        ^
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/h2/stream.py", line 894, in send_headers
        frames = self._build_headers_frames(
            bytes_headers, encoder, hf, hdr_validation_flags,
        )
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/h2/stream.py", line 1298, in _build_headers_frames
        encoded_headers = encoder.encode(headers)
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/hpack/hpack.py", line 276, in encode
        header_block.append(self.add(new_header, sensitive, huffman))
                            ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/hpack/hpack.py", line 301, in add
        match = self.header_table.search(name, value)
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/hpack/table.py", line 186, in search
        for (i, (n, v)) in enumerate(self.dynamic_entries):
                           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^

The code (in mb_httpx.py) that (rarely) causes httpx to crash is the following:

@backoff.on_exception(
    wait_gen=backoff.expo,
    exception=httpx.HTTPError,
    max_time=5,
    on_backoff=log_backoff,
)
@backoff.on_predicate(
    wait_gen=backoff.expo,
    predicate=lambda x: x.status_code in {408, 429, 500, 502, 503, 504},
    max_time=5,
    on_backoff=log_backoff,
)
def get_with_retry_5s(
    url: httpx._types.URLTypes,
    c: httpx.Client = default_client,
    *,
    params: httpx._types.QueryParamTypes | None = None,
    headers: httpx._types.HeaderTypes | None = None,
    cookies: httpx._types.CookieTypes | None = None,
    auth: httpx._types.AuthTypes | httpx._client.UseClientDefault = httpx._client.USE_CLIENT_DEFAULT,
    follow_redirects: bool | httpx._client.UseClientDefault = httpx._client.USE_CLIENT_DEFAULT,
    timeout: httpx._types.TimeoutTypes | httpx._client.UseClientDefault = httpx._client.USE_CLIENT_DEFAULT,
    extensions: httpx._types.RequestExtensions | None = None,
) -> httpx.Response:
    """
    Get url, retrying for up to 5 seconds with truncated exponential backoff if an HTTPError is encountered or
    any of the following response status codes is returned:

    * 408 Request Timeout
    * 429 Too Many Requests
    * 500 Internal Server Error
    * 502 Bad Gateway
    * 503 Service Unavailable
    * 504 Gateway Timeout
    """
    timeout = timeout or 5
    return c.get(
        url,
        params=params,
        headers=headers,
        cookies=cookies,
        auth=auth,
        follow_redirects=follow_redirects,
        timeout=timeout,
        extensions=extensions,
    )

From the logs, coincidental to the Exception is the recording of a httpx.ReadError: [Errno 11] Resource temporarily unavailable) by log_backoff (the logger from encountering a backoff condition), but I cannot determine whether this takes place before the Exception or is simply a byproduct of it.

Root cause found and fixed on a tested branch (I was unable to open a pull request from this account — GitHub returned "Shanmukh-l33t does not have the correct permissions to execute CreatePullRequest" — so please treat the compare link below as the proposed change).

Root cause

When a synchronous Client(http2=True) is shared between threads, httpcore's connection pool multiplexes concurrent requests over a single HTTP2Connection. That connection's h2/hpack state — including the HPACK encoder's dynamic table (collections.deque in hpack/table.py) — is not thread-safe, and HTTP2Connection._send_request_headers calls self._h2_state.send_headers(...) without any synchronization. Two threads encoding headers on the same connection interleave HeaderTable.search()'s iteration over dynamic_entries with HeaderTable.add()'s appendleft()/evictions, giving exactly RuntimeError: deque mutated during iteration. The same shared-state race also produces the companion symptoms seen in #3190 / #3002 (KeyError, LocalProtocolError: Invalid input ... in state ..., corrupted HPACK blocks, spurious RemoteProtocolError/timeouts).

HTTP/1.1 connections already avoid this: HTTP11Connection.handle_request raises ConnectionNotAvailable for concurrent requests, so the pool assigns each in-flight request its own connection. HTTP/2 connections instead accept concurrent requests (for stream multiplexing) with no locking. The hpack maintainers confirmed synchronization is the consumer's responsibility (python-hyper/hpack#275); the async client is unaffected because tasks on one event loop only interleave at await points and the h2 state is only touched by synchronous code in between.

Proposed fix

Compare: master...Shanmukh-l33t:httpx:fix-3633-h2-thread-safety
Branch: fix-3633-h2-thread-safety on https://github.com/Shanmukh-l33t/httpx (commit 7fa2707f4633492b8235b45db83001b04e7b5406)

The sync transport wraps every pooled connection in a _SingleFlightConnection guard that admits at most one in-flight request (send → response close) and raises httpcore.ConnectionNotAvailable for concurrent attempts, so the pool assigns those requests separate connections (its existing reassignment loop already handles this). This restores the HTTP/1.1 invariant: shared HTTP/2 state is only ever touched by one thread. The trade-off is that concurrently in-flight sync requests use separate connections (bounded by Limits.max_connections) instead of multiplexing onto one — per-thread parallelism is unaffected. This also complements encode/httpcore#1118 (serializing the HTTP/2 send path in httpcore), which is the deeper fix; once that ships in a released httpcore this guard can be revisited.

Validation

Regression test (tests/test_thread_safety.py): a minimal prior-knowledge HTTP/2 test server + 16 threads sharing one sync client (2,400 barrier-synchronized requests, tightened GIL switch interval). On the unmodified parent commit it fails 3/3 runs — first collected error: RuntimeError('deque mutated during iteration'), plus hpack.exceptions.InvalidTableIndex / h2.exceptions.ProtocolError wire-corruption artifacts. With the fix: passes 3/3.
Unit tests cover the guard's exclusive-use contract and release-on-error paths.
Full suite (Python 3.12): 1419 passed, 1 skipped; tests/test_timeouts.py::test_write_timeout[trio] fails identically on the unmodified parent commit in my environment (pre-existing trio ResourceWarning flake), unrelated to this change.
scripts/check (ruff format, mypy strict, ruff check) passes; coverage 100% (--fail-under=100).
Docs: short thread-safety note added to docs/http2.md.

Happy to reshape the approach or split commits if maintainers prefer a different fix location (e.g. waiting on the httpcore-side serialization and adding only the regression test here).

Navigation Menu

Repository navigation

BUG: RuntimeError: deque mutated during iteration #3633

Root cause

Proposed fix

Validation

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

BUG: RuntimeError: deque mutated during iteration #3633

Description

Activity

Shanmukh-l33t commented on Nov 11, 2026

Root cause

Proposed fix

Validation

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions