Skip to content

fix: WebSocket reconnect semantics - #3

Closed
chipkent wants to merge 9 commits into
masterfrom
fix/ws-reconnect
Closed

chipkent wants to merge 9 commits into
masterfrom
fix/ws-reconnect

Conversation

@chipkent

@chipkent chipkent commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

Preview only. This is a draft in the fork showing the exact text of the PR planned for massive-com/client-python. It will be closed, not merged, once the upstream PR is opened.

Stacked on the WebSocket subscriptions PR (F1), which is itself stacked on the free-threaded CI PR (F7, for the fake test server). Please review those first; this PR only adds the five commits below.

Summary

Five changes to how WebSocketClient.connect() handles disconnects, one commit each. The last
commit is separable.

  1. Reconnect after a clean server close.
    • Problem: connect() returned on ConnectionClosedOK. The server closes cleanly after
      statuses such as slow_consumer or max_connections, so the client ended for good.
    • Fix: it reconnects unless close() or request_close() was called. The desired
      subscriptions are kept across the reconnect. If connect() gives up (see 3), it re-raises
      the error that ended the last connection.
  2. Backoff, and retrying only what can succeed.
    • Problem: reconnects were immediate. Massive's documentation asks clients to wait at least
      15 seconds before reconnecting
      (knowledge base).
      The websockets iterator also retried every error, including ones that can never succeed,
      such as HTTP 401/403/404 or an unverifiable certificate.
    • Fix: connect() makes each attempt itself, with a new websockets connection object per
      attempt; one is built up front so that invalid arguments raise immediately. Only transient
      failures are retried: OSError (including DNS and TLS errors other than certificate
      verification), the connection closing before a response (as reported by websockets 14 and
      15), and HTTP 408, 429 and 5xx. Each retry logs a WARNING with the delay. The wait after the
      n-th consecutive failure is uniform(b, min(1.2·b, max)) with
      b = min(reconnect_backoff·2ⁿ, max_reconnect_backoff), so every wait lies within the
      configured bounds. process_exception, which only the iterator used, is rejected with
      TypeError.
    • New parameters: reconnect_backoff=15.0 and max_reconnect_backoff=120.0.
  3. max_reconnects counts consecutive failures.
    • Problem: the limit was a lifetime total, so with the default of 5 a long-running client
      raised on its sixth outage, even if every reconnect had succeeded.
    • Fix: it counts consecutive failures and resets after a successful authentication. The
      default becomes None, retrying forever. Because non-transient errors raise at once (2),
      run() is safe for long-lived processes.
  4. request_close().
    • Problem: close() is a coroutine and is not thread-safe.
    • Fix: request_close() can be called from any thread. It stops connect() promptly,
      including while it is connecting or waiting to reconnect, and no further attempt starts.
      Cancelling connect() also closes a connection opened in the meantime.
  5. Surface max_connections received during the handshake (separable).
    • Problem: the status was only logged.
    • Fix: max_connections_reached is set, and in raw mode the frame is passed to the
      processor, so callers can tell a full connection slot from a network drop.

Breaking changes

Commits 1–3 are marked fix!: with BREAKING CHANGE: footers:

  • Clean close: connect() no longer returns when the server closes the connection cleanly;
    it reconnects until close()/request_close() is called or max_reconnects is exceeded.
  • Backoff: reconnects wait at least 15 s by default instead of happening immediately. Errors
    that cannot pass (HTTP 4xx other than 408/429, certificate failures, invalid arguments) are
    raised at once instead of being retried forever.
  • max_reconnects: defaults to None (retry forever) instead of 5, and counts consecutive
    failed attempts rather than all reconnects.

Test plan

  • New test_websocket/test_reconnect.py, covering:
    • subscription changes made during an outage;
    • a clean close, then reconnect;
    • backoff bounds (pure tests of the delay function with random.uniform patched, plus measured
      waits against the fake server with tolerances for Windows timer resolution);
    • transient and fatal classification, including the websockets 14 and 15 forms;
    • consecutive versus lifetime counting;
    • request_close() from another thread, while waiting to reconnect, during a stuck handshake,
      before connect(), and as a handshake completes (no further attempt);
    • cancellation cleanup;
    • handshake max_connections.
  • make test passes on 3.10 to 3.14 (ubuntu and windows), and on 3.13t and 3.14t with
    PYTHON_GIL=0; the suite also passes with websockets 14.0.
  • make lint passes.

chipkent and others added 2 commits October 23, 2026 15:21
FakeServer listens on a free port and records every client message per
connection. It speaks the Massive protocol (connected, auth_success or
auth_failed, per-topic success acks) and can be scripted: replace the auth
reply, reject topics or handshakes, send status and data messages, and
close or drop connections with a chosen code. Tests cover connect, auth,
subscribe, reconnect, and independent clients in parallel threads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
16 threads make 200 requests each through one client against a local
HTTP server and check every response matches its request and per-request
options don't leak into the client's default headers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 23, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@chipkent
chipkent force-pushed the fix/ws-reconnect branch 3 times, most recently from d35a158 to 639baa1 Compare October 24, 2026 02:39
@chipkent chipkent changed the title [CI only] fix/ws-reconnect fix: WebSocket reconnect semantics Oct 24, 2026
chipkent and others added 2 commits October 23, 2026 20:55
Replace the deprecated [tool.poetry.dev-dependencies] section with a
`test` group (pook, all the unit tests need) and a `dev` group (linters,
type stubs, Sphinx, orjson for the examples). `poetry install` still
installs both, so local and CI workflows are unchanged, while
`poetry install --only main,test` installs just what `make test` needs.

Versions are unchanged. Re-locking also merged a stale duplicate entry
for typing-extensions (4.12.2 below Python 3.15, 4.16.0 from 3.15) into
4.16.0, a transitive dependency of the dev tools.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a job that runs `make test` on 3.13t and 3.14t with PYTHON_GIL=0, and
extend the regular matrix to 3.12, 3.13 and 3.14. The free-threaded job
installs only the main and test dependency groups: the dev group is not
needed to run the tests, and orjson in it does not support free-threaded
Python.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chipkent
chipkent force-pushed the fix/ws-reconnect branch 2 times, most recently from 67febb0 to f5d4b4b Compare October 24, 2026 03:16

@chipkent chipkent left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assessment: not ready to merge — one likely blocker and a few medium-risk edge concerns.

PR summary

  • The design direction is strong: reconnect on clean close, preserve desired subscriptions, transient/fatal retry split, thread-safe shutdown path, and substantial test expansion.
  • Test coverage is materially improved (especially reconnect and fake-server behavior).
  • Main risk is in reconnect-loop mechanics and close/cancel races.

Blocking issue

  • _open() likely reuses a single-shot awaitable across reconnect attempts.
    In connect(), connector = connect(...) is created once and reused in the loop via _open(connector). _open() wraps it with asyncio.ensure_future(connector). If connector is coroutine-like/single-use, later attempts can fail or become undefined.
    Why this is blocking: reconnect reliability depends on this path; failure here breaks core semantics in production outages.
    What to change: create a fresh connect awaitable each attempt (factory per loop iteration), rather than storing one reusable awaitable.

Other adversarial findings (non-blocking but important)

  • Class-level mutable _late_closes can cross-contaminate instances.
    Make it instance state (self._late_closes) to avoid shared global task bookkeeping across multiple clients/loops.

  • Exception taxonomy may be brittle across websockets versions.
    _is_transient() treats InvalidMessage as transient only when __cause__ is EOFError. Equivalent transient handshake failures can surface differently by version/platform. Consider broadening classification or pinning/version-gating behavior with tests.

  • close()/request_close() contract needs one explicit invariant in docs/tests:
    “After close request, no further reconnect attempts occur, even if handshake completes concurrently.”
    You test many races already; add a direct assertion on attempt-count stability after close request.

Design correctness check

  • Subscription persistence logic: looks correct (scheduled_subs preserved, subs reset).
  • Reconnect after clean close: correctly implemented.
  • Backoff bounds and defaults: coherent and well-tested.
  • max_reconnects as consecutive failures: semantics are improved and match long-lived process expectations.
  • max_connections surfacing: good improvement and tested.

Conventions / project fit

  • Overall style and test structure match repo conventions (unittest + isolated async tests).
  • Docs updated for request_close(): good.
  • Minor cleanup nit: duplicated import logging in websocket module area.

Merge readiness

  • Status: Changes requested
  • Risk: Medium-high until reconnect awaitable lifecycle is fixed
  • Decision: Fix blocker, rerun websocket suite + matrix, then re-review

connect() returned on ConnectionClosedOK, so a clean close from the
server, which it sends e.g. after a slow_consumer or max_connections
status, ended the client for good. Treat it like any other disconnect
and reconnect, unless close() was called.

On any disconnect, keep scheduled_subs, the desired set: it used to be
replaced with the topics sent so far, so a subscribe() or unsubscribe()
made just before the drop was lost. Reset only subs, so the next
connection subscribes exactly the desired set.

When connect() gives up after max_reconnects, it raises the error that
ended the last connection, ConnectionClosedOK included. The status test
that relied on connect() returning now sets max_reconnects=0.

BREAKING CHANGE: connect() no longer returns when the server closes the
connection cleanly; it reconnects until close() is called or
max_reconnects is exceeded.
Reconnects were immediate after a disconnect: the websockets connect()
iterator resets its delay after every successful connection, and its own
delay for refused handshakes starts below 5 s. Massive asks clients to
wait at least 15 s. The iterator also retried every error, including
ones that cannot pass, such as HTTP 401 or an unverifiable certificate.

connect() now makes each attempt itself, with a new websockets
connection object per attempt; one is built up front so that invalid
arguments still raise at once. Only transient failures are retried:
OSError (refused, unreachable, timed out, a DNS failure, or a TLS error
other than certificate verification), the connection closing before a
response (EOFError in websockets 14, InvalidMessage caused by EOFError
or OSError in websockets 15), and HTTP 408, 429 and 5xx. Anything else
is raised at once. A process_exception argument, which only the
iterator used, is rejected with TypeError.

After the n-th consecutive failure the client logs a warning and waits
uniform(b, min(1.2 * b, max_reconnect_backoff)) seconds, where
b = min(reconnect_backoff * 2**n, max_reconnect_backoff). Jitter only
lengthens a wait and every wait lies in
[reconnect_backoff, max_reconnect_backoff]. The count resets once a
connection authenticates. close() during the wait returns at once,
without the "no websocket open to close" warning.

New constructor parameters: reconnect_backoff (default 15.0) and
max_reconnect_backoff (default 120.0).

BREAKING CHANGE: reconnects wait at least 15 s by default (they used to
be immediate); pass reconnect_backoff to change it. Errors that cannot
pass, e.g. HTTP 4xx other than 408 and 429, are raised instead of
retried.
max_reconnects limited the reconnects over the client's lifetime, so with
the default of 5 a long-running client raised on its sixth outage even if
every reconnect had succeeded. Count consecutive failed attempts instead
(refused connections included) and start over whenever a connection
authenticates.

The default becomes None, retrying forever. Only transient failures are
retried, with a growing wait, and other errors are raised at once, so
run() is safe in a long-lived process.

BREAKING CHANGE: max_reconnects now defaults to None (retry forever)
instead of 5, and counts consecutive failed attempts rather than all
reconnects.
close() is a coroutine that uses the websocket from whatever thread calls
it, and while connect() was between connections it only logged "no
websocket open to close". request_close() can be called from any thread:
it sets the close flag and, through loop.call_soon_threadsafe, wakes
connect() and closes the socket, so connect() returns while connected,
while connecting, or while waiting to reconnect. close() is kept.

A request made while no connect() runs stays pending until a connect()
returns because of it, and a stale wake-up queued for an earlier
connect() is ignored.

If connect() is cancelled while an attempt is in flight, the attempt is
cancelled and awaited (websockets bounds it by open_timeout), and a
connection it opened meanwhile is closed.
When the account has too many connections open, the server answers auth
with a max_connections status and then closes the connection. The client
only logged it, so the caller could not tell it from a network drop.

Now connect() sets max_connections_reached until a later handshake
succeeds and, in raw mode, passes the message to the processor, as it
does for statuses after the handshake. Such a handshake does not reset
the failure count, so the backoff keeps growing while the limit is hit.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Initial handshake rejection handling and overlapping shutdown cancellation still have correctness gaps.

2 open issues
What changed in this PR

This PR revises the Massive Python client’s WebSocket reconnect behavior for long-running connections, with supporting tests, documentation, and CI updates.

Changes:

  • Reconnects after clean closes, adds bounded backoff, and counts consecutive failures.
  • Adds thread-safe request_close() and exposes handshake connection-limit statuses.
  • Expands local-server tests and Python coverage, including free-threaded builds.
File Description
test_websocket/​test_status.py Updates clean-close expectations.
test_websocket/​test_reconnect.py Tests retries, backoff, shutdown, and handshake statuses.
test_websocket/​test_fake_server.py Tests client behavior against the fake server.
test_websocket/​fake_server.py Adds a scriptable local WebSocket server.
test_rest/​test_concurrency.py Tests a REST client shared across threads.
pyproject.toml Separates test and development dependencies.
poetry.lock Synchronizes dependency groups and resolution.
massive/​websocket/​__init__.py Implements reconnect and shutdown semantics.
docs/​source/​WebSocket.rst Documents request_close().
.github/​workflows/​test.yml Expands Python and free-threaded test coverage.

Review effort: Balanced

Comment on lines +242 to +246
for frame, parsed in ((msg, msg_parsed), (auth_msg, auth_msg_parsed)):
if any(m.get("status") == "max_connections" for m in parsed):
self.max_connections_reached = True
if self.raw:
await processor(frame) # type: ignore
closing.cancel()
if not stop:
return attempt.result()
await self._abandon(attempt)
@chipkent

Copy link
Copy Markdown
Owner Author

Superseded by smaller PRs split by content: #19, #23, #24. Closing this preview; the replacements carry the same changes, opt-in where behaviour changes.

@chipkent chipkent closed this Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants