Feishu frontend cannot recover when the SDK event loop stalls while the process stays alive

[ytahml]

Problem

The Feishu frontend's existing retry loop only runs after cli.start() returns or raises. If synchronous work stalls the SDK event-loop thread, the process can remain alive while it stops processing incoming events. An exit-based process supervisor cannot recover it in that state.

This report concerns event-loop stalls, not every form of silent or half-open connection failure. One concrete trigger is synchronous endpoint discovery during startup or reconnection without an explicit HTTP timeout; see larksuite/oapi-sdk-python#169.

Reproduction

1. Run the Feishu frontend under a supervisor that restarts it after failure.
2. Stall the WebSocket event-loop thread during initial connection or reconnection.
3. Observe that the process remains alive, incoming events stop being handled, and the frontend's outer retry loop does not run.

Local HTTP/WebSocket checks using real lark-oapi 1.6.8 and 1.7.3 clients reproduce an endpoint-discovery stall after an initial event and a simulated disconnection. A fresh process can connect and receive events once the endpoint accepts new requests.

Expected behavior and proposed scope

A supervised frontend should expose a persistently stalled event loop as a process failure, allowing the supervisor to restart it. A small application-owned watchdog around cli.start() can monitor loop progress from an independent thread without requiring an SDK upgrade or duplicating endpoint-discovery code.

The proposed defaults are a 5-second heartbeat/check interval and a 180-second stall threshold. Existing reconnect backoff would stay unchanged. Normal idle time and asynchronous retry waits must not be treated as failures, and watchers must be cleaned up when start() returns or raises.

This recovery would interrupt in-flight tasks and require an external restart policy. It would not detect cases where the event loop remains responsive but the connection or business logic is unhealthy. Feedback on this limited frontend-level recovery approach is welcome.

[aron-intframe]

Reproduced this against real lark-oapi 1.6.8 and 1.7.3, and the stall has a single site: ws.Client._get_conn_url() is a synchronous requests.post with no timeout=, called from inside the _connect() coroutine, so it parks the event-loop thread rather than only that coroutine.

Startup phase, against a local endpoint that accepts the TCP connection and never answers. A heartbeat coroutine scheduled onto lark_oapi.ws.client.loop measures loop progress:

t+ 4.0s  start()_returned=False  raised=None  loop_ticks=0  loop_running=True  process_alive=True
t+12.0s  start()_returned=False  raised=None  loop_ticks=0  loop_running=True  process_alive=True

  File ".../lark_oapi/ws/client.py", line 162, in start
    loop.run_until_complete(self._connect())
  File ".../lark_oapi/ws/client.py", line 194, in _connect
    conn_url = self._get_conn_url()
  File ".../lark_oapi/ws/client.py", line 237, in _get_conn_url
    response = requests.post(
  File ".../urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
  File ".../urllib3/connectionpool.py", line 540, in _make_request
    response = conn.getresponse()
  File "/usr/lib/python3.12/socket.py", line 707, in readinto
    return self._sock.recv_into(b)

Reconnect phase, which is the case you describe: one event handled, connection dropped, reconnect discovery hangs.

[10:40:26] handler ran, events=1
[10:40:27] reconnecting -> endpoint discovery POST
t+ 2.0s events=1 loop_ticks=  1 start()_returned=False raised=None conn=None process_alive=True
t+12.0s events=1 loop_ticks=  1 start()_returned=False raised=None conn=None process_alive=True

So the structural claim holds. cli.start() stays inside loop.run_until_complete(_select()), fsapp.main()'s while True never runs again, and the process stays alive with loop_ticks frozen. The missing timeout is identical in 1.7.3 (current latest), so an SDK upgrade does not remove the need for an application-level watchdog:

lark-oapi 1.6.8 _get_conn_url has timeout= False
lark-oapi 1.7.3 _get_conn_url has timeout= False

I then ran a watchdog whose first rule is the same as #826's (loop.call_later beat, monotonic staleness check on a separate thread, os._exit) against the real SDK in both phases, with thresholds shortened to check=1s / stall=6s, plus two more scenarios:

[watchdog] event loop made no progress for 6s -> exit 1
startup-stall: watchdog_fired=True start()_still_parked=True start()_raised=None process_alive=True
[watchdog] event loop made no progress for 6s -> exit 1
reconnect-stall: watchdog_fired=True start()_still_parked=True start()_raised=None process_alive=True
[watchdog] reconnect started 6s ago and never completed -> exit 1
deaf: watchdog_fired=True start()_still_parked=True start()_raised=None process_alive=True
healthy-idle: watchdog_fired=False start()_still_parked=True start()_raised=None process_alive=True

Loop progress covers both stall phases. The deaf row is the gap I would fold into #826 before merge, and it is reachable with the frontend's current settings (auto_reconnect defaults to True). If reconnect-time endpoint discovery returns a non-retryable client code, _try_connect re-raises it, _reconnect propagates, and the entire _receive_message_loop task dies. Local endpoint answering HTTP 200 with {"code":99991663,...}:

lark_oapi.ws.exception.ClientException: 99991663: app ticket is invalid
t+ 3.0s start()_returned=False raised=None loop_ticks=  6 conn=None process_alive=True
t+ 9.0s start()_returned=False raised=None loop_ticks= 18 conn=None process_alive=True

loop_ticks keeps advancing at full rate while _conn stays None forever, so a loop-progress watchdog reports healthy on a frontend that can no longer receive anything. Your own docstring already says a live loop does not prove a healthy connection, and the SDK gives you the signal for free in both 1.6.8 and 1.7.3: cli.on_reconnecting / cli.on_reconnected. Chaining them onto beat() is a few lines:

prev_ing, prev_ed = cli.on_reconnecting, cli.on_reconnected
recon = [None]
cli.on_reconnecting = lambda: (recon.__setitem__(0, time.monotonic()), prev_ing())
cli.on_reconnected = lambda: (recon.__setitem__(0, None), prev_ed())
# inside watch(), next to the last_beat check:
if recon[0] and time.monotonic() - recon[0] >= timeout:
    ...  # reconnect started and never completed

That is what the deaf row fired on, and healthy-idle (connected, zero events for 14s against a 6s threshold) stayed quiet, so it does not violate the idle requirement in your proposed scope.

One more freeze that no progress-based check can see, as an upstream note rather than something for this PR: _connect() acquires self._lock and then returns early when self._conn is not None, before entering the try whose finally releases it.

B after _connect() early return: lock_held=True
B send: blocked on the leaked lock (timed out after 2s)
B disconnect: blocked on the leaked lock (timed out after 2s)

I reached that by awaiting _connect() directly with _conn already set, so I have not proven that a live session races into it. If it does, the loop keeps ticking while every send and _disconnect() blocks forever.

All of the above needs no Feishu credentials: a local socket that accepts and never answers, a local HTTP handler returning the error envelope, and a small fake conn object whose recv() raises.

The shape here (process alive, supervisor satisfied, work stopped) is the failure class our own scheduled agents keep producing, and the check we use reports process state and log staleness together instead of answering "is it running" (ours): https://github.com/aron-intframe/agent-guards

[ytahml]

@aron-intframe Thanks for the detailed reproduction! The startup and reconnect stall results are helpful confirmation for #826.

For this PR, I’m keeping the scope limited to event-loop stalls. The deaf case you found is useful additional context, but detecting it would introduce a separate reconnect recovery policy, so I’d prefer to discuss that independently if the maintainers are interested.