Skip to content

net: SocketServer keeps accepting after running out of file descriptors - #848

Closed
christianparpart wants to merge 2 commits into
masterfrom
fix/accept-loop-transient-errors
Closed

christianparpart wants to merge 2 commits into
masterfrom
fix/accept-loop-transient-errors

Conversation

@christianparpart

@christianparpart christianparpart commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

SocketServer::acceptLoop() returned on any exception from TcpSocket::tryAccept(), and tryAccept() throws for every accept(2) errno except EINTR and EAGAIN/EWOULDBLOCK. So a single EMFILE ended the accept thread for the rest of the server's life, however soon descriptors were free again. The listener stayed bound, so the kernel went on completing handshakes into a backlog nobody drained, and clients hung in their Upgrade read — the consequence docs/spec/core/backend.md already names for a listener nobody accepts on. The existing EMFILE case drove the loop into exactly this state and asserted only that close() then returned, which holds whether the loop ended or kept serving.

Measured, on Linux (WSL2, clang 22.1.2, clang-debug with MORPH_BUILD_NET=ON):

  • The strengthened case is red on the first commit, which changes only the test on top of 9d001052: reported for: false, served for: false. A connection that queued while RLIMIT_NOFILE was exhausted is never served after the limit is restored.
  • It is green with the fix.
  • The full ctest passes, 1865/1865, with 0 build warnings under the strict -Weverything set.
  • Setting the EMFILE row to ListenerUnusable turns all three new or changed cases red.
  • Having tryAccept() throw a plain std::runtime_error again turns the server case and the tryAccept case red.

Changes

  • A. Exhaustion backs off instead of ending the loop. kAcceptErrors in detail/tcp_socket.hpp sorts each errno accept(2) can answer into three outcomes, and acceptFailureOf() reads it.
    • Exhausted (EMFILE, ENFILE, ENOBUFS, ENOMEM): the loop waits 10 ms, doubling to at most 1 s, and resets once an accept stops failing. The wait is a poll() on the wakeup alone, so it cannot spin on the still-readable listener and close() still ends it at once. The first failure of a run is logged at warn.
    • ListenerUnusable (EBADF, ENOTSOCK, EINVAL, EFAULT, and any unnamed errno): still ends the loop, as before, and now logs why.
    • tryAccept() throws std::system_error carrying the errno. That is still a std::runtime_error, with the same message text.
  • B. Connections that failed while pending are skipped like EAGAIN. This part is separable; drop it if you would rather not take it. Linux accept(2) documents that it hands back network errors already pending on the new connection (ENETDOWN, EPROTO, ENOPROTOOPT, EHOSTDOWN, ENONET, EHOSTUNREACH, EOPNOTSUPP, ENETUNREACH), and says to treat them like EAGAIN by retrying. EPERM is a firewall refusing that one connection. These are the NoConnection rows, and tryAccept() answers them std::nullopt.
  • The spec's SocketServer section describes the three outcomes, and there is a CHANGELOG.md entry under Fixed.
  • The EMFILE case exhausts through FdLimitClamp, promoted from test_tcp_socket.cpp to tests/net/fd_limit_clamp.hpp so both files share it. It fills every free descriptor. Lowering RLIMIT_NOFILE to one above the highest open fd, which the old case did, leaves any hole below that fd free, and a sanitizer or coverage runtime leaves such holes: accept(2) took one and never met EMFILE. The old case's assertion could not see that, and the new one's first gate did, on this PR's first CI run. It is reproduced without a sanitizer by starting the binary with fd 50 open.
  • scripts/branch_partial_allowlist.json: two socket_server.hpp line hints move with the code. The if (!clientSocket) entry is retired, because the backoff now takes that arm on every exhausted accept, which is the retirement condition the entry named.

morph reaches none of the code core-cpp v0.5.1 changed: its accept loop and socket-error tables. morph uses only core-cpp's event loop, timers, wakeup, base64 and async, so this is independent of the core-cpp pin.

…it has them

The EMFILE case drove the accept loop into exhaustion and asserted only
that close() then returned promptly, which holds whether the loop ended
or kept serving -- so it could not tell the defect from the fix, and its
comment described the loop ending as the expected outcome.

It now asserts the property a server needs: once RLIMIT_NOFILE is
restored, a connection that queued during the exhaustion completes its
WebSocket Upgrade handshake. The probe is a raw fd with poll() bounds,
so a loop that has stopped fails the check instead of hanging the case.
It also waits on the loop's own report of the exhaustion rather than a
fixed sleep.

Red on this commit: acceptLoop() returns on the first EMFILE.

Signed-off-by: Christian Parpart <christian@parpart.family>
@christianparpart christianparpart added bug Something isn't working area: core Subsystem: core labels Sep 29, 2026
@christianparpart
christianparpart force-pushed the fix/accept-loop-transient-errors branch 2 times, most recently from ca48bf6 to 148ac29 Compare September 29, 2026 18:43
…e descriptors

SocketServer's accept loop returned on any exception from tryAccept(),
which throws for every accept(2) errno but EINTR and EAGAIN. A single
EMFILE therefore ended the accept thread for the rest of the server's
life, however soon descriptors were free again, while the listener stayed
bound and the kernel went on completing handshakes into a backlog nobody
drained: clients hung in their Upgrade read.

accept(2) answers for three different things, and kAcceptErrors in
detail/tcp_socket.hpp now sorts each errno into one of them:

- Exhausted (EMFILE, ENFILE, ENOBUFS, ENOMEM): the loop backs off, 10 ms
  doubling to 1 s, waiting on the wakeup alone so close() still ends it
  at once, and serves again once the next accept stops failing. The
  first failure of a run is logged at warn.
- NoConnection: the network errors Linux accept(2) hands back for a
  connection that failed while pending, which its man page says to treat
  like EAGAIN, and a firewall's EPERM. tryAccept() answers them nullopt,
  so they take the loop's existing EAGAIN branch; no new branch.
  ECONNABORTED has no row: a reset pending connection is still accepted
  on Linux, measured, and its reset reaches recvSome() as a close.
- ListenerUnusable (EBADF, ENOTSOCK, EINVAL, EFAULT, and any errno no
  row names): the loop ends, as before, and now logs why.

tryAccept() throws std::system_error carrying the errno, still a
std::runtime_error, so the loop can tell these apart. The spec's
SocketServer section says what each outcome does.

The EMFILE case exhausts descriptors through FdLimitClamp, now shared from
tests/net/fd_limit_clamp.hpp, rather than by lowering RLIMIT_NOFILE to one
above the highest open fd: a hole below that fd -- which a sanitizer
runtime leaves -- let accept(2) succeed, so the loop never met EMFILE.
Reproduced by starting the binary with fd 50 open.

scripts/branch_partial_allowlist.json: two socket_server.hpp line hints
move with the code, and the `if (!clientSocket)` entry is retired. It
recorded that arm as never taken; the backoff now takes it on every
exhausted accept, which the EMFILE case drives -- the retirement
condition the entry itself names.

Signed-off-by: Christian Parpart <christian@parpart.family>
@christianparpart
christianparpart force-pushed the fix/accept-loop-transient-errors branch from 148ac29 to 31c1775 Compare September 29, 2026 18:55
@codecov

codecov Bot commented Sep 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 61.53846% with 15 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
include/morph/net/socket_server.hpp 56.25% 7 Missing and 7 partials ⚠️
include/morph/net/detail/tcp_socket.hpp 85.71% 0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@christianparpart

Copy link
Copy Markdown
Member Author

Closing this in favour of a different direction, decided by the maintainer: rather than hardening a second accept loop here, morph's socket layer (`TcpSocket` / `SocketServer`) will move onto core-cpp's `core::net` networking, which morph already depends on. core-cpp v0.5.1 already carries the transient-accept-error handling this PR added (exhaustion backs off; only a dead listener ends the loop). A tracking issue for that migration will follow once its plan is settled.

One finding from this PR is worth keeping regardless: the existing EMFILE test clamps `RLIMIT_NOFILE` to highest-open-fd + 1. When the process starts with a hole below that descriptor (which sanitizer and coverage runtimes leave), `accept(2)` succeeds and EMFILE never happens, so the test cannot reliably produce the condition it is named for. The `FdLimitClamp` in `test_tcp_socket.cpp`, which fills every free descriptor, does not have this flaw.

The branch is left in place for reference.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: core Subsystem: core bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant