net: SocketServer keeps accepting after running out of file descriptors - #848
christianparpart wants to merge 2 commits into
Conversation
…it has them The EMFILE case drove the accept loop into exhaustion and asserted only that close() then returned promptly, which holds whether the loop ended or kept serving -- so it could not tell the defect from the fix, and its comment described the loop ending as the expected outcome. It now asserts the property a server needs: once RLIMIT_NOFILE is restored, a connection that queued during the exhaustion completes its WebSocket Upgrade handshake. The probe is a raw fd with poll() bounds, so a loop that has stopped fails the check instead of hanging the case. It also waits on the loop's own report of the exhaustion rather than a fixed sleep. Red on this commit: acceptLoop() returns on the first EMFILE. Signed-off-by: Christian Parpart <christian@parpart.family>
ca48bf6 to
148ac29
Compare
…e descriptors SocketServer's accept loop returned on any exception from tryAccept(), which throws for every accept(2) errno but EINTR and EAGAIN. A single EMFILE therefore ended the accept thread for the rest of the server's life, however soon descriptors were free again, while the listener stayed bound and the kernel went on completing handshakes into a backlog nobody drained: clients hung in their Upgrade read. accept(2) answers for three different things, and kAcceptErrors in detail/tcp_socket.hpp now sorts each errno into one of them: - Exhausted (EMFILE, ENFILE, ENOBUFS, ENOMEM): the loop backs off, 10 ms doubling to 1 s, waiting on the wakeup alone so close() still ends it at once, and serves again once the next accept stops failing. The first failure of a run is logged at warn. - NoConnection: the network errors Linux accept(2) hands back for a connection that failed while pending, which its man page says to treat like EAGAIN, and a firewall's EPERM. tryAccept() answers them nullopt, so they take the loop's existing EAGAIN branch; no new branch. ECONNABORTED has no row: a reset pending connection is still accepted on Linux, measured, and its reset reaches recvSome() as a close. - ListenerUnusable (EBADF, ENOTSOCK, EINVAL, EFAULT, and any errno no row names): the loop ends, as before, and now logs why. tryAccept() throws std::system_error carrying the errno, still a std::runtime_error, so the loop can tell these apart. The spec's SocketServer section says what each outcome does. The EMFILE case exhausts descriptors through FdLimitClamp, now shared from tests/net/fd_limit_clamp.hpp, rather than by lowering RLIMIT_NOFILE to one above the highest open fd: a hole below that fd -- which a sanitizer runtime leaves -- let accept(2) succeed, so the loop never met EMFILE. Reproduced by starting the binary with fd 50 open. scripts/branch_partial_allowlist.json: two socket_server.hpp line hints move with the code, and the `if (!clientSocket)` entry is retired. It recorded that arm as never taken; the backoff now takes it on every exhausted accept, which the EMFILE case drives -- the retirement condition the entry itself names. Signed-off-by: Christian Parpart <christian@parpart.family>
148ac29 to
31c1775
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
Closing this in favour of a different direction, decided by the maintainer: rather than hardening a second accept loop here, morph's socket layer (`TcpSocket` / `SocketServer`) will move onto core-cpp's `core::net` networking, which morph already depends on. core-cpp v0.5.1 already carries the transient-accept-error handling this PR added (exhaustion backs off; only a dead listener ends the loop). A tracking issue for that migration will follow once its plan is settled. One finding from this PR is worth keeping regardless: the existing EMFILE test clamps `RLIMIT_NOFILE` to highest-open-fd + 1. When the process starts with a hole below that descriptor (which sanitizer and coverage runtimes leave), `accept(2)` succeeds and EMFILE never happens, so the test cannot reliably produce the condition it is named for. The `FdLimitClamp` in `test_tcp_socket.cpp`, which fills every free descriptor, does not have this flaw. The branch is left in place for reference. |
SocketServer::acceptLoop()returned on any exception fromTcpSocket::tryAccept(), andtryAccept()throws for everyaccept(2)errno exceptEINTRandEAGAIN/EWOULDBLOCK. So a singleEMFILEended the accept thread for the rest of the server's life, however soon descriptors were free again. The listener stayed bound, so the kernel went on completing handshakes into a backlog nobody drained, and clients hung in their Upgrade read — the consequencedocs/spec/core/backend.mdalready names for a listener nobody accepts on. The existing EMFILE case drove the loop into exactly this state and asserted only thatclose()then returned, which holds whether the loop ended or kept serving.Measured, on Linux (WSL2, clang 22.1.2,
clang-debugwithMORPH_BUILD_NET=ON):9d001052:reported for: false,served for: false. A connection that queued whileRLIMIT_NOFILEwas exhausted is never served after the limit is restored.-Weverythingset.EMFILErow toListenerUnusableturns all three new or changed cases red.tryAccept()throw a plainstd::runtime_erroragain turns the server case and thetryAcceptcase red.Changes
kAcceptErrorsindetail/tcp_socket.hppsorts eacherrnoaccept(2)can answer into three outcomes, andacceptFailureOf()reads it.Exhausted(EMFILE,ENFILE,ENOBUFS,ENOMEM): the loop waits 10 ms, doubling to at most 1 s, and resets once an accept stops failing. The wait is apoll()on the wakeup alone, so it cannot spin on the still-readable listener andclose()still ends it at once. The first failure of a run is logged at warn.ListenerUnusable(EBADF,ENOTSOCK,EINVAL,EFAULT, and any unnamed errno): still ends the loop, as before, and now logs why.tryAccept()throwsstd::system_errorcarrying the errno. That is still astd::runtime_error, with the same message text.EAGAIN. This part is separable; drop it if you would rather not take it. Linuxaccept(2)documents that it hands back network errors already pending on the new connection (ENETDOWN,EPROTO,ENOPROTOOPT,EHOSTDOWN,ENONET,EHOSTUNREACH,EOPNOTSUPP,ENETUNREACH), and says to treat them likeEAGAINby retrying.EPERMis a firewall refusing that one connection. These are theNoConnectionrows, andtryAccept()answers themstd::nullopt.ECONNABORTEDis deliberately not a row, per TcpSocket::accept()/tryAccept(): the ECONNABORTED no-retry rationale is stale once the accept loop has its own wakeup #465's measurement: 8000 reset races, zeroECONNABORTEDon Linux. It keeps its current behaviour through the default.SocketServersection describes the three outcomes, and there is aCHANGELOG.mdentry under Fixed.FdLimitClamp, promoted fromtest_tcp_socket.cpptotests/net/fd_limit_clamp.hppso both files share it. It fills every free descriptor. LoweringRLIMIT_NOFILEto one above the highest open fd, which the old case did, leaves any hole below that fd free, and a sanitizer or coverage runtime leaves such holes:accept(2)took one and never metEMFILE. The old case's assertion could not see that, and the new one's first gate did, on this PR's first CI run. It is reproduced without a sanitizer by starting the binary with fd 50 open.scripts/branch_partial_allowlist.json: twosocket_server.hppline hints move with the code. Theif (!clientSocket)entry is retired, because the backoff now takes that arm on every exhausted accept, which is the retirement condition the entry named.morph reaches none of the code core-cpp v0.5.1 changed: its accept loop and socket-error tables. morph uses only core-cpp's event loop, timers, wakeup, base64 and async, so this is independent of the core-cpp pin.