Repository navigation
bug(sandbox): a loopback listener keeps accepting connections after the workload closes it #4407
Description
Problem
In an OpenShell 0.1.2 sandbox, a TCP listener on 127.0.0.1 that the workload has closed keeps accepting connections. New connects to the port succeed and then sit unaccepted. A port that was only bound, or never used, refuses as expected. This looks like the network broker keeping its own copy of the listening socket it injected.
Reproduction
Inside a sandbox (openshell sandbox exec --name <sb> --no-tty -- python3 -):
import socket
def connects(p):
c = socket.socket(); c.settimeout(1)
try:
c.connect(("127.0.0.1", p)); return True
except OSError:
return False
finally:
c.close()
s = socket.socket(); s.bind(("127.0.0.1", 0)); p = s.getsockname()[1]; s.close()
print("bound only, closed:", connects(p)) # False (expected)
s = socket.socket(); s.bind(("127.0.0.1", 0)); s.listen(); p = s.getsockname()[1]; s.close()
print("listened, closed:", connects(p)) # True (expected False)
print("never used:", connects(59999)) # False (expected)Observed effects
- After a PostgreSQL server in the sandbox exited,
/proc/net/tcpstill showed its127.0.0.1:5432socket in LISTEN, owned by the sandbox uid, with a pending connection. A new postmaster on 5432 then failed:could not bind IPv4 address "127.0.0.1": Address already in use. No workload process held the socket. - Every later
openshell sandbox execchild inherited that socket as an open descriptor (fd 19,socket:[…]matching the stale listener's inode). The exec child's parent is pid 1 (openshell-sandbox --bootstrap …), so the supervisor appears to hold the socket withoutO_CLOEXEC./proc/1/fdisn't readable from the workload, so I couldn't confirm that directly. - Tests that check "a closed listener's port refuses connections" fail inside the sandbox.
Expected
Once the workload closes a listener, the port refuses connections and can be listened on again. Descriptors the supervisor holds aren't inherited by exec'd workload processes.
Environment
- OpenShell 0.1.2 (gateway, supervisor and sandbox runtime), Docker compute driver
- Host: Linux x86_64. Workload: python:3.14-slim-bookworm-based image, non-root uid 999
- Related but distinct: bug(sandbox): network broker logs expected socket races as warnings #3526 (broker socket-race warnings)
Activity
johntmyers commentedon Oct 12, 2026
Confirmed the core listener-lifetime bug on the running
docker-devgateway, usingopenshell sandbox execinocsf-network-demo. Sandbox and supervisor both report0.1.4-dev.35+ge585b4917; workload uid 1000, Linux7.0.12-linuxkit, aarch64. Python is absent in this image, so I used installed Perl with nativesocket,bind,listen,close, andconnectcalls and a bounded connection timeout, preserving the reported three cases:After close,
/proc/net/tcpstill showed port 46279 inLISTEN(inode 543994) with a queued connection. A separate listener closed before its exec process exited also remained inLISTENin the next exec (port 37433, inode 544670). Rebinding the closed listener's port succeeded; I reclaimed the test listeners this way.The descriptor-inheritance observation was not reproduced in this test: the subsequent exec had zero descriptors matching inode 544670, despite the stale listener remaining visible. This does not exclude inheritance with other socket creation flags or on the reported 0.1.2 runtime. The exact Python/custom-image reproduction and PostgreSQL restart failure were not tested here.
Current source supports the retained-descriptor explanation:
listen_socketcallslistenonretained_preconnect()and recordsListeningwithout releasing the broker-owned descriptor. Reclamation remains operation/pressure driven rather than following workload close immediately.The core behavior duplicates #4331, which already covers listener close, owner exit, misleading probes, and the successful-rebind workaround. #3526 concerns broker diagnostics rather than listener lifetime. No PR referencing #4331 or #4407 was found. This report provides first-hand context through the PostgreSQL failure and concrete expected behavior.
Ready for maintainer review: decide whether to consolidate the core lifetime fix under #4331 and retain descriptor inheritance as a separately verified regression case. No acceptance or implementation decision made; existing assignment preserved.
johntmyers commentedon Oct 12, 2026
Outcome and scope
Fix #4407 so closing the last workload reference to a TCP listener, or exiting its owning process, removes the listener immediately. Subsequent connections must fail with
ECONNREFUSED, and the port must be reusable without triggering broker garbage collection. Broker-owned descriptors must never leak into exec'd workloads, including when the workload deliberately requests a socket withoutSOCK_CLOEXEC.The listener-lifetime failure reproduces on docker-dev with runtime
0.1.4-dev.35+ge585b4917. Descriptor inheritance was not reproduced with the Perl fixture, so its regression must explicitly exercise socket creation withoutSOCK_CLOEXEC. #4331 reports the same listener-lifetime bug and should share the regression coverage. No active PR referencing either issue was found. #4407 remains assigned to johntmyers and ready for human review; this direct request authorizes planning only.Complexity: medium. Confidence: high in the retained-listener root cause; repeated operations and FD ownership need careful regression coverage.
Design
crates/openshell-sandbox/src/network_broker.rs, keep the firstlistenoperation mediated. After nativelistensucceeds and the local address is recorded, transition the registry entry toListeningand release its broker-owned descriptor before completing the notification. Keep lightweight metadata for subsequent mediation. Failedlistenmust retain the pre-connect descriptor and its previous state.listenon an established listener execute natively, preserving backlog changes and notification restart behavior. Simply addingrelease_preconnect()is insufficient: the current handler always dereferences the retained descriptor. Audit every other consumer, especiallybind_socketandconnect_socket(repeated_connect_outcomecurrently treatsListeningas pre-connect), so operations on a listener preserve appropriate kernel/error behavior without accidentally entering a connection/relay path or returning an artificial missing-descriptor error. If exact error handling requires descriptor access, use a validated, short-lived close-on-exec duplicate, never a permanently retained listener reference. Preserve the existing enforcement of mutable connect addresses and protected control endpoints.SOCK_CLOEXEC, including the EMFILE retry. Independently preserve the workload's requested close-on-exec flag in metadata andadd_fd_and_send. This prevents a source descriptor created without close-on-exec from leaking through a concurrent workload fork/exec before it is released. Do not change intentional inheritance of a workload-owned non-CLOEXEC descriptor.crates/openshell-sandbox/src/linux/socket_registry.rs, document and enforce that successful listening, like successful connection, relinquishes the retained source. Aligncommit_with_state's retention rules with that contract and retain bounded metadata collection. Keep nativeaccept/accept4, loopback confinement, and denied socket options intact.Regression coverage
LISTENand cause immediate connection refusal before any rebinding or descriptor-pressure collection. Include bound-only and unused-port controls. Test IPv4 and IPv6 where supported.listenwith a different backlog, implicit bind throughlisten, failedlisten, and bind/connect attempts on listening sockets. Compare applicable errnos with Linux behavior; do not weaken network enforcement to preserve error behavior.SOCK_CLOEXEC: verify the broker source always hasFD_CLOEXEC, while the injected workload descriptor follows the requested flag. Start an unrelated exec while the source is still retained and assert that no descriptor with its socket inode leaks. Separately verify intentional inheritance from the owning workload still works.e2e/rust/tests/listener_lifetime.rsfixture registered with the existing E2E harness. Use existing workload tools and ephemeral ports; check close, process exit, subsequent exec visibility, connection refusal, and rebind. Keep the fixture independent of PostgreSQL installation.Verification and documentation
During implementation, run repository formatting and scoped lint/build checks, the Linux
openshell-sandboxtests, existing native-accept/confinement coverage, and the relevant Docker E2E lane (mise run e2e:docker, scoped using the existing harness where appropriate). macOS-only compilation cannot validate Linux seccomp behavior: execute the kernel-facing tests in the supported Linux environment. Re-run the three-case reproducer on docker-dev with the newly built runtime and confirm all three report refusal after close. Do not claim success against the pre-fix running image.Document the descriptor ownership contract in
crates/openshell-sandbox/README.mdand update relevant Rust comments. No new CLI, configuration, middleware, or published-doc workflow is needed; this restores ordinary socket semantics inside the runtime. Review related contributor/public skills through thesync-agent-inframaintenance map during implementation and change them only if their instructions become inaccurate.Alternatives and risks
Periodic collection or intercepting every
closewould introduce delayed cleanup or additional mediation complexity across dup/fork/exit. Releasing the reference after successfullistenlets the kernel track the actual workload ownership. Removinglistenmediation entirely would require a broader registry/state redesign. Middleware and gateway interceptors cannot repair runtime FD ownership.The main risks are repeated-listen regressions, changed errors for operations on a listener, and confusing broker CLOEXEC with workload CLOEXEC. Cover these explicitly. Maintainers may consolidate tracking under #4331; this plan does not close either issue or change acceptance/roadmap state.
Revision 1 — plan only; implementation and test execution pending.
johntmyers commentedon Oct 12, 2026
Implemented in #4444 on
fix/4407-listener-lifetime/johntmyers(signed-off commit9520604c8). Successful listening now releases the broker reference, repeatedlistenremains native, bind/connect on a listener preserve Linux errors, and broker source sockets always use close-on-exec independently of workload flags.All four new Linux unit tests, the 38-test focused broker suite, both new Docker CLI E2E tests, scoped lint, and the full pre-commit hook pass. The lifetime E2E test fails on the original docker-dev runtime and passes with the branch-built musl runtime. Broader validation caveats (Docker's AF_ALG errno, existing musl test-only ABI assumptions, and an initial restart failure that passed on rerun) are recorded in the PR.
The user's existing docker-dev sandbox remains healthy on its original runtime; temporary validation gateways and sandboxes were cleaned up. Updated the crate README. No issue acceptance or roadmap labels changed.