worker: keep reserved host ports bound until the proxy takes them - #1940
Merged
Merged
Conversation
luke-lombardi
force-pushed
the
worker/hold-reserved-host-ports
branch
from
September 28, 2026 23:09
1709cc5 to
aa1e4f9
Compare
ReservePorts bound an ephemeral port, read the number, and closed the socket; the port proxy re-bound it only once the container answered. On a busy host the kernel could hand the number to an outbound connection in between, so the proxy failed to bind and the container came up unreachable. Keep the reservation's listener and pass it through the exposure to the proxy, which now always receives its listener instead of binding one itself. Where kernel forwarding serves the port the socket is released once the rule is in, so DNAT workers are unchanged. Unexposed reservations and exposures closed before a proxy starts release the socket.
luke-lombardi
force-pushed
the
worker/hold-reserved-host-ports
branch
from
September 28, 2026 23:15
aa1e4f9 to
de1569e
Compare
jackf-beamcloud
approved these changes
Sep 29, 2026
jackf-beamcloud
left a comment
Contributor
There was a problem hiding this comment.
Holding the reserved listener and handing it to the proxy closes the bind-to-rebind race cleanly; connect() autobind skips ports held by a bound socket, so outbound connections can't take it anymore. Ownership looks right on every path: DNAT success releases, DNAT failure falls back to the proxy with the socket still held, close/ReleasePortReservations release, and startProxy under proxyMu can't double-start. Non-blocking: DNAT-mode workers now briefly hold a 0.0.0.0 listener until the rules land, so an early connection sits in backlog and gets reset on release instead of refused; harmless.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On agent-hosted workers the container port proxy is the only path from the gateway to a container (
forcePortProxyfor a loopback pod address).ReservePortspicked a host port by binding:0, reading the number and closing the socket; the proxy re-bound that port only once the container answered, seconds later. In that gap the kernel can assign the same number to an outbound connection. The proxy then logsfailed to start container port proxy ... bind: address already in useand gives up, and the container runs healthy but unreachable.Change
ReservePortskeeps the listener it bound, on the network/address the proxy uses for the worker's address family.startProxyhands it to the proxy; where kernel forwarding serves the port, the exposure releases it once the DNAT rule is in, so k8s workers are unchanged.run()only waits for the backend and accepts.startProxybinds a fresh listener only for exposures created without a reservation (worker restart on a persistent machine).ReleasePortReservations, deferred for the container's lifetime inspawn) and exposures closed before a proxy starts release the socket.Notes
Testing
go test ./pkg/worker/passes;-raceon the port and proxy tests passes.TestReservePortsHoldsPortUntilReleased: a reserved port cannot be bound by another socket until it is released.