Skip to content

worker: keep reserved host ports bound until the proxy takes them - #1940

Merged
luke-lombardi merged 1 commit into
mainfrom
worker/hold-reserved-host-ports
Sep 29, 2026
Merged

luke-lombardi merged 1 commit into
mainfrom
worker/hold-reserved-host-ports

Conversation

@luke-lombardi

@luke-lombardi luke-lombardi commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Problem

On agent-hosted workers the container port proxy is the only path from the gateway to a container (forcePortProxy for a loopback pod address). ReservePorts picked a host port by binding :0, reading the number and closing the socket; the proxy re-bound that port only once the container answered, seconds later. In that gap the kernel can assign the same number to an outbound connection. The proxy then logs failed to start container port proxy ... bind: address already in use and gives up, and the container runs healthy but unreachable.

Change

  • ReservePorts keeps the listener it bound, on the network/address the proxy uses for the worker's address family.
  • The exposure receives the listener. Where the proxy is the path, startProxy hands it to the proxy; where kernel forwarding serves the port, the exposure releases it once the DNAT rule is in, so k8s workers are unchanged.
  • The proxy always receives its listener now, so run() only waits for the backend and accepts. startProxy binds a fresh listener only for exposures created without a reservation (worker restart on a persistent machine).
  • Unexposed reservations (ReleasePortReservations, deferred for the container's lifetime in spawn) and exposures closed before a proxy starts release the socket.

Notes

  • One fd per reserved port for the container's lifetime on proxy-mode workers; nothing extra on DNAT workers.
  • On proxy-mode workers a connection arriving before the proxy starts waits in the listen backlog instead of being refused. Only pods and sandboxes can receive traffic that early; endpoints and taskqueues are routed after readiness.

Testing

  • go test ./pkg/worker/ passes; -race on the port and proxy tests passes.
  • New TestReservePortsHoldsPortUntilReleased: a reserved port cannot be bound by another socket until it is released.
  • Existing proxy tests now hand the proxy its listener instead of picking a free port and letting the proxy re-bind it.

@luke-lombardi
luke-lombardi force-pushed the worker/hold-reserved-host-ports branch from 1709cc5 to aa1e4f9 Compare September 28, 2026 23:09
ReservePorts bound an ephemeral port, read the number, and closed the socket;
the port proxy re-bound it only once the container answered. On a busy host
the kernel could hand the number to an outbound connection in between, so the
proxy failed to bind and the container came up unreachable.

Keep the reservation's listener and pass it through the exposure to the proxy,
which now always receives its listener instead of binding one itself. Where
kernel forwarding serves the port the socket is released once the rule is in,
so DNAT workers are unchanged. Unexposed reservations and exposures closed
before a proxy starts release the socket.
@luke-lombardi
luke-lombardi force-pushed the worker/hold-reserved-host-ports branch from aa1e4f9 to de1569e Compare September 28, 2026 23:15

@jackf-beamcloud jackf-beamcloud left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Holding the reserved listener and handing it to the proxy closes the bind-to-rebind race cleanly; connect() autobind skips ports held by a bound socket, so outbound connections can't take it anymore. Ownership looks right on every path: DNAT success releases, DNAT failure falls back to the proxy with the socket still held, close/ReleasePortReservations release, and startProxy under proxyMu can't double-start. Non-blocking: DNAT-mode workers now briefly hold a 0.0.0.0 listener until the rules land, so an early connection sits in backlog and gets reset on release instead of refused; harmless.

@luke-lombardi
luke-lombardi merged commit de6e16c into main Sep 29, 2026
5 checks passed
@luke-lombardi
luke-lombardi deleted the worker/hold-reserved-host-ports branch September 29, 2026 00:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants