Skip to content

T-8292 [Fable] Atomic unix socket takeover + inode-guarded cleanup - #8

Merged
alistairjevans merged 1 commit into
masterfrom
alistair/atomic-socket-takeover
Aug 14, 2026
Merged

T-8292 [Fable] Atomic unix socket takeover + inode-guarded cleanup#8
alistairjevans merged 1 commit into
masterfrom
alistair/atomic-socket-takeover

Conversation

@alistairjevans

@alistairjevans alistairjevans commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Problem

After a complete docker shutdown/restart, unix sockets sometimes never recover.

Root cause is a cross-generation race in this gem's socket lifecycle:

  1. start() did exists → remove_filebind, silently stealing the path from a still-running older server, with a window where the path doesn't exist at all.
  2. stop() unconditionally unlinked the bind path by name, with no check that the file was still the socket this server bound.

On a normal rolling deploy (swarm stop-first) generations never overlap, so this never bit. After a full docker restart, swarm reconciliation can start the new task while the old container is still draining (5s DRAIN_TIME): the new server binds the socket, then the old server's stop() deletes the new server's socket file. The new server keeps listening on an unlinked inode forever — haproxy gets ENOENT on every connect and nothing ever recreates the file, until the replica is manually restarted.

The ticket's WARN Failed to remove socket file: NotFound log line is that manual restart: the victim discovering at shutdown that its socket file had been missing for its whole run. The harmful deletion itself was silent.

Fix

  • start(): bind to a unique temp path ({path}.{pid}.{seq}.tmp) and atomically rename(2) it over the target. The path always points at a live socket, and takeover of a still-bound path is a single step.
  • stop(): record the bound socket file's (dev, ino) at bind time (lstat'd on the temp path, so it can't race) and only unlink the path if it still matches. An older generation can no longer remove a newer generation's socket; if the path was taken over it's left in place with an info log.

Error-message change: startup failures at the target path now raise Failed to install Unix socket file … (rename step) instead of Failed to remove existing Unix socket file ….

Tests

  • test_unix_socket_takeover_preserves_new_generation_socket — old server stops while a new server owns the path: socket file survives, is connectable, and the new server's own stop still cleans it up. Reproduces the production race (and the ticket's NotFound warning) on unfixed code.
  • test_unix_socket_stop_leaves_foreign_file — stop leaves a file it didn't bind untouched.
  • Updated the undeletable-file/directory error tests for the new failure point.

Full suite: 39 runs, 0 failures (1 skip = sudo-gated test).

After a full docker restart, a replacement ingester can bind its unix
socket while the old container is still draining; the old server's
stop() then unlinked the path by name, deleting the new server's socket.
The new server keeps listening on an unlinked inode and never recovers
(BetterStackHQ Linear T-8292).

- start() now binds to a unique temp path and atomically renames it over
  the target, so the path always points at a live socket and takeover of
  a still-bound path is a single step.
- stop() records the bound socket file's (dev, ino) and only unlinks the
  path if it still matches, so an older generation never removes a newer
  generation's socket.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@alistairjevans
alistairjevans merged commit 13390c8 into master Aug 14, 2026
3 checks passed
@alistairjevans
alistairjevans deleted the alistair/atomic-socket-takeover branch August 14, 2026 16:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant