fix(server): self-update no longer rolls itself back on restart - #5095
Conversation
A boot-service self-update runs `systemctl --user restart` from inside
the service it restarts. systemd's default KillMode=control-group sends
SIGTERM to the whole cgroup, so the systemctl child died mid-restart,
the updater read that as a restart failure, and its rollback restored
the previous unit while the old server was still shutting down. Result:
download and install succeed, then the server resumes on the old
version ("The server did not resume on t3@<target>").
Queue the restart with --no-block instead: systemctl exits before the
stop signal lands, so its exit code only reflects whether systemd
accepted the job. The rollback path now fires only on genuine
rejections, which it was written for.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 513de4c. Configure here.
ApprovabilityVerdict: Approved 001b2c4 This is a targeted bug fix for the self-update mechanism. The change adds You can customize Macroscope's approvability policy. Learn more. |
With --no-block the restart fiber completes while the old process is still shutting down; the unconditional ensuring released the in-flight lock at that point, letting a second update rewrite the unit mid-teardown. Release the lock only on the failure path, after the rollback has restored the previous unit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Problem
Remote server updates on boot-service (systemd) machines fail at the resume step: download and install succeed, then the client reports "The server did not resume on t3@" and the server comes back on the old version.
The updater runs
systemctl --user restartfrom inside the service being restarted. systemd's defaultKillMode=control-groupSIGTERMs the entire cgroup — including thatsystemctlchild — so the blocking restart call dies mid-flight. The updater interprets the dead child as a failed restart and its rollback handler rewrites the unit back to the previous version while the old server process is still in its multi-second graceful shutdown. systemd then starts the unit pointing at the old install. The freshly installed target version is left on disk, unused, and every retry hits the same race.Observed on bb-1 (two consecutive failures,
0.0.32-nightly.20260730.956 → 0.0.32-nightly.20260731.963): the boot log shows "Server self-update installed; restarting boot service" followed ~2s later by "Server self-update could not restart the boot service", and the unit file's mtime matches the rollback write.Fix
Queue the restart with
systemctl --user restart --no-block. It enqueues the job and exits before the stop signal reaches the cgroup, so its exit code only reflects whether systemd accepted the job. The existing rollback path is unchanged and now fires only for genuine rejections (bad unit, unreachable manager) — the case it was written for — instead of for the updater's own death.Updated the two test assertions on the restart argv; the rollback-on-failure and retry tests pass unchanged.
🤖 Generated with Claude Code (Claude Fable 5)
Note
Medium Risk
Changes critical remote self-update handoff on systemd hosts (unit rewrite + restart timing and concurrency lock), though scope is limited to the boot-service path and rollback behavior is intentionally preserved for real failures.
Overview
Fixes boot-service self-updates that rolled back to the old version after install succeeded. The updater ran a blocking
systemctl --user restartfrom inside the unit being restarted; systemd SIGTERM’d the shared cgroup and killed thatsystemctlchild, which was treated as restart failure and restored the previous unit file.Restart now uses
systemctl --user restart --no-blockso the job is queued and the command exits before the stop signal. Rollback on non-zero exit still applies only when systemd genuinely rejects the job.The in-flight update lock is no longer cleared on every restart fiber completion (
Effect.ensuringremoved). It is released only when restart fails after rollback, because--no-blockreturns while this process is still shutting down—holding the lock blocks a second update from rewriting the unit mid-teardown. Tests assert--no-blockon restart argv and that a concurrent update is rejected after a successful queued restart.Reviewed by Cursor Bugbot for commit 001b2c4. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Fix self-update lock to remain held during systemd restart, preventing rollback on process exit
systemctl restartinvocation inselfUpdate.tsto use--no-block, so the process exits before systemd restarts it rather than waiting for the restart to complete.finallyrelease of the in-flight lock; the lock is now only released on error (after rollback), while a queued restart keeps the lock held until process termination.Macroscope summarized 001b2c4.
Summary by CodeRabbit