Skip to content

fix(server): self-update no longer rolls itself back on restart - #5095

Merged
t3dotgg merged 2 commits into
mainfrom
fix/self-update-restart-race
Jul 31, 2026
Merged

fix(server): self-update no longer rolls itself back on restart#5095
t3dotgg merged 2 commits into
mainfrom
fix/self-update-restart-race

Conversation

@t3dotgg

@t3dotgg t3dotgg commented Jul 31, 2026

Copy link
Copy Markdown
Member

Problem

Remote server updates on boot-service (systemd) machines fail at the resume step: download and install succeed, then the client reports "The server did not resume on t3@" and the server comes back on the old version.

The updater runs systemctl --user restart from inside the service being restarted. systemd's default KillMode=control-group SIGTERMs the entire cgroup — including that systemctl child — so the blocking restart call dies mid-flight. The updater interprets the dead child as a failed restart and its rollback handler rewrites the unit back to the previous version while the old server process is still in its multi-second graceful shutdown. systemd then starts the unit pointing at the old install. The freshly installed target version is left on disk, unused, and every retry hits the same race.

Observed on bb-1 (two consecutive failures, 0.0.32-nightly.20260730.956 → 0.0.32-nightly.20260731.963): the boot log shows "Server self-update installed; restarting boot service" followed ~2s later by "Server self-update could not restart the boot service", and the unit file's mtime matches the rollback write.

Fix

Queue the restart with systemctl --user restart --no-block. It enqueues the job and exits before the stop signal reaches the cgroup, so its exit code only reflects whether systemd accepted the job. The existing rollback path is unchanged and now fires only for genuine rejections (bad unit, unreachable manager) — the case it was written for — instead of for the updater's own death.

Updated the two test assertions on the restart argv; the rollback-on-failure and retry tests pass unchanged.


🤖 Generated with Claude Code (Claude Fable 5)


Open in Devin Review

Note

Medium Risk
Changes critical remote self-update handoff on systemd hosts (unit rewrite + restart timing and concurrency lock), though scope is limited to the boot-service path and rollback behavior is intentionally preserved for real failures.

Overview
Fixes boot-service self-updates that rolled back to the old version after install succeeded. The updater ran a blocking systemctl --user restart from inside the unit being restarted; systemd SIGTERM’d the shared cgroup and killed that systemctl child, which was treated as restart failure and restored the previous unit file.

Restart now uses systemctl --user restart --no-block so the job is queued and the command exits before the stop signal. Rollback on non-zero exit still applies only when systemd genuinely rejects the job.

The in-flight update lock is no longer cleared on every restart fiber completion (Effect.ensuring removed). It is released only when restart fails after rollback, because --no-block returns while this process is still shutting down—holding the lock blocks a second update from rewriting the unit mid-teardown. Tests assert --no-block on restart argv and that a concurrent update is rejected after a successful queued restart.

Reviewed by Cursor Bugbot for commit 001b2c4. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Fix self-update lock to remain held during systemd restart, preventing rollback on process exit

  • Changes the systemctl restart invocation in selfUpdate.ts to use --no-block, so the process exits before systemd restarts it rather than waiting for the restart to complete.
  • Removes the unconditional finally release of the in-flight lock; the lock is now only released on error (after rollback), while a queued restart keeps the lock held until process termination.
  • Concurrent update attempts during the shutdown window now fail with an 'already in progress' error instead of being allowed to proceed.
  • Behavioral Change: the in-flight lock is no longer released on successful update — it stays held until the process is killed by systemd.

Macroscope summarized 001b2c4.

Summary by CodeRabbit

  • Bug Fixes
    • Improved service restart handling during updates by allowing restarts to proceed asynchronously.
    • Updated restart behavior to reduce failures during the handoff and retry process.

A boot-service self-update runs `systemctl --user restart` from inside
the service it restarts. systemd's default KillMode=control-group sends
SIGTERM to the whole cgroup, so the systemctl child died mid-restart,
the updater read that as a restart failure, and its rollback restored
the previous unit while the old server was still shutting down. Result:
download and install succeed, then the server resumes on the old
version ("The server did not resume on t3@<target>").

Queue the restart with --no-block instead: systemctl exits before the
stop signal lands, so its exit code only reflects whether systemd
accepted the job. The rollback path now fires only on genuine
rejections, which it was written for.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the server self-update rollback fix and matches the main change.
Description check ✅ Passed The description clearly explains the problem, root cause, fix, scope, and test impact, but omits the template headings and checklist.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added size:S 10-29 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. labels Jul 31, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 513de4c. Configure here.

Comment thread apps/server/src/cloud/selfUpdate.ts
@macroscopeapp

macroscopeapp Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved 001b2c4

This is a targeted bug fix for the self-update mechanism. The change adds --no-block to systemctl restart to prevent SIGTERM race conditions that caused erroneous rollbacks, and adjusts lock release timing to prevent concurrent updates during shutdown. Well-scoped with clear intent and updated tests.

You can customize Macroscope's approvability policy. Learn more.

With --no-block the restart fiber completes while the old process is
still shutting down; the unconditional ensuring released the in-flight
lock at that point, letting a second update rewrite the unit
mid-teardown. Release the lock only on the failure path, after the
rollback has restored the previous unit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@t3dotgg
t3dotgg merged commit ef4ec2a into main Jul 31, 2026
17 checks passed
@t3dotgg
t3dotgg deleted the fix/self-update-restart-race branch July 31, 2026 10:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:S 10-29 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant