[TakGN]

What was broken

roles/common writes 20auto-upgrades and 51-ipfs-unattended, but no managed box was installing security updates on its own. Found with read-only checks on 2026-11-11:

ipfs_nodes (Ubuntu 24.04)

• Cherry's image ships /etc/apt/apt.conf.d/docker-disable-periodic-update with APT::Periodic::Enable "0". It belongs to no package and isn't in this repo. It sorts after 20auto-upgrades and 99*, so it wins.
• As a result, apt-daily and apt-daily-upgrade start and exit at once, /var/log/unattended-upgrades/ doesn't exist, and the package lists are weeks old. There is a sizeable backlog of -security upgrades on each box.
• The origins are fine. Cherry's Ubuntu mirror keeps Origin: Ubuntu, Suite: noble-security, so Ubuntu's default -security pattern matches. Only the Periodic switch is broken.
• needrestart had no conf.d override (default mode i).

indexers (Debian 13)

• unattended-upgrades runs but logs "No packages found" every day while trixie-security updates are pending.
• Cherry's mirror rewrites its Release files (Label: debian trixie-security, Codename: trixie-security, and likewise for trixie and trixie-updates). Debian's defaults require label=Debian / label=Debian-Security, so they match nothing. Only this repo's extra origin=Debian,codename=${distro_codename}-updates pattern matched.

What changes

Every managed box (roles/common/tasks/main.yml)

• Deletes docker-disable-periodic-update. That's cleaner than an override file of ours that would have to sort after docker-*.
• Installs needrestart with $nrconf{restart} = 'l' (/etc/needrestart/conf.d/50-ipni.conf), moved out of reboot_reporting.yml so it applies everywhere. needrestart then lists services left on old libraries and restarts none of them.
• Adds a guard in the style of hardening's sshd -T check. It runs apt-config shell enable APT::Periodic::Enable unattended APT::Periodic::Unattended-Upgrade and asserts that enable='0' is absent and unattended='1' is present. It's skipped in check mode, because in CI's dry run the file hasn't really been removed yet.

ipfs_nodes: the effect is the three changes above. Allowed origins are unchanged (Ubuntu's default, -security only).

indexers (group_vars/indexers/main.yml): match by origin and codename, which hold on Cherry's mirror and on Debian's own:

• origin=Debian,codename=${distro_codename}
• origin=Debian,codename=${distro_codename}-security
• origin=Debian,codename=${distro_codename}-updates

Backports still don't match. common_reboot_reporting now only controls the node_reboot_required metric.

README, "Package updates and reboots": the table covers both groups. It notes the two provider pitfalls, the new guard, and what the guard can't see: an origin pattern that matches nothing.

Rollout risk (please read before approving the real run)

site.yml applies from CI, one box at a time, after a required reviewer approves the dry run.

Once applied, the next apt-daily-upgrade on each box installs its whole backlog: dozens of packages on each Ubuntu box. CI applies one box at a time, but the runs finish minutes apart, so all three Ubuntu boxes may well upgrade the same morning.

• needrestart is in list mode, so it restarts nothing itself. Packages that restart their own service in postinst still do (ssh, chrony, fail2ban and similar). Established SSH sessions survive.
• k3s isn't an apt package, so apt never restarts k3s, containerd or needle.
• Kernels are installed but stay inactive until someone reboots by hand. A reboot restarts needle, which stays degraded for tens of minutes after it's Ready.

Recommendation: before merging, have an operator catch up each Ubuntu box by hand, one at a time, so the merge doesn't land a large backlog on all three at once:

1. On one box, with needrestart in list mode (create /etc/needrestart/conf.d/50-ipni.conf with $nrconf{restart} = 'l'; first): sudo apt-get update && sudo apt-get upgrade
2. Check that SSH and needle are healthy, then do the next box.
3. Plan a reboot of each box for its new kernel, one at a time, only once needle is warm on the other two.

The first unattended run after the merge then has little left to do.

Checks

• python3 -m yamllint .: clean
• python3 -m ansiblelint: Passed: 0 failure(s), 0 warning(s) ... 'production'
• ansible-playbook --syntax-check for site, routing, bootstrap, content, k3s-upgrade, platform and cve: ok
• scripts/render-manifests.yml plus kubectl kustomize on each rendered dir: ok (no manifest changes)

Nothing has been run on the boxes.

🤖 Generated with Claude Code

[TakGN]

@claude review opus

[github-actions]

Claude review failed to run. See https://github.com/ipni/workers/actions/runs/43033577328

[github-actions]

Claude review failed to run. See https://github.com/ipni/workers/actions/runs/43033577328

[TakGN]

Reviewed at 1054476. I found two rollout issues that can affect security patching and one documentation error.

1. [P2] Configure the restart policy before enabling periodic upgrades. The role removes the provider’s APT disable file before installing and configuring needrestart. If apt-daily-upgrade.timer fires in that window, Ubuntu can restart services under its default policy. The same exposure persists if the play fails after deleting the file. Install the list-only configuration before enabling unattended upgrades. Task · Ubuntu restart behavior
2. [P2] Verify that package indexes refresh and the APT timers run. The new assertion still passes if a later drop-in sets APT::Periodic::Update-Package-Lists to 0, leaving scheduled upgrades on stale indexes, or if the systemd timers are inactive. Verify the effective refresh interval and that both APT timers are active and enabled. Assertion · APT scheduling documentation
3. [P3] Document Ubuntu’s inherited allowlist. The role inherits Ubuntu’s default allowlist, which includes noble and the ESM security origins alongside noble-security. Document the inherited allowlist so operators know which sources are eligible. README · Ubuntu default origins

I reviewed the diff directly. git diff --check, the local site.yml syntax check, and GitHub’s site-dry-run passed. The dry run skips the effective-settings assertion, so that check and live timer behavior remain unverified. GitHub dry run

[TakGN]

Thanks. All three points are addressed in 882ce17:

1. Restart policy first. needrestart and /etc/needrestart/conf.d/50-ipni.conf (restart = 'l') are now the first tasks in the "Automatic security updates" section. They run before 20auto-upgrades and before the provider's switch file is deleted, so no unattended run, and no play that fails partway, is left with Ubuntu's restart-by-default policy.
2. List refresh and timers. apt-daily.timer and apt-daily-upgrade.timer are kept enabled and started. The guard now also requires APT::Periodic::Update-Package-Lists to be 1, alongside Enable not 0 and Unattended-Upgrade 1. On all three Ubuntu boxes, both timers are already enabled and active, and apt-config currently shows e='0' u='1' a='1', so the assert passes once the provider's file is gone.
3. Allowlist documented. The README table now lists Ubuntu's inherited origins: noble, noble-security, noble-apps-security and noble-infra-security. It notes that noble never changes after release and that the ESM origins apply only under Ubuntu Pro. pro status shows no box is attached, and none has ESM sources.

Checks: yamllint is clean, ansible-lint passes (0 failures, 0 warnings, production profile), --syntax-check passes on every playbook, and git diff --check is clean. The live assert still runs only in the real apply, not in CI's dry run.