From 85964c426cae4dbc4b09a02c808b7f14be6da83b Mon Sep 17 00:00:00 2001 From: Takwa Date: Wed, 11 Nov 2026 14:51:14 +0100 Subject: [PATCH 1/3] common: make unattended upgrades actually run on every box Cherry's images stopped unattended-upgrades on both groups, silently. Ubuntu (ipfs_nodes): the image ships /etc/apt/apt.conf.d/docker-disable-periodic-update with APT::Periodic::Enable "0". It sorts after 20auto-upgrades, so apt-daily and apt-daily-upgrade did nothing. Remove it, and assert afterwards via apt-config that periodic apt is on and Unattended-Upgrade is 1. Debian (indexers): Cherry's mirror rewrites Label:, so Debian's default label= patterns match nothing. Match the release, -security and -updates by origin and codename instead. needrestart in list-only mode moves from reboot_reporting.yml to every box, so the first unattended run on the IPFS boxes restarts nothing beyond packages' own postinst restarts. Co-Authored-By: Claude Opus 5.5 --- README.md | 39 +++++++++++---- group_vars/indexers/main.yml | 20 +++++--- roles/common/defaults/main.yml | 5 +- roles/common/tasks/main.yml | 63 ++++++++++++++++++++++++- roles/common/tasks/reboot_reporting.yml | 17 +------ 5 files changed, 110 insertions(+), 34 deletions(-) diff --git a/README.md b/README.md index dd28134..047098b 100644 --- a/README.md +++ b/README.md @@ -336,12 +336,32 @@ would fail until the box is built and pinned. If root takes neither your key nor ### Package updates and reboots -| | Policy on an indexer | -|---|---| -| What updates | `unattended-upgrades`, daily: Debian stable, its security archive and `-updates`. Never backports. | -| Reboots | Never automatic. | -| Services | Packages that restart their own service on upgrade still do (ssh, chrony, fail2ban). Established SSH sessions survive. k3s is not an apt package, so apt never restarts k3s or containers. `needrestart` only lists what is left running on old libraries. | -| Kernels | Installed but inactive until a reboot. | +| | IPFS boxes (Ubuntu) | Indexers (Debian) | +|---|---|---| +| What updates | `unattended-upgrades`, daily: `noble-security` only, Ubuntu's default. | `unattended-upgrades`, daily: Debian stable, its security archive and `-updates`. Never backports. | +| Reboots | Never automatic. | Never automatic. | +| Services | Packages that restart their own service on upgrade still do (ssh, chrony, fail2ban). Established SSH sessions survive. k3s is not an apt package, so apt never restarts k3s or containers. `needrestart` only lists what is left running on old libraries. | Same. | +| Kernels | Installed but inactive until a reboot. | Same. | + +Two things on Cherry's images silently stopped this, both found on 2026-11-11: + +- **Ubuntu: periodic apt switched off.** The image ships + `/etc/apt/apt.conf.d/docker-disable-periodic-update` with + `APT::Periodic::Enable "0"`. It sorts after everything of ours, so the + timers fired and did nothing. `common` deletes it. +- **Debian: relabelled suites.** Cherry's mirror rewrites `Label:` (to + `debian trixie-security` and so on), so Debian's default + `label=Debian-Security` patterns matched nothing and unattended-upgrades + reported "No packages found" every day. `group_vars/indexers` matches by + origin and codename instead, which hold on Cherry's mirror and on Debian's. + +`common` then asks apt what it settled on (`apt-config shell` for +`APT::Periodic::Enable` and `APT::Periodic::Unattended-Upgrade`) and fails the +run if periodic apt is off or unattended upgrades are not on, the way +`hardening` checks `sshd -T`. The check is skipped in a dry run. It cannot see +an origin pattern that matches nothing; for that, look at +`/var/log/unattended-upgrades/unattended-upgrades.log` and +`apt list --upgradable` on the box. A pending reboot shows in three places: `/var/run/reboot-required`, the `reboot:` line in every `site.yml` summary, and the `node_reboot_required` metric, which an @@ -349,9 +369,12 @@ hourly timer writes to `/var/lib/node_exporter/textfile/reboot.prom`. The metric reaches Prometheus only once ipni/infra's node-exporter is given that directory (`--collector.textfile.directory`, with a host mount) and an alert is added. -To reboot: one box at a time. Every container restarts. +The metric is written on indexers only (`common_reboot_reporting`); the IPFS +boxes have the marker file and the summary line. -The IPFS boxes keep their existing update policy. They only gain the `reboot:` summary line. +To reboot: one box at a time. Every container restarts, needle included, and +needle stays degraded for tens of minutes after it is Ready, so reboot an IPFS +box only once needle is warm on the other two. ## needle diff --git a/group_vars/indexers/main.yml b/group_vars/indexers/main.yml index 146bfc4..2084e19 100644 --- a/group_vars/indexers/main.yml +++ b/group_vars/indexers/main.yml @@ -11,15 +11,23 @@ hardening_removed_accounts: - ubuntu - debian -# Point releases as well as security fixes, so packages keep up with Debian -# stable and not only with its advisories. Debian's own defaults already allow -# the release itself and Debian-Security; this adds -updates. Backports are -# never matched. See "Package updates and reboots" in the README. +# The release itself, its security archive and its point releases (-updates), +# so packages keep up with Debian stable and not only with its advisories. +# Backports are never matched. +# +# Matched by origin and codename only. Debian's own defaults also require +# label=Debian / label=Debian-Security, and Cherry's mirror rewrites the labels +# (`Label: debian trixie-security`, `Codename: trixie-security`), so on +# indexers-chic1 they matched nothing: unattended-upgrades logged "No packages +# found" daily with 9 trixie-security updates pending (2026-11-11). Origin and +# codename hold on Cherry's mirror and on Debian's own. See "Package updates +# and reboots" in the README. common_unattended_extra_origins: + - "origin=Debian,codename=${distro_codename}" + - "origin=Debian,codename=${distro_codename}-security" - "origin=Debian,codename=${distro_codename}-updates" -# needrestart reports and restarts nothing, and the reboot state goes to a -# file node-exporter can read. +# The reboot state goes to a file node-exporter can read. common_reboot_reporting: true # Cherry's Debian image ships 50-cloud-init.conf with PasswordAuthentication diff --git a/roles/common/defaults/main.yml b/roles/common/defaults/main.yml index 1db15a9..e3cd75c 100644 --- a/roles/common/defaults/main.yml +++ b/roles/common/defaults/main.yml @@ -46,8 +46,9 @@ common_distro_packages: # Origins allowed on top of the distribution's own 50unattended-upgrades. common_unattended_extra_origins: [] -# needrestart in list-only mode, and a metric saying a reboot is pending. Off on -# the IPFS boxes, whose behaviour this must not change. +# The node_reboot_required metric (an hourly timer writing a textfile for +# node-exporter). needrestart's list-only mode is not part of this: every box +# gets it. Off on the IPFS boxes, whose node-exporter does not read the file. common_reboot_reporting: false # node-exporter reads *.prom from here once it is given diff --git a/roles/common/tasks/main.yml b/roles/common/tasks/main.yml index 5782920..050a6fc 100644 --- a/roles/common/tasks/main.yml +++ b/roles/common/tasks/main.yml @@ -199,8 +199,10 @@ APT::Periodic::Unattended-Upgrade "1"; APT::Periodic::AutocleanInterval "7"; -# With the defaults this writes what it always has: the policy on the IPFS boxes -# is unchanged. An indexer widens the origins. +# With the defaults this keeps the distribution's own allowed origins, which on +# Ubuntu (Cherry's mirror included) is -security only. An indexer adds +# origin-and-codename patterns, because Debian's label-based defaults match +# nothing on Cherry's mirror: see group_vars/indexers. - name: Configure unattended-upgrades to security only, no auto-reboot ansible.builtin.copy: dest: /etc/apt/apt.conf.d/51-ipfs-unattended @@ -220,6 +222,63 @@ }; {% endif %} +# Cherry's Ubuntu image ships this file with `APT::Periodic::Enable "0"`. It +# belongs to no package and is not ours, and it sorts after 20auto-upgrades and +# 99*, so it wins: apt-daily and apt-daily-upgrade start and exit at once, and +# nothing above ever runs. Found 2026-11-11 on sing-1, lith-1 and chic-1: no +# /var/log/unattended-upgrades at all, package lists weeks old, openssh-server +# one security release behind and 72-80 -security upgrades pending on each. +# Removed rather than overridden, so no file of ours has to sort after it. +- name: Remove the provider's switch that turns periodic apt off + ansible.builtin.file: + path: /etc/apt/apt.conf.d/docker-disable-periodic-update + state: absent + +# A package upgrade on a box can restart services through needrestart. `l` only +# lists them: k3s and everything it runs restart when a person decides, not at +# 06:00 on every box at once. Set before the first unattended run picks it up. +- name: Install needrestart + ansible.builtin.apt: + name: needrestart + state: present + +- name: Keep needrestart to listing what needs a restart + ansible.builtin.copy: + dest: /etc/needrestart/conf.d/50-ipni.conf + mode: "0644" + content: | + # Managed by Ansible. + $nrconf{restart} = 'l'; + +# apt reads every file under apt.conf.d and the last value wins, so a file +# someone else drops in can quietly switch the timers off again, as above. Ask +# apt what it settled on. Skipped in a dry run, where the removal has not +# happened and the answer would still be the old one. +- name: Read the periodic apt settings apt is actually using + ansible.builtin.command: + argv: + - apt-config + - shell + - enable + - APT::Periodic::Enable + - unattended + - APT::Periodic::Unattended-Upgrade + register: apt_periodic_effective + changed_when: false + check_mode: false + when: not ansible_check_mode + +- name: Fail unless unattended upgrades really run + ansible.builtin.assert: + that: + - "\"enable='0'\" not in apt_periodic_effective.stdout_lines" + - "\"unattended='1'\" in apt_periodic_effective.stdout_lines" + fail_msg: >- + {{ inventory_hostname }}: another file under /etc/apt/apt.conf.d turns + periodic apt off; apt-config shows {{ apt_periodic_effective.stdout_lines }}. + quiet: true + when: not ansible_check_mode + - name: Report pending reboots and restarts ansible.builtin.import_tasks: reboot_reporting.yml when: common_reboot_reporting | bool diff --git a/roles/common/tasks/reboot_reporting.yml b/roles/common/tasks/reboot_reporting.yml index c6b6854..4392316 100644 --- a/roles/common/tasks/reboot_reporting.yml +++ b/roles/common/tasks/reboot_reporting.yml @@ -1,22 +1,7 @@ --- # Nothing here reboots or restarts anything. It makes "this box needs a reboot" # something a person or an alert can see. - -- name: Install needrestart - ansible.builtin.apt: - name: needrestart - state: present - -# The default in a non-interactive apt run can be to restart services. `l` only -# lists them. Restarting k3s would be survivable, but this box decides nothing -# on its own. -- name: Keep needrestart to listing what needs a restart - ansible.builtin.copy: - dest: /etc/needrestart/conf.d/50-ipni.conf - mode: "0644" - content: | - # Managed by Ansible. - $nrconf{restart} = 'l'; +# needrestart itself, in list-only mode, is installed on every box in main.yml. - name: Create the node-exporter textfile directory ansible.builtin.file: From 1054476770946574fb3ed068f9c36509954153d3 Mon Sep 17 00:00:00 2001 From: Takwa Date: Wed, 11 Nov 2026 14:52:36 +0100 Subject: [PATCH 2/3] common: keep per-box package state out of comments Co-Authored-By: Claude Opus 5.5 --- group_vars/indexers/main.yml | 6 +++--- roles/common/tasks/main.yml | 6 ++---- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/group_vars/indexers/main.yml b/group_vars/indexers/main.yml index 2084e19..07c76eb 100644 --- a/group_vars/indexers/main.yml +++ b/group_vars/indexers/main.yml @@ -17,9 +17,9 @@ hardening_removed_accounts: # # Matched by origin and codename only. Debian's own defaults also require # label=Debian / label=Debian-Security, and Cherry's mirror rewrites the labels -# (`Label: debian trixie-security`, `Codename: trixie-security`), so on -# indexers-chic1 they matched nothing: unattended-upgrades logged "No packages -# found" daily with 9 trixie-security updates pending (2026-11-11). Origin and +# (`Label: debian trixie-security`, `Codename: trixie-security`), so they +# matched nothing: unattended-upgrades logged "No packages found" daily +# (found 2026-11-11). Origin and # codename hold on Cherry's mirror and on Debian's own. See "Package updates # and reboots" in the README. common_unattended_extra_origins: diff --git a/roles/common/tasks/main.yml b/roles/common/tasks/main.yml index 050a6fc..b9d1841 100644 --- a/roles/common/tasks/main.yml +++ b/roles/common/tasks/main.yml @@ -225,10 +225,8 @@ # Cherry's Ubuntu image ships this file with `APT::Periodic::Enable "0"`. It # belongs to no package and is not ours, and it sorts after 20auto-upgrades and # 99*, so it wins: apt-daily and apt-daily-upgrade start and exit at once, and -# nothing above ever runs. Found 2026-11-11 on sing-1, lith-1 and chic-1: no -# /var/log/unattended-upgrades at all, package lists weeks old, openssh-server -# one security release behind and 72-80 -security upgrades pending on each. -# Removed rather than overridden, so no file of ours has to sort after it. +# nothing above ever runs. Found 2026-11-11 on the IPFS boxes: no +# /var/log/unattended-upgrades at all and package lists weeks old. Removed rather than overridden, so no file of ours has to sort after it. - name: Remove the provider's switch that turns periodic apt off ansible.builtin.file: path: /etc/apt/apt.conf.d/docker-disable-periodic-update From 882ce1755a10bf366fa883a426b655a96933c5e1 Mon Sep 17 00:00:00 2001 From: Takwa Date: Thu, 12 Nov 2026 15:18:54 +0100 Subject: [PATCH 3/3] common: list-only needrestart first, check the list refresh and timers Review on #36: - Install needrestart and its list-only setting before anything turns unattended upgrades on, so no run in between, and no failed play, leaves Ubuntu's restart-by-default policy in charge. - Keep apt-daily.timer and apt-daily-upgrade.timer enabled and running, and make the guard also require APT::Periodic::Update-Package-Lists "1", so upgrades never run on stale lists. - README: document Ubuntu's inherited allowlist (noble, noble-security and the ESM security origins, which apply only under Ubuntu Pro). Co-Authored-By: Claude Opus 5.5 --- README.md | 14 +++++--- roles/common/tasks/main.yml | 65 ++++++++++++++++++++++++------------- 2 files changed, 51 insertions(+), 28 deletions(-) diff --git a/README.md b/README.md index 047098b..3221c49 100644 --- a/README.md +++ b/README.md @@ -338,7 +338,7 @@ would fail until the box is built and pinned. If root takes neither your key nor | | IPFS boxes (Ubuntu) | Indexers (Debian) | |---|---|---| -| What updates | `unattended-upgrades`, daily: `noble-security` only, Ubuntu's default. | `unattended-upgrades`, daily: Debian stable, its security archive and `-updates`. Never backports. | +| What updates | `unattended-upgrades`, daily, with Ubuntu's default allowlist: `noble`, `noble-security`, and the ESM security origins `noble-apps-security` and `noble-infra-security`. In practice that is security fixes: `noble` is the release pocket, which does not change after release, and the ESM origins only apply on a box attached to Ubuntu Pro, which these are not. Never `noble-updates`. | `unattended-upgrades`, daily: Debian stable, its security archive and `-updates`. Never backports. | | Reboots | Never automatic. | Never automatic. | | Services | Packages that restart their own service on upgrade still do (ssh, chrony, fail2ban). Established SSH sessions survive. k3s is not an apt package, so apt never restarts k3s or containers. `needrestart` only lists what is left running on old libraries. | Same. | | Kernels | Installed but inactive until a reboot. | Same. | @@ -355,10 +355,14 @@ Two things on Cherry's images silently stopped this, both found on 2026-11-11: reported "No packages found" every day. `group_vars/indexers` matches by origin and codename instead, which hold on Cherry's mirror and on Debian's. -`common` then asks apt what it settled on (`apt-config shell` for -`APT::Periodic::Enable` and `APT::Periodic::Unattended-Upgrade`) and fails the -run if periodic apt is off or unattended upgrades are not on, the way -`hardening` checks `sshd -T`. The check is skipped in a dry run. It cannot see +`common` keeps `apt-daily.timer` (refreshes the lists) and +`apt-daily-upgrade.timer` (installs) enabled and running. It then asks apt what +it settled on (`apt-config shell` for `APT::Periodic::Enable`, +`APT::Periodic::Update-Package-Lists` and `APT::Periodic::Unattended-Upgrade`) +and fails the run if periodic apt is off, the lists are not refreshed daily or +unattended upgrades are not on, the way `hardening` checks `sshd -T`. needrestart's +list-only setting goes in before any of this, so no unattended run on a box ever +uses Ubuntu's restart-everything default. The check is skipped in a dry run. It cannot see an origin pattern that matches nothing; for that, look at `/var/log/unattended-upgrades/unattended-upgrades.log` and `apt list --upgradable` on the box. diff --git a/roles/common/tasks/main.yml b/roles/common/tasks/main.yml index b9d1841..912b083 100644 --- a/roles/common/tasks/main.yml +++ b/roles/common/tasks/main.yml @@ -190,6 +190,24 @@ # --------------------------------------------------------------------------- # Automatic security updates # --------------------------------------------------------------------------- +# A package upgrade can restart services through needrestart, and Ubuntu 24.04's +# default in a non-interactive run is to restart them. `l` only lists them: k3s +# and everything it runs restart when a person decides, not at 06:00 on every +# box at once. First in this section, so it is in place before anything below +# turns unattended upgrades on, and stays in place if the play fails later. +- name: Install needrestart + ansible.builtin.apt: + name: needrestart + state: present + +- name: Keep needrestart to listing what needs a restart + ansible.builtin.copy: + dest: /etc/needrestart/conf.d/50-ipni.conf + mode: "0644" + content: | + # Managed by Ansible. + $nrconf{restart} = 'l'; + - name: Enable unattended security upgrades ansible.builtin.copy: dest: /etc/apt/apt.conf.d/20auto-upgrades @@ -199,10 +217,11 @@ APT::Periodic::Unattended-Upgrade "1"; APT::Periodic::AutocleanInterval "7"; -# With the defaults this keeps the distribution's own allowed origins, which on -# Ubuntu (Cherry's mirror included) is -security only. An indexer adds -# origin-and-codename patterns, because Debian's label-based defaults match -# nothing on Cherry's mirror: see group_vars/indexers. +# With the defaults this keeps the distribution's own allowed origins. On Ubuntu +# that is noble, noble-security and the two ESM security origins (see "Package +# updates and reboots" in the README). An indexer adds origin-and-codename +# patterns, because Debian's label-based defaults match nothing on Cherry's +# mirror: see group_vars/indexers. - name: Configure unattended-upgrades to security only, no auto-reboot ansible.builtin.copy: dest: /etc/apt/apt.conf.d/51-ipfs-unattended @@ -226,31 +245,28 @@ # belongs to no package and is not ours, and it sorts after 20auto-upgrades and # 99*, so it wins: apt-daily and apt-daily-upgrade start and exit at once, and # nothing above ever runs. Found 2026-11-11 on the IPFS boxes: no -# /var/log/unattended-upgrades at all and package lists weeks old. Removed rather than overridden, so no file of ours has to sort after it. +# /var/log/unattended-upgrades at all and package lists weeks old. Removed +# rather than overridden, so no file of ours has to sort after it. - name: Remove the provider's switch that turns periodic apt off ansible.builtin.file: path: /etc/apt/apt.conf.d/docker-disable-periodic-update state: absent -# A package upgrade on a box can restart services through needrestart. `l` only -# lists them: k3s and everything it runs restart when a person decides, not at -# 06:00 on every box at once. Set before the first unattended run picks it up. -- name: Install needrestart - ansible.builtin.apt: - name: needrestart - state: present - -- name: Keep needrestart to listing what needs a restart - ansible.builtin.copy: - dest: /etc/needrestart/conf.d/50-ipni.conf - mode: "0644" - content: | - # Managed by Ansible. - $nrconf{restart} = 'l'; +# apt-daily refreshes the package lists, apt-daily-upgrade installs. Both ship +# enabled, but nothing above helps if either is masked or stopped. +- name: Keep the apt timers enabled and running + ansible.builtin.systemd: + name: "{{ item }}" + enabled: true + state: started + loop: + - apt-daily.timer + - apt-daily-upgrade.timer # apt reads every file under apt.conf.d and the last value wins, so a file -# someone else drops in can quietly switch the timers off again, as above. Ask -# apt what it settled on. Skipped in a dry run, where the removal has not +# someone else drops in can quietly switch this off again, as above: all of it, +# or just the list refresh, which would leave upgrades running on stale lists. +# Ask apt what it settled on. Skipped in a dry run, where the removal has not # happened and the answer would still be the old one. - name: Read the periodic apt settings apt is actually using ansible.builtin.command: @@ -259,6 +275,8 @@ - shell - enable - APT::Periodic::Enable + - update + - APT::Periodic::Update-Package-Lists - unattended - APT::Periodic::Unattended-Upgrade register: apt_periodic_effective @@ -266,10 +284,11 @@ check_mode: false when: not ansible_check_mode -- name: Fail unless unattended upgrades really run +- name: Fail unless the package lists refresh and unattended upgrades run ansible.builtin.assert: that: - "\"enable='0'\" not in apt_periodic_effective.stdout_lines" + - "\"update='1'\" in apt_periodic_effective.stdout_lines" - "\"unattended='1'\" in apt_periodic_effective.stdout_lines" fail_msg: >- {{ inventory_hostname }}: another file under /etc/apt/apt.conf.d turns