common: take Ubuntu's security pocket from security.ubuntu.com

README.md

@@ -367,6 +367,59 @@ an origin pattern that matches nothing; for that, look at
 `/var/log/unattended-upgrades/unattended-upgrades.log` and
 `apt list --upgradable` on the box.
 
+**Where the IPFS boxes' packages come from.** `common` writes
+`/etc/apt/sources.list.d/ubuntu.sources` (`templates/ubuntu.sources.j2`): the
+release, `-updates` and `-backports` from Cherry's mirror, signed with
+Cherry's key, as the image has them, and `<release>-security` from
+`security.ubuntu.com`, signed with Ubuntu's own key (`ubuntu-keyring`), as on a
+stock Ubuntu install. `repo.cherryservers.com` resolves to a mirror inside each
+datacenter, and one of them stopped updating for weeks. `apt-get update` kept
+"hitting" the old index without an error, so unattended-upgrades on that box
+quietly found nothing to install while the other boxes kept getting fixes.
+Taking the security pocket from upstream means a fix never waits on a mirror.
+It costs a little outbound traffic. If `security.ubuntu.com` is unreachable,
+unattended-upgrades finds nothing new that night and catches up on the next
+run, but a `site.yml` run stops on the box where the refresh fails (after
+three tries) rather than carrying on without a security index.
+
+A security update that needs a version only `-updates` has stays pending:
+unattended-upgrades does not install from `-updates` (see the table above),
+whether or not the mirror has it. Upgrade that one by hand. Ubuntu builds
+security updates against the release and security pockets, so this is rare.
+
+The URIs and keyrings are `common_apt_*` in `roles/common/defaults/main.yml`.
+The indexers keep their Debian image's sources.
+
+**How old the package lists are.** An apt hook writes two timestamps to
+`/var/lib/node_exporter/textfile/apt-lists.prom` after every package-list
+refresh, on every box:
+
+| Metric | What it is | Old means |
+|---|---|---|
+| `node_apt_security_lists_timestamp_seconds` | When the newest `<release>-security` index on the box was published (the `Date:` of its `InRelease`; 0: none). | The box's security source is stale or unreachable, or the archive has simply published nothing new. |
+| `node_apt_lists_refresh_timestamp_seconds` | The last refresh in which at least one source answered (`common_apt_refresh_stamp`, touched by apt's `Post-Invoke-Success`, which runs even when another source failed; 0: none yet). | The refresh is not running at all, or no source answers. One unreachable source does not show here. |
+
+They are timestamps rather than ages, so they keep getting older when the
+lists stop refreshing; the age is `time() - <metric>`. A quiet archive also
+makes the first one grow, and how quiet `noble-security` can get has not been
+measured yet. Three alerts between them cover the failures:
+
+- one IPFS box's security index more than a day behind another's (they all
+  read the same upstream index): one box's source is stale or unreachable;
+- the refresh timestamp more than a day or two old: the refresh has stopped;
+- the security index older than a generous fixed age, a week say: the backstop
+  for `security.ubuntu.com` being unreachable from every box at once, which
+  the first alert cannot see. Tighten it once the `Date:` has been watched for
+  a while.
+
+Like `node_reboot_required`, these reach Prometheus only once
+ipni/infra's node-exporter reads the textfile directory, and the alerts belong
+with the fleet rules there. Until then, on the box:
+
+```bash
+ansible ipfs_nodes -u ipni -m shell -a 'cat /var/lib/node_exporter/textfile/apt-lists.prom'
+```
+
 A pending reboot shows in three places: `/var/run/reboot-required`, the `reboot:`
 line in every `site.yml` summary, and the `node_reboot_required` metric, which an
 hourly timer writes to `/var/lib/node_exporter/textfile/reboot.prom`. The metric

roles/common/defaults/main.yml

@@ -54,3 +54,16 @@ common_reboot_reporting: false
 # node-exporter reads *.prom from here once it is given
 # --collector.textfile.directory (ipni/infra kubernetes/platform/node-exporter).
 common_textfile_dir: /var/lib/node_exporter/textfile
+
+# Ubuntu's package sources (tasks/apt_sources.yml): the release, -updates and
+# -backports from the provider's mirror, signed with the provider's key; the
+# security pocket from Ubuntu, signed with Ubuntu's.
+common_apt_mirror_uri: http://repo.cherryservers.com/ubuntu/
+common_apt_mirror_keyring: /usr/share/keyrings/cherryservers-signing-public.gpg
+common_apt_security_uri: http://security.ubuntu.com/ubuntu/
+common_apt_security_keyring: /usr/share/keyrings/ubuntu-archive-keyring.gpg
+common_apt_components: [main, universe, restricted, multiverse]
+
+# Touched by an apt hook after each successful package-list refresh, for
+# node_apt_lists_refresh_timestamp_seconds (tasks/apt_lists_metric.yml).
+common_apt_refresh_stamp: /var/lib/apt/periodic/ipni-update-success-stamp

roles/common/tasks/apt_lists_metric.yml

@@ -0,0 +1,87 @@
+---
+# Two timestamps, for node-exporter's textfile collector (common_textfile_dir):
+#
+# node_apt_security_lists_timestamp_seconds: when the newest security index
+#   this box has was published (the `Date:` of its <release>-security
+#   InRelease; 0: none). A mirror that stops updating still answers `apt-get
+#   update` without an error, so this is how a box that has quietly stopped
+#   seeing security fixes shows up. It also grows when the archive simply
+#   publishes nothing new, so compare it across boxes that read the same
+#   source rather than only against a fixed age.
+# node_apt_lists_refresh_timestamp_seconds: the last package-list refresh in
+#   which at least one source answered (0: none since this was installed).
+#   That is what apt's Post-Invoke-Success means: it still runs when one
+#   source fails and another answers. So old means the refresh is not running
+#   at all, or no source answers; it does not show one source failing (the
+#   security index above does).
+#
+# Timestamps rather than ages: an age is frozen at whatever it was when the
+# file was last written, so it would stop growing in exactly the case it is
+# for, a box whose lists stop refreshing. Alert on `time() - <metric>`.
+#
+# Rewritten after every package-list refresh (an apt hook) and on every run of
+# this role.
+
+- name: Create the node-exporter textfile directory
+  ansible.builtin.file:
+    path: "{{ common_textfile_dir }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Install the package-list age reporter
+  ansible.builtin.copy:
+    dest: /usr/local/sbin/ipni-apt-lists-metric
+    mode: "0755"
+    owner: root
+    group: root
+    content: |
+      #!/bin/sh
+      # Managed by Ansible. Writes node_apt_security_lists_timestamp_seconds and
+      # node_apt_lists_refresh_timestamp_seconds for node-exporter.
+      set -eu
+      newest=0
+      for f in /var/lib/apt/lists/*_dists_{{ ansible_distribution_release }}-security_InRelease; do
+        [ -e "$f" ] || continue
+        t=$(stat -c %Y "$f")
+        d=$(sed -n 's/^Date: //p' "$f" | head -n 1)
+        if [ -n "$d" ] && dt=$(date -d "$d" +%s 2>/dev/null); then t=$dt; fi
+        if [ "$t" -gt "$newest" ]; then newest=$t; fi
+      done
+      refreshed=0
+      stamp={{ common_apt_refresh_stamp }}
+      [ -e "$stamp" ] && refreshed=$(stat -c %Y "$stamp")
+      tmp="{{ common_textfile_dir }}/.apt-lists.prom.$$"
+      {
+        printf '# HELP node_apt_security_lists_timestamp_seconds When the newest security package index on this box was published (0: none).\n'
+        printf '# TYPE node_apt_security_lists_timestamp_seconds gauge\n'
+        printf 'node_apt_security_lists_timestamp_seconds %s\n' "$newest"
+        printf '# HELP node_apt_lists_refresh_timestamp_seconds When a package-list refresh last got an answer from at least one source (0: never).\n'
+        printf '# TYPE node_apt_lists_refresh_timestamp_seconds gauge\n'
+        printf 'node_apt_lists_refresh_timestamp_seconds %s\n' "$refreshed"
+      } > "$tmp"
+      chmod 0644 "$tmp"
+      mv "$tmp" "{{ common_textfile_dir }}/apt-lists.prom"
+
+# Our own stamp: Ubuntu's update-success-stamp comes from update-notifier,
+# which Debian does not have. Post-Invoke-Success touches it and reports; the
+# Post-Invoke entry reports after a refresh apt counts as failed too, when the
+# lists left behind are still worth reporting. Neither fails the refresh.
+- name: Report the package lists' age after every refresh
+  ansible.builtin.copy:
+    dest: /etc/apt/apt.conf.d/80ipni-apt-lists-metric
+    mode: "0644"
+    owner: root
+    group: root
+    content: |
+      // Managed by Ansible.
+      APT::Update::Post-Invoke-Success { "touch {{ common_apt_refresh_stamp }}; /usr/local/sbin/ipni-apt-lists-metric || true"; };
+      APT::Update::Post-Invoke { "/usr/local/sbin/ipni-apt-lists-metric || true"; };
+
+# So the file exists from the first run, not only after the next refresh.
+# Skipped in a dry run, where the reporter may not be installed yet.
+- name: Report the package lists' age now
+  ansible.builtin.command: /usr/local/sbin/ipni-apt-lists-metric
+  changed_when: false
+  when: not ansible_check_mode

roles/common/tasks/apt_sources.yml

@@ -0,0 +1,90 @@
+---
+# Ubuntu only (main.yml): where the package lists come from. The indexers keep
+# their Debian image's own sources.
+#
+# Cherry's image takes every pocket, -security included, from Cherry's mirror,
+# re-signed with Cherry's key. repo.cherryservers.com resolves to a mirror
+# inside each datacenter, and one of them stopped updating for weeks: `apt-get
+# update` kept "hitting" the old index without an error, so that box quietly
+# stopped getting security fixes while the others did not. This takes the
+# security pocket from security.ubuntu.com, signed by Ubuntu's own key, as a
+# stock Ubuntu install does, and keeps the rest on the mirror.
+#
+# The whole file is ours, so if a reinstall or cloud-init rewrites it, the next
+# run puts it back. Runs before main.yml's "Update apt cache", so everything
+# after it installs from the new lists.
+
+- name: Find the keyrings the package sources are signed with
+  ansible.builtin.stat:
+    path: "{{ item }}"
+  loop:
+    - "{{ common_apt_mirror_keyring }}"
+    - "{{ common_apt_security_keyring }}"
+  register: common_apt_keyrings
+
+# A source whose keyring is missing makes every `apt-get update` fail for it.
+- name: Fail if a keyring is missing
+  ansible.builtin.assert:
+    that: common_apt_keyrings.results | rejectattr('stat.exists') | list | length == 0
+    fail_msg: >-
+      {{ inventory_hostname }}: missing
+      {{ common_apt_keyrings.results | rejectattr('stat.exists') | map(attribute='item') | join(', ') }};
+      the security keyring comes from the ubuntu-keyring package.
+    quiet: true
+
+- name: Take the security pocket from Ubuntu and the rest from the mirror
+  ansible.builtin.template:
+    src: ubuntu.sources.j2
+    dest: /etc/apt/sources.list.d/ubuntu.sources
+    owner: root
+    group: root
+    mode: "0644"
+  register: common_apt_sources
+
+# apt names the file after the URI: security.ubuntu.com_ubuntu_dists_... It
+# adds a missing trailing slash first, so this does too.
+- name: Work out where apt keeps the security index from the configured source
+  ansible.builtin.set_fact:
+    common_apt_security_index_path: >-
+      /var/lib/apt/lists/{{ (common_apt_security_uri | regex_replace('^[a-z]+://', '') | regex_replace('/+$', '')
+      ~ '/') | replace('/', '_') }}dists_{{ ansible_distribution_release }}-security_InRelease
+
+- name: Look for the security index from the configured source
+  ansible.builtin.stat:
+    path: "{{ common_apt_security_index_path }}"
+  register: common_apt_security_index
+
+# "Update apt cache" in main.yml skips a cache less than an hour old, which
+# would leave the rest of the play on the old lists. Keyed on the index as well
+# as the file, so a run that wrote the file and then failed here refreshes on
+# the next run too. Not reported as a change, like that task: the change is the
+# file above, which a dry run shows.
+- name: Refresh the package lists from the new sources
+  ansible.builtin.apt:
+    update_cache: true
+  register: common_apt_refresh
+  retries: 3
+  delay: 10
+  until: common_apt_refresh is succeeded
+  changed_when: false
+  when: common_apt_sources is changed or not common_apt_security_index.stat.exists
+
+# apt treats a source it could not reach as a warning, not a failure, as long
+# as another one answered. So check the index is really there now, and stop
+# the run on this box if it is not, rather than carry on without it. The next
+# run refreshes again (above). Skipped in a dry run, which refreshes nothing.
+- name: Look again for the security index after the refresh
+  ansible.builtin.stat:
+    path: "{{ common_apt_security_index_path }}"
+  register: common_apt_security_index_after
+  when: not ansible_check_mode
+
+- name: Fail unless the security index from the configured source is there
+  ansible.builtin.assert:
+    that: common_apt_security_index_after.stat.exists
+    fail_msg: >-
+      {{ inventory_hostname }}: no {{ common_apt_security_index_path }} after
+      `apt-get update`; {{ common_apt_security_uri }} was probably unreachable.
+      The sources file is in place; re-run once it answers.
+    quiet: true
+  when: not ansible_check_mode

roles/common/tasks/main.yml

@@ -23,12 +23,25 @@
   when: current_tz.stdout | trim != 'UTC'
   changed_when: true
 
+# Both before the first refresh below: the hook reporting the lists' age is in
+# place for it, and it reads the sources the box should have.
+- name: Report how old the security package lists are
+  ansible.builtin.import_tasks: apt_lists_metric.yml
+
+- name: Take Ubuntu's security pocket from Ubuntu itself
+  ansible.builtin.import_tasks: apt_sources.yml
+  when: ansible_distribution == 'Ubuntu'
+
 # Refreshing the package lists changes nothing installed. Reported as a change,
 # it would be one on every run, and in every dry run an approver reads.
 - name: Update apt cache
   ansible.builtin.apt:
     update_cache: true
     cache_valid_time: 3600
+  register: apt_cache
+  retries: 3
+  delay: 10
+  until: apt_cache is succeeded
   changed_when: false
 
 # Gated: see common_apt_upgrade in defaults/main.yml for why this does not run

roles/common/templates/ubuntu.sources.j2

@@ -0,0 +1,18 @@
+# Managed by Ansible (roles/common). Changes made here are overwritten.
+#
+# The release, -updates and -backports come from the provider's mirror. The
+# security pocket comes from Ubuntu itself, as on a stock Ubuntu install, so a
+# mirror that stops updating cannot hold security fixes back. See "Package
+# updates and reboots" in the README.
+{% set release = ansible_distribution_release %}
+Types: deb
+URIs: {{ common_apt_mirror_uri }}
+Suites: {{ release }} {{ release }}-updates {{ release }}-backports
+Components: {{ common_apt_components | join(' ') }}
+Signed-By: {{ common_apt_mirror_keyring }}
+
+Types: deb
+URIs: {{ common_apt_security_uri }}
+Suites: {{ release }}-security
+Components: {{ common_apt_components | join(' ') }}
+Signed-By: {{ common_apt_security_keyring }}