[TakGN]

Why

Cherry's Ubuntu image takes every pocket, -security included, from repo.cherryservers.com, re-signed with Cherry's own key. That name resolves to a mirror inside each datacenter. One of those mirrors stopped updating for weeks. apt-get update kept "hitting" the old index without an error, so unattended-upgrades on that box quietly found nothing to install while the other boxes kept getting fixes. Nothing in the repo or in monitoring could show it.

Ubuntu's own default takes the security pocket from security.ubuntu.com, so fixes never wait on a mirror, and that pocket is signed by Ubuntu's own archive key.

What

• roles/common/tasks/apt_sources.yml, Ubuntu only: templates /etc/apt/sources.list.d/ubuntu.sources (templates/ubuntu.sources.j2).
  • <release>, -updates and -backports stay on Cherry's mirror with Cherry's keyring, exactly as now.
  • <release>-security comes from http://security.ubuntu.com/ubuntu/, Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg (from ubuntu-keyring, already on the boxes), with the same components.
  • The suite is ansible_distribution_release. URIs, keyrings and components are common_apt_* role defaults.
  • Owning the whole file means a reinstall or cloud-init rewrite is put back on the next run.
  • It fails early if either keyring is missing. It runs before the existing Update apt cache (whose cache_valid_time: 3600 would skip a recent cache). It forces a refresh when the file changed or when the upstream security index is missing, so an interrupted run fixes itself on the next one. That refresh and Update apt cache retry three times. Because apt only warns when one source fails and another answers, the role then checks that the upstream security index is really there, and stops the run on that box if it is not. A dry run shows the file's diff and changes nothing.
  • The indexers (Debian) keep their image's sources.
• roles/common/tasks/apt_lists_metric.yml, every box: two timestamps in /var/lib/node_exporter/textfile/apt-lists.prom, written by /usr/local/sbin/ipni-apt-lists-metric. Apt hooks run it after every package-list refresh, and the role runs it once per run (not in a dry run).
  • node_apt_security_lists_timestamp_seconds is when the newest security index was published: the Date: of the newest *_dists_<release>-security_InRelease (or its mtime), 0 if there is none.
  • node_apt_lists_refresh_timestamp_seconds is the last refresh apt reported as a success, from a stamp our own Post-Invoke-Success hook touches (Debian has no update-success-stamp). apt runs that hook when any source answered, so this shows a refresh that stopped running or reaches nothing. It does not show one source failing; the security index metric does.
  • They are timestamps rather than ages: an age written into a file freezes when refreshes stop, which is the case they are for. The age is time() - <metric>.
• README, "Package updates and reboots": why the security pocket comes from upstream, and the metric.

The rendered file for noble differs from what the boxes have only in the security stanza's URIs and Signed-By. The metric's logic, run read-only on the boxes, prints the age you'd expect from each one's lists.

Trade-offs

• A little outbound traffic to security.ubuntu.com per box per day.
• If upstream is unreachable, that night's unattended run finds nothing new and the next one catches up. A site.yml run stops on the box where the refresh still fails after three tries, rather than carrying on without a security index.
• unattended-upgrades never installs from -updates, so a security update that needs a version only -updates has stays pending until someone upgrades it by hand. That was already true before this change, and it is rare, since Ubuntu builds security updates against the release and security pockets.

Alerting

This repo has no alert rules: monitoring lives in ipni/infra. The IPFS boxes' node-exporter doesn't read the textfile directory yet, the same gap node_reboot_required has. Two things are needed there:

• node-exporter (kubernetes/platform/node-exporter): --collector.textfile.directory with a read-only host mount of /var/lib/node_exporter/textfile.
• Alerts next to the fleet rules (rules/fleet.yaml):
  • one IPFS box's node_apt_security_lists_timestamp_seconds more than a day behind another's (they all read the same upstream index): one box's source is stale or unreachable;
  • time() - node_apt_lists_refresh_timestamp_seconds over a day or two: the refresh has stopped;
  • time() - node_apt_security_lists_timestamp_seconds over a generous fixed age, a week say: the backstop for upstream being unreachable from every box at once. Tighten it once someone has watched how often the Date: changes, since a quiet archive makes it grow too.

Interaction with #36

This builds on #36 (merged), which turned the nightly unattended runs back on. #36 makes them run; this makes sure they read a current index. Neither depends on the other being applied first.

Rollout

site.yml from CI, one box at a time as usual. The real change on each Ubuntu box is the new ubuntu.sources, one apt-get update, a new apt hook and the metric script. Nothing is installed or restarted, and k3s and needle aren't touched.

Checks

ansible-playbook --syntax-check on all seven playbooks, yamllint ., ansible-lint (production profile, 0 failures). Nothing that renders manifests changed.

🤖 Generated with Claude Code

[claude]

Claude review

I found no blocking problems. I read the template, both new task files, the main.yml wiring, the existing unattended-upgrades config from #36 and the README text. The sources file only touches the security stanza, and the keyring check runs before the template is written. The refresh task forces an apt update only when the file changed, so the rest of the play installs from the new lists. The metric script handles the cases I traced:

• A missing glob match gives 0.
• An unparseable Date: falls back to the file mtime.
• The write is atomic.
• Ubuntu's ${distro_id}:${distro_codename}-security default origin pattern still matches packages from security.ubuntu.com, so unattended-upgrades keeps working.

I did not run any of it, and I did not touch the boxes.

No issues found. Two caveats, neither of which blocks the merge:

• Nit — roles/common/tasks/apt_lists_metric.yml:60: the metric is the Date: of the newest <release>-security InRelease. That is when the archive last published, not when the box last refreshed. Once the source is upstream, a quiet security pocket makes the age grow even though the box is current. The README's "3 days" alert threshold and its claim that the pocket gets a new index "every few days at most" are the author's assumption, and I could not confirm them from the repo. This matters most for the Debian indexers, since security.debian.org regenerates its index less often. Check how often the Date: really changes on a box before the ipni/infra alert goes live. Otherwise it may fire on a healthy box. You could also export the lists' mtime as a second metric to tell "refresh failing" from "nothing new upstream".
• Nit — roles/common/tasks/apt_sources.yml:142: this is the only task in the play that now depends on security.ubuntu.com being reachable. It has no retries, and ansible.builtin.apt with update_cache fails on a fetch failure. So a transient upstream blip during site.yml stops the one-box-at-a-time rollout at the first box. That is probably what you want, but it differs from the README's "an upstream outage only delays one night's run", which is true only for unattended-upgrades. Adding retries/until here and on Update apt cache would make it less brittle.

_{claude-sonnet-5-5 at high effort · requested by TakGN · re-run with @claude review [sonnet|opus|fable] [low|medium|high|xhigh|max]}

[TakGN]

Two findings from a static review of the APT source migration. I did not run the playbook against the fleet.

[TakGN]

[P2] Retry the APT refresh if the source migration was interrupted. The task writes ubuntu.sources before refreshing APT, and the refresh runs only when the template changes. If the refresh fails after the write, a rerun skips it. The regular cache task can also skip it if the old cache is less than an hour old, so the play can continue without an upstream security index. Also trigger the refresh when the configured upstream index is missing.

[TakGN]

Fixed in 3fc54a6. The refresh now runs when ubuntu.sources changed or when /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_<release>-security_InRelease is missing (the path is derived from common_apt_security_uri). So a run that wrote the file and then failed to refresh refreshes on the next run, whatever the cache age. It also retries three times (10 s apart), as does Update apt cache. If it still fails, the run stops on that box rather than carrying on without a security index.

[TakGN]

[P3] Correct the documented behavior for dependencies available only in -updates. The fleet’s unattended-upgrades policy excludes noble-updates, and unattended-upgrades pins disallowed origins out of dependency resolution. A security upgrade that needs a version available only in -updates can therefore remain pending after Cherry’s mirror catches up. unattended-upgrades implementation

[TakGN]

Right, thanks. unattended-upgrades never installs from -updates here, so a security update that needs a version only -updates has stays pending until someone upgrades it by hand, whether or not the mirror has caught up. Fixed in 3fc54a6 in the README and the PR description. It also says this was already true before this change, and that it is rare, since Ubuntu builds security updates against the release and security pockets.

[TakGN]

Both nits from the Claude review are addressed in 3fc54a6.

• The metric is the archive's publish time, not the last refresh. It now has a second metric, node_apt_lists_refresh_timestamp_seconds. That is the last refresh apt reported as a success, from a stamp our own Post-Invoke-Success hook touches. Debian has no update-success-stamp, so we can't rely on Ubuntu's. Both metrics go to apt-lists.prom.
  • A plain mtime metric wouldn't have helped: apt sets the InRelease mtime to the same Date:.
  • The README no longer claims how often the security index changes. The recommended alert is now one IPFS box's security index more than a day behind another's, since they all read the same upstream. A fixed age should wait until the Date: cadence has been watched.
• No retries on the forced refresh. It and Update apt cache now retry three times. The README now says an upstream outage delays one night's unattended run but stops a site.yml run on the box where the refresh still fails.

[TakGN]

Follow-up static review. The README now correctly describes -updates dependencies, and reruns retry when the upstream index is missing. Two findings remain.

[TakGN]

[P2] Verify that the upstream security index exists after refreshing. The new stat runs only before the refresh. If Cherry responds but security.ubuntu.com has a transient network failure, APT can return success with a warning. On a first rollout, the play can then continue without an upstream security index. Check for the expected index after the refresh and fail if it is still missing. APT return behavior

[TakGN]

Fixed in the latest commit. After the refresh, the role stats the upstream index again (the same path, now one set_fact) and asserts it exists. If it does not, the run stops on that box, with a message saying the sources file is in place and to re-run. The next run refreshes again because the index is missing. Both checks are skipped in a dry run, which refreshes nothing.

[TakGN]

[P2] The refresh metric can report a fresh timestamp during a partial outage. APT runs Post-Invoke-Success when any download succeeds, even if another source fails. If Cherry keeps responding while the Ubuntu security source is unreachable, the stamp keeps advancing. The README also relies on comparing security index dates across hosts, which misses an outage affecting all of them. Either measure the security source’s refresh separately or document that the stamp records partial success. APT hook behavior

[TakGN]

Agreed. Rather than measure each source on its own, I documented what the stamp is: a refresh ran and at least one source answered. That is in the task comment, the metric HELP text, the README table and the PR description. For an outage on every box, which the cross-box comparison cannot see, the README and the PR now add a third alert: the security index older than a generous fixed age (a week), as a backstop to tighten once the Date: cadence is known. One box failing alone still shows in the cross-box comparison.

[TakGN]

[P2] Normalize the security URI before checking the index path. APT accepts http://security.ubuntu.com/ubuntu and adds the trailing slash. With that override, this expression looks for ubuntudists_…, while APT writes ubuntu_dists_…. The new assertion then stops site.yml even after a successful refresh. Normalize the trailing slash before deriving the filename so both forms work. APT URI normalization

[TakGN]

Fixed in 645216d. The path now strips any trailing slashes and adds exactly one before replacing / with _. Rendered through the role's own set_fact, http://security.ubuntu.com/ubuntu/, http://security.ubuntu.com/ubuntu and https://security.ubuntu.com/ubuntu// all give security.ubuntu.com_ubuntu_dists_noble-security_InRelease. A first attempt with regex_replace('/*$', '/') doubled the slash, because Python's re.sub also matches the empty string at the end.