[TakGN]

Part of libp2p/go-libp2p#3564 and ipfs/kubo#11480 (the inbound-slot leak that #10 and #11 gave headroom for).

Why

The go-libp2p fix, libp2p/go-libp2p#3568, is merged to master (da148e11) but is in no release yet. Upstream tested it with synthetic loopback traffic and asked operators who see the leak to try it on production nodes, to find out whether it is the whole leak.

Our content kubos still leak, so a restart is due anyway. Leaked inbound slots on 2026-11-11, measured as system − transient − live inbound, out of a 65536 limit:

	sing-1	lith-1	chic-1
leaked slots	29,005	29,529	32,272
per day since 2026-10-28	~775	~765	~840

Running one box with the fix and two without gives a clean comparison under the same traffic. On the old build, a box leaks about 30 slots an hour, so a few hours already show which is which.

The bootstrappers see far more short-lived connections from DHT clients, so they are the more demanding test: if the fix misses a leak path, it should show there. chic-1 runs two bootstrappers, so bootstrap-b gets the fix and bootstrap on the same box is the control. Since their restart on 2026-11-02, the bootstrappers have leaked about 400–690 slots a day each, far below the 262144 limit, so this restart is only for the test.

The image

ghcr.io/ipni/kubo:v0.43.1-rcmgr-fix (sha256:44f66498…) is ipfs/kubo v0.43.1 (acc53c0) built by kubo's own Dockerfile. The only change is to go-libp2p:

• kubo v0.43.1 does not use a go-libp2p release. Its go.mod replaces go-libp2p with lidel/go-libp2p@9c4ee60 (v0.49.0 plus a quic-go bump).
• The image uses that same commit plus the five lines of #3568 in p2p/net/upgrader/listener.go. The surrounding code is identical to upstream, and the new upstream test TestAcceptDoesNotBlockOnClose fails without the fix and passes with it.
• go version -m on the binary matches the deployed upstream image except for the go-libp2p line: same Go 1.26.5, same CGO setting, same build settings and dependencies. The version string is unchanged (0.43.1-acc53c0), so the box and the image digest are what tell the canary apart.
• It is amd64 only and was pushed by hand. Anyone can pull it without logging in. The build recipe is not in a repo yet.

Change

• k8s/content/kustomization.yaml → kustomization.yaml.j2: the kubo and ipfs-cluster images come from role variables.
• roles/content/defaults/main.yml: content_kubo_image and content_cluster_image, with the same digests and comments as before.
• host_vars/sing-1/content.yml: sing-1 overrides content_kubo_image with the canary. lith-1 and chic-1 stay on upstream v0.43.1 as the control.
• scripts/recover-upstream-pinset.sh: reads the images from the role defaults instead of the kustomization, so it always uses upstream kubo.
• k8s/bootstrap/kustomization.yaml.j2, roles/bootstrap/defaults/main.yml, roles/bootstrap/tasks/main.yml: a bootstrapper instance can set its own kubo_image in its bootstrap_extra_instances entry. The default (bootstrap_default_kubo_image) is the same v0.43.1 digest as before.
• host_vars/chic-1/bootstrap.yml: bootstrap-b runs the canary.
• README.md: a short Images note under "Content cluster node".

Checked

• scripts/render-manifests.yml, built with kubectl kustomize: the default render of content is byte-identical to main, and no other directory changes. With sing-1's vars, only the two kubo image: lines change (the init container and the daemon).
• Each bootstrapper instance's kustomization, rendered through bootstrap_instances the way the role does: only chic-1 bootstrap-b gets ghcr.io/ipni/kubo; sing-1, lith-1 and chic-1 bootstrap keep ipfs/kubo@sha256:b293…. scripts/bootstrap-dns.py prints the same records as on main.
• The image keeps kubo's peerlog plugin, which ProbeLab's data comes from (compiled in, as upstream).
• ansible-playbook --syntax-check content.yml, plus a YAML parse of the changed files and the script's image lookup.
• Not run locally: yamllint and ansible-lint. CI runs them.

After merging

Merging deploys nothing; content.yml and bootstrap.yml are run from a controller.

Since #38 merged, main matches the content boxes again: a dry run of content.yml from #38 on all three boxes left the gateway, the pinning proxy, the Secrets and the certificates unchanged. It does put back what the 2026-10-21 deploy had rolled back from #9 and #10 (ipfs-cluster metrics, IPFS_FD_MAX and MaxFileDescriptors 131072), so the first run restarts each content kubo once. On sing-1 that restart also brings in the canary.

bootstrap-b on chic-1 has run this branch since 2026-11-11 15:35 UTC (bootstrap.yml -l chic-1 from the branch). Until this merges, a bootstrap.yml run on chic-1 from main would put it back on upstream kubo.

ansible-playbook content.yml -l sing-1 --check --diff
ansible-playbook content.yml -l sing-1
ansible-playbook content.yml        # no change on lith-1/chic-1; checks peering

ansible-playbook bootstrap.yml -l chic-1 --check --diff
ansible-playbook bootstrap.yml -l chic-1

The rollout restarts sing-1's content pod, which also clears its leaked slots. Then compare the leaked-slot gauge across the three boxes for 24–48 hours (the ContentInboundSlotsLeaking expression in ipni/infra) and report back upstream.

Rollback: remove the override from host_vars/sing-1/content.yml and run content.yml -l sing-1, or remove kubo_image from bootstrap-b and run bootstrap.yml -l chic-1. Remove it too once a kubo release has the fix.

🤖 Generated with Claude Code

[TakGN]

@claude review opus

[github-actions]

Claude review failed to run. See https://github.com/ipni/workers/actions/runs/43038176591

[github-actions]

Claude review failed to run. See https://github.com/ipni/workers/actions/runs/43038176591