Skip to content

Gateway origin for dist.ipfs.tech, POST /add on the pinning API - #38

Merged
TakGN merged 5 commits into
mainfrom
content-gateway-and-cluster-add
Nov 11, 2026
Merged

TakGN merged 5 commits into
mainfrom
content-gateway-and-cluster-add

Conversation

@nymd

@nymd nymd commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Commits

  1. Use the default callback with YAML results. ansible.cfg replaces the deprecated yaml stdout callback with default + callback_result_format = yaml.
  2. POST /add on pin.ipni.io:9443. This is for ipfs-cluster-ctl add, which ipfs-deploy-action runs. Envoy sends only POST /add to the cluster REST API on loopback 9094, behind HTTP basic auth.
    • The basic-auth users are vault_content_pin_users. The role computes their SHA1 htpasswd entries on the controller.
    • Other routes are unchanged; ipfs-cluster still authenticates them.
    • The stream idle timeout goes from 60s to 300s so slow CAR uploads finish.
    • Each deploy checks: anonymous POST /add gets 401; PUT/DELETE /add, POST /peers and POST /pins/<cid> get 403.
  3. Let dist.ipfs.tech past the size gate. dist.ipfs.tech is ~128 GB and the claim is 100Gi, but local-path neither enforces nor expands the claim. The real limit is the NVMe (~820 GB free per box). The entry is pinned by hand with ipfs-cluster-ctl.
  4. Read-only gateway origin at dist-origin.ipfs.io:2053. kubo's gateway runs on 127.0.0.1:8181 with NoFetch and NoDNSLink. A new Envoy (k8s/content-gateway) publishes it on every box as the origin behind the dist.ipfs.tech Cloudflare Worker.
    • Only GET/HEAD under /ipfs/ get through; everything else gets 403.
    • TLS comes from cert-manager by DNS-01 in the ipfs.io zone.
    • Ports 2053, 8181 and 9905 are reserved in group_vars/managed_boxes; 2053/tcp is opened.
  5. Render placeholder. CI's scripts/render-manifests.yml has no vault, so it gets one placeholder entry for content_pin_basic_users.

Checked locally

  • --syntax-check passes on every playbook.
  • yamllint and ansible-lint report no errors in the changed files.
  • scripts/render-manifests.yml renders all manifests and kubectl kustomize builds every directory.
  • envoy --mode validate passes on the rendered content-pin and content-gateway configs, using the pinned distroless-v1.39.1 image.

Before deploying

  • Commit 4 uses vault_cloudflare_dns_token_ipfs_io, which is already in group_vars/ipfs_nodes/vault.yml. On 2026-11-11 Cloudflare reported the token active and able to see the ipfs.io zone. DNS edit permission isn't confirmed yet; the first certificate issuance will show whether it has it.
  • Turning on kubo's gateway and NoFetch changes the kubo config, so the content node restarts.

🤖 Generated with Claude Code

nymd and others added 5 commits November 11, 2026 09:03
community.general's yaml stdout callback is deprecated; ansible-core's default
callback prints results as YAML with callback_result_format = yaml.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ipfs-deploy-action uploads with `ipfs-cluster-ctl add`, which speaks
ipfs-cluster's REST API, not the Pinning Service API. Envoy on pin.ipni.io:9443
now routes POST /add, and only that, to the REST API on loopback 9094.

- The REST API has no auth of its own, so Envoy checks HTTP basic auth on that
  route against the same vault_content_pin_users. The role hashes them to
  htpasswd SHA1 entries on the controller, sorted so an unchanged user list
  does not restart the proxy; passwords go through the environment only.
- Every other route disables basic auth; ipfs-cluster still authenticates them.
- The stream idle timeout goes from 60s to 300s and /add has no route timeout,
  so a slow CAR upload finishes.
- Every deploy checks that POST /add without credentials gets 401 and that
  PUT/DELETE /add, POST /peers and POST /pins/<cid> get 403.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dist.ipfs.tech (~128 GB) is larger than the 100Gi content-repo claim, but the
claim is only a label: k3s's local-storage class cannot expand volumes and
does not enforce the size, so the NVMe under /data is the real bound (about
820 GB free per box on 2026-10-21). The entry is allowed by name and pinned
directly with ipfs-cluster-ctl, not through the playbook's pin timeout.
pvc.yaml now says why the size is not raised.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The content kubo's gateway comes on at 127.0.0.1:8181 with NoFetch (it serves
only blocks the node already holds) and NoDNSLink. Envoy (k8s/content-gateway)
publishes it over TLS on every box as the origin behind the dist.ipfs.tech
Cloudflare Worker, passing only GET and HEAD under /ipfs/ and refusing the rest
with 403.

- 2053 is a port Cloudflare can reach as an origin (8443 is taken on lith-1).
- The certificate comes from cert-manager by DNS-01 in the ipfs.io zone, with
  its own issuers and a new vault secret, vault_cloudflare_dns_token_ipfs_io:
  the ipni.io token cannot see ipfs.io. The role refuses to deploy without it.
- 2053, 8181 and the Envoy admin port 9905 are reserved; 2053/tcp is open.
- Every deploy checks from the controller that health answers 200 over TLS and
  that POST /ipfs/, /ipns/ and /api/ get 403.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…th users

k8s/content-pin/envoy.yaml.j2 now lists content_pin_basic_users, which a deploy
hashes from the vault. scripts/render-manifests.yml (CI's manifests job) has no
vault, so it could not render the template; it now passes one placeholder
entry in the same htpasswd SHA1 form.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Claude review failed to run. See https://github.com/ipni/workers/actions/runs/43055174035

@TakGN
TakGN merged commit dc56ae3 into main Nov 11, 2026
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants