Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -88,22 +88,22 @@ jobs:
kubectl kustomize "$dir" > /dev/null
done

# The instance count is a knob (someguy_instances). It is only a real one
# The instance count is a knob (needle_instances). It is only a real one
# if the Deployments, their snapshots and Envoy's endpoint list move
# together, which is the whole point of rendering them from one table.
- name: someguy instance count drives all the manifests
- name: needle instance count drives all the manifests
run: |
for count in 1 2 4; do
ansible-playbook scripts/render-manifests.yml \
-e dest="rendered-$count" -e "someguy_instances=$count" > /dev/null
someguy_build=$(kubectl kustomize "rendered-$count/someguy")
deployments=$(printf '%s\n' "$someguy_build" | grep -c '^kind: Deployment')
pvcs=$(printf '%s\n' "$someguy_build" | grep -c '^kind: PersistentVolumeClaim')
-e dest="rendered-$count" -e "needle_instances=$count" > /dev/null
needle_build=$(kubectl kustomize "rendered-$count/needle")
deployments=$(printf '%s\n' "$needle_build" | grep -c '^kind: Deployment')
pvcs=$(printf '%s\n' "$needle_build" | grep -c '^kind: PersistentVolumeClaim')
endpoints=$(kubectl kustomize "rendered-$count/route-origin" \
| grep -c 'port_value: 819')
echo "someguy_instances=$count -> $deployments Deployments, $pvcs PVCs, $endpoints endpoints"
echo "needle_instances=$count -> $deployments Deployments, $pvcs PVCs, $endpoints endpoints"
if [ "$deployments" -ne "$count" ] || [ "$pvcs" -ne "$count" ] || [ "$endpoints" -ne "$count" ]; then
echo "::error::someguy_instances=$count produced $deployments Deployments, $pvcs PVCs and $endpoints Envoy endpoints"
echo "::error::needle_instances=$count produced $deployments Deployments, $pvcs PVCs and $endpoints Envoy endpoints"
exit 1
fi
done
Expand Down
38 changes: 19 additions & 19 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,8 @@ it and there is no flag to open it. Two ways in:

```bash
# 1. straight through ssh, no setup
ansible sing-1 -u ipni -m shell -a 'sudo k3s kubectl get pods -n someguy -o wide'
ssh ipni@5.199.165.77 'sudo k3s kubectl -n someguy logs deploy/someguy --tail=50'
ansible sing-1 -u ipni -m shell -a 'sudo k3s kubectl get pods -n needle -o wide'
ssh ipni@5.199.165.77 'sudo k3s kubectl -n needle logs deploy/needle --tail=50'

# 2. an ssh tunnel plus the fetched kubeconfig, for real kubectl
./scripts/kubectl-tunnel.sh sing-1 # prints the KUBECONFIG line to use
Expand All @@ -47,30 +47,30 @@ pkill -f "^ssh -f -N .*-L 6443:127.0.0.1:6443" # stop it

`kubeconfigs/` holds cluster-admin credentials and is gitignored.

## someguy
## needle

Four instances per box, `hostNetwork`, all on loopback. Defined in
`roles/someguy/defaults/main.yml` as `someguy_running_instances`:
`roles/needle/defaults/main.yml` as `needle_running_instances`:

| Instance | HTTP API | libp2p |
|-----------|------------------|--------|
| someguy | 127.0.0.1:8190 | 4004 |
| someguy-b | 127.0.0.1:8191 | 4005 |
| someguy-c | 127.0.0.1:8192 | 4006 |
| someguy-d | 127.0.0.1:8193 | 4007 |
| needle | 127.0.0.1:8190 | 4004 |
| needle-b | 127.0.0.1:8191 | 4005 |
| needle-c | 127.0.0.1:8192 | 4006 |
| needle-d | 127.0.0.1:8193 | 4007 |

Loopback-only, so metrics have to be curled **on the box**:

```bash
ansible sing-1 -u ipni -m shell -a \
'curl -s http://127.0.0.1:8190/debug/metrics/prometheus | grep ^someguy_dht'
'curl -s http://127.0.0.1:8190/debug/metrics/prometheus | grep ^needle_dht'
```

Each instance has an `<instance>-data` PVC mounted at `/data/someguy`, holding
Each instance has an `<instance>-data` PVC mounted at `/data/needle`, holding
the autoconf cache and the `*.ndjson` snapshots. It is backed by k3s
`local-path`, pointed at the NVMe by `default-local-storage-path`
(`roles/k3s/tasks/main.yml`), so on the box the data is under
`/data/local-path-provisioner/<pv>_someguy_<instance>-data`. Find it by claim
`/data/local-path-provisioner/<pv>_needle_<instance>-data`. Find it by claim
rather than guessing:

```bash
Expand All @@ -92,10 +92,10 @@ ansible-playbook routing.yml -l sing-1 # one box
ansible-playbook routing.yml -e production_rollout=true # one box at a time, warmed
```

someguy stays degraded for tens of minutes after a restart, long after the pod
needle stays degraded for tens of minutes after a restart, long after the pod
reports Ready — measured at 200 req/s: 11 minutes in, p95 2904 ms with 2937
rejected lookups; by 69 minutes, p95 588–691 ms. That is what
`production_rollout` and `roles/someguy/tasks/wait_warm.yml` exist for, and why
`production_rollout` and `roles/needle/tasks/wait_warm.yml` exist for, and why
`-l <box>` is the habit rather than the exception.

## Locally built images
Expand All @@ -109,12 +109,12 @@ into `ImagePullBackOff` — an outage for that box, on all four instances.
# build once, on the controller, and ship the result - never build per box,
# or the boxes end up running different bytes under the same tag
docker buildx build --platform linux/amd64 --load --provenance=false --sbom=false \
-t someguy:<tag> .
docker save someguy:<tag> -o someguy-snap.tar
-t needle:<tag> .
docker save needle:<tag> -o needle-snap.tar

scp someguy-snap.tar ipni@<ip>:/tmp/
ssh ipni@<ip> 'sudo k3s ctr images import /tmp/someguy-snap.tar'
ssh ipni@<ip> 'sudo k3s ctr images ls | grep someguy' # digest must match everywhere
scp needle-snap.tar ipni@<ip>:/tmp/
ssh ipni@<ip> 'sudo k3s ctr images import /tmp/needle-snap.tar'
ssh ipni@<ip> 'sudo k3s ctr images ls | grep needle' # digest must match everywhere
```

`--provenance=false --sbom=false` matters: buildx otherwise attaches an
Expand Down Expand Up @@ -170,7 +170,7 @@ for d in rendered/*/; do [ -f "$d/kustomization.yaml" ] && kubectl kustomize "$d

`rendered/` is gitignored and is the way to see what a change actually does to
the manifests before it reaches a box. Diffing `kubectl kustomize
rendered/someguy` against the same render on `main` is how you check a manifest
rendered/needle` against the same render on `main` is how you check a manifest
change does only what you meant.

**Two traps when linting locally**, both of which make a clean branch look
Expand Down
Loading
Loading