fix(msb): detect stale ssh-agent --vsock route after a host reboot

[mogul]

AI-assisted (OpenCode). Human-owned and reviewed per repo AI-contribution policy.

Fixes #413.

Context

On the msb backend, after a host reboot (especially an unclean one that did not stop the sandbox first), resuming the sandbox and re-attaching leaves the guest unable to reach the host ssh-agent, so git signing fails — even though SSH_AUTH_SOCK is set on both host and guest, the host agent holds the key, and the in-guest socat bridge is running.

This is a distinct failure from #392. #392 fixed the SSH_AUTH_SOCK env-var injection on running-reattach (restart the bridge + write the /var/lib/acq/ssh-auth-sock marker) — that works. This is one layer deeper: the --vsock HOST_PATH:3552 route is emitted only at create time and pins HOST_PATH to the host's SSH_AUTH_SOCK path captured then; it is persisted and never re-derived on msb start / re-attach. A reboot restarts the host ssh-agent under a new socket path, so the persisted route's host endpoint is dead. The bridge faithfully restarts and connects to that dead endpoint (socat … VSOCK-CONNECT → "Connection reset by peer"), leaving a silent dead bridge behind a present marker.

Change

The --vsock route is create-time only and cannot be refreshed on a running sandbox, so acq cannot transparently heal this — the honest remedy is a recreate. Instead of failing silently (repo no-silent-failure rule), add a liveness probe in _acq_msb_start_ssh_agent_bridge (_acq_msb_warn_if_agent_unreachable) that runs after the bridge is (re)started and, when the forwarded agent is unreachable, prints an actionable warning naming the host-reboot cause and the acq rm + re-run remedy.

Key correctness detail (surfaced in review): the probe classifies on the ssh-add -l message, not the exit code alone. When the socat listener socket exists but its vsock backend is dead, ssh-add connects and then the agent protocol fails — it exits 1 with communication with agent failed, the same exit code as a healthy-but-empty agent (The agent has no identities.). Exit 2 is only a missing socket, which is not this scenario. So:

• exit 0 (has keys) → quiet
• exit 1 with "no identities" → quiet (healthy, just no keys loaded)
• anything else on failure (the "communication with agent failed" text, or a bare exit 2) → warn

Other robustness fixes from review:

• The warn-return is || true-guarded at the call site so it can never abort a verb under set -euo pipefail (the bridge starter is the last statement of acq_backend_start, called bare by start/restart). Regression test 10c20b asserts this.
• A short bounded retry (3×0.3s) replaces a fixed sleep 1, absorbing a fresh-create race without taxing the common already-running reattach.
• The probe skips silently when ssh-add is absent in the guest.

No auto-refresh of the route is attempted; that is left as future work pending an msb capability to update a --vsock route in place (noted in the ADR).

Files

• acq.backends/msb.sh — _acq_msb_warn_if_agent_unreachable + call site (guarded).
• scripts/test-acq-lib.sh — faithful ssh-add stub knobs (STUB_AGENT_UNREACHABLE = exit 1 + "communication…"; STUB_AGENT_NO_SOCKET = exit 2; STUB_AGENT_NO_KEYS = exit 1 + "no identities").
• test/bats/115-ssh-agent-forward.bats — 10c19 (reboot dead-bridge warns), 10c19b (missing socket warns), 10c20 (healthy with/without keys quiet), 10c20b (no abort under set -e), 10c21 (ssh-add-absent skip).
• docs/adr/0021-msb-host-ssh-agent-forwarding-via-vsock.md — new "Stale route after a host reboot (detect-and-report)" section.
• docs/KNOWN_FAILURE_MODES.md — §34 new variant with the VSOCK-CONNECT "Connection reset by peer" fingerprint and the recreate remedy.

Verification

Check	Result
shellcheck --severity=warning acq.backends/msb.sh scripts/test-acq-lib.sh	clean
markdown lint (repo files)	0 issues
scripts/test-acq-bats	380 passed, 0 failed
gitleaks protect --staged	no leaks
adversarial code-review sub-agent	2 blockers found (exit-code classification; set -e abort) — both fixed & re-verified

Real ssh-add exit-code contract was confirmed live: dead-socket-backend → exit 1 "communication with agent failed"; empty keyring → exit 1 "no identities"; missing socket → exit 2.

Rollback

Revert the commit. The change is additive (a new helper + one guarded call site in the bridge starter); reverting restores the prior behavior (silent dead bridge after reboot). No config/data migration.

Security impact

Touches the ssh-agent forwarding path (ADR-0021). No new external service or data-classification change. The probe is read-only (ssh-add -l), consent/opt-in unchanged, and never widens the forward. It cannot forward an agent into a sandbox that wasn't created for one.

Live-verify still needed

The probe's behavior against a real rebooted msb guest (a genuinely stale route) has not been exercised end-to-end on a KVM/HVF host (cannot run in CI / inside a sandbox). Suggest verifying on the next sandbox-capable-host pass; the offline suite covers the classification logic against faithful ssh-add stubs.

[wz-gsa]

Adversarial review — LGTM, verified the classification logic and injection surface directly

Root cause is correctly diagnosed and distinct from #392 (which fixed the env-var injection on reattach; this is one layer deeper — the --vsock route itself is create-time-only and never re-derived, so a host reboot that rotates the host ssh-agent's socket leaves a bridge that faithfully restarts and connects to a dead endpoint).

Verified the exit-code/message classification against the real binary, not just the PR's claim:

$ strings $(command -v ssh-add) | grep -i identities
The agent has no identities.

The *"no identities"* substring check matches this exactly, and I found no other "empty keyring" message variant on this system that could slip past it — the classification is complete, not just plausible.

Verified the injection surface is clean. $name/$_sock are interpolated unquoted into the sh -c body, which would normally be a flag — but both are acq-internal constants at this call site: $_sock traces back through _acq_msb_ssh_auth_sock_for/ACQ_MSB_SSH_AGENT_GUEST_SOCK, never from untrusted external input in this path. The comment correctly identifies this.

The 10c20b regression is the right kind of test — it doesn't just assert the || true guard exists, it empirically proves the bridge-starter still returns 0 under set -euo pipefail when the probe warns, which is the actual property that matters (a silent-failure fix that itself crashes the verb would be worse than the bug it fixes).

The bash32-compat.yml mirror-fallback is a disclosed second commit, not a silent rider like the ones flagged on #391/#410 — its own commit message, and it's the same fix class I'd suggested for the signing-key keyserver flake (#332/#395), applied here to the tarball fetch itself: try ftpmirror.gnu.org first, fall back to the canonical host, with the correct note that integrity doesn't depend on which mirror served the bytes (the OpenPGP signature check is unchanged and still authoritative).

Test coverage is thorough and matches the stated scenarios 1:1: reboot-dead-bridge warns (10c19), missing-socket warns (10c19b), healthy-with/without-keys stays quiet (10c20), no-abort-under-set-e (10c20b), ssh-add-absent skips silently (10c21).

Honest about its own limit — the PR body correctly flags that the probe's behavior against a genuinely stale real vsock route hasn't been exercised on a live KVM/HVF host (can't run in CI or inside a sandbox); the offline suite covers the classification logic against faithful stubs. That's the right scope for what can be verified without hardware access, and it's stated rather than implied.

All 5 CI checks green (380 passed offline). Approve.

AI-assisted (OpenCode).