Standardize host-side tooling delivery with onebox-kit

[vishr]

Context

ob runs from the operator machine and manages one Linux host over SSH. Onebox is agentless: it must not add a resident daemon or listening control port to that host. Linux, SSH, Docker, and Buildx are already prerequisites.

Today, host-side artifacts arrive through different paths:

1. The proxy stack is a multi-arch OCI image.
2. Durable scheduled executions use an embedded Python helper, copied over SSH to <appdir>/schedule/execution-v1.py; it requires /usr/bin/python3 3.8+.
3. wal-g is downloaded on the operator machine, checked against a hand-maintained per-architecture SHA-256 table, then uploaded over SSH and mounted into Postgres.
4. onebox-postgres is already a custom multi-arch OCI image.

Workload volume backup will require Onebox-owned host-side logic plus a backup engine. Adding another uploaded binary or interpreter would create another delivery, verification, architecture, upgrade, and test path.

Problem

Settle a durable installation model for Onebox-owned host-side tooling, including workload-volume backup, without violating the agentless contract.

The design must cover:

• delivery and verification across amd64 and arm64;
• registry-mirrored or preloaded hosts as well as normal registry access;
• versioning, upgrades, rollback, retention, and cleanup;
• the Python durable-execution helper and its host prerequisite;
• wal-g, which PostgreSQL must execute inside its own container;
• Docker-unavailable diagnostics; and
• a small, stable public runtime interface rather than an unbounded utility image.

Proposal

Use OCI images as the standard delivery mechanism. Add a multi-arch, digest-pinned companion image:

ghcr.io/labstack/onebox-kit@sha256:…

onebox-kit is a short-lived runtime, never a daemon. ob pulls or verifies the exact image digest during an explicit apply/enable operation; generated units and durable state record that digest. Scheduled invocations use the installed image and must not fetch from a registry.

The image contains only:

• onebox-kit, a Go program replacing the embedded Python helper and implementing the durable-execution state protocol plus volume-backup orchestration;
• Docker CLI and Compose plugin while the execution protocol still uses Docker/Compose inspection and control through the mounted host Docker socket;
• one folder-backup engine, initially expected to be restic; and
• CA certificates and timezone data.

Its interface is intentionally small:

onebox-kit execution …
onebox-kit volume …
onebox-kit version

Do not expose the backup engine directly as the Onebox protocol. Do not add SSH, package management, credentials, a generic administration toolbox, a listener, or a resident process.

Package wal-g in the existing multi-arch onebox-postgres image instead of materializing it from onebox-kit: wal-g runs from PostgreSQL archive commands, so the database image is its natural runtime location. This removes the uploaded per-architecture binary and checksum map without adding a host-side installer or tool-volume garbage collector.

Keep Docker-independent execution evidence available: list/inspect should read and validate durable records over SSH without requiring the Docker daemon. Mutating execution paths can require Docker; read-only diagnosis must not regress.

Delivery and maintenance contract

• Compile the immutable image index digest into the matching ob release; human-readable tags are informational only.
• Build/test/publish amd64 and arm64 variants in the normal release train, with SBOM/provenance attestations and signature verification if publisher provenance is a product requirement. A digest establishes content identity, not publisher identity.
• Start with one repository and release train, while retaining separate Go packages and tests for execution and backup. Split release cadence only if an operational need becomes concrete.
• Before rewriting units or backup state, verify the pinned digest is locally available or pull it. Fail before mutation if it is unavailable.
• For air-gapped or registry-restricted hosts, support the existing mirror/preload model and qualify the exact Docker stores/versions used. Do not add an SSH-push fallback, and do not assume arbitrary docker save/load preserves the source multi-arch digest reference without testing it.
• Retain images referenced by installed units, recorded execution state, or live backup state. Remove only specifically identified, unreferenced digests; never use broad Docker prune as lifecycle management.
• Give each invocation the least authority needed: execution receives the Docker socket and only required host state; backup receives only its declared source/destination mounts and read-only credentials/CA material.

Non-goals

• Installing a host package, interpreter, or resident Onebox agent.
• A new control port.
• A generic remote administration container.
• Hiding workload-backup consistency semantics; that contract is separate and must be explicit.

Decision criteria

The result should replace, rather than add to, the current artifact-delivery matrix; remove host Python and architecture tables for Onebox-owned logic; preserve safe rollback and diagnostics; and keep Onebox agentless.