Graceful gateway shutdown can lose a pending driver operation, leaving sandboxes undeletable (caller gone, or SIGTERM during startup recovery) #4409

User Story

I run openshell-gateway (Docker driver, single replica, SQLite store) on a lab host and a pilot VM as the runtime of a self-hosted agent platform. The gateway runs as a supervised child process: when a dependency of the platform fails, the supervisor stops the gateway with SIGTERM (escalating to SIGKILL after 15 s), keeps it down, then starts it again and deletes every sandbox through the API. New work resumes only once ListSandboxes is empty. Stopping the gateway is therefore a normal operation for me, not a rare crash. While evaluating v0.1.3 I reproduced the lost-ownership case described in #4038 and found two graceful-SIGTERM paths that reach it.

Problem Statement

#4038 documents that if the gateway owning a driver request crashes before recording its result, driver_operation_pending stays set and start, stop and deletion are blocked, with no API to resolve it. On v0.1.3 the same state is also reached on a graceful SIGTERM:

1. The caller has already disconnected. The detached provisioning worker is not awaited by shutdown: the gateway exits in about 50 ms while the driver call is still in flight, and the claim survives restart.
2. SIGTERM arrives during startup recovery. start_persisted_sandboxes_with_authentication runs before the SIGTERM handler is installed (crates/openshell-server/src/lib.rs:936-958 vs :979 at v0.1.3), so the default disposition kills the process at once. The StartSandbox operation the recovery sweep had just claimed for a previously Ready sandbox stays pending with provisioning.deadline: null, so no timeout path revisits it.

With the caller still connected, SIGTERM behaves correctly: shutdown waits on the per-sandbox lifecycle gate, records the driver answer, and the flag clears.

Impact / Why This Matters

• Restarting or stopping the gateway is an ordinary operation (service restart, host maintenance, a supervisor reacting to a dependency failure), and every restart runs a StartSandbox per persisted running sandbox. On my host StartSandbox takes 0.66 / 0.72 / 2.6 s (min / median / max), so a stop shortly after a start has a window of seconds per sandbox, not milliseconds.
• Once hit, the sandbox can never be stopped, started or deleted through the API (FailedPrecondition: previous driver operation is still pending; retry and deletion are blocked). In my runs it stayed so for more than 33 minutes and 52 gateway restarts, past the image-preparation deadline (that only moves it to Error / ImagePreparationTimedOut with a reclamation that never completes). After a StartSandbox case, its exited container cannot be removed through the API either.
• Any workflow that needs an empty sandbox list after a restart (mine does, before resuming work) cannot complete without out-of-band changes to the gateway's state, which I do not want to do.
• The workaround I am implementing: treat such records as dead after verifying in Docker that none of their containers exist, and retry their deletion at every restart. It only works around the symptom, and the records accumulate.

Acceptance Criteria

• A SIGTERM received while a driver operation is in flight, whether or not its caller is still connected, lets the operation settle and records its result before the process exits, within the shutdown budget; or, if it cannot, the record is left in a state that a later start, stop or delete can resolve.
• A SIGTERM received during the startup recovery sweep triggers the same graceful shutdown as after startup (no immediate termination, no newly claimed operation left pending).
• After either case and a restart, openshell sandbox delete <name> succeeds.

Reproduction Steps

Gateway started directly with a TOML config: compute_driver = "docker", a unique sandbox_label, image_pull_policy = "if_not_present", default SQLite store, mTLS, no providers. Policy file p.yaml:

version: 1
filesystem_policy:
  include_workdir: true
  read_only: [/bin, /usr, /lib, /proc, /dev/urandom, /app, /etc, /var/log]
  read_write: [/sandbox, /tmp, /dev/null]
landlock:
  compatibility: best_effort

The windows are short, so to make them deterministic I point [openshell.drivers.docker] socket_path at a small Python (standard library only) Unix-socket relay to /var/run/docker.sock. While a file hold exists and contains a substring, the relay holds the first Docker API request whose request line contains it (here sandbox-id, the driver's per-sandbox container lookup); every other request passes; deleting the file releases it. The script (78 lines) is included at the end of this report. Without the relay, case 0 below occurred naturally in 1 of 14 SIGKILL timings (kill 16 ms after CreateSandbox).

0. Crash (the documented case, for reference). echo sandbox-id > hold; openshell sandbox create --name s0 --detach --no-tty --policy p.yaml &; openshell sandbox get s0 -o json shows provisioning.driver_operation_pending: true; kill -KILL <gateway>; rm hold; restart the gateway. openshell sandbox delete s0 fails with FailedPrecondition.
1. Graceful SIGTERM, caller gone. echo sandbox-id > hold; openshell sandbox create --name s1 --detach --no-tty --policy p.yaml &; once driver_operation_pending: true, kill the CLI process; kill -TERM <gateway>. The gateway exits in about 50 ms (log: Failed to stop sandbox during gateway shutdown … sandbox not found, then gateway shutdown cleanup failed). rm hold; restart. s1 has driver_operation_pending: true; openshell sandbox delete s1 fails with FailedPrecondition.
2. SIGTERM during startup recovery. openshell sandbox create --name s2 --detach --no-tty --policy p.yaml and wait for Ready. kill -TERM <gateway> (the shutdown sweep stops s2). echo sandbox-id > hold; start the gateway; its recovery sweep claims a StartSandbox for s2 (driver_operation_pending: true, new driver_operation_id). kill -TERM <gateway>: the process exits at once and never logs Shutdown signal received; stopping gateway. rm hold; restart. s2 (Ready before) now has driver_operation_pending: true, phase Error, provisioning.deadline: null; stop, start and delete fail with FailedPrecondition.
3. Control (works). As step 1, but keep the CLI connected: after kill -TERM, the gateway logs Shutdown signal received; stopping gateway, waits for the held call, records Created sandbox: …, stops the sandbox and exits; after restart driver_operation_pending: false.

Relevant source at v0.1.3 (e1f3c82c), as I read it: the claim is persisted before dispatch and cleared only on an actual driver response (compute/provisioning_operation.rs:42-51,121-192,254-312); shutdown waits on lifecycle gates for running-phase sandboxes (compute/mod.rs:3121-3130) but does not drain detached provisioning workers, which are dropped when run_server returns (openshell-gateway/src/main.rs:4-8); the recovery sweep runs before shutdown_signal() is awaited (lib.rs:936-958,979,1193-1222).

Suggested UX (if applicable)

None for the shutdown fix. Separately from this report: a supported way for an operator to resolve a lost claim after a real crash, for example an explicit, journaled "abandon the pending operation" on a stopped record once the driver confirms the backend objects are absent, would remove the need for out-of-band state changes.

Environment

• OpenShell v0.1.3 from openshell_0.1.3-1_amd64.deb (sha256 64000b75aaf7d23763ad43aef6d9d6386325cba7bb636a64539a13ce98f88108, matches the release checksum file), gateway run directly with --config, not the packaged unit.
• Supervisor image ghcr.io/nvidia/openshell/supervisor:0.1.3@sha256:4ab2b970cdb02c7518ba4ccbbadb4861ae9a8d0b8770e5b160697f0453b0a8e4; sandbox images alpine:3.21 and ghcr.io/nvidia/openshell-community/sandboxes/base@sha256:aeef1c63f00e2913ea002ccb3aaf925f338b5c5d70e63576f0d95c16a138044e, both cached.
• Debian 13 LXC on Proxmox, kernel 7.0.14, x86_64, Docker Engine 29.8.1 (API 1.56).

Logs

Case 2: the gateway log ends on an ordinary request line, with no Shutdown signal received; stopping gateway line, although SIGTERM was sent during the recovery sweep. After restart, for each stuck record:

WARN openshell_server::compute: Retaining pending driver operation during gateway recovery sandbox_id="0ef00e7a-…"
$ openshell sandbox delete s1
! Failed to delete sandbox s1: code: 'The system is not in a state required for the operation's execution', message: "previous driver operation is still pending; retry and deletion are blocked"
Error:   × failed to delete 1 sandbox: s1

#!/usr/bin/env python3
"""Private Docker API relay for one disposable OpenShell gateway.

While the file <root>/run/hold exists and contains a substring P, the FIRST
client->daemon chunk whose request line contains P is held (that Docker API
request stalls); every other request passes. Removing the file releases the
held bytes in order and re-arms the one-shot. Bodies are never logged; only a
request-line prefix.
"""

import asyncio
import sys
import time
from pathlib import Path

ROOT = Path(sys.argv[1])
ENDPOINT = ROOT / "run/docker.sock"
HOLD = ROOT / "run/hold"
LOG = ROOT / "relay.log"


def log(msg):
    with LOG.open("a") as f:
        f.write(f"{time.time():.3f} {msg}\n")


HELD_ONCE = {"armed": True}


async def handle(reader, writer):
    source, upstream = await asyncio.open_unix_connection("/var/run/docker.sock")

    async def up():
        while data := await reader.read(65536):
            if not HOLD.exists():
                HELD_ONCE["armed"] = True
            else:
                pattern = HOLD.read_text().strip().encode()
                line = data.split(b"\r\n", 1)[0]
                first = line[:160] + (b"...[" + str(len(line)).encode() + b"B]" if len(line) > 160 else b"")
                if HELD_ONCE["armed"] and pattern and pattern in line:
                    HELD_ONCE["armed"] = False
                    log(f"HOLD {first!r}")
                    while HOLD.exists():
                        await asyncio.sleep(0.01)
                    log(f"RELEASE {first!r}")
            upstream.write(data)
            await upstream.drain()

    async def down():
        while data := await source.read(65536):
            writer.write(data)
            await writer.drain()

    tasks = [asyncio.create_task(up()), asyncio.create_task(down())]
    try:
        await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
    except (ConnectionError, OSError):
        pass
    finally:
        for t in tasks:
            t.cancel()
        await asyncio.gather(*tasks, return_exceptions=True)
        writer.close()
        upstream.close()


async def main():
    if ENDPOINT.exists():
        ENDPOINT.unlink()
    server = await asyncio.start_unix_server(handle, path=str(ENDPOINT))
    ENDPOINT.chmod(0o600)
    print("relay ready", flush=True)
    async with server:
        await server.serve_forever()


asyncio.run(main())