Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .agents/skills/debug-openshell-cluster/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,13 @@ For source checkout development, restart the local gateway with:
mise run gateway:docker
```

During a graceful gateway restart, Docker, Podman, and VM sandboxes with
running intent should stop before the gateway exits and restart after it
returns. Check for `Stopped sandbox during gateway shutdown` and `Started
sandbox during gateway startup` in gateway logs. A sandbox explicitly stopped
through the CLI remains stopped. Kubernetes sandboxes are cluster-owned and do
not follow this local gateway lifecycle.

### Step 5: Check Podman-Backed Gateways

```bash
Expand Down
9 changes: 9 additions & 0 deletions architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,15 @@ compute to zero, and VM retains its launch request and writable overlay beside
a stop marker. Delete remains a separate operation that removes these
resources.

On graceful gateway shutdown, persisted running intent for Docker, Podman, and
VM is stopped through the shared `StopSandbox` RPC before any gateway-managed
driver process exits. The gateway does not persist `Stopped` for this
infrastructure event. On startup, it reconciles the retained intent through the
shared idempotent `StartSandbox` RPC before watch processing begins. Explicitly
`Stopped` sandboxes are excluded from both sweeps. Kubernetes workloads are
cluster-owned and continue running without gateway shutdown or startup
lifecycle calls.

## Deletion Lifecycle

Lifecycle requests use per-sandbox gates to serialize stop, start, and
Expand Down
5 changes: 5 additions & 0 deletions crates/openshell-driver-docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,11 @@ policy. Start starts that same container, so files in the resolved OCI
workspace remain available. A durably stopped sandbox is excluded from
gateway startup recovery and stays stopped across gateway restarts. Delete
continues to force-remove the container and clean up driver-owned material.
Graceful gateway shutdown sends `StopSandbox` for each sandbox whose persisted
phase requires running compute without changing that persisted intent. On
startup, the gateway sends an idempotent `StartSandbox` request for the same
sandboxes, restarting their retained containers. Explicitly stopped sandboxes
remain excluded.

Before creating the container, the driver inspects the final sandbox image and
captures its immutable image ID, raw OCI `Config.User`, and OCI
Expand Down
72 changes: 0 additions & 72 deletions crates/openshell-driver-docker/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1055,69 +1055,6 @@ impl DockerComputeDriver {
}
}

pub async fn stop_managed_containers_on_shutdown(&self) -> Result<usize, Status> {
let containers = self.list_managed_container_summaries().await?;
let targets = containers
.into_iter()
.filter_map(|container| {
let state = container.state.unwrap_or(ContainerSummaryStateEnum::EMPTY);
if container_state_needs_shutdown_stop(state) {
summary_container_target(&container)
} else {
None
}
})
.collect::<Vec<_>>();
let target_count = targets.len();
let mut stopped = 0usize;
let mut failures = Vec::new();
let stop_timeout_secs = self.config.stop_timeout_secs;

let mut stop_results = futures::stream::iter(targets.into_iter().map(|target| {
let docker = self.docker.clone();
async move {
let result = docker
.stop_container(
&target,
Some(
StopContainerOptionsBuilder::default()
.t(docker_stop_timeout_secs(stop_timeout_secs))
.build(),
),
)
.await;
(target, result)
}
}))
.buffer_unordered(16);

while let Some((target, result)) = stop_results.next().await {
match result {
Ok(()) => {
stopped += 1;
}
Err(err) if is_not_found_error(&err) || is_not_modified_error(&err) => {}
Err(err) => {
warn!(
container = %target,
error = %err,
"Failed to stop Docker sandbox container during shutdown"
);
failures.push(target);
}
}
}

if !failures.is_empty() {
return Err(Status::internal(format!(
"failed to stop {} of {target_count} Docker sandbox containers during shutdown",
failures.len()
)));
}

Ok(stopped)
}

async fn reserve_pending_sandbox(&self, sandbox: &DriverSandbox) -> Result<(), Status> {
let mut pending = self.pending.lock().await;
if pending
Expand Down Expand Up @@ -3200,15 +3137,6 @@ fn summary_container_target(summary: &ContainerSummary) -> Option<String> {
.or_else(|| summary_container_name(summary))
}

fn container_state_needs_shutdown_stop(state: ContainerSummaryStateEnum) -> bool {
matches!(
state,
ContainerSummaryStateEnum::RUNNING
| ContainerSummaryStateEnum::RESTARTING
| ContainerSummaryStateEnum::PAUSED
)
}

/// States from which a managed container can be brought back to running by
/// `start_container`. Skip `Restarting` (already coming up), `Removing`,
/// `Dead` (terminal), `Paused` (needs `unpause`, not `start`), and
Expand Down
6 changes: 6 additions & 0 deletions crates/openshell-driver-podman/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,12 @@ volume. Stopped managed containers remain visible through list and watch
reconciliation. Delete remains responsible for removing the container,
driver-owned secrets, and workspace volume.

Graceful gateway shutdown sends `StopSandbox` for each sandbox whose persisted
phase requires running compute without changing that persisted intent. On
startup, the gateway sends an idempotent `StartSandbox` request for the same
sandboxes, restarting their retained containers. Explicitly stopped sandboxes
remain excluded.

## Architecture

The Podman driver communicates with the Podman daemon over a Unix socket and
Expand Down
9 changes: 5 additions & 4 deletions crates/openshell-driver-vm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,10 +194,11 @@ during the first prepare.

The driver also writes the accepted `DriverSandbox` launch request to
`<state-dir>/sandboxes/<id>/sandbox.pb`. If the gateway restarts, it starts a
new VM driver process; that process scans the sandbox state directories,
restarts each persisted VM launcher, and preserves any existing `overlay.ext4`
instead of cloning a fresh overlay template. If a restart happened before the
overlay was created, the driver creates it during the start attempt.
new VM driver process. During graceful shutdown, the gateway first sends the
shared `StopSandbox` request for each persisted running-intent sandbox, which
stops its launcher while retaining the launch request and `overlay.ext4`.
After driver initialization, the gateway sends the idempotent `StartSandbox`
request for that retained intent. Explicitly stopped sandboxes remain excluded.

Stop writes a marker in the sandbox state directory before terminating
the launcher and releasing host GPU and network allocations. It retains
Expand Down
Loading
Loading