Skip to content

refactor(compute): unify gateway restart reconciliation - #2743

Open
drew wants to merge 3 commits into
mainfrom
refactor/2417-unify-driver-restart/drew
Open

refactor(compute): unify gateway restart reconciliation#2743
drew wants to merge 3 commits into
mainfrom
refactor/2417-unify-driver-restart/drew

Conversation

@drew

@drew drew commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

Unify graceful gateway shutdown and startup for local compute through the public ComputeDriver lifecycle RPCs. Shutdown calls StopSandbox for running-intent Docker, Podman, and VM sandboxes without persisting an explicit user stop; startup calls the idempotent StartSandbox RPC for that retained intent. Kubernetes remains cluster-owned.

Related Issue

Part of #2417

Changes

  • stop running-intent Docker, Podman, and VM compute through ComputeDriver::StopSandbox during graceful gateway shutdown
  • run independent shutdown stop RPCs with bounded concurrency of 16 while retaining per-sandbox lifecycle serialization
  • preserve persisted lifecycle intent so gateway shutdown remains distinct from an explicit openshell sandbox stop
  • terminate a gateway-managed VM driver process only after the sandbox stop sweep, and continue cleanup after individual stop failures
  • reconcile retained running intent through ComputeDriver::StartSandbox before startup watch processing
  • exclude explicitly stopped, stopping, deleting, and error states and leave Kubernetes compute running
  • add unit coverage for phase filtering, phase preservation, driver selection, bounded concurrency, failure continuation, missing resources, and start failures
  • update Docker, Podman, and VM restart E2Es to assert compute is stopped while the gateway is down and restarted afterward
  • document the shared stop-on-shutdown/start-on-startup contract

Testing

  • mise run pre-commit
  • cargo test -p openshell-server shutdown_stop_sweep --lib
  • cargo test -p openshell-server start_persisted_sandboxes --lib
  • Docker, Podman, and VM restart E2E targets compile with their respective features
  • OPENSHELL_E2E_DOCKER_TEST=gateway_start mise run e2e:docker
  • Podman runtime E2E not run because this host has no Podman CLI
  • VM runtime E2E remains blocked by the pre-existing supervisor/SSH relay readiness failure before the restart test
  • mise run test: all completed suites passed except the unrelated gateway_completer_returns_empty_when_no_config, which reproduces in isolation because this host has registered system gateway configuration

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture and driver documentation updated

@github-actions

Copy link
Copy Markdown

@drew
drew force-pushed the refactor/2417-unify-driver-restart/drew branch from 2f74bca to a036b59 Compare August 17, 2026 06:05
@drew
drew requested a review from sjenning as a code owner August 17, 2026 06:05
@drew
drew removed the request for review from maxamillion August 18, 2026 06:29
@drew
drew force-pushed the refactor/2417-unify-driver-restart/drew branch from b103f36 to f8ce804 Compare August 19, 2026 01:26
@drew drew added the test:e2e Requires end-to-end coverage label Aug 19, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for f8ce804. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

drew added 2 commits August 18, 2026 21:05
Remove the Docker-specific gateway shutdown cleanup and reconcile persisted running intent through ComputeDriver::StartSandbox for Docker, Podman, and VM drivers. Explicitly stopped sandboxes remain stopped.

Closes #2417

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew
drew force-pushed the refactor/2417-unify-driver-restart/drew branch from f8ce804 to 9cad488 Compare August 19, 2026 04:09
@drew

drew commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 9cad488

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant