feat(k8s): support ephemeral emptyDir-backed sandbox workspaces as an alternative to the default PVC

[kon-angelo]

User Story

As a platform operator running OpenShell on a managed Kubernetes cluster (ROSA, EKS, AKS, GKE, Gardener) or on a short-lived CI cluster, I want to opt sandboxes into an ephemeral, emptyDir-backed workspace instead of the per-sandbox PVC, so that I can run OpenShell in environments where PVC provisioning is unavailable, unreliable, expensive, or disallowed by cluster policy — without giving up sandbox lifecycle correctness.

Problem Statement

The Kubernetes compute driver unconditionally injects a workspace PVC into every sandbox pod via volumeClaimTemplates on the agents.x-k8s.io/Sandbox CR (crates/openshell-driver-kubernetes/src/driver.rs:2041-2176, constants at :413-431). After PR #2088 removed SandboxTemplate.volume_claim_templates from the public API, there is no supported way to run a sandbox pod without a PVC. The only knobs available today are workspace_default_storage_size (#1436) and workspace_storage_class (#2442, PR #2463) — both change which PVC gets created, not whether one is created.

An emergent workaround exists (has_explicit_sandbox_data_mount disables default PVC injection when the operator supplies any driver_config subPath mount under /sandbox), but it is undocumented, requires supplying an unrelated PVC just to trigger the branch, and does not deliver emptyDir semantics. It is an implementation side-effect, not a contract.

The agents.x-k8s.io/Sandbox CRD does not require the workspace to be a PVC — volumeClaimTemplates is optional in api/v1beta1. This is entirely an OpenShell-side rendering assumption, so the change is scoped to this repo.

Impact / Why This Matters

Without an ephemeral option, users of the Kubernetes driver today must:

• Provide a StorageClass in every target cluster, even when the sandbox does not need pod-reschedule survival. On clusters without a default StorageClass and without workspace_storage_class set, the workspace PVC stays Pending and the sandbox never starts. This blocks OpenShell adoption on short-lived CI clusters, edge clusters, restricted managed clusters, and Gardener-style shoots where dynamic provisioning is not always installed.
• Pay the provisioning cost and lifecycle risk of a PVC per sandbox, even for short-lived agent tasks that never restart. The PVC + init-container path adds first-start latency (called out as a stopgap in the code comment at driver.rs:399-411) and, as documented in Warm-pooled sandboxes for the Kubernetes compute driver #1879, leaves orphaned PVCs holding written user data on teardown paths that skip the finalizer (default agent-sandbox shutdownPolicy: Retain).
• Give up the fail-safe property emptyDir provides — kubelet-guaranteed reclaim, no external object to orphan, no dependence on cleanup being correct. Warm-pooled sandboxes for the Kubernetes compute driver #1879 already identifies this as a valid workspace model, but scopes it to warm-pooled sandboxes only, leaving cold-path sandboxes without the option.
• Rely on an undocumented driver-config side-effect (has_explicit_sandbox_data_mount) to skip the PVC — not a stable contract and requires supplying a foreign PVC just to disable default injection.

The maintainer-authored design for the current PVC path (#743) explicitly anticipated this: "Opt-out mechanism: some users may want ephemeral sandboxes. Consider --no-persist flag or server env var. Can be a follow-up." No follow-up has been filed.

Proposed Design

Add a workspace backend selector to the Kubernetes driver configuration, mirroring the shape of workspace_storage_class (#2442) exactly.

Gateway-wide default (crates/openshell-driver-kubernetes/src/config.rs):

#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub enum WorkspaceBackend {
    /// Per-sandbox PVC via `volumeClaimTemplates` on the Sandbox CR.
    /// Survives pod rescheduling. Requires a StorageClass.
    #[default]
    Pvc,
    /// `emptyDir` volume mounted at `/sandbox`. No PVC created.
    /// Ephemeral — deleted with the pod.
    Ephemeral,
}

pub struct KubernetesComputeConfig {
    // ...existing fields...
    pub workspace_backend: WorkspaceBackend,      // new
    pub workspace_default_storage_size: String,   // ignored when Ephemeral
    pub workspace_storage_class: String,          // ignored when Ephemeral
    // ...
}

• Same driver-config → gateway TOML → Helm value fan-out that the existing storage-class field uses (server.workspaceBackend).
• Default is Pvc, preserving current behavior byte-for-byte.

Per-sandbox override — no new proto or CLI surface required. The existing SandboxTemplate.driver_config envelope (proto/openshell.proto:862) is already the driver-keyed opaque config the gateway forwards to the compute driver, and the CLI already exposes it as --driver-config-json. The only change is: the Kubernetes driver reads workspace_backend from the per-request struct as it already does for other overridable fields.

# Ephemeral sandbox on a gateway defaulted to pvc:
openshell sandbox create --name ephemeral-task \
  --driver-config-json '{"kubernetes":{"workspace_backend":"ephemeral"}}' \
  -- claude

# Persistent sandbox on a gateway defaulted to ephemeral:
openshell sandbox create --name durable-task \
  --driver-config-json '{"kubernetes":{"workspace_backend":"pvc","workspace_default_storage_size":"10Gi"}}' \
  -- claude

Rendering (crates/openshell-driver-kubernetes/src/driver.rs, apply_workspace_persistence):

• Pvc → existing path unchanged.
• Ephemeral → skip volumeClaimTemplates entirely; add a pod-spec emptyDir volume named workspace at /sandbox (WORKSPACE_MOUNT_PATH); the workspace-init init container still seeds /sandbox from the image on every pod start (sentinel unnecessary — volume is always empty at pod start).

Observable behavior

• sandbox create in Ephemeral mode succeeds on a cluster with no StorageClass installed.
• sandbox stop + sandbox start on an ephemeral sandbox starts with a fresh /sandbox — documented as the ephemeral contract, complementary to the PVC-backed lifecycle from feat(sandbox): add storage-preserving suspend and resume lifecycle #2652.
• Pod eviction / rescheduling on an ephemeral sandbox loses /sandbox state (documented). PVC-backed sandboxes retain today's behavior unchanged.
• kubectl get pvc -n <ns> shows zero workspace PVCs for ephemeral sandboxes.

Acceptance Criteria

• driver_config.kubernetes.workspace_backend accepts "pvc" (default) and "ephemeral"; unknown values rejected at config parse.
• With workspace_backend = "ephemeral", the rendered pod spec contains an emptyDir volume at /sandbox and the Sandbox CR contains no volumeClaimTemplates entry.
• With workspace_backend = "ephemeral", sandbox creation succeeds on a cluster that has no default StorageClass and no explicit workspace_storage_class.
• The workspace-init init container seeds /sandbox from the image on every pod start in ephemeral mode.
• Default (unset) behavior is byte-for-byte identical to today's PVC path — asserted by a render-diff test.
• Per-sandbox override via --driver-config-json '{"kubernetes":{"workspace_backend":"..."}}' works and takes precedence over the gateway default.
• Combining workspace_backend = "ephemeral" with workspace_default_storage_size or workspace_storage_class fails validation with a clear error.
• Helm value server.workspaceBackend and the gateway TOML field render correctly through the deployment path and are documented in docs/reference/gateway-config.mdx and docs/reference/sandbox-compute-drivers.mdx.
• Kubernetes e2e: create an ephemeral sandbox, write a marker to /sandbox, delete the pod, verify (a) no PVC is left behind and (b) a re-created sandbox with the same identity starts with a fresh /sandbox.

Alternatives Considered

• Do nothing; document the has_explicit_sandbox_data_mount side-effect. Keeps the emergent workaround as the answer. Requires operators to supply an unrelated PVC just to disable default injection, does not deliver emptyDir semantics, and remains contract-less.
• Generic Ephemeral Volumes (ephemeral.volumeClaimTemplate on the pod spec). Still requires a StorageClass; does not solve the "cluster has no provisioner" case. Complementary; could be added later as a third workspace_backend variant.
• tmpfs for /sandbox. Bounded by pod memory limits and interacts poorly with the image-seed path (image content can be arbitrarily large). Not a fit for the default case.
• Wait for the container-snapshotting replacement referenced by driver.rs:399-411. There is no tracking issue for that work; the ephemeral option is complementary and useful regardless of when snapshotting lands.
• Reintroduce SandboxTemplate.volume_claim_templates. Explicitly reverted by PR refactor(api): remove SandboxTemplate.volume_claim_templates #2088. This proposal instead follows the current direction (customize through driver_config, not by opting the whole PVC surface out via raw CRs).

Agent Investigation

• Kubernetes driver injects the workspace PVC unconditionally: crates/openshell-driver-kubernetes/src/driver.rs:413-431 (constants), :2041-2139 (apply_workspace_persistence), :2150-2176 (default_workspace_volume_claim_templates).
• Emergent bypass via subPath mounts: has_explicit_sandbox_data_mount disables default injection for any driver-config mount at-or-under /sandbox (noted by @elezar in PR feat(kubernetes): support PVC subPath driver config #2034 review).
• PR refactor(api): remove SandboxTemplate.volume_claim_templates #2088 removed SandboxTemplate.volume_claim_templates from the public API, closing the previous raw-CR escape hatch and consolidating storage configuration behind driver_config.kubernetes.
• Existing driver-keyed override pipe: SandboxTemplate.driver_config (proto/openshell.proto:862) is already the opaque per-request envelope the gateway forwards to the compute driver — no proto change required for the per-sandbox override.
• CLI already exposes the override generically: --driver-config-json on sandbox create (crates/openshell-cli/src/main.rs:1386-1392, parser in crates/openshell-cli/src/run.rs:262) — no CLI change required.
• feat(sandbox): persist /sandbox directory across gateway restarts via init container + PVC #743 (design spike for the current PVC path) explicitly deferred the ephemeral opt-out as a follow-up that never materialized.
• Warm-pooled sandboxes for the Kubernetes compute driver #1879 (warm-pooled sandboxes) proposes emptyDir as one of two workspace models — but only on the warm path, leaving cold-path sandboxes without the option.
• Precedent for the config-knob shape and rollout: feat(k8s): support configuring storageClassName for the default workspace PVC #2442 → PR feat(k8s): support configuring workspace PVC storageClassName #2463 (workspace_storage_class) — identical driver/TOML/Helm/docs fan-out.
• Managed-cluster context: feat: Support restricted SecurityContextConstraints for managed Kubernetes platforms #899 (restricted SCC on OpenShift) confirms the managed-Kubernetes audience is present upstream; discussion Using openshell-sandbox standalone on edge devices (automotive/robotics) #2626 (edge devices) is an adjacent audience for the same "PVC undesirable" case.
• The agents.x-k8s.io/Sandbox CRD (kubernetes-sigs/agent-sandbox, api/v1beta1) does not require volumeClaimTemplates — the field is optional, so this proposal needs no operator or CRD change.

Checklist

• I've reviewed existing issues and the architecture docs
• This is a design proposal, not a "please build this" request