Skip to content

runtime-backends.md

Latest commit

 

History

History
168 lines (145 loc) · 8.45 KB

runtime-backends.md

File metadata and controls

168 lines (145 loc) · 8.45 KB
Raw

Runtime backends: the pod-template contract

On the Pod path, upstream's agent-sandbox controller turns a Sandbox into a Pod from spec.podTemplate (or from a SandboxTemplate's blueprint). Nothing in this repository mutates that Pod. Everything a microVM node needs — the node selector, the toleration and the provider's annotations — is written by the template author, verbatim, and reaches the provider unchanged.

Three backends serve a sandbox Pod today.

standard kubelet sandboxd (vk-sandbox) vk-cocoon
node selector none, or your own sandbox.cocoonstack.io/runtime: sandboxd cocoonstack.io/pool: <pool>
toleration none virtual-kubelet.io/provider, Exists, NoSchedule same key
required annotations none sandbox.cocoonstack.io/template vm.cocoonstack.io/name
isolation the cluster's RuntimeClass (Kata, gVisor) a pre-booted microVM claimed from the node's warm pool a Cocoon microVM booted or cloned for this Pod
delivery Pod scheduling and image pull node-local ownership transfer, 0.2–0.7 ms VM boot or snapshot clone

The two virtual nodes advertise their labels and taint themselves: vk-sandbox from --node-labels (default sandbox.cocoonstack.io/runtime=sandboxd) plus the taint virtual-kubelet.io/provider=sandboxd:NoSchedule, vk-cocoon from VK_NODE_POOL (default pool default) plus virtual-kubelet.io/provider=cocoon:NoSchedule. Neither provider reads spec.runtimeClassName: it names a CRI handler, and a virtual node runs no container runtime.

sandboxd

apiVersion: agents.x-k8s.io/v1beta1
kind: Sandbox
metadata:
  name: sandboxd-example
spec:
  podTemplate:
    metadata:
      annotations:
        sandbox.cocoonstack.io/runtime: sandboxd
        sandbox.cocoonstack.io/template: ghcr.io/cocoonstack/sandbox/rt:24.04
        sandbox.cocoonstack.io/net: none
        sandbox.cocoonstack.io/size: small
    spec:
      nodeSelector:
        sandbox.cocoonstack.io/runtime: sandboxd
      tolerations:
        - key: virtual-kubelet.io/provider
          operator: Exists
          effect: NoSchedule
      containers:
        - name: agent
          image: ghcr.io/cocoonstack/sandbox/rt:24.04
Annotation Effect
sandbox.cocoonstack.io/runtime must be sandboxd; absent is read as sandboxd, any other value fails the create
sandbox.cocoonstack.io/template required — the sandboxd pool template. CreatePod fails without it
sandbox.cocoonstack.io/net pool network lane (none, egress); empty takes sandboxd's default
sandbox.cocoonstack.io/size pool size class (small, medium, large); empty takes sandboxd's default
sandbox.cocoonstack.io/ttl-seconds claim lease; absent means 86400 (sandboxd's maximum for an ordinary claim), an explicit 0 takes sandboxd's five-minute default, and a negative or non-integer value fails the create
sandbox.cocoonstack.io/claim-id written back by the provider: the claim id backing the Pod

The container image is not the claim axis — template is. Keep the two equal unless the pool is built from a different reference, or the Pod claims from a pool nobody provisioned. The pool key the warm-pool driver provisions is (template, net, size); a Pod naming a key with no warm capacity gets a typed CreatePod failure and stays Pending; under restartPolicy: Never virtual-kubelet marks it Failed instead and stops retrying.

Three things set how fast a drained warm pool refills (performance):

  • Spread the Pods. vk-sandbox nodes look identical to the scheduler, which then packs one of them. Label the pool's Pods and add a hostname topologySpreadConstraint, as the example does.
  • Run a vk-sandbox whose pod queues follow --kube-api-qps (vk-sandbox#19). The virtual-kubelet default holds each node to 10 Pod creates and status pushes a second.
  • Size both pools. SandboxWarmPool replicas cover the burst you expect, and each node's sandboxd keeps enough warm microVMs to refill them.

The full provider contract, including what happens when a Pod is deleted while its Sandbox lives, is in vk-sandbox's pod contract.

vk-cocoon

apiVersion: agents.x-k8s.io/v1beta1
kind: Sandbox
metadata:
  name: cocoon-example
spec:
  podTemplate:
    metadata:
      annotations:
        vm.cocoonstack.io/name: sandbox-default-cocoon-example
        cocoonset.cocoonstack.io/managed: "true"
        cocoonset.cocoonstack.io/mode: run
        cocoonset.cocoonstack.io/image: ubuntu-dev-base
        cocoonset.cocoonstack.io/os: linux
    spec:
      nodeSelector:
        cocoonstack.io/pool: default
      tolerations:
        - key: virtual-kubelet.io/provider
          operator: Exists
          effect: NoSchedule
      containers:
        - name: agent
          image: ubuntu-dev-base
Annotation Effect
vm.cocoonstack.io/name required — the VM's name on the node. CreatePod fails without it, and a Pod that reuses the name of a live VM adopts that VM instead of booting one
cocoonset.cocoonstack.io/managed "true" lets cocoon create the VM; an unmanaged Pod must carry a pre-assigned vm.cocoonstack.io/id and vm.cocoonstack.io/ip
cocoonset.cocoonstack.io/mode run boots the image, clone clones a snapshot
cocoonset.cocoonstack.io/image the image in the Cocoon image library the VM boots from
cocoonset.cocoonstack.io/os guest OS family (linux, windows, macos); empty means linux

vk-cocoon reads more annotations than these — storage, network, snapshot policy, probe port, hypervisor backend. They are listed in cocoon-common's key set and documented in vk-cocoon.

Two constraints have no equivalent on the sandboxd path:

  • The VM name is per sandbox, so a SandboxTemplate cannot supply it. A template's pod template is copied verbatim, with no per-sandbox substitution, so every Sandbox in a warm pool would carry one vm.cocoonstack.io/name — and the second Pod would adopt the first Pod's VM instead of booting its own. Write the annotation per Sandbox; there is no vk-cocoon warm-pool example for this reason.
  • cocoon-webhook gates the cocoon toleration. Where that webhook runs, a Pod carrying virtual-kubelet.io/provider or vm.cocoonstack.io/name is denied unless it is owned by a CocoonSet and its creator is listed in the webhook's POD_CREATORS (default: the cocoon-operator service account). A Sandbox Pod is neither, so that cluster needs the agent-sandbox controller's service account added to POD_CREATORS.

What replaced the mutator

Until the upstream import this repo shipped a Pod mutator that filled these fields in and rejected conflicting ones at admission. It is gone, and so is admission-time rejection. A wrong pod template now fails later, and differently:

Mistake What happens
No node selector, or one matching no node The scheduler never binds the Pod; it stays Pending with an unschedulable condition
Selector set, toleration missing The virtual node's NoSchedule taint keeps the Pod off it; still Pending
spec.nodeName pinned to a virtual node The Pod bypasses the scheduler and reaches the provider; it succeeds or fails on the annotations alone
Missing required annotation The provider's CreatePod fails; the Pod stays Pending and the error is on the Pod's events. Under restartPolicy: Never virtual-kubelet marks the Pod Failed instead and never retries it
sandbox.cocoonstack.io/runtime set to something other than sandboxd on a vk-sandbox node CreatePod refuses it: that node serves one runtime

The failure is visible on the Pod rather than on the API call that created the Sandbox. Read kubectl describe pod when a sandbox stays Pending or its Pod fails.

Standard kubelet

A Pod with no virtual-node selector schedules onto an ordinary node and is a plain container workload. This is upstream's own scope: agent-sandbox delegates isolation to a sandbox runtime through spec.runtimeClassName (Kata Containers, gVisor). No cocoon-specific behaviour is available on this path.