ate-setup: port microvm-deps installation from shell to Go

.github/workflows/pr-workflow.yaml

@@ -192,9 +192,8 @@ jobs:
     env:
       E2E_ATENET_DATAPLANE: ${{ matrix.dataplane }}
       # The installer requires a credential provider selection, and deploys the
-      # bundled Kubernetes Secrets provider with the control plane. Job-wide
-      # because every ate-setup invocation reads it, including the control
-      # plane redeploy in run-microvm-demo-kind.sh.
+      # bundled Kubernetes Secrets provider with the control plane. Job-wide so
+      # every ate-setup invocation in this job sees the same selection.
       ATE_CREDENTIAL_PROVIDER: '{"name":"k8s.io"}'
       # Job-wide for the same reason. Empty is an explicit "no preview
       # features", not an unset value. The e2e steps pass it to the suites as
@@ -258,13 +257,13 @@ jobs:
       id: microvm-assets
       uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
       with:
-        # Assembling the assets (download kata-static + cloud-hypervisor + the
-        # prebuilt virtiofsd — all downloads on amd64) is fully pinned by assemble.sh,
+        # Assembling the assets (download kata-static + cloud-hypervisor for
+        # amd64 and arm64) is fully pinned by cmd/ate-setup/internal/steps/microvm.go,
         # so key the cache on its hash. The push-to-main run populates the
         # default-branch cache PRs inherit; the weekly schedule refreshes it.
-        # run-microvm-demo-kind.sh skips assembling when these are present.
-        path: bin/microvm-assets/amd64
-        key: microvm-assets-amd64-${{ hashFiles('hack/microvm-assets/assemble.sh') }}
+        # ate-setup skips assembling when these are present.
+        path: bin/microvm-assets
+        key: microvm-assets-${{ hashFiles('cmd/ate-setup/internal/steps/microvm.go') }}
     - name: Enable KVM
       # Grant the runner access to /dev/kvm so create-kind-cluster.sh mounts it into
       # the node and labels it for the micro-VM sandbox class.
@@ -302,12 +301,14 @@ jobs:
         KUBECTL_CONTEXT: "kind-kind"
       run: go run ./cmd/ate-setup setup csi nfs
     - name: Deploy micro-VM counter demo
-      # Stages the (cached) assets into the cluster's rustfs and deploys the
-      # counter-microvm demo onto the control plane installed above. The demo
-      # redeploys the control plane, so retain the selected dataplane.
       env:
-        ATE_ATENET_DATAPLANE: ${{ matrix.dataplane }}
-      run: hack/run-microvm-demo-kind.sh
+        NO_DEV_ENV: "true"
+        KO_DOCKER_REPO: "localhost:5001"
+        ATE_INSTALL_KIND: "true"
+        BUCKET_NAME: "ate-snapshot"
+        KIND_CLUSTER_NAME: "kind"
+        KUBECTL_CONTEXT: "kind-kind"
+      run: go run ./cmd/ate-setup deploy demo counter-microvm
     - name: Deploy gVisor counter demo
       env:
         NO_DEV_ENV: "true"

api/infra/k8s/ate/v1/sandboxconfig_validation_test.go

@@ -352,7 +352,7 @@ func TestShippedSandboxConfigManifests(t *testing.T) {
 
 	for _, path := range []string{
 		"../../../../../manifests/ate-install/sandboxconfig-gvisor.yaml",
-		"../../../../../manifests/microvm/sandboxconfig-microvm.yaml.tmpl",
+		"../../../../../manifests/ate-install/sandboxconfig-microvm.yaml.tmpl",
 	} {
 		t.Run(path, func(t *testing.T) {
 			raw, err := os.ReadFile(path)

benchmarking/automation/README.md

@@ -20,17 +20,17 @@ the router capacity benchmark — see
    and/or `${KO_DOCKER_REPO}/nighthawk-ingress-test:<commit>`.
 5. `hack/install-ate.sh deploy ate-system` (with `ATE_CREDENTIAL_PROVIDER`
    in `.ate-dev-env.sh`, or `--credential-provider` in the test's
-   `ateArgs`) + `benchmarking/workloads/deploy.sh
+   `ateArgs`; passes `--skip-microvm-deps` unless `sandboxClass` is `microvm`,
+   in which case `deploy ate-system` also stages the microVM guest assets to
+   `<BUCKET_NAME>-assets` and applies the cluster-wide `microvm`
+   `SandboxConfig`) + `benchmarking/workloads/deploy.sh
    --deploy --sandbox-class <class>` (these build & push substrate / workload
    images via `ko` as part of their deploy steps — there's no separate
    image build step). The install sets `ATE_API_AUTHZ_BOOTSTRAP_OWNERS` to
    the `ate-client` and `benchmarking/benchmark-runner` ServiceAccounts so the
    runner's calls pass [authorization](../../docs/authorization.md), and sets
    `ATE_INGRESS_ALLOWED_SPIFFE_IDS` to `benchmarking/benchmark-runner` for the
-   Envoy ingress router. For a `microvm` test the orchestrator also
-   runs `hack/install-microvm-deps.sh --install` between the two, which
-   stages kata + cloud-hypervisor + virtiofsd assets to the cluster's object
-   store bucket and applies the cluster-wide `microvm` SandboxConfig.
+   Envoy ingress router.
    For a `nighthawk-ingress` test the orchestrator additionally patches the
    `atenet-router` Deployment right after `deploy_substrate`: `envoyCpu`
    is the benchmark's independent variable and the shipped manifest sets
@@ -47,7 +47,7 @@ the router capacity benchmark — see
      variable (a JSON object) if set. A `--metadata` in the test's own
      `flags` is merged in too, its keys winning on conflict.
    - Polls until complete/failed/timeout; tails logs; deletes the Job.
-   - Tears down workloads + microVM deps (if any) + substrate.
+   - Tears down workloads + substrate.
    - If not the last test, redeploys them so the next run starts clean.
 
 ## Choosing a sandbox class

benchmarking/automation/orchestrator.py

@@ -391,10 +391,15 @@ def validate_and_normalize_tests(tests: list[dict[str, Any]]) -> None:
 
 def substrate_args(test: dict[str, Any]) -> list[str]:
     """Return the hack/install-ate.sh deploy ate-system arguments for a
-    tests.yaml entry: its ateArgs, then the fake data plane flag its isolate
-    block asks for. The teardown needs no matching flag: delete ate-system
-    removes the fake data plane whether or not it was installed."""
-    return [*test.get("ateArgs", []), *isolate_install_args(test)]
+    tests.yaml entry: its ateArgs, the fake data plane flag its isolate block
+    asks for, and --skip-microvm-deps when sandboxClass is not microvm so
+    gVisor runs do not need write access to <BUCKET_NAME>-assets. The teardown
+    needs no matching flag: delete ate-system removes the fake data plane and
+    any microvm SandboxConfig whether or not they were installed."""
+    args = [*test.get("ateArgs", []), *isolate_install_args(test)]
+    if test.get("sandboxClass", "gvisor") != "microvm" and "--skip-microvm-deps" not in args:
+        args.append("--skip-microvm-deps")
+    return args
 
 
 # ate-api-server enforces authorization, and the runner Job calls it with its
@@ -420,20 +425,6 @@ def teardown_substrate() -> None:
     run_no_check(["hack/install-ate.sh", "delete", "ate-system"])
 
 
-def install_microvm_deps() -> None:
-    """Stage kata/cloud-hypervisor assets and apply the cluster-wide
-    microvm SandboxConfig. Required before a microvm WorkerPool can run
-    actors; must run after deploy_substrate() (which installs the CRDs)."""
-    run(["hack/install-microvm-deps.sh", "--install"])
-
-
-def teardown_microvm_deps() -> None:
-    """Remove the microvm SandboxConfig. Must run before
-    teardown_substrate(), which deletes the SandboxConfig CRD (and would
-    prevent this from succeeding via kubectl)."""
-    run_no_check(["hack/install-microvm-deps.sh", "--delete"])
-
-
 def deploy_workloads(
     worker_count: int = 1,
     sandbox_class: str = "gvisor",
@@ -632,11 +623,8 @@ def main() -> None:
             # fire that crashed mid-test (or any other process that left
             # state behind) would otherwise leak its substrate + workloads
             # into this run. All teardowns use --ignore-not-found, so
-            # this is cheap on a clean cluster. Order matters:
-            # microvm-deps deletes a SandboxConfig CR, which requires the
-            # SandboxConfig CRD that teardown_substrate removes.
+            # this is cheap on a clean cluster.
             teardown_workloads()
-            teardown_microvm_deps()
             teardown_substrate()
 
             sandbox_class = test.get("sandboxClass", "gvisor")
@@ -646,10 +634,6 @@ def main() -> None:
             try:
                 deploy_substrate(substrate_args(test))
                 TYPES[ttype].pre_test(test)
-                # install-microvm-deps needs the CRDs from deploy_substrate;
-                # deploy_workloads needs the microvm SandboxConfig.
-                if sandbox_class == "microvm":
-                    install_microvm_deps()
                 deploy_workloads(
                     test.get("workerCount", 1),
                     sandbox_class,
@@ -678,10 +662,7 @@ def main() -> None:
             finally:
                 # Always tear down, even if deploy or run failed, so the
                 # next test (and the next CronJob fire) starts clean.
-                # microvm-deps must go before substrate for the same reason
-                # as above.
                 teardown_workloads()
-                teardown_microvm_deps()
                 teardown_substrate()
             duration = time.time() - start_time
             results.append((test["name"], status, duration, failure_msg))

benchmarking/automation/test_orchestrator.py

@@ -40,7 +40,7 @@ class SubstrateArgsTest(unittest.TestCase):
     def test_ate_args_only(self):
         self.assertEqual(
             orchestrator.substrate_args({"ateArgs": ["--cluster-size=size10"]}),
-            ["--cluster-size=size10"],
+            ["--cluster-size=size10", "--skip-microvm-deps"],
         )
 
     def test_isolate_installs_the_fake_data_plane(self):
@@ -50,14 +50,26 @@ def test_isolate_installs_the_fake_data_plane(self):
                 "isolate": {"delay": "500ms"},
             }),
             # The delay tunes fake-atelet through deploy.sh, not the install.
-            ["--cluster-size=size10", "--fake-data-plane"],
+            ["--cluster-size=size10", "--fake-data-plane", "--skip-microvm-deps"],
         )
 
     def test_neither(self):
-        self.assertEqual(orchestrator.substrate_args({}), [])
+        self.assertEqual(orchestrator.substrate_args({}), ["--skip-microvm-deps"])
 
     def test_bare_isolate_key(self):
-        self.assertEqual(orchestrator.substrate_args({"isolate": None}), ["--fake-data-plane"])
+        self.assertEqual(
+            orchestrator.substrate_args({"isolate": None}),
+            ["--fake-data-plane", "--skip-microvm-deps"],
+        )
+
+    def test_microvm_keeps_microvm_deps(self):
+        self.assertEqual(
+            orchestrator.substrate_args({
+                "sandboxClass": "microvm",
+                "ateArgs": ["--cluster-size=size10"],
+            }),
+            ["--cluster-size=size10"],
+        )
 
 
 class DeploySubstrateTest(unittest.TestCase):

benchmarking/deploy_locust.sh

@@ -45,8 +45,7 @@ usage() {
   echo "  --deploy                Deploy workloads, build/push locust image, then deploy locust"
   echo "  --delete                Delete locust and then workloads"
   echo "  --worker-count N        Number of WorkerPool replicas (default: 1)"
-  echo "  --sandbox-class CLASS   Sandbox runtime for the WorkerPool: gvisor | microvm (default: gvisor)."
-  echo "                          microvm requires hack/install-microvm-deps.sh --install to have run."
+  echo "  --sandbox-class CLASS   Sandbox runtime for the WorkerPool: gvisor | microvm (default: gvisor)"
   echo "  --otlp-endpoint URL     Forwarded to workloads/deploy.sh. The address to which an"
   echo "                          instrumented actor container sends telemetry."
   echo "  --actor-memory SIZE     Forwarded to workloads/deploy.sh. Memory limit for the"

benchmarking/workloads/deploy.sh

@@ -93,8 +93,7 @@ usage() {
   echo "  --deploy                    Substitute env vars and deploy workloads to the cluster using ko apply"
   echo "  --delete                    Substitute env vars and delete workloads from the cluster"
   echo "  --worker-count N            Number of WorkerPool replicas (default: 1)"
-  echo "  --sandbox-class CLASS       Sandbox runtime for the WorkerPool: gvisor | microvm (default: gvisor)."
-  echo "                              microvm requires hack/install-microvm-deps.sh --install to have run."
+  echo "  --sandbox-class CLASS       Sandbox runtime for the WorkerPool: gvisor | microvm (default: gvisor)"
   echo "  --actor-memory SIZE         Memory limit for the benchmark ActorTemplates (default: 256Mi)"
   echo "  --actor-cpu CPU             CPU limit for the benchmark ActorTemplates (default: 1)"
   echo "  --worker-memory SIZE        Memory request and limit for each WorkerPool pod"

cmd/ate-setup/commands.md

@@ -109,18 +109,22 @@ that already names a manifest is used as written, and is not looked up.
 |---|---|
 | `deploy ate-system` | `--deploy-ate-system` |
 | `deploy ate-system --setup-csi=nfs` | `--deploy-ate-system --setup-csi=nfs` |
+| `deploy ate-system --skip-microvm-deps` | (no shell equivalent) |
 | `deploy atelet` | `--deploy-atelet` |
 | `deploy apiserver` | `--deploy-ate-apiserver` |
 | `deploy ate-controller` | (no shell equivalent) |
 | `deploy atenet` | `--deploy-atenet` |
 | `deploy podcertificate-controller` | (no shell equivalent) |
-| `deploy sandboxconfig` | (no shell equivalent) |
+| `deploy sandboxconfig [--skip-microvm-deps]` | (no shell equivalent) |
 | `deploy postgres` | (no shell equivalent) |
 
 `deploy ate-system` is the whole control plane: CRDs, RBAC, the store, the
-apiserver, the controller, atenet, and atelet. It creates every `create`
-resource below on the way, so those subcommands are only needed to redo one on
-a running cluster.
+apiserver, the controller, atenet, and atelet. Unless `--skip-microvm-deps` is
+passed (also accepted by `deploy sandboxconfig`), it also stages the microVM
+guest assets into the asset bucket (`rustfs` on Kind, `<BUCKET_NAME>-assets` on
+GKE) and applies the `microvm` cluster-wide `SandboxConfig` alongside
+`gvisor-default`. It creates every `create` resource below on the way, so those
+subcommands are only needed to redo one on a running cluster.
 
 ## Publish
 
@@ -210,9 +214,9 @@ See
 |---|---|---|
 | `deploy demo counter` | `--deploy-demo-counter` | A counter actor exercising snapshot, resume, and atenet ingress |
 | `deploy demo counter --with-external-volume [--storage-class NAME]` | `--deploy-demo-counter-with-external-volume` (`STORAGE_CLASS=NAME`) | The same, plus an external volume and a pre-seeded file to validate (requires `--preview=ExternalVolumes`). Run `setup csi` first and name the class it created, e.g. `csi-nfs-sc`; defaults to `standard` |
-| `deploy demo counter-microvm` | `--deploy-demo-counter-microvm` | The counter demo on microVM workers. Run `hack/install-microvm-deps.sh --install` first |
+| `deploy demo counter-microvm` | `--deploy-demo-counter-microvm` | The counter demo on microVM workers |
 | `deploy demo egress` | `--deploy-demo-egress` | Egress policy enforcement through atenet; the actors trust the egress gateway's CA |
-| `deploy demo egress-microvm` | `--deploy-demo-egress-microvm` | The same on microVM workers. Run `hack/install-microvm-deps.sh --install` first |
+| `deploy demo egress-microvm` | `--deploy-demo-egress-microvm` | The same on microVM workers |
 | `deploy demo jupyter` | `--deploy-demo-jupyter` | A Jupyter notebook server per actor, reached through atenet ingress |
 | `deploy demo sandbox` | `--deploy-demo-sandbox` | An on-demand sandbox actor driven by the sandbox client |
 | `deploy demo multi-template` | `--deploy-demo-multi-template` | Two ActorTemplates sharing one WorkerPool |

cmd/ate-setup/differences.md

@@ -1,13 +1,12 @@
 # ate-setup vs. the install shell scripts
 
 `ate-setup` is a Go port of `hack/install-ate.sh`, the seven
-`hack/install-demo-*.sh` scripts it sourced, and `hack/setup-csi-*-kind.sh`.
+`hack/install-demo-*.sh` scripts it sourced, `hack/setup-csi-*-kind.sh`, and
+`hack/install-microvm-deps.sh`.
 
-The port has replaced them. The demo scripts are deleted, and
-`hack/install-ate.sh` is a shim holding no install logic of its own: it
-translates the flags and environment variables the installer has always
-accepted onto `ate-setup` commands, so existing command lines keep working. See
-[`commands.md`](commands.md) for the flag-by-flag mapping.
+The port has replaced them. The demo, CSI, and microVM asset scripts are
+deleted, and `hack/install-ate.sh` forwards its arguments directly to
+`ate-setup`. See [`commands.md`](commands.md) for the historical flag mapping.
 
 This document covers what changed *behind* that mapping: the design decisions
 of the port, against the shell it replaced. For what a user of
@@ -85,15 +84,22 @@ Still required, and why:
 | `ko` | building and publishing images (`ko resolve` only); not used with `--image-repo` |
 | `go` | locating the pinned `ko` tool, exactly as `hack/run-tool.sh` does; not used with `--image-repo` |
 | `git` | the `git describe` version stamp passed to ko; not used with `--image-repo` |
-| `docker` | the kind CSI setup (`docker exec` into the node) and the claude-code-multiplex workload build |
-| `gcloud` | GKE `get-credentials`, only when `PROJECT_ID` is set and no context was given |
+| `docker` | the kind CSI setup (`docker exec` into the node), staging microVM assets into Kind's `rustfs`, and the claude-code-multiplex workload build |
+| `gcloud` | GKE `get-credentials` (when `PROJECT_ID` is set and no context was given) and `gcloud storage` when staging microVM assets on GKE |
 | `bash` | sourcing `.ate-dev-env.sh` |
 
-Two shell scripts are still invoked rather than reimplemented, because they
-orchestrate image builds, asset assembly, and object-store staging that are out
-of scope for an installer: `benchmarking/deploy_locust.sh` and
-`hack/install-microvm-deps.sh`. They receive `Config.ScriptEnv()`, which
-reconstructs the environment the shell installer would have exported to them.
+One shell script is still invoked rather than reimplemented, because it
+orchestrates benchmark image builds and Locust stack deployment:
+`benchmarking/deploy_locust.sh`. It receives `Config.ScriptEnv()`, which
+reconstructs the environment the shell installer would have exported to it, and
+is fronted by `deploy`/`delete benchmarks`.
+
+The microVM asset assembly (`kata-static` `.tar.zst` extraction and
+`cloud-hypervisor` download), object-store staging (`rustfs` on Kind,
+`<BUCKET_NAME>-assets` on GKE), and `microvm` `SandboxConfig` lifecycle now run
+in Go as part of `deploy ate-system` (and `deploy sandboxconfig`), with
+`delete ate-system` removing the `microvm` `SandboxConfig` alongside the rest
+of the control plane.
 
 `ko` is no longer asked to apply anything. The scripts ran `run_ko apply`, which
 made ko shell out to kubectl and forced the awkward `-- --context=` special case

cmd/ate-setup/internal/cmd/benchmarks.go

@@ -82,11 +82,7 @@ See benchmarking/README.md for the walkthrough and customization options.`,
 var deleteBenchmarksCmd = &cobra.Command{
 	Use:   "benchmarks",
 	Short: "Delete the locust load test stack and the benchmark workloads",
-	Long: `Delete the locust load test stack and the benchmark workloads.
-
-Pass --sandbox-class=microvm to also remove the micro-VM SandboxConfig; it is
-cluster-wide, so it is left in place by default.`,
-	Args: cobra.NoArgs,
+	Args:  cobra.NoArgs,
 	RunE: func(cmd *cobra.Command, _ []string) error {
 		opts, err := benchmarkOptions()
 		if err != nil {