Skip to content

On GKE, micro-VM assets move to their own bucket and nodes lose access to the snapshot bucket - #3037

Merged
Michelle Au (msau42) merged 2 commits into
agent-substrate:mainfrom
ILLUM1N0X:microvm-asset-bucket
Oct 21, 2026
Merged

Michelle Au (msau42) merged 2 commits into
agent-substrate:mainfrom
ILLUM1N0X:microvm-asset-bucket

Conversation

@ILLUM1N0X

@ILLUM1N0X ILLUM1N0X commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #777

setup-gcp gave atelet object admin on the snapshot bucket and on the whole project, so root on a node could read and write every snapshot without the credential exchange. atelet only needs its own identity for micro-VM sandbox assets, which lived in the snapshot bucket. On GKE they now move to a bucket of their own, <BUCKET_NAME>-assets, which setup-gcp creates and lets atelet read, and atelet gets no other storage role. Snapshot transfers already run on per-actor tokens from the exchange.

Breaking change

On GKE with micro-VM workers, recreate the cluster. Substrate does not upgrade a cluster in place, and a node keeps fetching micro-VM assets from the snapshot bucket, since their digests do not change when they move. Run go run ./tools/setup-gcp create bucket and create iam before hack/install-microvm-deps.sh --install or a micro-VM demo, since assets are now staged in <BUCKET_NAME>-assets, which those commands create. On any existing install, gVisor-only included, run create bucket before rerunning create iam, which now also grants on the asset bucket and stops if it is missing. setup-gcp also refuses a BUCKET_NAME longer than 56 characters, since the asset bucket name would exceed GCS's limit.

Existing installs keep atelet's old grants, since setup-gcp only adds bindings. Once Substrate runs a release whose node plugin gets snapshot credentials from the credential exchange, and micro-VM assets are in the asset bucket, remove them:

ATELET="principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${PROJECT_ID}.svc.id.goog/subject/ns/ate-system/sa/atelet"
gcloud projects remove-iam-policy-binding "${PROJECT_ID}" --member="${ATELET}" --role=roles/storage.objectAdmin --condition=None
gcloud storage buckets remove-iam-policy-binding "gs://${BUCKET_NAME}" --member="${ATELET}" --role=roles/storage.objectAdmin
gcloud storage buckets remove-iam-policy-binding "gs://${BUCKET_NAME}" --member="${ATELET}" --role=roles/storage.bucketViewer

Tested on a GKE cluster with atelet's storage grants removed: atelet's identity was refused on the snapshot bucket and could read the asset bucket, a micro-VM actor booted from the asset bucket, suspended to a completed durable snapshot and resumed, and the full e2e suite passed.

AI use: AI-assisted, I reviewed it.

  • Tests pass
  • Docs updated

@ILLUM1N0X
ILLUM1N0X force-pushed the microvm-asset-bucket branch from f663983 to 9f1e703 Compare October 21, 2026 02:24
@ILLUM1N0X
ILLUM1N0X marked this pull request as ready for review October 21, 2026 02:24
@ILLUM1N0X
ILLUM1N0X force-pushed the microvm-asset-bucket branch from dd88ecb to 34cbe03 Compare October 21, 2026 02:29
@ILLUM1N0X ILLUM1N0X added the breaking-change PRs with breaking changes label Oct 21, 2026
@ILLUM1N0X
ILLUM1N0X force-pushed the microvm-asset-bucket branch from 34cbe03 to f9df791 Compare October 21, 2026 02:58
@ILLUM1N0X
ILLUM1N0X force-pushed the microvm-asset-bucket branch 3 times, most recently from 11869e1 to 4c56497 Compare October 21, 2026 15:04
Nodes read micro-VM sandbox assets with their own identity, and the assets
lived in the snapshot bucket, so setup-gcp granted atelet object admin on it
and on the whole project. On GKE, install-microvm-deps.sh now stages the
assets in BUCKET_NAME-assets, which setup-gcp creates and lets atelet read.
atelet gets no other storage role, since snapshot transfers on GCS run on
per-actor tokens. teardown.sh deletes the asset bucket, and the docs cover
upgrading an existing install.
@ILLUM1N0X
ILLUM1N0X force-pushed the microvm-asset-bucket branch from 4c56497 to 1d68199 Compare October 21, 2026 18:30
Comment thread manifests/microvm/sandboxconfig-microvm.yaml.tmpl
Comment thread tools/setup-gcp/README.md
Comment thread tools/setup-gcp/README.md
Comment thread hack/install-microvm-deps.sh
A node indexes a sandbox runtime by the digests of its assets, and those
do not change when the assets move to the asset bucket, so a node keeps
fetching them from the snapshot bucket. The upgrade section now says to
recreate such clusters, and lists the node service account's cleanup
command beside atelet's. The SandboxConfig template test also substitutes
${ASSET_BUCKET_NAME}, and the benchmarking README names the asset bucket.
@hdp617

Copy link
Copy Markdown
Contributor

cc noahlwest for #2092.

@msau42
Michelle Au (msau42) added this pull request to the merge queue Oct 21, 2026
Merged via the queue into agent-substrate:main with commit e494343 Oct 21, 2026
15 checks passed
Steven Shriver (stshrive) added a commit to stshrive/substrate-gke that referenced this pull request Oct 26, 2026
Pulls in the separate micro-VM asset bucket (`<BUCKET_NAME>-assets`) and the removal of atelet's snapshot storage roles from [Agent Substrate PR 3037](agent-substrate/substrate#3037).

Upstream drops atelet's storage.objectAdmin on the project and its
objectAdmin and bucketViewer on the snapshot bucket, since upstream main
moves snapshots on per-actor tokens from the credential exchange. No
Substrate release has that yet, and the installer defaults to
v0.4.0-gke.0, so those grants now sit behind --atelet-snapshot-access
(ATELET_SNAPSHOT_ACCESS), on by default, for bootstrap and create iam.
Turn it off for a revision with per-actor snapshot credentials. The
asset bucket and atelet's objectViewer on it are always created; an old
release leaves the bucket empty. The installer does not set the flag:
the revision it installs can be any commit, and detecting the feature
would need a probe of the fetched tree.

Other differences from upstream:

- Bucket policies are still read and written at version 3 with addBinding,
  so conditional bindings survive (ai-on-gke#55).
- teardown.sh: bucket_gone takes a bucket name and caches per bucket, and
  the binding and bucket steps cover the asset bucket. cleanup-gcp's help
  and confirmation text name it.
- Installer: bootstrap's checklist and dry-run lines follow the new step
  text; the micro-VM upload row no longer names the snapshot bucket.
- Installer: the derived bucket name is capped at 56 characters, matching
  setup-gcp's new limit. This changes the default name for clusters whose
  derivation is 57 to 63 characters long. setup-gcp would refuse the old
  name now anyway.
- README: the upgrade section explains when atelet's old grants can be
  removed. The node service account part is already in the IAM table
  footnote from the #3042 pull.

docs/screenshots/complete.svg still shows an untruncated derived bucket
name; `make screenshots` regenerates it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking-change PRs with breaking changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants