Repository navigation
On GKE, micro-VM assets move to their own bucket and nodes lose access to the snapshot bucket - #3037
Merged
Michelle Au (msau42) merged 2 commits intoOct 21, 2026
Conversation
2 tasks done
ILLUM1N0X
force-pushed
the
microvm-asset-bucket
branch
from
October 21, 2026 02:24
f663983 to
9f1e703
Compare
ILLUM1N0X
marked this pull request as ready for review
October 21, 2026 02:24
ILLUM1N0X
force-pushed
the
microvm-asset-bucket
branch
from
October 21, 2026 02:29
dd88ecb to
34cbe03
Compare
ILLUM1N0X
force-pushed
the
microvm-asset-bucket
branch
from
October 21, 2026 02:58
34cbe03 to
f9df791
Compare
ILLUM1N0X
requested review from
Benjamin Elder (BenTheElder),
Taahir Ahmed (ahmedtd) and
Michelle Au (msau42)
October 21, 2026 02:58
ILLUM1N0X
force-pushed
the
microvm-asset-bucket
branch
3 times, most recently
from
October 21, 2026 15:04
11869e1 to
4c56497
Compare
Nodes read micro-VM sandbox assets with their own identity, and the assets lived in the snapshot bucket, so setup-gcp granted atelet object admin on it and on the whole project. On GKE, install-microvm-deps.sh now stages the assets in BUCKET_NAME-assets, which setup-gcp creates and lets atelet read. atelet gets no other storage role, since snapshot transfers on GCS run on per-actor tokens. teardown.sh deletes the asset bucket, and the docs cover upgrading an existing install.
ILLUM1N0X
force-pushed
the
microvm-asset-bucket
branch
from
October 21, 2026 18:30
4c56497 to
1d68199
Compare
A node indexes a sandbox runtime by the digests of its assets, and those
do not change when the assets move to the asset bucket, so a node keeps
fetching them from the snapshot bucket. The upgrade section now says to
recreate such clusters, and lists the node service account's cleanup
command beside atelet's. The SandboxConfig template test also substitutes
${ASSET_BUCKET_NAME}, and the benchmarking README names the asset bucket.
Contributor
Michelle Au (msau42)
approved these changes
Oct 21, 2026
This was referenced Oct 25, 2026
Steven Shriver (stshrive)
added a commit
to stshrive/substrate-gke
that referenced
this pull request
Oct 26, 2026
Pulls in the separate micro-VM asset bucket (`<BUCKET_NAME>-assets`) and the removal of atelet's snapshot storage roles from [Agent Substrate PR 3037](agent-substrate/substrate#3037). Upstream drops atelet's storage.objectAdmin on the project and its objectAdmin and bucketViewer on the snapshot bucket, since upstream main moves snapshots on per-actor tokens from the credential exchange. No Substrate release has that yet, and the installer defaults to v0.4.0-gke.0, so those grants now sit behind --atelet-snapshot-access (ATELET_SNAPSHOT_ACCESS), on by default, for bootstrap and create iam. Turn it off for a revision with per-actor snapshot credentials. The asset bucket and atelet's objectViewer on it are always created; an old release leaves the bucket empty. The installer does not set the flag: the revision it installs can be any commit, and detecting the feature would need a probe of the fetched tree. Other differences from upstream: - Bucket policies are still read and written at version 3 with addBinding, so conditional bindings survive (ai-on-gke#55). - teardown.sh: bucket_gone takes a bucket name and caches per bucket, and the binding and bucket steps cover the asset bucket. cleanup-gcp's help and confirmation text name it. - Installer: bootstrap's checklist and dry-run lines follow the new step text; the micro-VM upload row no longer names the snapshot bucket. - Installer: the derived bucket name is capped at 56 characters, matching setup-gcp's new limit. This changes the default name for clusters whose derivation is 57 to 63 characters long. setup-gcp would refuse the old name now anyway. - README: the upgrade section explains when atelet's old grants can be removed. The node service account part is already in the IAM table footnote from the #3042 pull. docs/screenshots/complete.svg still shows an untruncated derived bucket name; `make screenshots` regenerates it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #777
setup-gcp gave atelet object admin on the snapshot bucket and on the whole project, so root on a node could read and write every snapshot without the credential exchange. atelet only needs its own identity for micro-VM sandbox assets, which lived in the snapshot bucket. On GKE they now move to a bucket of their own,
<BUCKET_NAME>-assets, which setup-gcp creates and lets atelet read, and atelet gets no other storage role. Snapshot transfers already run on per-actor tokens from the exchange.Breaking change
On GKE with micro-VM workers, recreate the cluster. Substrate does not upgrade a cluster in place, and a node keeps fetching micro-VM assets from the snapshot bucket, since their digests do not change when they move. Run
go run ./tools/setup-gcp create bucketandcreate iambeforehack/install-microvm-deps.sh --installor a micro-VM demo, since assets are now staged in<BUCKET_NAME>-assets, which those commands create. On any existing install, gVisor-only included, runcreate bucketbefore rerunningcreate iam, which now also grants on the asset bucket and stops if it is missing. setup-gcp also refuses aBUCKET_NAMElonger than 56 characters, since the asset bucket name would exceed GCS's limit.Existing installs keep atelet's old grants, since setup-gcp only adds bindings. Once Substrate runs a release whose node plugin gets snapshot credentials from the credential exchange, and micro-VM assets are in the asset bucket, remove them:
Tested on a GKE cluster with atelet's storage grants removed: atelet's identity was refused on the snapshot bucket and could read the asset bucket, a micro-VM actor booted from the asset bucket, suspended to a completed durable snapshot and resumed, and the full e2e suite passed.
AI use: AI-assisted, I reviewed it.