Repository navigation
Sync upstream runtime, AgentGateway, and documentation updates - #104
Merged
Merged
Conversation
…gent-substrate#3182) Lower atelet's snapshot upload delay in kind so e2e suites exercise background snapshot upload instead of leaving asynchronous checkpoints pending for the full test run. Update TestUpdateTemplateLifecycle and RevertResumeActor to stop assuming an asynchronous suspend never uploads in the background. TestUpdateTemplateLifecycle now reverts before its asynchronous suspend. RevertResumeActor accepts either snapshot its revert of the paused actor can keep, depending on whether the upload was reported first. The control plane returns Aborted while it applies atelet's upload report, so the tests retry Aborted on actor RPCs that follow an asynchronous suspend. AI use: AI-assisted, I reviewed and tested it - [x] Tests pass - [x] Docs updated Signed-off-by: msau42 <msau@google.com>
…imes fails (agent-substrate#3146) Fixes agent-substrate#3145 When `workerSandboxSyncer` in `cmd/atelet` fails to build or write a worker pod's `sandbox-runtimes.json`, `atelet` only logs a warning on the node. `atecontroller` emits events on the `WorkerPool` when a `SandboxConfig` reference does not resolve, and `ateapi` only hears from `ateworker` after `sandbox-runtimes.json` changes, so node-side failures (unreachable asset URLs, SHA256 mismatches, missing architecture assets, or failed `runsc cpu-features` probes) are only visible in `atelet` container logs. Pass an `events.k8s.io/v1` `EventRecorder` into `workerSandboxSyncer` and emit a `Warning` `SandboxRuntimeSyncFailed` (`action: SyncSandboxRuntimes`) `Event` on the worker `Pod` in `finish` for `errPermanent`, `errIncomplete`, non-`errUnresolved` retries, and `errUnresolved` when giving up at `syncMaxRetries`, attaching the resolved `SandboxConfig` (or `WorkerPool` when its config has not resolved) as `related` so config edits start a new event series with the updated error message. Truncate event notes at the 1024-byte `events.k8s.io/v1` limit on a UTF-8 rune boundary, and grant the `atelet` `ClusterRole` `create` and `patch` on `events.k8s.io` `events`. AI-assisted, I reviewed and tested it. <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Shrutiya <shrutiyam@google.com>
…gent-substrate#3130) Fixes agent-substrate#2934 Builds on agent-substrate#2956 by @ericcurtin, with additional fixes when appling network policy. ### Why `NetworkPolicyName` kept only the first 30 characters of the pool name plus 5 hex digits of its SHA-256. Two pools in the same namespace could get the same NetworkPolicy name. The controller applies with `ForceOwnership`, so the second pool took over the first pool's policy and rewrote its pod selector. ### What changed - The name is now `resources.NetworkPolicyNamePrefix` (`substrate-`) plus the pool name. The pool name is also the `ate.dev/worker-pool` label value, which Kubernetes caps at 63 characters, so the name needs no truncation. On upgrade, each existing pool gets a second policy under the new name. The old one has the same rules and is deleted with the pool. - Before applying, the controller reads the existing policy. It refuses to take it over if another object controls it, or if nothing controls it and its `ate.dev/worker-pool` label isn't the pool's name. A refusal returns an error and emits a `NetworkPolicyConflict` Warning event on the WorkerPool. ### Upgrade The new ate-controller creates `substrate-<pool>` for each pool, but leaves the old `substrate-<truncated>-<hash>` policy in place until the pool is deleted. Policies add together, so if a later release tightens the rule, the old policy keeps admitting the old peers. If you upgrade in place, wait until the new ate-controller has created `substrate-<pool>` for each pool, then delete the old policies: ```sh kubectl get networkpolicy -A -o go-template='{{range .items}}{{$np := .}}{{range .metadata.ownerReferences}}{{if and .controller (eq .kind "WorkerPool") (ne $np.metadata.name (printf "substrate-%s" .name))}}{{$np.metadata.namespace}} {{$np.metadata.name}}{{"\n"}}{{end}}{{end}}{{end}}' | while read -r ns name; do kubectl -n "$ns" delete networkpolicy "$name"; done ``` This deletes only policies a WorkerPool controls whose name isn't `substrate-<pool>`. Run the `kubectl get` part alone first to see the list. AI use: AI-assisted (Claude Code). Code reviewed by myself. - [x] Tests pass - [ ] Docs updated (none needed) --------- Signed-off-by: Yufan Su <yufans@google.com> Co-authored-by: Eric Curtin <eric.curtin@docker.com>
…arties accept it (agent-substrate#3251) New installs generated an Ed25519 actor identity CA, which many external relying parties don't accept as a trust anchor. - `ate-setup` now generates `actor-id-ca-pool` with ECDSA P-256, the same way it already generated `egress-mitm-ca-pool`. Cluster-internal pools (pod identity, service DNS, postgres) stay Ed25519. - `kubectl ate admin make-ca-pool --key-type` now defaults to `ECDSAP256`. Pass `--key-type ED25519` for pools that are only validated inside the cluster. - Existing pools are kept on upgrade, so installed clusters keep their current root until it is rotated. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs: I don't think this needs a docs change since existing installs are unchanged. Signed-off-by: Greg Castle <gcastle@google.com>
…ent-substrate#3220) Kind installs select the bundled Kubernetes Secrets credential provider, but releases do not publish its image, so `ate-setup deploy --image-repo ghcr.io/agent-substrate/substrate --credential-provider='{"name":"k8s.io"}'` fails. This adds it to `released`. Part of agent-substrate#3217. envoy-dataplane, the other image agent-substrate#3217 needs, is left for a separate PR because the release job has no buildx. After the first release that includes it, an org owner has to make the new `ghcr.io/agent-substrate/substrate/kubernetes-secrets` package public, as `docs/dev/releasing.md` describes. AI use: I used Claude Code to find the affected tests and docs and to draft the change. I reviewed the diff and ran the ate-setup tests. - [x] Tests pass - [x] Docs updated Signed-off-by: Grant McCloskey <gmccloskey@google.com>
…#3237) <!-- The PR title becomes the release note. Write it for users. --> Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) Part of cncf/sandbox#542 I have enabled DCO enforcement, and will shortly drop the Google CLA enforcement (I wanted to get these docs up and double check the DCO is working as expected). <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: Yes, drafted by AI. I reviewed it. <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [x] Docs updated Signed-off-by: Benjamin Elder <bentheelder@google.com>
…rate#3183) `TestSandboxCrashIsReported` flaked a few times recently. Each time `UpdateEphemeralContainers` returned a 409 Conflict. This PR resolves the image first and retries the get and update on a conflict to deflake. AI assisted and human reviewed. - [ ] Tests pass - [ ] Docs updated --------- Signed-off-by: Haven Xia <haoyuxia@google.com>
The GKE quickstart in `README.md` says the required Kubernetes beta APIs can only be enabled at cluster creation. The `setup-gcp` README says otherwise, and `setup-gcp create cluster` enables them on an existing cluster. Fixes agent-substrate#3233 `README.md` now says GKE 1.37 and higher serve the APIs by default. On 1.36, one can enable the beta APIs and then replace any 1.36 node pools created before that. - [ ] Tests pass - [x] Docs updated Signed-off-by: Haven Xia <haoyuxia@google.com>
…bstrate#3262) `ate-api-server` installed the authz interceptor only for unary RPCs. Streams passed after authentication alone, so the docs' "an RPC with no rule is denied" didn't hold for them. In the bug bash, a ServiceAccount with no grants could list and describe every service through gRPC reflection. This adds `authz.StreamServerInterceptor`, which denies every stream not in `streamRules`. `ateapi.proto` already can't define streaming RPCs (`TestDefaultRPCPermissions_CoversEveryServedRPC`), so the allowlist holds only gRPC reflection. Reflection stays open to any authenticated caller, the same way Kubernetes discovery is open to `system:authenticated`, since it returns only the schema that's public in `api/ateapi/v1/ateapi.proto`. Unauthenticated callers still get `Unauthenticated` from `apiauthn`, which runs first. Fixes agent-substrate#3231 I used an AI agent (Jetski) to find the gap during the bug bash, draft the change, and run the tests. I reviewed the diff. Signed-off-by: Shruti Nair <shrutinair@google.com>
To match the other lifecycle enums Breaking change: `SnapshotStorage.status` is renamed to `state`, and the `SnapshotStorageStatus` enum to `SnapshotStorageState`. On upgrade: - Generated clients: rename `SnapshotStorageStatus` to `SnapshotStorageState`, `SNAPSHOT_STORAGE_STATUS_*` to `SNAPSHOT_STORAGE_STATE_*`, and in Go, `.Status`/`GetStatus()` on `SnapshotStorage` to `.State`/`GetState()`. - JSON consumers, such as scripts that parse `kubectl ate get -o json`: read `durableSnapshot.state` instead of `durableSnapshot.status`, and expect `SNAPSHOT_STORAGE_STATE_PENDING` and `SNAPSHOT_STORAGE_STATE_COMPLETED`. Stored actors and tags need no migration. Signed-off-by: Luiz Oliveira <luizaoj@google.com>
…gent-substrate#3310) <!-- The PR title becomes the release note. Write it for users. --> Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) FYI @destijl, follow-up on agent-substrate#2939 (comment) Besides forwarding, a new namespace picks up the node's: - `accept_redirects` - `route_localnet` - `rp_filter` - `accept_local` - `accept_source_route` - ... and has IPv6 on with `accept_ra=1`. NOTE: `SetSysctls` now writes every key, since turning forwarding off sets `conf/all/accept_redirects` back to 1. The sysctls are set before any link is added, a conf/default write doesn't reach existing links for keys the node set explicitly. Fixes agent-substrate#2939 <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: Assisted with identifying the gaps (with upstream sources cited), drafting the change and testing. I read through this before submitting and iterated a bit. <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [x] Docs updated Signed-off-by: Benjamin Elder <bentheelder@google.com>
…ts (agent-substrate#3315) Following the README order on a fresh cluster, `deploy.sh` fails: Error from server (NotFound): error when creating ".../nighthawk-server.yaml": namespaces "nighthawk" not found agent-substrate#2656 moved `nighthawk-server` and `traffic-driver-invoker` into namespace `nighthawk`, but only `deploy-traffic-driver.sh` (README step 2) applied the Namespace, through the WorkerPool template. The Namespace moves to `namespace.yaml` and both scripts apply it, so either script works on its own. Neither `--delete` path removes it, since the namespace is shared between the two scripts; the README says to delete it by hand after `deploy-traffic-driver.sh --delete`. The README target URLs, the port-forward namespace, the `deploy.sh` messages, and the manifest comments that still said `ate-system` now say `nighthawk`. AI use: AI-assisted. An agent found the failure while running the benchmark and drafted the change and this description. I wrote the commit message and reviewed both. - [ ] Tests pass (`bash -n` and the boilerplate check pass locally; the end-to-end rerun on a fresh kind cluster is pending) - [x] Docs updated Signed-off-by: Yuan Gao <ypgao@google.com>
…er it starts (agent-substrate#3144) `acquireConn` kept pooling HTTP/2 connections to `atenet-egress` after `MintWorkerCertificate` renewed the actor certificate. Because the gateway authenticates each `CONNECT` with the certificate the TLS handshake presented, every new tunnel on an old connection failed with 403 once the first certificate expired (agent-substrate#3142). `runDial` now records the `*tls.Certificate` the handshake presented on `h2ClientConn`. When `GetClientCertificate` returns a different certificate, `acquireConn` drops the connection from `h2Conns` and `retire` installs a `SetStateHook` that closes it once `InFlight` reaches 0. The switch happens at renewal (90% of the 1-hour lifetime), so existing connections have the remaining 10% to drain. Fixes agent-substrate#3142 AI use: AI-assisted to draft the fix and unit test. I reproduced on GKE, reviewed, and tested it. - [x] Tests pass - [ ] Docs updated --------- Signed-off-by: Chenyi Wang <chenyiwang@google.com>
…o match gVisor (agent-substrate#1904) A micro-VM container that drops `NET_BIND_SERVICE` could not bind a port below 1024; the same container on gVisor could, since its network stack has no privileged-port limit. The kata agent now receives the spec's sysctls, and ShapeMicroVM sets `net.ipv4.ip_unprivileged_port_start=0`, so both runtimes behave the same. Neither sets `no_new_privileges`; gVisor runs with `--allow-suid` since agent-substrate#2361, so setuid binaries work on both. The capabilities e2e suite gains a drop-ALL template whose probe listens on port 80, and the probe reports `NoNewPrivs`, asserted off on both lanes. - [x] Tests pass: `go test -race ./internal/ocispec/...`, `GOOS=linux go vet` on kata and ocispec, `go vet ./internal/e2e/...`; e2e on both lanes in CI. - [x] Appropriate changes to documentation are included in the PR: docs/api-guide.md. --------- Signed-off-by: Davanum Srinivas <davanum@gmail.com>
<!-- The PR title becomes the release note. Write it for users. --> Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) Fixes the agentgateway postsubmit by updating the agentgateway implementation after some substrate contract changes. As part of this, I expanded some tests to be less Envoy-specific in their assertions. <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: AI assisted; I reviewed and ran local e2e tests <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [X] Tests pass - [X] Docs updated --------- Signed-off-by: Keith Mattix II <keithmattix2@gmail.com> Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io> Co-authored-by: Eitan Yarmush <eitan.yarmush@solo.io>
…ubstrate#3324) `ReportActorCrashed` was missing from the `WorkerService` table in `docs/authorization.md`. Fixes agent-substrate#3234 I used an AI agent to find the missing row and check it against registry.go and workerservice/crash.go. Signed-off-by: Shruti Nair <shrutinair@google.com>
… actor's runtime overhead (agent-substrate#3104) Today the microVM worker takes a flat 128MiB out of each actor's memory limit for the VMM, which is too much for small guests and too little for large ones, and nothing accounts for what an actor costs outside its sandbox when placing it. - Workers report what an actor costs on top of its limits on each runtime (`WorkerSandboxRuntime.actor_overhead`: a fixed amount plus an amount per GiB of the actor's memory limit), and placement books it with the actor, so it shows in the worker's allocated resources - MicroVM guests get their whole memory limit, and the worker reports 128MiB + 8MiB/GiB per actor as overhead (cloud-hypervisor, virtiofsd, host page tables, and the page cache of snapshot writes). Once a follow-up caps snapshot writes this drops to 4MiB + 8MiB/GiB - Both workers keep 64MiB of their memory limit out of the capacity they report, for the worker process and in-flight snapshot writes agent-substrate#3091 proposed wrapping `SandboxRuntime`, but agent-substrate#3022 already split the worker-side runtime (`WorkerSandboxRuntime`) from the stamp on snapshots and assignments (`ActorSandboxRuntime`), so the overhead is a field on the former. Same intent, and it stays out of compatibility matching. Fixes agent-substrate#3091 TODO: For microVM, actors still lose some memory inside the guest limit to various things, including the kernel and kata-agent. Similarly, in gVisor they lose memory to the sentry, which is inside the actor's cgroup. As such, we currently leave gVisor overhead accounting at zero. AI use: AI-assisted, I reviewed and tested it. Tested with a fresh install on GKE (gVisor and microVM e2e lanes). Breaking change: worker pools hold a little less. A worker's memory capacity is now its memory limit minus 64MiB, and each microVM actor books its limit plus 128MiB + 8MiB/GiB. A pool sized to exactly fit its actors (e.g. pod memory limit equal to one actor's limit) needs its `limits.memory` raised, roughly 64MiB per worker plus the microVM overhead per actor. MicroVM guests also now see their whole memory limit, 128MiB more than before. **These values are subject to tuning in the future**. - [x] Tests pass - [x] Docs update --------- Signed-off-by: Benjamin Elder <bentheelder@google.com>
Part of agent-substrate#3217. The default atenet dataplane runs envoy-dataplane in the router and egress gateways, but releases do not publish it, so a pre-built install has to use `--atenet-dataplane=agentgateway` or build from source. agent-substrate#2112 dropped it because building Rust under QEMU was slow. agent-substrate#3152 made the Dockerfile cross-compile, so the release job now builds it with buildx and no QEMU. Released Dockerfile images are an allowlist (`releasedDockerfiles`) next to the ko allowlist. The release jobs pin BuildKit by digest. Release builds take each Dockerfile image's digest from buildx's `--metadata-file`, build Dockerfile images before the ko images, ignore `DOCKER_BUILD_FLAGS`, and label every image with its source, revision and version. With every image a default install deploys now published, the release notes give the install command. After the first release that includes it, an org owner has to make the new `ghcr.io/agent-substrate/substrate/envoy-dataplane` package public. ### Validation `release.yaml` from this branch, run in my fork at 619f0d9 with a throwaway tag `v0.0.0-envoytest.1`: - [`images-dry-run`](https://github.com/MushuEE/substrate/actions/runs/40562367990), dispatched with `dry_run_images`, built all eleven release images cold on `ubuntu-latest` and passed the platform and digest checks in 11m35s, well inside the 30-minute timeout. - The [tag push run](https://github.com/MushuEE/substrate/actions/runs/40561810523) created the draft release. The images job skips itself outside agent-substrate/substrate, as designed. Earlier, at bbd8614, I ran the same publish locally against a throwaway registry, as the `images-dry-run` job does: ```sh docker run -d --rm --name ate-release-test -p 5000:5000 \ registry:3@sha256:ddf754342cfc8acc51a56d5d0ab6af06826461864460636d8bd5c546dab2a7b8 printf '[registry."localhost:5000"]\n http = true\n' > /tmp/buildkitd.toml docker buildx create --name ate-release --driver docker-container \ --driver-opt network=host --buildkitd-config /tmp/buildkitd.toml --use VERSION=v0.0.0-local.1 KO_DEFAULTPLATFORMS=linux/amd64,linux/arm64 \ go run ./cmd/ate-setup publish release-images --ko-docker-repo localhost:5000/substrate docker buildx imagetools inspect localhost:5000/substrate/envoy-dataplane:v0.0.0-local.1 docker buildx imagetools inspect localhost:5000/substrate/envoy-dataplane:v0.0.0-local.1 \ --format '{{json (index .Image "linux/amd64").Config.Labels}}' ``` - buildx cross-compiled the Rust module (`linux/amd64->arm64`), with no QEMU. - The envoy-dataplane digest from `--metadata-file` matched what the tag resolved to. - envoy-dataplane had `linux/amd64` and `linux/arm64`, and the `source` and `revision` labels. - A rerun built nothing. That run showed envoy-dataplane inheriting `org.opencontainers.image.version: "22.04"` from the Ubuntu base of `envoyproxy/envoy`. Release images now set that label to the release tag. AI use: I used Claude Code to trace the code agent-substrate#2112 removed, draft the change, address review comments, and script the local and fork validation. I reviewed the diff and ran the ate-setup tests and the validation above. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Grant McCloskey <gmccloskey@google.com>
…gent-substrate#3218) Follow-up to agent-substrate#2836. A Checkpoint retry still failed with `runsc checkpoint: exit status 128` when atelet died before the snapshot committed: while staging the local snapshot or after publishing it (both reported by @eren23), or while the worker was still capturing the workload, which the worker then canceled. - atelet stages the snapshot files as hard links, so checkpoint-state keeps them until the commit, and a retry replaces an uncommitted local snapshot. - The workers finish CheckpointWorkload under the caller's deadline but not its cancellation. TestRetriedCheckpointAfterUncommittedStaging and TestCheckpointContext cover the change, and the TestCheckpointBoundaryAudit fixture from agent-substrate#2836 passes all 15 cases, up from 9. The "pause move fails" row of TestFailedCheckpointKeepsLocalSnapshot is removed, since a leftover local snapshot directory is now replaced. AI use: AI-assisted. I reviewed and tested it. - [x] Tests pass --------- Signed-off-by: Chenyi Wang <chenyiwang@google.com>
… its node pool label to ate-system-infra (agent-substrate#3210) `--cordon-control-plane` also pins atenet-router and atenet-egress, which are data plane components, so the name is confusing. Renamed: | Old | New | |---|---| | `--cordon-control-plane` | `--cordon-system-infra-components` | | `ATE_INSTALL_CORDON_CONTROL_PLANE` | `ATE_INSTALL_CORDON_SYSTEM_INFRA_COMPONENTS` | | `cordonControlPlane` (config file key) | `cordonSystemInfraComponents` | | `ate.dev/workloadType=ate-control-plane` (shared pool label and taint) | `ate.dev/workloadType=ate-system-infra` | | `manifests/ate-install/components/cordon-control-plane` | `manifests/ate-install/components/cordon-system-infra-components` | The postgres pool label `ate.dev/workloadType=ate-postgres` is unchanged. The missing postgres pool error linked to `docs/upgrade.md#cordoned-control-plane-postgres-pool`, which doesn't exist. It now links to the System Infra Isolation section in `docs/architecture.md`. AI use: I used Claude Code to make the rename and update the docs and comments, and I reviewed the diff. ## Breaking change Installs that don't use `--cordon-control-plane` need no action. Installs that do: 1. Replace the old flag, env var or config file key with the new one in your scripts and config. The old flag and config key fail with an error. **The old env var is ignored silently**, and the install would stop cordoning. 2. Relabel and retaint the shared pool from `ate-control-plane` to `ate-system-infra`. Running pods keep running, because node selectors and `NoSchedule` taints only apply at scheduling. On GKE: ```bash gcloud container node-pools update <shared pool> --cluster $CLUSTER --location $LOCATION \ --node-labels ate.dev/workloadType=ate-system-infra gcloud container node-pools update <shared pool> --cluster $CLUSTER --location $LOCATION \ --node-taints ate.dev/workloadType=ate-system-infra:NoSchedule ``` Both flags replace the pool's whole set of labels or taints, so include any others the pool carries. gcloud asks for confirmation. Add `--quiet` to skip it. GKE updates the existing nodes in place. On other providers, update the pool's node template the same way. For nodes labeled by hand, overwrite the label and taint in place: ```bash kubectl label node <node> ate.dev/workloadType=ate-system-infra --overwrite kubectl taint node <node> ate.dev/workloadType=ate-system-infra:NoSchedule --overwrite ``` Don't remove the old taint before adding the new one. While the node has no taint, DaemonSets such as atelet schedule pods onto it. 3. Redeploy with `ate-setup deploy ate-system --cordon-system-infra-components`. The new pods schedule onto the relabeled pool. Do steps 2 and 3 close together. A system infra pod rescheduled between them stays Pending until step 3. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Haiyan Meng <haiyanmeng@google.com>
…substrate#3318) Move atelet's e2e-specific snapshot upload delay out of the kind overlay into a dedicated kustomize component at manifests/ate-install/components/e2e, selected with --e2e (or ATE_INSTALL_E2E=true / config.e2e.enabled). This keeps regular kind development and production installs on the default atelet snapshot timings while allowing both kind and other e2e runs to opt into the shorter e2e delays. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated Signed-off-by: msau42 <msau@google.com>
Allow a user to land traffic on an actor from a dev machine. The `kubectl-ate` CLI now authenticates to `atenet` by minting a certificate (via `PodCertificateRequest`) for `spiffe://cluster.local/ns/ate-system/sa/atenet-client`. Ideally, atenet should support JWT based auth and there would be no need for a certificate minting. But that support is not in place yet. The `kubectl ate proxy actor` command also improves local dev ergonomics as it attachjes the actor header so clients don't need to be "substrate-aware". For example, a web browser can just point to the local proxy. Fixed agent-substrate#3194 and agent-substrate#3187 --------- Signed-off-by: Julian Gutierrez Oschmann <juliangut@google.com>
gvisor-default gains v6, release 20261026.0, as its default version. v5 stays enabled, so actors with memory snapshots taken on v5 still restore with their memory. In a sandbox which ateworker-gvisor runs, the new release's restore picks each container's real init process. The old one could take an orphan, so signals reached the wrong process and runsc reported a running container as stopped. The new gVisor release includes some reliability fixes such as: - Fix nil pointer crash in restoreDone on the sandbox restore path. - Fix no-root-container-mode sandboxes: shared PID namespace could panic the Sentry. - Fix kernel state dependency-cycle which could panic or recurse forever. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated Signed-off-by: Nayana Bidari <nybidari@google.com>
agent-substrate#3333 Signed-off-by: Luiz Oliveira <luizaoj@google.com>
…gent-substrate#3338) Verify that RevertActor returns an actor to its most recent completed durable snapshot from RUNNING, paused SUSPENDED after background upload, and CRASHED states, and reverts to the template golden snapshot when the actor has no snapshot of its own. Fixes part of agent-substrate#3147 AI use: AI-assisted, I reviewed and tested it. No breaking change - [x] Tests pass - [x] Docs updated Signed-off-by: Nayana Bidari <nybidari@google.com>
…ster (agent-substrate#3296) `hack/teardown.sh --all` deletes resources that every Substrate cluster in the project shares, so cleaning up one cluster broke image pulls and snapshot transfers on the others. Fixes agent-substrate#3227 - `--cluster-scoped` deletes only the cluster, its buckets and the bindings on them. - `--all` still deletes everything setup-gcp created, including the dashboards, the transfer service account and the project-level IAM bindings. The usage text lists those steps as project-wide. - The README says to use `--cluster-scoped` while other clusters share the project. Assisted by Claude Code. I reviewed the diff. --------- Signed-off-by: Haven Xia <haoyuxia@google.com>
…te#2499) A dense run meets limits the demo path never reaches. One new section lists them with the setting for each: reaching the API and router without `kubectl port-forward`, PostgreSQL slots and probe timeouts, the parking budget, synchronized bursts against the ext_proc breaker, the kernel neighbor table, restores under full load, kubelet CPU reservation, and what to watch during the run. Docs only. --------- Signed-off-by: Davanum Srinivas <davanum@gmail.com>
agent-substrate#3339) `TestActorEgressMultiplexing` previously sent 250 concurrent requests in `/mux` to overflow the 100-stream per-connection limit and scraped Envoy admin metrics to check HTTP/2 multiplexing. Include `protocol=%PROTOCOL%` in the Envoy egress CONNECT access log (`manifests/ate-install/atenet-egress.yaml`), assert `protocol=HTTP/2` in `assertEgressGatewayConnect`, and reduce `muxRequestCount` in `demos/egress` from 250 to 10 concurrent requests. This verifies that the cluster negotiated HTTP/2 and multiplexed the concurrent streams without needing to overflow 100 streams or scrape Envoy admin metrics in `TestActorEgressMultiplexing`. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated Signed-off-by: Bo-Cheng Chu <bochengchu@google.com>
…lugins/credprovider-gcp-secret-manager (agent-substrate#3365) We're keeping a set of basic functionality that allows us to boot up a kind cluster and get substrate working in core, but additional plugins / implementations are moving out. This is in-line with the cloud-provider decoupling proposal. We're doing this now, during the GA push, so that we don't change the publish locations underneath users. We have not yet published any pre-built binaries, we will for the next tag cut. Following the naming agreed in agent-substrate#2595, the GCP Secret Manager credential provider now lives in [agent-substrate-plugins/credprovider-gcp-secret-manager](https://github.com/agent-substrate-plugins/credprovider-gcp-secret-manager), with module path `plugins.go.ate.dev/credprovider-gcp-secret-manager`. It's git history moved with it, with full attribution (direct push for initial setup, so I am not the committer, the original authors arE) Part of agent-substrate#3161 AI use: AI-assisted, I used Claude Code to assist with staging the new repository and prepare this change, and reviewed both. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Benjamin Elder <bentheelder@google.com>
Signed-off-by: Scott Funkenhauser <sfunkenhauser@google.com>
agent-substrate#3333 Signed-off-by: Luiz Oliveira <luizaoj@google.com>
Add a dedicated guide describing how to augment or replace Substrate's ingress and egress gateways via ext_proc callouts or bring-your-own gateways. Replaces the outdated extensibility section in `docs/security.md` and links tracking issues agent-substrate#3258 and agent-substrate#3259. AI use: some help with drafting. I reviewed and iterated. - [x] Tests pass: no code - [x] Appropriate changes to documentation are included in the PR --------- Signed-off-by: Greg Castle <gcastle@google.com>
…rate#3382) agent-substrate#3333 Signed-off-by: Luiz Oliveira <luizaoj@google.com>
…the actor (agent-substrate#3363) Adds a note to `docs/egress-traffic.md` with the errors to expect and the command to check the actor's egress policy. Part of agent-substrate#3228. - [ ] Tests pass - [x] Docs updated Signed-off-by: Yufan Su <yufans@google.com>
This document covers proposed SLIs, SLOs, and non-SLO alerts to monitor essential surfaces in Substrate. Some lower-priority surfaces are still tbd but this is likely good enough for v1.0, as sophisticated operators can likely self-unblock given existing observability. I tested the configurations shown here on GKE 1.35 using GMP, and there are some idiosyncrasies in the code samples as a result. I've tried to call these all out in comments to make it clear how operators can clean them up, either now (e.g. if they're using Prometheus directly and not GMP) or in the future (when the idiosyncrasies are fixed). Fixes agent-substrate#574 - [ n/a ] Tests pass - [ x ] Appropriate changes to documentation are included in the PR Signed-off-by: Adrian Ludwin <adrian.ludwin@gmail.com>
…bstrate#3342) agent-substrate#3333 Signed-off-by: Luiz Oliveira <luizaoj@google.com>
…ate#3353) A suspend writes about as much page cache as the actor has in memory, and it's charged to the worker pod. In a pod sized close to its actors that runs the pod out of memory, and the kernel OOM kills the worker along with every actor on it. Part of agent-substrate#3092. - While a microVM snapshot or gVisor checkpoint is written, its cgroup gets `memory.high`, so the page cache can only grow into what the actor booked and isn't using, plus what the pod has spare to lend. If the pod can lend all of it there's no limit, so pods with room still suspend at full speed. Tight pods slow down to disk speed instead. - A worker writes at most 2 snapshots at once, and a newly placed actor waits for lent memory to be written back before it boots - The rootfs upper and durable-dir tars write out and drop their page cache every 8MiB, so they can't pile up dirty pages either. This waits on the disk, so it gives back some of the speed agent-substrate#3026 got by dropping the fsync. Happy to tune it. - microVM actor overhead goes from 128MiB back to 4MiB + 8MiB/GiB, the 128MiB in agent-substrate#3104 was headroom for these writes. Both workers keep 32MiB plus 16MiB per write slot (64MiB) for themselves. Tested on GKE with a fresh install: - A 1GiB microVM actor filled to 893MiB in a 1132Mi pod survived 4/4 suspend/resume cycles - four 256Mi gVisor actors filled to 225MiB in a 1088Mi pod survived 6/6 concurrent suspend waves. - Without the cap that gVisor case OOM-killed the worker. Neither worker restarted. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Benjamin Elder <bentheelder@google.com>
List the version skew rules a code change is most likely to break, so developers' agents follow them without opening the policy. They apply from v1.0.0. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Haven Xia <haoyuxia@google.com> Co-authored-by: Davanum Srinivas <davanum@gmail.com>
…us (agent-substrate#2644) When the actor used to build a template crashes, the template status now includes the actor's crash message. This makes failures such as a bad image volume or failed restore visible directly on the template. Overlaps with agent-substrate#2453, which also adds crash details to template failures. - [x] Tests pass: `go test -race ./cmd/ateapi/internal/controlapi/`; `go vet`. - [x] Appropriate changes to documentation are included in the PR: none needed. Signed-off-by: Davanum Srinivas <davanum@gmail.com>
… template by default (agent-substrate#3366) Remove the default `ATE_ENV_IMAGE` digest and exclude `ate-env` from the default `WORKLOAD_TEMPLATES` list. `deploy.sh` now deploys `ate-env` only when `ATE_ENV_IMAGE` is set (or `WORKLOAD_TEMPLATES` includes `ate-env`), and always includes it on `--delete` so teardown cleans up the template. - [x] Tests pass - [x] Docs updated Signed-off-by: Sairaj Pokale <sairajpokale@google.com>
…ate#3354) Since agent-substrate#2718, both scripts only forward to `go run ./cmd/ate-setup`. This PR deletes them, and every caller, CI step and doc now runs `ate-setup` directly. CI passes `--kind` in place of `install-ate-kind.sh`. The kind Quickstart sets the same option once with `export ATE_INSTALL_KIND=true`, so the demo READMEs no longer mention kind or GKE. Fix agent-substrate#1503 ### Breaking Change There are some behavior changes: - `ate-setup` exits 1 on an unknown subcommand such as `deploy demo demo-counter`, instead of printing help and exiting 0, this fixes agent-substrate#3209 - `--setup-csi` no longer accepts the old shell spellings `true`, `false` or an empty value because they are returned `unknown CSI driver`, the previous params are left for compatiblity with shell script. - `hack/setup-csi-hostpath-kind.sh` is replaced by `ate-setup --kind setup csi hostpath`. ### Testing Ran every demo README as written on fresh kind and GKE clusters. Every `ate-setup` step works on both. claude-code-multiplex, parking and the counter microVM script still fail, as they did before this PR (agent-substrate#2487, agent-substrate#3287; agent-substrate#2092 rewrites the microVM script). Claude help review the list and do the replacement, I reviewed. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Haven Xia <haoyuxia@google.com>
…ion (agent-substrate#3394) `benchmarking/automation/README.md` only covers running tests from the CronJob. Running the cold resume tests once from a workstation takes steps that aren't written down: the node pools for `--cordon-system-infra-components`, a `file://` clone, a separate `SUBSTRATE_DIR` and `KUBECONFIG`, and the variables the install needs in the target cluster file. Step 6 sets `orchestrator.SUBSTRATE_DIR` from a small wrapper script, because the orchestrator has no flag for it. I can add a `--substrate-dir` flag in a follow-up. AI use: Claude Code helped write the guide and this description, and ran the steps on a GKE cluster. - [x] Tests pass - [x] Docs updated Signed-off-by: Haiyan Meng <haiyanmeng@google.com>
…substrate#3213) Follow-up to agent-substrate#3015. Part of agent-substrate#2836. A retried CheckpointWorkload reuses the worker's record of a completed checkpoint. The worker does not sync the snapshot files, so after a node reboot the record and the files can survive with torn contents. The retry then reused them, and atelet stamped the local snapshot with the new boot ID, so its pre-reboot check no longer catches them. - RecordCheckpoint records the node's boot ID, and ReuseCheckpoint skips a record from another boot. After a reboot, the retry fails instead of committing files that may be torn. - atelet's boot ID reader moves to internal/bootid so the workers can use it. TestRecordAndReuseCheckpoint covers a record from another boot. AI use: AI-assisted. I reviewed and tested it. - [x] Tests pass --------- Signed-off-by: Chenyi Wang <chenyiwang@google.com>
…substrate#3123) Partially addresses agent-substrate#3046. Draining a node under sync-suspend load crashed actors with `suspend failed: atelet Checkpoint: while calling worker CheckpointWorkload: rpc error: code = Unavailable ... dial unix /var/lib/ate/worker/...`. Checkpoint wrapped the worker's error with fmt.Errorf, so atelet's interceptor returned Internal and ateapi crashed the actor. Run already keeps a worker's Unavailable through `workerCallError`; Checkpoint now does the same, so ateapi returns Unavailable and keeps the actor SUSPENDING with its assignment for a retry. A worker whose pod is gone, or that restarted and registered again, still crashes its actors through the worker workflows. Unlike Restore there is no retry limit: a crash after a failed checkpoint loses the running actor's state. A worker that restarts and never registers (agent-substrate#3118) therefore leaves the actor SUSPENDING, with ResumeActor refused, until that issue is fixed; before this change the next suspend crashed it. Tests: `TestCheckpointKeepsWorkerUnavailable` (fails without the change), Canceled and DeadlineExceeded rows in `TestWorkerCallError`. `go test -race -count=3 -shuffle=on ./cmd/atelet/...`, gofmt and go vet are clean. AI use: code and tests written with Claude Code; a second Claude session reviewed the diff adversarially and its findings were applied. I reviewed the final diff and the test results by hand. --------- Signed-off-by: Aditya Shantanu <aditya-shantanu@users.noreply.github.com> Co-authored-by: Aditya Shantanu <aditya-shantanu@users.noreply.github.com>
…3376) Part of agent-substrate#3351. This is a workaround. The real fix is for the caches to flush their writes to disk. atelet doesn't sync cache writes, so an unclean reboot can leave cache entries, image layers and bundle overlay specs empty or truncated. atelet then serves them, every restore on the node fails, and the image cache GC stops on the unreadable spec. At startup, atelet now compares the node's boot ID with the one it recorded when the caches were last written (`/var/lib/ate/cache-boot-id`). If the node rebooted since, it empties the image, sandbox asset and snapshot caches, and removes each actor's `bundles` dir and each worker's `default-runtime` dir. Those hold specs naming cached layers and hard links to cached assets, which atelet would otherwise reuse. Run and the worker sandbox syncer recreate them. This is the approach agent-substrate#3050 took for local snapshots. The rest of an actor's dir stays. Run and Restore rebuild it before use, and Terminate of an actor that crashed in the reboot names no worker, so it doesn't read the actor's sandbox record or asset links. The cost is cold caches after any reboot. When the record is missing, atelet only records the boot, since actors may be running on the caches the first time it starts with this check. So a node that is already broken stays broken after the upgrade until it reboots again. Reboot or replace such a node once after upgrading. On a GKE node, I filled the snapshot and image caches, waited 10s and hard-reset the VM. On main this tore 92 image cache and 10 snapshot cache files, and both a golden restore and a new actor from the torn image failed with `unexpected end of JSON input`. With this change, the caches were cleared at startup and both resumes succeeded. I used an AI agent (Claude Code) to write this change and its test, test it on a GKE node with a hard reset, and draft this description. --------- Signed-off-by: Ivy <igooch@google.com>
…-substrate#3357) Follow-up to agent-substrate#2544, from review comment agent-substrate#2544 (comment). `snapshotConfig.storageLocation` was accepted and ignored while the `PerTemplateSnapshotLocation` gate was off. A user could name a bucket, get a success response, and have snapshots written under the cluster `--snapshot-location`. ### What changed? Tested on kind with the cluster `--snapshot-location=gs://ate-snapshots/`. `<loc>` is the template's `storageLocation`. | Gate | Case | Before (`7e6476cbd`) | This PR | |---|---|---|---| | ON | with `storageLocation`, `--preview` | created, snapshot under `<loc>` | created, snapshot under `<loc>` | | ON | with `storageLocation`, no `--preview` | rejected: `Forbidden: requires a client that opts in to preview features ...` | rejected: `Forbidden` | | ON | without `storageLocation` | created, snapshot under `gs://ate-snapshots/` | same | | OFF | with `storageLocation`, `--preview` | created, but snapshot under `gs://ate-snapshots/`, not `<loc>` | rejected: `Forbidden` | | OFF | with `storageLocation`, no `--preview` | created, field ignored | rejected: `Forbidden` | | OFF | without `storageLocation` | created, snapshot under `gs://ate-snapshots/` | same | ### API validation - With preview off, a template that sets `storageLocation` is rejected with `InvalidArgument`. Before, it was accepted and silently ignored. - The old custom check is replaced by the standard ifDisabled forbidden rule that other preview fields use. ### Demos and benchmarks - No demo or benchmark template sets `storageLocation` any more. They all store snapshots under the cluster `--snapshot-location`. - `${BUCKET_NAME}` templating is gone from the demo rendering and the benchmark deploy scripts. The traffic-driver script loses `--bucket-name`. ### Install (ate-setup) - The comments and operator docs now say that the demos, benchmarks and e2e need `SNAPSHOT_LOCATION` even with the gate on. ### E2E fixtures and suites - With the gate on, the fixtures, combinedvolumes, and the demo, golden and template-update tests each set their own per-template location through `e2e.SnapshotLocation`. With it off, they leave the field out. - Templates copied from a demo also get their own location with the gate on. This brings per-template coverage back to the networkpolicy and egress suites. ### Unit tests - Validation tests now expect `Forbidden` when preview is off. Length and URI checks run with preview on. - The control API tests no longer rely on a default `storageLocation`. They set the cluster default the way nearby tests do. - The demo manifest tests and the kubectl-ate template tests fail if a shipped manifest sets `storageLocation` again. ### Docs - The API guide, operator install docs, demo READMEs, benchmarking README and the preview gate doc no longer mention bucket templating. They describe the field as preview-only. --------- Signed-off-by: Sneha Aradhey <snehaaradhey@google.com>
…tup sets it from ATE_EGRESS_GATEWAY_EXTRA_SPIFFE_IDS (agent-substrate#3360) Why is this needed? Credential plugins may need an actor JWT to exchange for other credentials, e.g. through an RFC 8693 exchange. Today only `egress_gateway` can call `MintActorJwt`; ate-api-server grants that role to one SPIFFE ID. It's not possible to grant permissions to generate Actor JWTs to any other identity. This makes `--egress-gateway-spiffe-id` repeatable and grants `egress_gateway` to every ID passed. An operator who deploys a plugin lists its SPIFFE ID in ate-setup's new `ATE_EGRESS_GATEWAY_EXTRA_SPIFFE_IDS` setting (`--egress-gateway-extra-spiffe-ids`). ate-setup then passes `atenet-egress`'s ID first, since setting the flag replaces the default, followed by the listed IDs. The default is still `atenet-egress` alone, and an empty ID still fails startup. There is no API or manifest change. A listed ID gets the whole `egress_gateway` role: besides minting actor JWTs, it can get and list actors, read their egress policies and mint actor certificates. Part of agent-substrate#1661. AI use: AI-assisted draft using Claude. - [ ] Tests pass - [x] Docs updated --------- Signed-off-by: Max Thompson <maxthomp@google.com>
Fixes agent-substrate#1863 > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR Github Pages URL from the fork: https://moficodes.github.io/substrate/ Signed-off-by: Mofi Rahman <mofi@google.com>
…te#2887) Probe prctl(PR_SCHED_CORE_GET) once and pass --enable-core-tags to runsc when core scheduling is available on the host kernel to isolate gVisor sandboxes across SMT siblings. Fixes agent-substrate#2136 *AI disclosure: AI tooling assisted in writing and testing parts of this change.* - [X] Tests pass Signed-off-by: Fabricio Voznika <fvoznika@google.com> Signed-off-by: Fabricio Voznika <fvoznika@google.com>
…ent-substrate#3409) Part of agent-substrate#1661. We plan an out-of-tree credential provider that exchanges an actor's JWT for a cloud access token (RFC 8693), so actors can call Google Cloud APIs as themselves. Clusters will also run Secret Manager or Kubernetes Secrets providers. Today the egress gateway dials one provider and refuses URIs naming any other, which prevents us from supporting multiple credential providers. - atenet's `--credential-provider-name`, `-address` and `-server-name` repeat once per provider. The gateway sends each `ate-secret://` URI to the provider its host names, and still fails closed on any other. - ate-setup's `--credential-provider` also takes a JSON array, rendered for both the Envoy and agentgateway dataplanes. The single-object JSON and single flags behave as before, except that atenet now rejects an empty `--credential-provider-address`, which used to turn injection off, and an address without a `--credential-provider-name` (ate-setup always passes both; `--credential-provider-name=` keeps the unchecked dev mode). Providers share one CA and client certificate, as they do today. Docs follow in agent-substrate#3410. Tested with unit tests. Not run: kind e2e with two providers, which is worth a follow-up, or agentgateway with more than one entry. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated (in agent-substrate#3410) --------- Signed-off-by: Max Thompson <maxthomp@google.com>
…Hub button (agent-substrate#3426) Aligns the documentation website footer with the [CNCF Website Guidelines](https://github.com/cncf/foundation/blob/main/policies-guidance/website-guidelines.md) as part of CNCF Sandbox onboarding (agent-substrate#3060) and updates the top navbar GitHub link: - **CNCF Sandbox Attribution & Logo (Guideline 6):** Adds *"We are a Cloud Native Computing Foundation sandbox project."* alongside the official CNCF horizontal SVG logo in the homepage footer. - **Linux Foundation Series LLC Copyright & Policy Link (Guidelines 5 & 7):** Updates the homepage and docs footers with `Copyright © 2026 Agent Substrate a Series of LF Projects, LLC. For website terms of use, trademark policy and other project policies please see policy.` linking to `https://lfprojects.org/policies/`. - **Subtle Origin Statement (Guideline 2):** Adds a muted *"Agent Substrate was originally created by Google."* line beneath the brand tagline in the homepage footer. - **Navbar GitHub Button:** Changes the top-right navbar button from `Star on GitHub` (star icon) to `GitHub` with the GitHub mark icon. Signed-off-by: Mofi Rahman <mofi@google.com>
…actorauthz (agent-substrate#3138) Step 1 of agent-substrate#3095. No behavior change. `ExchangeActorJwt` runs the checks that guard the GCS credential exchange inline. A second provider's exchange would have to copy them and could drift from them. - New package `internal/objectstoreplugin/actorauthz`. `Authorizer.Authorize` verifies the worker-for-actor JWT, looks the actor up, checks it is still on the token's node, and derives its grants. Its `Decision` carries only `ActorUID` and `Grants`. Codes and messages are unchanged. - `exchange.Verifier` and `exchange.Lookup` move to actorauthz and are passed to `actorauthz.New`. `exchange.New` takes the one-method `exchange.Authorizer`, which `*actorauthz.Authorizer` satisfies, and `exchange.Server` calls `Authorize`, then the issuer. - `exchange.ActorJWTAudience` becomes `actorauthz.Audience`, with the same value. atelet, the plugin and the snapshotcredentials e2e suite use it. - `hack/verify/package-deps.sh` replaces `hack/verify/ateapi-deps.sh`. It keeps the ate-api-server check and adds one that actorauthz depends on no provider package or cloud SDK, on every platform in `.ko.yaml`. Tested with `go test -race` on `./internal/objectstoreplugin/...`, `./cmd/objectstore-plugin/...`, `./cmd/atelet/...` and `./internal/resources/...` on Linux. - [x] Tests pass - [x] Docs updated AI use: I used Claude Code to implement and review this change, and reviewed the diff myself. --------- Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>
… down its worker (agent-substrate#3429) A microVM actor's cloud-hypervisor and virtiofsd run in the worker pod with no memory limit of their own. If they grow past what the actor booked, they take the pod to its limit, and the OOM kill takes out the worker and every actor on it. agent-substrate#3106 fixed this for gVisor, this PR fixes it for microVM. Part of agent-substrate#3092. - each actor's cgroup gets `memory.high` at what it books (its memory limit plus the runtime overhead from agent-substrate#3104). The guest can't use more than its memory limit, so anything past that is cloud-hypervisor or virtiofsd. - the worker kills a cgroup stuck at its `memory.high` with `cgroup.kill`, using the same rule as agent-substrate#3106, and reports the actor crashed with "killed for exceeding its memory limit". Other actors keep running. - the watch pauses during snapshot writes. When the pod can lend the whole write, the cgroup's `memory.high` is lifted for the write and restored after, same as agent-substrate#3353. - without PSI the worker sets no limit and warns at startup, like gVisor - the gVisor watch from agent-substrate#3106 moves into a shared `cgroup.MemoryGuard` that both workers use, gVisor's behavior is unchanged Tested on GKE with a fresh install: - a 1GiB actor filled to 925MiB peaked at 1007MiB against a `memory.high` of 1036MiB, and survived 3/3 suspend/resume cycles with no kills - a 400MiB process added to one 512Mi actor's cgroup got that actor killed and reported crashed. Its neighbour kept answering and the worker didn't restart. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Benjamin Elder <bentheelder@google.com>
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bring the fork up to
agent-substrate/substrate@89475d52bb3f29197b56391afa5c2d8955a7f94d, preserving upstream history and the Helm deployment.Includes checkpoint and node-reboot recovery fixes, microVM memory accounting, gVisor core tagging, installer support for multiple egress credential providers, AgentGateway v1.7.0, and the documentation site. The chart now grants atelet permission to report Events and uses gVisor release
20261026.0. Helm E2E uses the current installer commands and the snapshot upload delay required by the new revert tests.Upgrade: configure
atenetRouter.ingressAllowedSPIFFEIDsand use mTLS on ports 8443/8444. Plaintext router listeners are removed, and an empty allowlist denies ingress. Upgrade custom gVisor configurations with the worker; existing memory snapshots require a compatible runtime. This needs thebreaking-changelabel.Update generated clients and JSON consumers from
SnapshotStorage.status/SnapshotStorageStatustostate/SnapshotStorageState. Templates that setsnapshotConfig.storageLocationnow require thePerTemplateSnapshotLocationpreview gate; remove the field to use the global snapshot location.Local validation: repository verifiers, manifest/CRD parity, migration immutability, Helm lint, and 40 Helm tests passed. All four Helm E2E lanes passed with previews enabled: 242 tests passed and four skipped. Two snapshot-credential tests require GCS; the cluster used S3. The microVM clock test skipped on gVisor, and external-volume node migration skipped on microVM. The microVM lane ran the demo suite, which does not include the clock test. The full Go race suite (
go test -race -p=1 ./...) passed.Merge with a merge commit to retain upstream ancestry. The branch contains one sync merge with
origin/mainas its first parent and upstream as its second parent.AI use: Codex prepared the merge, reviewed integration changes, and ran validation. A separate agent reviewed the integration diff.