ate-setup: add a `render` subcommand for kustomize generation by stshrive · Pull Request #3411 · agent-substrate/substrate

Steven Shriver (stshrive) · 2026-10-28T21:44:30Z

Release artifacts, Helm charts and provider CI need the manifests that deploy ate-system applies, without access to a cluster. render ate-system writes them to --output-dir, one file per apply step, numbered in apply order. It requires --image-repo and --image-tag and pins every image to a digest.

It also writes the settings ConfigMaps that deploy creates through the API, and ate-api-server-secret-envvars for the bundled database. An external database's connection strings are not written.

It leaves out the CA and JWT pool Secrets, which each hold a private key generated for one cluster: rendering them would put the keys wherever the output goes, and every apply would replace the pools and invalidate the certificates and tokens issued from them. deploy creates each pool only when it is absent, and actor-id-ca-certs is copied from the actor-id-ca-pool Secret in the cluster. It also leaves out the credential provider's namespace policy, the bundled PostgreSQL identities and the CSI drivers.

deploy ate-system now builds the podcertificate-controller manifest, the ate-api-server settings and the service account issuer through the same helpers as render.

Part of #3240.

Related provider-neutral core work:

manifests: remove the GKE-specific install content #3399 removes ate-setup's Cloud SQL support and the GKE content in manifests/ate-install/base. render relies on its config-load rejection of ATE_API_POSTGRES_CLOUDSQL_*, and its output has no GKE content once it lands.
tools: remove setup-gcp and hack/teardown.sh #3396 removes tools/setup-gcp and hack/teardown.sh.
--component for render follows ate-setup deploy accepts --component, and control plane sizing moves into kustomize Components #3179.

Outside kind, render still needs the GCS snapshot location and transfer service account, because the object store defaults to GCS. A later core change makes the backend explicit.

tk (tryambak2019) · 2026-10-29T00:50:45Z

Hi Steven Shriver (@stshrive) ,

RenderAteSystem omits prerequisites that DeployAteSystem creates, such as CA pools, JWT, actor-identity Secrets, egress MITM pool. As these are runtime prerequisites, a fresh-cluster workflow must generate them before apply.
The PostgreSQL choice checks only the external DSN. It ignores an explicitly configured non-empty Cloud SQL instance and renders bundled PostgreSQL.
With an external database, render omits the environment Secret and calculates the Deployment hash without its data. Injecting or rotating that Secret therefore will not update the hash or trigger a rollout.
The documented minimal command fails without a credential-provider choice and, outside Kind, GCP transfer settings. I assume “neutral defaults for every setting” means omitted provider settings must produce portable output.

Separately, --component compatibility remains dependent on #3179.

Release artifacts, Helm charts and provider CI need the manifests that deploy ate-system applies, without access to a cluster. render ate-system writes them to --output-dir, one file per apply step, numbered in apply order. It requires --image-repo and --image-tag and pins every image to a digest. It also writes the settings ConfigMaps that deploy creates through the API, and ate-api-server-secret-envvars for the bundled database. An external database's connection strings are not written. It leaves out the CA and JWT pool Secrets, which each hold a private key generated for one cluster: rendering them would put the keys wherever the output goes, and every apply would replace the pools and invalidate the certificates and tokens issued from them. deploy creates each pool only when it is absent, and actor-id-ca-certs is copied from the actor-id-ca-pool Secret in the cluster. It also leaves out the credential provider's namespace policy, the bundled PostgreSQL identities and the CSI drivers. deploy ate-system now builds the podcertificate-controller manifest, the ate-api-server settings and the service account issuer through the same helpers as render. Signed-off-by: Steven Shriver <6007067+stshrive@users.noreply.github.com>

Signed-off-by: Steven Shriver <6007067+stshrive@users.noreply.github.com>

Steven Shriver (stshrive) · 2026-10-29T01:41:18Z

RenderAteSystem omits prerequisites that DeployAteSystem creates, such as CA pools, JWT, actor-identity Secrets, egress MITM pool. As these are runtime prerequisites, a fresh-cluster workflow must generate them before apply.

This is intentional. I updated the commit message and the PR description to make this a bit more clear. Additional comments in the code were also added.

2. The PostgreSQL choice checks only the external DSN. It ignores an explicitly configured non-empty Cloud SQL instance and renders bundled PostgreSQL.

this is a known, interim gap. #3399 closes it by rejecting ATE_API_POSTGRES_CLOUDSQL_* for every command.

3. With an external database, render omits the environment Secret and calculates the Deployment hash without its data. Injecting or rotating that Secret therefore will not update the hash or trigger a rollout.

This is true and documented in the code. A rotated external Secret won't change the hash or trigger a rollout. Worth a tracking issue? Ultimately the bootstrap work (#3175) owns this Secret.

4. The documented minimal command fails without a credential-provider choice and, outside Kind, GCP transfer settings. I assume “neutral defaults for every setting” means omitted provider settings must produce portable output.

This is dependent on a few things, also added to the PR description to clarify a bit, but some other PRs are removing a bunch of content that impacts render directly. The Cited PRs are generally higher priority and this will land after a few of the others and will need a rebase for a few things. Any lingering provider-neutral issues will be addressed then.

Additionally, I have made a few tweaks to documentation to address this issue.

haiyanmeng · 2026-10-29T12:32:24Z

+	issuer := e.configuredJWTIssuer()
+	if issuer == "" {
+		issuer = inClusterIssuer
+	}


🤖 blocking 🔴 – With --kind this writes https://kubernetes.default.svc, but kind (kubeadm) issues tokens as https://kubernetes.default.svc.cluster.local. deploy reads the real issuer from the cluster, and buildAuthenticationConfig already handles both values. The output of the documented --kind example would make ate-api-server reject every service account token, including the ate-client one kubectl ate uses. Please use the .cluster.local issuer when Kind is set, and have the kind test row assert it.

haiyanmeng · 2026-10-29T12:32:24Z

+// renderCredentialProvider is the configured credential provider, or the
+// bundled Kubernetes Secrets provider when none is set. The bundled provider
+// runs in the install namespace and needs nothing outside the cluster, which
+// is why kind installs use it too.
+func (e *Env) renderCredentialProvider() (config.CredentialProvider, error) {
+	if e.Cfg.CredentialProviderJSON == "" {
+		log.Infof("no credential provider set; rendering the bundled %s provider", config.K8sCredentialProviderName)
+		return config.CredentialProvider{
+			Name:    config.K8sCredentialProviderName,
+			Address: config.K8sCredentialProviderAddress,
+		}, nil
+	}
+	return e.Cfg.CredentialProvider()


🤖 should-fix 🟡 – deploy ate-system refuses to run without --credential-provider, on kind too. Kind installs pass {"name":"k8s.io"} explicitly, so the comment's "which is why kind installs use it too" and the help text's "as kind installs do" (internal/cmd/render.go:72) are misleading. Here render quietly picks the bundled provider, which gets cluster-wide Secret read, and it does so for production renders as well. I'd call e.Cfg.CredentialProvider() directly, the way deploy does, and drop this helper, its test row, and the help text bullet.

haiyanmeng · 2026-10-29T12:32:24Z

+
+// apiServerConfigVars is the ate-api-server-envvars ConfigMap payload without
+// the Cloud SQL settings, which only CreateAPIServerEnvVars adds. render ate-system
+// writes this payload, and Cloud SQL is not part of the core install.


🤖 question 🟢 – #3399 is still open, so on this branch render accepts ATE_API_POSTGRES_CLOUDSQL_* and silently leaves those settings out. Should this PR wait for #3399, or reject Cloud SQL settings itself until then?

haiyanmeng · 2026-10-29T12:32:24Z

+ate-setup create podcertificate-controller-cas
+kubectl apply -f DIR/01-crds.yaml
+kubectl wait --for=condition=Established crd --all
+kubectl apply -f DIR


🤖 nit 🟢 – The bundled provider needs k8s-credential-provider-namespace-policy to start, but this block leaves it out, so following these steps as written leaves the provider unable to start. Could you add the kubectl apply -f manifests/egress-credential-injection/namespace-policy.yaml step here, before the last apply? Then the prose below would not need to repeat it.

haiyanmeng · 2026-10-29T13:59:28Z

+var renderAteSystemCmd = &cobra.Command{
+	Use:   "ate-system",
+	Short: "Write the manifests deploy ate-system applies into a directory",
+	Long: `Write the manifests "deploy ate-system" applies to a fresh cluster into


a fresh cluster -> a cluster, since this cluster may already have Substrate installed.

haiyanmeng · 2026-10-29T14:00:47Z

+  ate-setup render ate-system --kind --image-repo REPO --image-tag TAG \
+    --output-dir DIR
+
+Other clusters store snapshots on GCS today, so they also need the snapshot


Other clusters means non-KIND clusters?

haiyanmeng · 2026-10-29T14:06:45Z

+		Key: "render.outputDir", Env: "ATE_RENDER_OUTPUT_DIR", Flag: "output-dir",
+		Kind:     config.KindString,
+		Commands: []string{renderAteSystemPath},
+		Usage:    "Directory the rendered manifests are written to; it must be new or empty",


Requiring the directory to be new or empty is safe, but may make it difficult to diff the config change.

haiyanmeng · 2026-10-29T14:20:56Z

 resource below on the way, so those subcommands are only needed to redo one on
 a running cluster.

+## Render


🤖 nit 🟢 – This section repeats the Long help of render ate-system almost word for word, and the RenderAteSystem doc comment in steps/render.go lists the same exclusions a third time. The copies have already drifted, for example in how they describe the --gcp-* issuer settings. Other sections in this file are mapping tables with short notes. Could this keep the table row with a pointer to ate-setup render ate-system --help, and could the Go doc comment shrink to what the function writes?

haiyanmeng · 2026-10-29T14:42:53Z

+		if err != nil {
+			return nil, err
+		}
+		out.WriteString("---\n")


🤖 nit 🟢 – kube.ReadPath already prepends "---\n" and guarantees a trailing newline for each file, so out.WriteString("---\n") here produces duplicate ---\n---\n separators in 03-sandboxconfig.yaml.

Haven Xia (HavenXia) · 2026-10-29T22:46:25Z

+// default gVisor SandboxConfig, the order DeploySandboxConfig applies them in.
+func (e *Env) sandboxConfigManifest() ([]byte, error) {
+	var out bytes.Buffer
+	for _, name := range []string{"sandboxconfig-validation.yaml", "sandboxconfig-gvisor.yaml"} {


#3254 is replacing sandboxconfig-gvisor.yaml with sandboxconfig-gvisor.yaml.tmpl, which deploy fills in with a substrate-<hash>-gvisor name, a created-by-version label a the class-default annotation.

Zoe Zhao (@zoez7) for vis, either PR merge later need to be careful on this.

Haven Xia (HavenXia) · 2026-10-29T22:53:53Z

+ate-setup create podcertificate-controller-cas
+kubectl apply -f DIR/01-crds.yaml
+kubectl wait --for=condition=Established crd --all
+kubectl apply -f DIR


This one from Claude: Same gap as the namespace policy, for the bundled database: ate-api-server logs in as substrate_readwrite_user, which only the SQL in setupBundledPostgres creates, so ate-api-server can't connect after this block. Could the block end with ate-setup deploy postgres when the database is bundled?

Haven Xia (HavenXia) · 2026-10-29T23:03:43Z

+later file holds a SandboxConfig:
+
+```bash
+kubectl apply -f DIR/00-namespace.yaml


On a cluster installed with these commands, a field that a later release removes (an env var, a volume, a sidecar) is still there after the runbook's ate-setup deploy steps.

Client-side kubectl apply records kubectl-client-side-apply as the owner of every field, and server-side apply only removes a field no other manager owns.

Asked claude to rerpo this on kind with a ConfigMap key. Could these commands use kubectl apply --server-side --field-manager=ate-setup, so the upgrade steps own the same fields?

Steven Shriver (stshrive) requested a review from haiyanmeng October 28, 2026 21:45

Steven Shriver (stshrive) force-pushed the dev/ate-setup-render branch from 90dd378 to ae8359e Compare October 28, 2026 21:56

Steven Shriver (stshrive) requested a review from Haven Xia (HavenXia) October 28, 2026 21:57

Steven Shriver (stshrive) force-pushed the dev/ate-setup-render branch from ae8359e to 4282ded Compare October 29, 2026 01:21

ate-setup render: neutral documentation update

cbbe58e

Signed-off-by: Steven Shriver <6007067+stshrive@users.noreply.github.com>

Steven Shriver (stshrive) force-pushed the dev/ate-setup-render branch from d3d0aad to cbbe58e Compare October 29, 2026 01:41

haiyanmeng suggested changes Oct 29, 2026

View reviewed changes

haiyanmeng reviewed Oct 29, 2026

View reviewed changes

Haven Xia (HavenXia) reviewed Oct 29, 2026

View reviewed changes

Conversation

Steven Shriver (stshrive) commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tk (tryambak2019) commented Oct 29, 2026

Uh oh!

Steven Shriver (stshrive) commented Oct 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Steven Shriver (stshrive) commented Oct 28, 2026 •

edited

Loading