From 80590c46c7d5be84432186306cb8ca46e2acf3b6 Mon Sep 17 00:00:00 2001 From: Yufan Su Date: Wed, 28 Oct 2026 23:43:20 -0700 Subject: [PATCH 1/4] update egress-credential-injection doc Signed-off-by: Yufan Su --- docs/guides/egress-credential-injection.md | 66 +++++++++++++--------- 1 file changed, 38 insertions(+), 28 deletions(-) diff --git a/docs/guides/egress-credential-injection.md b/docs/guides/egress-credential-injection.md index 610cc58171..b718a98ea4 100644 --- a/docs/guides/egress-credential-injection.md +++ b/docs/guides/egress-credential-injection.md @@ -8,15 +8,18 @@ description: > # Egress credential injection -Egress credential injection puts a secret into an actor's outbound HTTPS -requests without the actor ever holding it. It works on HTTPS the egress -gateway intercepts: an `https` rule in the actor's `EgressPolicy` has the -gateway terminate the actor's TLS for the names it lists and re-originate it -(see [egress-trust-bundle.md](egress-trust-bundle.md)). When that rule carries -a `replace_headers` effect and the actor's request includes a header it names, -the gateway resolves the referenced credential from a **credential provider** -and replaces the header's value with it (for example -`Authorization: Bearer `) before the request leaves the cluster. +Egress credential injection puts a secret into an actor's outbound HTTP and +HTTPS requests without the actor ever holding it. It works on requests the +egress gateway can read: cleartext HTTP that an `http` rule in the actor's +`EgressPolicy` allows, and HTTPS that an `https` rule allows. An `https` rule +has the gateway terminate the actor's TLS for the names it lists and +re-originate it (see [egress-trust-bundle.md](egress-trust-bundle.md)). + +When the rule carries a `replaceHeaders` effect and the actor's request +includes a header it names, the gateway resolves the referenced credential +from a **credential provider** and replaces the header's value with it (for +example `Authorization: Bearer `) before the request leaves the +cluster. The actor never holds the secret. It sends the header with a placeholder value, which the gateway discards, so it cannot read, snapshot, or exfiltrate @@ -61,7 +64,7 @@ flowchart LR terminates the actor's TLS — the actor trusts the gateway CA through a projected trust bundle — and decides the decrypted request against the actor's `EgressPolicy`. -2. If the deciding rule has `replace_headers` and the request carries one of +2. If the deciding rule has `replaceHeaders` and the request carries one of those headers, the gateway asks the credential provider for the credential that header names. 3. The provider decides whether that actor may have that credential, reads it @@ -85,7 +88,7 @@ Injection is declared in the `effects` of an `http` or `https` rule. This policy HTTPS to `api.example.com` on port 443, the default, and replaces the actor's `Authorization` header with a credential: -```bash +```yaml rules: - https: hostnames: ["api.example.com"] @@ -103,9 +106,8 @@ curl -H 'Authorization: placeholder' https://api.example.com/v1/items ``` This URI names the sample Secret deployed under [Enable it](#enable-it). The -sample namespace policy lets only actors in atespace `team-a` resolve it; an -actor in any other atespace gets a 403 until a cluster admin grants its -atespace access. +install's namespace policy grants nothing, so the actor gets a 403 until a +cluster admin grants its atespace access to the `ns1` namespace. * `header` names the request header to replace. * `prefix` is prepended verbatim to the credential — include the separator, @@ -114,9 +116,13 @@ atespace access. ### Credential URIs -A credential URI has the form `ate-secret:///`: the host -names the provider that resolves it, and the path is that provider's to -interpret. A URI the provider refuses denies the request with 403. +A credential URI has the form +`ate-secret:///`. The +host names the provider that resolves it, and the provider interprets the +path. The API requires the host to be a DNS-1123 subdomain and the path to +have at least two non-empty segments. It rejects a port, userinfo, query, fragment, +percent-encoding, or trailing slash. A URI the provider refuses denies the +request with 403. * **Kubernetes Secrets** (provider class `k8s.io`) — see [The reference provider](#the-reference-provider). @@ -196,8 +202,10 @@ no injection says so with `{"enabled":false}`. go run ./cmd/ate-setup deploy atenet --credential-provider='{"name":"k8s.io"}' ``` -`--credential-provider` (or `ATE_CREDENTIAL_PROVIDER` when the flag is -absent) is a JSON object with these keys; any other key is an error: +The setting can also come from `ATE_CREDENTIAL_PROVIDER` or from +`atenet.egress.credentialProvider` in the `--config` file (see +[Precedence](operator-install.md#precedence)). Its value is a JSON object with +these keys; any other key is an error: | Key | Value | |---|---| @@ -267,10 +275,11 @@ kubectl apply -f manifests/egress-credential-injection/sample-secret.yaml ``` **3. The actors.** Actors can now add `replaceHeaders` effects to their -`https` rules, as described under [The policy](#the-policy). Each such actor -also needs the projected egress trust bundle to do TLS through the gateway at -all — see [egress-trust-bundle.md](egress-trust-bundle.md). Actor JWT entries -need no provider; ateapi signs them with a key that every install creates. +`http` or `https` rules, as described under [The policy](#the-policy). An +actor with `https` rules also needs the projected egress trust bundle to do +TLS through the gateway at all (see +[egress-trust-bundle.md](egress-trust-bundle.md)). Actor JWT entries need no +provider; ateapi signs them with a key that every install creates. ### Verify @@ -285,7 +294,8 @@ Then give an actor an `https` rule for `httpbin.org` that replaces a header, have it request `https://httpbin.org/headers` with that header set to a placeholder, and look for the credential in the echoed response. -If the request is denied instead, the gateway logs the reason: +On the Envoy dataplane, the gateway's `ext-proc` container logs why a request +was denied: ```bash kubectl -n ate-system logs deployment/atenet-egress -c ext-proc | grep 'egress denied' @@ -325,7 +335,8 @@ a policy URI naming any other class fails closed rather than being sent to the wrong provider. The gateway reads only the URI host, to confirm the URI targets the provider it -serves. Everything after the host is the provider's to interpret. +serves. The provider interprets the path, within the rules under +[Credential URIs](#credential-uris). [credprovider-gcp-secret-manager](https://github.com/agent-substrate-plugins/credprovider-gcp-secret-manager) is such a provider for Google Cloud Secret Manager. @@ -364,6 +375,5 @@ may only resolve Secrets in namespaces explicitly granted to it. * `api/credprovider/v1/credprovider.proto` — the provider plugin API and its trust model. * `cmd/credential-provider/kubernetes-secrets` — the reference provider. -* `internal/e2e/suites/egresscredinject` — the e2e suite that proves the - behavior table above on both sandbox classes, against a cluster installed - with `--credential-provider='{"name":"k8s.io"}'`. +* `internal/e2e/suites/egresscredinject` — the e2e suite, run against a + cluster installed with `--credential-provider='{"name":"k8s.io"}'`. From 8a5b5865d8a85189c836802bd7c50c9d494d587d Mon Sep 17 00:00:00 2001 From: Yufan Su Date: Wed, 28 Oct 2026 15:29:44 -0700 Subject: [PATCH 2/4] update egress-trust-bundle doc Signed-off-by: Yufan Su --- docs/guides/egress-trust-bundle.md | 54 ++++++++++++++---------------- 1 file changed, 26 insertions(+), 28 deletions(-) diff --git a/docs/guides/egress-trust-bundle.md b/docs/guides/egress-trust-bundle.md index 94db7ffc2f..07cb29529c 100644 --- a/docs/guides/egress-trust-bundle.md +++ b/docs/guides/egress-trust-bundle.md @@ -48,40 +48,36 @@ rule, are answered `NXDOMAIN`. ## Configuring a workload to trust public and man-in-the-middle CAs -Add a `systemInfo` volume with a `trustBundle` data source, and mount it: +Add a `systemInfo` volume with a `trustBundle` data source to the actor +template (the manifest you pass to `kubectl ate create actor-template -f`), +and mount it in each container that makes TLS connections: ```yaml -apiVersion: ate.dev/v1 -kind: ActorTemplate -metadata: - name: my-actor - namespace: my-namespace -spec: - volumes: +volumes: +- name: system-info + systemInfo: + dataSources: + # This file will contain trust anchors for both the MITM CA and standard + # public WebPKI CAs, meaning it will work for actors that use `https` rules, + # `tls_passthrough` rules, or a mix. + - trustBundle: + names: + - egress-mitm.ate.dev + - system-roots.ate.dev + path: trust-bundle.pem +containers: +- name: app + # ... + volumeMounts: - name: system-info - systemInfo: - dataSources: - # This file will contain trust anchors for both the MITM CA and standard - # public WebPKI CAs, meaning it will work for actors that use `https` rules, - # `tls_passthrough` rules, or a mix. - - trustBundle: - names: - - egress-mitm.ate.dev - - system-roots.ate.dev - path: trust-bundle.pem - containers: - - name: app - image: ... - volumeMounts: - - name: system-info - mountPath: /run/ate/egress-trust-bundle/ + mountPath: /run/ate/egress-trust-bundle ``` Substrate only offers a few built-in trust bundles with well-known names: * `egress-mitm.ate.dev` — the egress gateway CA. * `system-roots.ate.dev` — A selection of public CA root certificates provided by Substrate. In official Substrate images, this is the Mozilla Root Store as - shipped on Debian (consumed via the distroless-static base image). + shipped on Debian, read from atelet's `distroless/base-debian13` base image. You then need to configure your application to use the trust-bundle file. In general, this is application specific, but many runtimes respect the @@ -117,7 +113,7 @@ this setup in your application's startup logic). See the [example](#example-e #!/bin/sh set -eu sys=/etc/ssl/certs/ca-certificates.crt # RHEL family: /etc/pki/tls/certs/ca-bundle.crt -ca=/run/ate/trust-bundle.pem +ca=/run/ate/egress-trust-bundle/trust-bundle.pem out=/tmp/ca-bundle.pem { cat "$sys"; echo; cat "$ca"; } > "$out" export SSL_CERT_FILE="$out" REQUESTS_CA_BUNDLE="$out" CURL_CA_BUNDLE="$out" GIT_SSL_CAINFO="$out" PIP_CERT="$out" @@ -130,7 +126,9 @@ Note: Do not bake the bundle into the image at build time. It will tie the image ## Verify `demos/egress/egress-template.yaml.tmpl` is a complete working template that -does exactly this. Deploy it: +projects both bundles to `/run/ate/egress-trust-bundle/trust-bundle.pem`, the +file this recipe uses. Its Go app finds the file through `SSL_CERT_DIR`. Deploy +it: ```bash go run ./cmd/ate-setup deploy demo egress @@ -145,7 +143,7 @@ bundle did the validating. * Problems with trust bundles will block actor startup. Error messages: - `... unknown trust bundle "foo"`: You tried to use an unknown trust bundle name. - - `... trust bundle "foo": unusable contents ...`: Your substrate installation is misconfigured. + - `... trust bundle "egress-mitm.ate.dev": ClusterTrustBundle "egress-mitm.ate.dev:mitm:primary-bundle" not found`: Your substrate installation is misconfigured. * Certificate errors are likely problems with the actor, not the gateway: If you see an error message like `... certificate signed by unknown authority ...`, the most likely cause is that your actor is not configured to trust the MITM From da4aab4979c9c9dc8a63efb3b52c697dec4d03af Mon Sep 17 00:00:00 2001 From: Yufan Su Date: Wed, 28 Oct 2026 23:40:25 -0700 Subject: [PATCH 3/4] update egress-traffic doc Signed-off-by: Yufan Su --- docs/guides/egress-traffic.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/docs/guides/egress-traffic.md b/docs/guides/egress-traffic.md index c71a529765..5ce15eee9b 100644 --- a/docs/guides/egress-traffic.md +++ b/docs/guides/egress-traffic.md @@ -29,7 +29,8 @@ within about 5 seconds. > the actor's egress policy has a rule for it with > `kubectl ate get egress-policy -a -o yaml`. -Other UDP and other traffic is filtered by nftables and never reaches the gateway. +Other UDP and other traffic is rejected by nftables and never reaches the egress +gateway. This applies to an actor's own telemetry too: an OpenTelemetry SDK inside the actor cannot reach the collector until the actor has an `http` egress rule for it. See @@ -39,11 +40,12 @@ cannot reach the collector until the actor has an `http` egress rule for it. See | Port | Traffic | Behavior | Path | What the actor sees when refused | | :---- | :---- | :---- | :---- | :---- | -| any | HTTP(S) 1.1 / 2 | Supported with policy controls | atunnel -> egress gateway -> origin | `403 Forbidden` | -| any | WebSocket | Blocked | n/a | `403 Forbidden` | -| any | Standard HTTP(S) CONNECT (forward-proxy tunnel) | Blocked | n/a | `403 Forbidden` | +| any | HTTP(S) 1.1 / 2 | Supported with policy controls | atunnel -> egress gateway -> origin | `403 Forbidden`. For HTTPS whose SNI matches no `https` or `tls_passthrough` rule, the gateway closes the connection during the TLS handshake, without an HTTP response. | +| any | TLS whose SNI matches a `tls_passthrough` rule | Relayed to the SNI hostname without inspection, so WebSocket, CONNECT, and other TCP inside it get through | atunnel -> egress gateway -> origin | Closed during the TLS handshake. | +| any | WebSocket | Blocked | atunnel -> egress gateway | `403 Forbidden` | +| any | Standard HTTP(S) CONNECT (forward-proxy tunnel) | Blocked | atunnel -> egress gateway | `403 Forbidden` | | 53 | DNS to the sandbox's gateway | Allowed for hostnames matching the actor's egress policy | sandbox DNS relay -> worker pod's resolvers | `NXDOMAIN` | -| any | Any other TCP | Blocked | n/a | The connection is accepted and then closed with no bytes returned. There is no status code. atunnel logs the failure. | +| any | Any other TCP | Blocked | atunnel -> egress gateway | A client that sends first gets `400 Bad Request` before the connection closes. If the client waits for the server to speak first, the connection closes after about 1 second with no data. Only the gateway logs the refusal. | See the [Network Egress Contract](network-egress.md#refusal-responses-and-policy-changes) for refusal status codes, policy-refresh semantics, and the distinction between DNS @@ -59,7 +61,8 @@ refusals and HTTP policy denials. ## Other protocols -Everything that is neither TCP nor UDP is blocked. Packets are dropped, not rejected: no ICMP port-unreachable is sent, so the client hangs until its own timeout. +Everything that is neither TCP nor UDP is rejected with ICMP network +unreachable. ## Requesting support From 2c9a31d5064e8b4681d5ecedf6f209b533c18582 Mon Sep 17 00:00:00 2001 From: Yufan Su Date: Fri, 30 Oct 2026 15:49:51 -0700 Subject: [PATCH 4/4] minor fix Signed-off-by: Yufan Su --- docs/guides/egress-credential-injection.md | 6 ++---- docs/guides/egress-trust-bundle.md | 25 +++++++++++++--------- 2 files changed, 17 insertions(+), 14 deletions(-) diff --git a/docs/guides/egress-credential-injection.md b/docs/guides/egress-credential-injection.md index b718a98ea4..632613e3bb 100644 --- a/docs/guides/egress-credential-injection.md +++ b/docs/guides/egress-credential-injection.md @@ -202,10 +202,8 @@ no injection says so with `{"enabled":false}`. go run ./cmd/ate-setup deploy atenet --credential-provider='{"name":"k8s.io"}' ``` -The setting can also come from `ATE_CREDENTIAL_PROVIDER` or from -`atenet.egress.credentialProvider` in the `--config` file (see -[Precedence](operator-install.md#precedence)). Its value is a JSON object with -these keys; any other key is an error: +`--credential-provider` (or `ATE_CREDENTIAL_PROVIDER` when the flag is +absent) is a JSON object with these keys; any other key is an error: | Key | Value | |---|---| diff --git a/docs/guides/egress-trust-bundle.md b/docs/guides/egress-trust-bundle.md index 07cb29529c..0baa4ab98d 100644 --- a/docs/guides/egress-trust-bundle.md +++ b/docs/guides/egress-trust-bundle.md @@ -53,6 +53,15 @@ template (the manifest you pass to `kubectl ate create actor-template -f`), and mount it in each container that makes TLS connections: ```yaml +metadata: + atespace: my-atespace + name: my-actor +containers: +- name: app + image: registry.example.com/app@sha256: + volumeMounts: + - name: system-info + mountPath: /run/ate/egress-trust-bundle volumes: - name: system-info systemInfo: @@ -65,19 +74,17 @@ volumes: - egress-mitm.ate.dev - system-roots.ate.dev path: trust-bundle.pem -containers: -- name: app - # ... - volumeMounts: - - name: system-info - mountPath: /run/ate/egress-trust-bundle +resources: + limits: + cpuMillicores: 1000 + memoryBytes: 536870912 ``` Substrate only offers a few built-in trust bundles with well-known names: * `egress-mitm.ate.dev` — the egress gateway CA. * `system-roots.ate.dev` — A selection of public CA root certificates provided by Substrate. In official Substrate images, this is the Mozilla Root Store as - shipped on Debian, read from atelet's `distroless/base-debian13` base image. + shipped on Debian (consumed via the distroless-static base image). You then need to configure your application to use the trust-bundle file. In general, this is application specific, but many runtimes respect the @@ -126,9 +133,7 @@ Note: Do not bake the bundle into the image at build time. It will tie the image ## Verify `demos/egress/egress-template.yaml.tmpl` is a complete working template that -projects both bundles to `/run/ate/egress-trust-bundle/trust-bundle.pem`, the -file this recipe uses. Its Go app finds the file through `SSL_CERT_DIR`. Deploy -it: +does exactly this. Deploy it: ```bash go run ./cmd/ate-setup deploy demo egress