diff --git a/docs/guides/egress-credential-injection.md b/docs/guides/egress-credential-injection.md index 610cc58171..632613e3bb 100644 --- a/docs/guides/egress-credential-injection.md +++ b/docs/guides/egress-credential-injection.md @@ -8,15 +8,18 @@ description: > # Egress credential injection -Egress credential injection puts a secret into an actor's outbound HTTPS -requests without the actor ever holding it. It works on HTTPS the egress -gateway intercepts: an `https` rule in the actor's `EgressPolicy` has the -gateway terminate the actor's TLS for the names it lists and re-originate it -(see [egress-trust-bundle.md](egress-trust-bundle.md)). When that rule carries -a `replace_headers` effect and the actor's request includes a header it names, -the gateway resolves the referenced credential from a **credential provider** -and replaces the header's value with it (for example -`Authorization: Bearer `) before the request leaves the cluster. +Egress credential injection puts a secret into an actor's outbound HTTP and +HTTPS requests without the actor ever holding it. It works on requests the +egress gateway can read: cleartext HTTP that an `http` rule in the actor's +`EgressPolicy` allows, and HTTPS that an `https` rule allows. An `https` rule +has the gateway terminate the actor's TLS for the names it lists and +re-originate it (see [egress-trust-bundle.md](egress-trust-bundle.md)). + +When the rule carries a `replaceHeaders` effect and the actor's request +includes a header it names, the gateway resolves the referenced credential +from a **credential provider** and replaces the header's value with it (for +example `Authorization: Bearer `) before the request leaves the +cluster. The actor never holds the secret. It sends the header with a placeholder value, which the gateway discards, so it cannot read, snapshot, or exfiltrate @@ -61,7 +64,7 @@ flowchart LR terminates the actor's TLS — the actor trusts the gateway CA through a projected trust bundle — and decides the decrypted request against the actor's `EgressPolicy`. -2. If the deciding rule has `replace_headers` and the request carries one of +2. If the deciding rule has `replaceHeaders` and the request carries one of those headers, the gateway asks the credential provider for the credential that header names. 3. The provider decides whether that actor may have that credential, reads it @@ -85,7 +88,7 @@ Injection is declared in the `effects` of an `http` or `https` rule. This policy HTTPS to `api.example.com` on port 443, the default, and replaces the actor's `Authorization` header with a credential: -```bash +```yaml rules: - https: hostnames: ["api.example.com"] @@ -103,9 +106,8 @@ curl -H 'Authorization: placeholder' https://api.example.com/v1/items ``` This URI names the sample Secret deployed under [Enable it](#enable-it). The -sample namespace policy lets only actors in atespace `team-a` resolve it; an -actor in any other atespace gets a 403 until a cluster admin grants its -atespace access. +install's namespace policy grants nothing, so the actor gets a 403 until a +cluster admin grants its atespace access to the `ns1` namespace. * `header` names the request header to replace. * `prefix` is prepended verbatim to the credential — include the separator, @@ -114,9 +116,13 @@ atespace access. ### Credential URIs -A credential URI has the form `ate-secret:///`: the host -names the provider that resolves it, and the path is that provider's to -interpret. A URI the provider refuses denies the request with 403. +A credential URI has the form +`ate-secret:///`. The +host names the provider that resolves it, and the provider interprets the +path. The API requires the host to be a DNS-1123 subdomain and the path to +have at least two non-empty segments. It rejects a port, userinfo, query, fragment, +percent-encoding, or trailing slash. A URI the provider refuses denies the +request with 403. * **Kubernetes Secrets** (provider class `k8s.io`) — see [The reference provider](#the-reference-provider). @@ -267,10 +273,11 @@ kubectl apply -f manifests/egress-credential-injection/sample-secret.yaml ``` **3. The actors.** Actors can now add `replaceHeaders` effects to their -`https` rules, as described under [The policy](#the-policy). Each such actor -also needs the projected egress trust bundle to do TLS through the gateway at -all — see [egress-trust-bundle.md](egress-trust-bundle.md). Actor JWT entries -need no provider; ateapi signs them with a key that every install creates. +`http` or `https` rules, as described under [The policy](#the-policy). An +actor with `https` rules also needs the projected egress trust bundle to do +TLS through the gateway at all (see +[egress-trust-bundle.md](egress-trust-bundle.md)). Actor JWT entries need no +provider; ateapi signs them with a key that every install creates. ### Verify @@ -285,7 +292,8 @@ Then give an actor an `https` rule for `httpbin.org` that replaces a header, have it request `https://httpbin.org/headers` with that header set to a placeholder, and look for the credential in the echoed response. -If the request is denied instead, the gateway logs the reason: +On the Envoy dataplane, the gateway's `ext-proc` container logs why a request +was denied: ```bash kubectl -n ate-system logs deployment/atenet-egress -c ext-proc | grep 'egress denied' @@ -325,7 +333,8 @@ a policy URI naming any other class fails closed rather than being sent to the wrong provider. The gateway reads only the URI host, to confirm the URI targets the provider it -serves. Everything after the host is the provider's to interpret. +serves. The provider interprets the path, within the rules under +[Credential URIs](#credential-uris). [credprovider-gcp-secret-manager](https://github.com/agent-substrate-plugins/credprovider-gcp-secret-manager) is such a provider for Google Cloud Secret Manager. @@ -364,6 +373,5 @@ may only resolve Secrets in namespaces explicitly granted to it. * `api/credprovider/v1/credprovider.proto` — the provider plugin API and its trust model. * `cmd/credential-provider/kubernetes-secrets` — the reference provider. -* `internal/e2e/suites/egresscredinject` — the e2e suite that proves the - behavior table above on both sandbox classes, against a cluster installed - with `--credential-provider='{"name":"k8s.io"}'`. +* `internal/e2e/suites/egresscredinject` — the e2e suite, run against a + cluster installed with `--credential-provider='{"name":"k8s.io"}'`. diff --git a/docs/guides/egress-traffic.md b/docs/guides/egress-traffic.md index c71a529765..5ce15eee9b 100644 --- a/docs/guides/egress-traffic.md +++ b/docs/guides/egress-traffic.md @@ -29,7 +29,8 @@ within about 5 seconds. > the actor's egress policy has a rule for it with > `kubectl ate get egress-policy -a -o yaml`. -Other UDP and other traffic is filtered by nftables and never reaches the gateway. +Other UDP and other traffic is rejected by nftables and never reaches the egress +gateway. This applies to an actor's own telemetry too: an OpenTelemetry SDK inside the actor cannot reach the collector until the actor has an `http` egress rule for it. See @@ -39,11 +40,12 @@ cannot reach the collector until the actor has an `http` egress rule for it. See | Port | Traffic | Behavior | Path | What the actor sees when refused | | :---- | :---- | :---- | :---- | :---- | -| any | HTTP(S) 1.1 / 2 | Supported with policy controls | atunnel -> egress gateway -> origin | `403 Forbidden` | -| any | WebSocket | Blocked | n/a | `403 Forbidden` | -| any | Standard HTTP(S) CONNECT (forward-proxy tunnel) | Blocked | n/a | `403 Forbidden` | +| any | HTTP(S) 1.1 / 2 | Supported with policy controls | atunnel -> egress gateway -> origin | `403 Forbidden`. For HTTPS whose SNI matches no `https` or `tls_passthrough` rule, the gateway closes the connection during the TLS handshake, without an HTTP response. | +| any | TLS whose SNI matches a `tls_passthrough` rule | Relayed to the SNI hostname without inspection, so WebSocket, CONNECT, and other TCP inside it get through | atunnel -> egress gateway -> origin | Closed during the TLS handshake. | +| any | WebSocket | Blocked | atunnel -> egress gateway | `403 Forbidden` | +| any | Standard HTTP(S) CONNECT (forward-proxy tunnel) | Blocked | atunnel -> egress gateway | `403 Forbidden` | | 53 | DNS to the sandbox's gateway | Allowed for hostnames matching the actor's egress policy | sandbox DNS relay -> worker pod's resolvers | `NXDOMAIN` | -| any | Any other TCP | Blocked | n/a | The connection is accepted and then closed with no bytes returned. There is no status code. atunnel logs the failure. | +| any | Any other TCP | Blocked | atunnel -> egress gateway | A client that sends first gets `400 Bad Request` before the connection closes. If the client waits for the server to speak first, the connection closes after about 1 second with no data. Only the gateway logs the refusal. | See the [Network Egress Contract](network-egress.md#refusal-responses-and-policy-changes) for refusal status codes, policy-refresh semantics, and the distinction between DNS @@ -59,7 +61,8 @@ refusals and HTTP policy denials. ## Other protocols -Everything that is neither TCP nor UDP is blocked. Packets are dropped, not rejected: no ICMP port-unreachable is sent, so the client hangs until its own timeout. +Everything that is neither TCP nor UDP is rejected with ICMP network +unreachable. ## Requesting support diff --git a/docs/guides/egress-trust-bundle.md b/docs/guides/egress-trust-bundle.md index 94db7ffc2f..0baa4ab98d 100644 --- a/docs/guides/egress-trust-bundle.md +++ b/docs/guides/egress-trust-bundle.md @@ -48,33 +48,36 @@ rule, are answered `NXDOMAIN`. ## Configuring a workload to trust public and man-in-the-middle CAs -Add a `systemInfo` volume with a `trustBundle` data source, and mount it: +Add a `systemInfo` volume with a `trustBundle` data source to the actor +template (the manifest you pass to `kubectl ate create actor-template -f`), +and mount it in each container that makes TLS connections: ```yaml -apiVersion: ate.dev/v1 -kind: ActorTemplate metadata: + atespace: my-atespace name: my-actor - namespace: my-namespace -spec: - volumes: +containers: +- name: app + image: registry.example.com/app@sha256: + volumeMounts: - name: system-info - systemInfo: - dataSources: - # This file will contain trust anchors for both the MITM CA and standard - # public WebPKI CAs, meaning it will work for actors that use `https` rules, - # `tls_passthrough` rules, or a mix. - - trustBundle: - names: - - egress-mitm.ate.dev - - system-roots.ate.dev - path: trust-bundle.pem - containers: - - name: app - image: ... - volumeMounts: - - name: system-info - mountPath: /run/ate/egress-trust-bundle/ + mountPath: /run/ate/egress-trust-bundle +volumes: +- name: system-info + systemInfo: + dataSources: + # This file will contain trust anchors for both the MITM CA and standard + # public WebPKI CAs, meaning it will work for actors that use `https` rules, + # `tls_passthrough` rules, or a mix. + - trustBundle: + names: + - egress-mitm.ate.dev + - system-roots.ate.dev + path: trust-bundle.pem +resources: + limits: + cpuMillicores: 1000 + memoryBytes: 536870912 ``` Substrate only offers a few built-in trust bundles with well-known names: @@ -117,7 +120,7 @@ this setup in your application's startup logic). See the [example](#example-e #!/bin/sh set -eu sys=/etc/ssl/certs/ca-certificates.crt # RHEL family: /etc/pki/tls/certs/ca-bundle.crt -ca=/run/ate/trust-bundle.pem +ca=/run/ate/egress-trust-bundle/trust-bundle.pem out=/tmp/ca-bundle.pem { cat "$sys"; echo; cat "$ca"; } > "$out" export SSL_CERT_FILE="$out" REQUESTS_CA_BUNDLE="$out" CURL_CA_BUNDLE="$out" GIT_SSL_CAINFO="$out" PIP_CERT="$out" @@ -145,7 +148,7 @@ bundle did the validating. * Problems with trust bundles will block actor startup. Error messages: - `... unknown trust bundle "foo"`: You tried to use an unknown trust bundle name. - - `... trust bundle "foo": unusable contents ...`: Your substrate installation is misconfigured. + - `... trust bundle "egress-mitm.ate.dev": ClusterTrustBundle "egress-mitm.ate.dev:mitm:primary-bundle" not found`: Your substrate installation is misconfigured. * Certificate errors are likely problems with the actor, not the gateway: If you see an error message like `... certificate signed by unknown authority ...`, the most likely cause is that your actor is not configured to trust the MITM