Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 34 additions & 26 deletions docs/guides/egress-credential-injection.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,18 @@ description: >

# Egress credential injection

Egress credential injection puts a secret into an actor's outbound HTTPS
requests without the actor ever holding it. It works on HTTPS the egress
gateway intercepts: an `https` rule in the actor's `EgressPolicy` has the
gateway terminate the actor's TLS for the names it lists and re-originate it
(see [egress-trust-bundle.md](egress-trust-bundle.md)). When that rule carries
a `replace_headers` effect and the actor's request includes a header it names,
the gateway resolves the referenced credential from a **credential provider**
and replaces the header's value with it (for example
`Authorization: Bearer <token>`) before the request leaves the cluster.
Egress credential injection puts a secret into an actor's outbound HTTP and
HTTPS requests without the actor ever holding it. It works on requests the
egress gateway can read: cleartext HTTP that an `http` rule in the actor's
`EgressPolicy` allows, and HTTPS that an `https` rule allows. An `https` rule
has the gateway terminate the actor's TLS for the names it lists and
re-originate it (see [egress-trust-bundle.md](egress-trust-bundle.md)).

When the rule carries a `replaceHeaders` effect and the actor's request
includes a header it names, the gateway resolves the referenced credential
from a **credential provider** and replaces the header's value with it (for
example `Authorization: Bearer <token>`) before the request leaves the
cluster.

The actor never holds the secret. It sends the header with a placeholder
value, which the gateway discards, so it cannot read, snapshot, or exfiltrate
Expand Down Expand Up @@ -61,7 +64,7 @@ flowchart LR
terminates the actor's TLS — the actor trusts the gateway CA through a
projected trust bundle — and decides the decrypted request against the
actor's `EgressPolicy`.
2. If the deciding rule has `replace_headers` and the request carries one of
2. If the deciding rule has `replaceHeaders` and the request carries one of
Comment thread
yufan-su marked this conversation as resolved.
Resolved
those headers, the gateway asks the credential provider for the credential
that header names.
3. The provider decides whether that actor may have that credential, reads it
Expand All @@ -85,7 +88,7 @@ Injection is declared in the `effects` of an `http` or `https` rule. This policy
HTTPS to `api.example.com` on port 443, the default, and replaces the actor's
`Authorization` header with a credential:

```bash
```yaml
Comment thread
yufan-su marked this conversation as resolved.
Resolved
rules:
- https:
hostnames: ["api.example.com"]
Expand All @@ -103,9 +106,8 @@ curl -H 'Authorization: placeholder' https://api.example.com/v1/items
```
This URI names the sample Secret deployed under [Enable it](#enable-it). The
sample namespace policy lets only actors in atespace `team-a` resolve it; an
actor in any other atespace gets a 403 until a cluster admin grants its
atespace access.
install's namespace policy grants nothing, so the actor gets a 403 until a
cluster admin grants its atespace access to the `ns1` namespace.

* `header` names the request header to replace.
* `prefix` is prepended verbatim to the credential — include the separator,
Expand All @@ -114,9 +116,13 @@ atespace access.

### Credential URIs

A credential URI has the form `ate-secret://<provider-class>/<path>`: the host
names the provider that resolves it, and the path is that provider's to
interpret. A URI the provider refuses denies the request with 403.
A credential URI has the form
`ate-secret://<provider-class>/<provider-specific-tail>`. The
host names the provider that resolves it, and the provider interprets the
path. The API requires the host to be a DNS-1123 subdomain and the path to
have at least two non-empty segments. It rejects a port, userinfo, query, fragment,
percent-encoding, or trailing slash. A URI the provider refuses denies the
request with 403.

* **Kubernetes Secrets** (provider class `k8s.io`) — see
[The reference provider](#the-reference-provider).
Expand Down Expand Up @@ -267,10 +273,11 @@ kubectl apply -f manifests/egress-credential-injection/sample-secret.yaml
```

**3. The actors.** Actors can now add `replaceHeaders` effects to their
`https` rules, as described under [The policy](#the-policy). Each such actor
also needs the projected egress trust bundle to do TLS through the gateway at
all — see [egress-trust-bundle.md](egress-trust-bundle.md). Actor JWT entries
need no provider; ateapi signs them with a key that every install creates.
`http` or `https` rules, as described under [The policy](#the-policy). An
actor with `https` rules also needs the projected egress trust bundle to do
TLS through the gateway at all (see
[egress-trust-bundle.md](egress-trust-bundle.md)). Actor JWT entries need no
provider; ateapi signs them with a key that every install creates.

### Verify

Expand All @@ -285,7 +292,8 @@ Then give an actor an `https` rule for `httpbin.org` that replaces a header,
have it request `https://httpbin.org/headers` with that header set to a
placeholder, and look for the credential in the echoed response.

If the request is denied instead, the gateway logs the reason:
On the Envoy dataplane, the gateway's `ext-proc` container logs why a request
was denied:

```bash
kubectl -n ate-system logs deployment/atenet-egress -c ext-proc | grep 'egress denied'
Expand Down Expand Up @@ -325,7 +333,8 @@ a policy URI naming any other class fails closed rather than being sent to the
wrong provider.

The gateway reads only the URI host, to confirm the URI targets the provider it
serves. Everything after the host is the provider's to interpret.
serves. The provider interprets the path, within the rules under
[Credential URIs](#credential-uris).

[credprovider-gcp-secret-manager](https://github.com/agent-substrate-plugins/credprovider-gcp-secret-manager) is such a provider for Google Cloud Secret Manager.

Expand Down Expand Up @@ -364,6 +373,5 @@ may only resolve Secrets in namespaces explicitly granted to it.
* `api/credprovider/v1/credprovider.proto` — the provider plugin API and
its trust model.
* `cmd/credential-provider/kubernetes-secrets` — the reference provider.
* `internal/e2e/suites/egresscredinject` — the e2e suite that proves the
behavior table above on both sandbox classes, against a cluster installed
with `--credential-provider='{"name":"k8s.io"}'`.
* `internal/e2e/suites/egresscredinject` — the e2e suite, run against a
cluster installed with `--credential-provider='{"name":"k8s.io"}'`.
15 changes: 9 additions & 6 deletions docs/guides/egress-traffic.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,8 @@ within about 5 seconds.
> the actor's egress policy has a rule for it with
> `kubectl ate get egress-policy <actor-name> -a <atespace> -o yaml`.

Other UDP and other traffic is filtered by nftables and never reaches the gateway.
Other UDP and other traffic is rejected by nftables and never reaches the egress
gateway.

This applies to an actor's own telemetry too: an OpenTelemetry SDK inside the actor
cannot reach the collector until the actor has an `http` egress rule for it. See
Expand All @@ -39,11 +40,12 @@ cannot reach the collector until the actor has an `http` egress rule for it. See

| Port | Traffic | Behavior | Path | What the actor sees when refused |
| :---- | :---- | :---- | :---- | :---- |
| any | HTTP(S) 1.1 / 2 | Supported with policy controls | atunnel -> egress gateway -> origin | `403 Forbidden` |
| any | WebSocket | Blocked | n/a | `403 Forbidden` |
| any | Standard HTTP(S) CONNECT (forward-proxy tunnel) | Blocked | n/a | `403 Forbidden` |
| any | HTTP(S) 1.1 / 2 | Supported with policy controls | atunnel -> egress gateway -> origin | `403 Forbidden`. For HTTPS whose SNI matches no `https` or `tls_passthrough` rule, the gateway closes the connection during the TLS handshake, without an HTTP response. |
| any | TLS whose SNI matches a `tls_passthrough` rule | Relayed to the SNI hostname without inspection, so WebSocket, CONNECT, and other TCP inside it get through | atunnel -> egress gateway -> origin | Closed during the TLS handshake. |
| any | WebSocket | Blocked | atunnel -> egress gateway | `403 Forbidden` |
| any | Standard HTTP(S) CONNECT (forward-proxy tunnel) | Blocked | atunnel -> egress gateway | `403 Forbidden` |
| 53 | DNS to the sandbox's gateway | Allowed for hostnames matching the actor's egress policy | sandbox DNS relay -> worker pod's resolvers | `NXDOMAIN` |
| any | Any other TCP | Blocked | n/a | The connection is accepted and then closed with no bytes returned. There is no status code. atunnel logs the failure. |
| any | Any other TCP | Blocked | atunnel -> egress gateway | A client that sends first gets `400 Bad Request` before the connection closes. If the client waits for the server to speak first, the connection closes after about 1 second with no data. Only the gateway logs the refusal. |

See the [Network Egress Contract](network-egress.md#refusal-responses-and-policy-changes)
for refusal status codes, policy-refresh semantics, and the distinction between DNS
Expand All @@ -59,7 +61,8 @@ refusals and HTTP policy denials.

## Other protocols

Everything that is neither TCP nor UDP is blocked. Packets are dropped, not rejected: no ICMP port-unreachable is sent, so the client hangs until its own timeout.
Everything that is neither TCP nor UDP is rejected with ICMP network
unreachable.

## Requesting support

Expand Down
51 changes: 27 additions & 24 deletions docs/guides/egress-trust-bundle.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,33 +48,36 @@ rule, are answered `NXDOMAIN`.

## Configuring a workload to trust public and man-in-the-middle CAs

Add a `systemInfo` volume with a `trustBundle` data source, and mount it:
Add a `systemInfo` volume with a `trustBundle` data source to the actor
template (the manifest you pass to `kubectl ate create actor-template -f`),
and mount it in each container that makes TLS connections:

```yaml
apiVersion: ate.dev/v1
kind: ActorTemplate
metadata:
atespace: my-atespace
name: my-actor
namespace: my-namespace
spec:
volumes:
containers:
- name: app
image: registry.example.com/app@sha256:<digest>
volumeMounts:
- name: system-info
systemInfo:
dataSources:
# This file will contain trust anchors for both the MITM CA and standard
# public WebPKI CAs, meaning it will work for actors that use `https` rules,
# `tls_passthrough` rules, or a mix.
- trustBundle:
names:
- egress-mitm.ate.dev
- system-roots.ate.dev
path: trust-bundle.pem
containers:
- name: app
image: ...
volumeMounts:
- name: system-info
mountPath: /run/ate/egress-trust-bundle/
mountPath: /run/ate/egress-trust-bundle
volumes:
- name: system-info
systemInfo:
dataSources:
Comment thread
yufan-su marked this conversation as resolved.
Resolved
# This file will contain trust anchors for both the MITM CA and standard
# public WebPKI CAs, meaning it will work for actors that use `https` rules,
# `tls_passthrough` rules, or a mix.
- trustBundle:
names:
- egress-mitm.ate.dev
- system-roots.ate.dev
path: trust-bundle.pem
resources:
limits:
cpuMillicores: 1000
memoryBytes: 536870912
```

Substrate only offers a few built-in trust bundles with well-known names:
Expand Down Expand Up @@ -117,7 +120,7 @@ this setup in your application's startup logic). See the [example](#example-e
#!/bin/sh
set -eu
sys=/etc/ssl/certs/ca-certificates.crt # RHEL family: /etc/pki/tls/certs/ca-bundle.crt
ca=/run/ate/trust-bundle.pem
ca=/run/ate/egress-trust-bundle/trust-bundle.pem
out=/tmp/ca-bundle.pem
{ cat "$sys"; echo; cat "$ca"; } > "$out"
export SSL_CERT_FILE="$out" REQUESTS_CA_BUNDLE="$out" CURL_CA_BUNDLE="$out" GIT_SSL_CAINFO="$out" PIP_CERT="$out"
Expand Down Expand Up @@ -145,7 +148,7 @@ bundle did the validating.

* Problems with trust bundles will block actor startup. Error messages:
- `... unknown trust bundle "foo"`: You tried to use an unknown trust bundle name.
- `... trust bundle "foo": unusable contents ...`: Your substrate installation is misconfigured.
- `... trust bundle "egress-mitm.ate.dev": ClusterTrustBundle "egress-mitm.ate.dev:mitm:primary-bundle" not found`: Your substrate installation is misconfigured.
* Certificate errors are likely problems with the actor, not the gateway: If you
see an error message like `... certificate signed by unknown authority ...`,
the most likely cause is that your actor is not configured to trust the MITM
Expand Down
Loading