docs: update and add security diagrams by destijl · Pull Request #3415 · agent-substrate/substrate

Greg Castle (destijl) · 2026-10-28T23:13:52Z

Why is this needed?

The existing threat model diagram was out of date and the security guide lacked a high-level overview diagram illustrating Substrate's core isolation boundaries.

This PR:

Adds docs/assets/security-boundaries.svg and a new "Security Boundaries" section to docs/security.md, defining the four core isolation boundaries (sandbox, gateways, control plane, storage) held by Substrate.
Replaces the single threat model diagram with two coordinated SVGs:
- docs/assets/actor-template-diagram.svg: Details actor template creation (T1–T10), from image pull and golden actor warm-up to suspend, snapshot upload, and template readiness.
- docs/assets/threat-model-diagram.svg: Details actor creation (A1–A2) and the lifecycle of an incoming request through outbound tool invocation (B1–B11).
Adds numbered step tables (Step | What happens | What's checked) below each diagram in docs/threat-model.md.
Refreshes the Key Components list in docs/threat-model.md to reflect current components (including ate-idp-server, k8s-credential-provider, and atunnel), sorted alphabetically.

AI use:
AI-assisted, I reviewed and tested it.

Tests pass
Docs updated

…diagram The threat model diagram predated most of the current components (atenet-egress, ate-idp-server, k8s-credential-provider, objectstore plugins, PostgreSQL) and mixed several arrow styles without a legend. Replace it with two diagrams that keep every component in the same place. The first shows how an actor template is prepared: a golden actor is run, snapshotted and kept. The second shows creating an actor from the template and serving a request, through to an egress call with an injected credential. Each numbered arrow says what it does, and a table under each diagram lists what happens at each step and what is checked. Connections outside the journeys are drawn in grey so the full attack surface still shows. Refresh the Key Components list to match. Add a much simpler diagram to security.md that shows only the four boundaries Substrate holds: sandbox, ingress/egress gateways, control plane isolation and snapshot storage. Signed-off-by: Greg Castle <gcastle@google.com>

Benjamin Elder (BenTheElder)

Agent assisted first pass :-)

Benjamin Elder (BenTheElder) · 2026-10-29T06:33:14Z

+1. **Sandbox.** Each actor runs in its own gVisor or microVM sandbox, with its own network namespace and storage. Actor code can't reach the node, the worker, or other actors.
+2. **Gateways.** Traffic reaches actors only through the ingress gateway, from authenticated callers. Traffic leaves only through the egress gateway, which allows only the destinations in policy and injects credentials, so sandboxes hold none.
+3. **Control plane.** Actors and worker pods have no network path to the Substrate API, its database, or the Kubernetes API. API callers are authorized per atespace.
+4. **Storage.** Sandboxes never touch snapshot storage. The node supervisor moves an actor's snapshots using credentials scoped to that one actor, and only the control plane copies or deletes them.


This is only true on GCS. On S3 the node plugin ignores the actor JWT and transfers with its pod's own credentials, so root on a node can read or overwrite any actor's snapshot. The Dual-Party Snapshot Authorization row below already lists per-actor tokens on S3 as future work.

We should scope this to GCS, and the same for T8, B4 and the Object Storage entry in threat-model.md, and the "per-actor credentials" label in security-boundaries.svg.

substrate/internal/objectstoreplugin/credentials.go

Lines 28 to 33 in 90afb21

// Credentials supplies the storage client a snapshot transfer runs with. The

// node plugin asks once per transfer with the actor JWT the request carries,

// and again with force set after storage refused a request, which means the

// credential behind the client no longer covers the actor. The GCS source is

// exchangeclient, which trades the JWT for a token scoped to the actor. The

// S3 source is Ambient, which ignores the JWT.

🤖 Good catch. Explicitly scoped per-actor snapshot credential isolation to GCS across boundary 4 in security.md, steps P8 and B4 and the Object Storage entry in threat-model.md, and updated the label in security-boundaries.svg to "per-actor credentials (GCS)".

Benjamin Elder (BenTheElder) · 2026-10-29T06:33:14Z

+
+1. **Sandbox.** Each actor runs in its own gVisor or microVM sandbox, with its own network namespace and storage. Actor code can't reach the node, the worker, or other actors.
+2. **Gateways.** Traffic reaches actors only through the ingress gateway, from authenticated callers. Traffic leaves only through the egress gateway, which allows only the destinations in policy and injects credentials, so sandboxes hold none.
+3. **Control plane.** Actors and worker pods have no network path to the Substrate API, its database, or the Kubernetes API. API callers are authorized per atespace.


I don't think this holds for worker pods. The WorkerPool NetworkPolicy is ingress only and egress is left unmanaged, and worker pods don't set automountServiceAccountToken: false, so a compromised ateworker can reach kube-apiserver with its pod's token.

substrate/cmd/atecontroller/internal/controllers/networkpolicy_controller.go

Lines 170 to 172 in 90afb21

// Egress is left unmanaged by Kubernetes NetworkPolicy for now while waiting for

// further progress on Egress API designs and because some egress rules may not be

// expressible at the Kubernetes layer.

The Substrate API and postgres parts do hold, through the ate-system ingress policies. (The existing No Access by Default row below has the same problem.)

🤖 Updated boundary 3 in security.md to state: "Actors have no network path to the Substrate API, its database, or the Kubernetes API. Worker pods cannot reach the Substrate API or database. API callers are authorized per atespace." I'm also working on follow-up PRs to manage worker egress and remove tokens from worker pods.

Benjamin Elder (BenTheElder) · 2026-10-29T06:33:14Z

+| A1 | The application asks `ate-api-server` to create an actor from a template. | The caller is authenticated, and OpenFGA allows it to create actors in the atespace and use the template. |
+| A2 | `ate-api-server` saves the actor as suspended. It points at the template's golden snapshot without copying it, and nothing runs until its first request. | Only `ate-api-server` can reach the database, with a client certificate. |
+| B1 | The application sends a request for `<atespace>/<actor>` to `atenet-router`. | The caller's mTLS identity must be on the router's allowlist. |
+| B2 | The router asks `ate-api-server` to resume the actor and return its worker. | Only the router's identity can resume actors this way. |


This is incorrect, can_resume also allows any editor of the atespace (and directly granted users), and they call the same ResumeActor the router does.

substrate/cmd/ateapi/internal/authz/model.fga

Line 175 in 90afb21

define can_resume: [user] or editor from parent_atespace or ingress_router from parent_atespace

🤖 Updated step B2 to: "OpenFGA allows the router (or an authorized user/editor) to resume actors in the atespace."

Benjamin Elder (BenTheElder) · 2026-10-29T06:33:14Z

+| Step | What happens | What's checked |
+| :---- | :---- | :---- |
+| A1 | The application asks `ate-api-server` to create an actor from a template. | The caller is authenticated, and OpenFGA allows it to create actors in the atespace and use the template. |
+| A2 | `ate-api-server` saves the actor as suspended. It points at the template's golden snapshot without copying it, and nothing runs until its first request. | Only `ate-api-server` can reach the database, with a client certificate. |


This only holds for the bundled postgres, which the postgres NetworkPolicy limits to ate-api-server. With a managed PostgreSQL service, reachability is whatever the provider's network allows, and the login can be an IAM or other provider identity instead of a client certificate. Same for the Substrate Database entry below.

substrate/manifests/ate-install/ate-system-networkpolicies.yaml

Lines 93 to 111 in 90afb21

# The bundled postgres is reached only by ate-api-server. The installer runs

# psql through `kubectl exec`, and the probes are exec probes, so neither needs

# network access.

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

name: postgres

namespace: ate-system

spec:

podSelector:

matchLabels:

app: postgres

policyTypes:

- Ingress

ingress:

- from:

- podSelector:

matchLabels:

app: ate-api-server

substrate/docs/postgres.md

Line 78 in 90afb21

The login users come from the database provider: they may be ordinary PostgreSQL users, IAM identities, or another provider-managed identity. Substrate does not create them or require specific login names.

🤖 Updated step A2 and the Substrate Database key component entry to distinguish bundled PostgreSQL (NetworkPolicy restricted to ate-api-server over TLS with a client certificate) from managed PostgreSQL services (governed by provider network and IAM/TLS controls).

Benjamin Elder (BenTheElder) · 2026-10-29T06:33:14Z

+1. **Sandbox.** Each actor runs in its own gVisor or microVM sandbox, with its own network namespace and storage. Actor code can't reach the node, the worker, or other actors.
+2. **Gateways.** Traffic reaches actors only through the ingress gateway, from authenticated callers. Traffic leaves only through the egress gateway, which allows only the destinations in policy and injects credentials, so sandboxes hold none.


Actor traffic does reach the worker, nftables send all of it to atunnel and the DNS relay in the Worker Pod (B8), and that's the actor's attack surface against a worker shared by many actors.

Also allowed DNS queries go from the relay straight to the pod's resolvers, not through the egress gateway, so "Traffic leaves only through the egress gateway" isn't quite true either.

substrate/internal/atunnel/dns/dnshandler.go

Lines 138 to 140 in 90afb21

// onUDPRequest inspects an incoming UDP DNS query packet from the actor and

// returns whether to drop it, reply immediately with a synthesized error, or

// record and forward it to an upstream resolver.

🤖 Updated boundary 1 in security.md to: "Actor code cannot reach the node or other actors; it can only talk through its gateway path and DNS relay, where policy applies." Boundary 2 now reads: "Outbound traffic leaves only through the egress gateway and DNS relay. The gateway enforces destination allowlists and injects credentials, so sandboxes hold none."

Benjamin Elder (BenTheElder) · 2026-10-29T06:33:15Z

+| T7 | Once the workload is ready and warmed up, the reconciler suspends the golden actor, and `atelet` checkpoints its memory and disk. | Only `ate-api-server` can order a suspend. |
+| T8 | `atelet` trades a JWT for this actor for storage credentials. | The credentials cover only this actor's snapshot location. |
+| T9 | `atelet` uploads the golden snapshot. | The sandbox never touches storage. |
+| T10 | The reconciler tags the snapshot, deletes the golden actor, and marks the template ready. | The worker's actor directories are wiped before it is reused. |


This reads like a worker runs one actor at a time. Workers host many actors and atelet removes directories per actor UID when the actor is deleted, so I think this is "the golden actor's directories are removed when it's deleted"?

🤖 Rephrased step P10 to: "The golden actor's local directories are removed when it is deleted."

Benjamin Elder (BenTheElder) · 2026-10-29T06:33:15Z

@@ -16,25 +16,71 @@ This threat model catalogs the security threats inherent to high-density statefu

 # Architecture of Substrate

-![Substrate Architecture Diagram](./assets/threat-model-diagram.svg)
+Two diagrams show the same components in the same positions. The first prepares an actor template. The second creates an actor from it and serves a request.


the template diagram shows a subset (no router, egress gateway, controller, IdP, credential provider, kube-apiserver or podcertcontroller), maybe "components both diagrams show are in the same positions"?

🤖 Updated the introduction to clarify that components both diagrams show are coordinated in the same positions.

Benjamin Elder (BenTheElder) · 2026-10-29T06:33:15Z

+
+| Step | What happens | What's checked |
+| :---- | :---- | :---- |
+| T1 | The application creates an actor template. `ate-api-server` stores it, and a background reconciler starts building its golden snapshot. | OpenFGA allows the caller to create templates in the atespace. Container images must be pinned by digest. |


nit: T1-T10 are easy to confuse with the threat IDs T-01..T-NN later in this doc, maybe a different prefix for the template steps?

🤖 Changed the template creation step prefix from T1–T10 to P1–P10 ("Prepare template") in both threat-model.md and actor-template-diagram.svg to avoid confusion with threat IDs.

Benjamin Elder (BenTheElder) · 2026-10-29T06:33:15Z

+  <rect x="1115" y="200" width="150" height="56" rx="8" fill="#f5f5f5" stroke="#616161" stroke-width="1.5"/>
+  <text x="1190.0" y="233.0" font-size="14" font-weight="bold" text-anchor="middle" fill="#212121">kube-apiserver</text>
+  <!-- podcertcontroller -->
+  <rect x="1115" y="300" width="150" height="64" rx="8" fill="#f5f5f5" stroke="#616161" stroke-width="1.5"/>


podcertcontroller is ours (cmd/podcertcontroller, installed from manifests/ate-install), but it's filled as "Outside Substrate" here.

🤖 Good catch. It's ours, deployed in its own podcertificate-controller-system namespace. Recolored it as a Substrate component (blue) and moved it out of the grey Kubernetes box into its own namespace box in actor-call-diagram.svg. The Key Components entry now names the namespace.

Benjamin Elder (BenTheElder) · 2026-10-29T06:33:15Z

+  <rect x="322" y="806" width="230" height="64" rx="8" fill="#fdecea" stroke="#c62828" stroke-width="1.5"/>
+  <text x="437.0" y="834.5" font-size="15" font-weight="bold" text-anchor="middle" fill="#212121">Golden actor</text>
+  <text x="437.0" y="851.5" font-size="12.5" text-anchor="middle" fill="#212121">built from the template</text>
+  <!-- Snapshot bucket -->


This diagram has no ate-api-server to snapshot bucket edge, but T10's tag is a copy the control plane makes (CreateTag -> CopySnapshot), and actor-call-diagram.svg does draw it. Every actor created from the template starts from that copy, so it's probably worth showing.

🤖 Added the edge from ate-api-server to Snapshot bucket for step P10 ("copy tag snapshot") in actor-template-diagram.svg, matching the copy operation performed during template tagging.

Signed-off-by: Greg Castle <gcastle@google.com>

Greg Castle (destijl) requested a review from Benjamin Elder (BenTheElder) October 28, 2026 23:14

Greg Castle (destijl) force-pushed the security-diagrams branch from 90afb21 to f845a5f Compare October 29, 2026 15:28

Benjamin Elder (BenTheElder) reviewed Oct 29, 2026

View reviewed changes

docs: address review feedback on security diagrams and boundaries

f39108b

Signed-off-by: Greg Castle <gcastle@google.com>

	// Credentials supplies the storage client a snapshot transfer runs with. The
	// node plugin asks once per transfer with the actor JWT the request carries,
	// and again with force set after storage refused a request, which means the
	// credential behind the client no longer covers the actor. The GCS source is
	// exchangeclient, which trades the JWT for a token scoped to the actor. The
	// S3 source is Ambient, which ignores the JWT.

	// Egress is left unmanaged by Kubernetes NetworkPolicy for now while waiting for
	// further progress on Egress API designs and because some egress rules may not be
	// expressible at the Kubernetes layer.

	# The bundled postgres is reached only by ate-api-server. The installer runs
	# psql through `kubectl exec`, and the probes are exec probes, so neither needs
	# network access.
	apiVersion: networking.k8s.io/v1
	kind: NetworkPolicy
	metadata:
	name: postgres
	namespace: ate-system
	spec:
	podSelector:
	matchLabels:
	app: postgres
	policyTypes:
	- Ingress
	ingress:
	- from:
	- podSelector:
	matchLabels:
	app: ate-api-server

		1. Sandbox. Each actor runs in its own gVisor or microVM sandbox, with its own network namespace and storage. Actor code can't reach the node, the worker, or other actors.
		2. Gateways. Traffic reaches actors only through the ingress gateway, from authenticated callers. Traffic leaves only through the egress gateway, which allows only the destinations in policy and injects credentials, so sandboxes hold none.

	// onUDPRequest inspects an incoming UDP DNS query packet from the actor and
	// returns whether to drop it, reply immediately with a synthesized error, or
	// record and forward it to an upstream resolver.

Conversation

Greg Castle (destijl) commented Oct 28, 2026

Uh oh!

Benjamin Elder (BenTheElder) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants