feat: sandbox-to-upstream mTLS via provider-bound client identities

[kon-angelo]

User Story

As an operator running OpenShell for agents that must reach enterprise upstreams requiring mutual TLS, I want to bind a provider credential representing a client identity (certificate + private key + optional CA) to specific destinations, so that the proxy presents the client certificate on the sandbox's behalf and the sandbox never holds the private key.

Problem Statement

Provider credentials in OpenShell today are string-shaped end to end. A credential is a string value, the only placeholder scheme (openshell:resolve:env:*) is designed to be substituted into HTTP headers, URL paths, query parameters, or JSON/form bodies after the proxy terminates TLS, and every auth_style (basic | bearer | header | path | query) operates at the HTTP layer.

This model has no answer for upstreams that authenticate the client at the TLS layer rather than the HTTP layer. The proxy always initiates the re-originated upstream handshake with no client auth, and there is no mechanism to bind a credential to a (host, port, path) endpoint as a client identity for that handshake.

Impact / Why This Matters

Today, an operator whose upstream requires client-certificate mTLS has exactly one option: set tls: skip on the endpoint and ship the client certificate and private key into the sandbox filesystem so the sandbox itself completes the handshake. The docs recommend this workaround explicitly.

That workaround is insufficient because:

• It abandons OpenShell's core value proposition. The sandbox now holds the private key in plaintext. Any tenant that doesn't already trust the sandboxed process with that key cannot use this path — which is most tenants, since the whole point of provider credentials is that agents don't see secret material.
• It disables the rest of the L7 pipeline on that endpoint. tls: skip turns off placeholder credential rewriting, dynamic token grant injection, and L7 inspection. The proxy just relays encrypted bytes.
• It doesn't compose with rotation. External rotation systems (cert-manager, Vault rotation pipelines, SPIRE) already deliver fresh material to a workspace-visible store on their own schedule; there is no path to hand that material to the proxy without pushing it into the sandbox.

This blocks real, common destinations: internal enterprise APIs behind private-CA mTLS, SaaS APIs that offer mTLS as a stronger alternative to bearer tokens, private inference endpoints fronted by service meshes that require workload identity, and any SPIFFE deployment planning to use X.509-SVIDs (rather than JWT-SVIDs).

Proposed Design

Extend the credential model so a single provider credential can represent a client identity (certificate + private key + optional CA chain) as a distinct shape from a string, and let the L7 network supervisor present that identity during the upstream TLS handshake on a per-destination basis. The sandbox never receives the material.

User-facing behavior:

• Cert-shaped credentials. A provider credential can be created and stored as a certificate bundle (cert + key + optional CA) rather than a string. Rotation is atomic — one write swaps all components — and the credential exposes an expiry so the supervisor can re-resolve before it lapses.
• Endpoint binding, not header substitution. A policy endpoint can reference a cert credential by a client_identity_ref on its binding. When the sandbox makes a request that matches, the proxy presents the referenced identity during the re-originated upstream TLS handshake for that endpoint. Endpoints without a client_identity_ref continue to handshake with no client auth, preserving current behavior.
• Dedicated, non-substitutable reference scheme. Cert credentials live in a distinct namespace (openshell:resolve:mtls:*) that is never eligible for HTTP-layer substitution. If a policy or template ever references a cert credential from a header, path, query, or body field, load fails closed.
• External rotation only. The credential-driver contract stays passive: drivers read whatever the external system (cert-manager writing kubernetes.io/tls Secrets, Vault rotation pipelines writing KV entries, SPIRE writing SVIDs, corporate PKI pipelines) has placed in the workspace-scoped backing store, and report the certificate's NotAfter as the expiry. The supervisor hot-swaps the client identity on the live TLS config when a new bundle appears, without dropping in-flight connections.
• Mutually exclusive with tls: skip. Policy validation rejects any endpoint that combines tls: skip with a client_identity_ref. Those are semantically opposite: skip means the proxy does not touch the handshake; a client identity means the proxy owns the handshake.
• Driver capability advertising. Drivers advertise whether they can store cert bundles and whether the private key is isolated in-process (HSM / PKCS#11 / KMS style, sign-only, never disclosed). The gateway uses these to reject impossible configurations at profile validation.

Non-goals (called out to bound scope):

• In-driver certificate issuance. No Vault PKI CSR flow, no ACME client, no cert-manager CertificateRequest creation. External systems own issuance; drivers only read.
• Cloudflare / JA3-fingerprint-preserving passthrough (see Credentialed Cloudflare-fronted upstreams have no working TLS mode: unconditional termination (#533) collides with Cloudflare rejecting the re-originated handshake #2455). Shares one small piece of infrastructure (per-endpoint outbound TLS config) but is a different problem and should stay separate.
• Rewriting cert material into HTTP requests. Explicitly out of scope; the whole point is that the material never crosses the HTTP layer.

Acceptance Criteria

• A provider credential can be created and stored as a certificate bundle (cert + key + optional CA) as a shape distinct from a string credential.
• A policy endpoint binding can reference a cert credential via client_identity_ref; the proxy presents that identity during the upstream TLS handshake for matching requests.
• The sandbox never observes the private key material, either in plaintext or as a resolvable placeholder value.
• Referencing a cert credential from an HTTP header, path, query, or body field is rejected at policy load with a clear error.
• Combining tls: skip with client_identity_ref on the same endpoint is rejected at policy load.
• Rotation of the backing store entry (K8s Secret, Vault KV) is picked up by the supervisor: new connections use the new identity without dropping in-flight ones, and expiry-driven re-resolution happens before NotAfter.
• Endpoints without client_identity_ref continue to handshake upstream with no client auth (no behavior change).
• Driver capabilities expose whether cert-bundle storage and key isolation are supported; profile validation rejects configurations the driver cannot satisfy.

Alternatives Considered

1. Static file mount in the sandbox (tls: skip + cert/key on disk). Works, but abandons OpenShell's core value proposition — the sandbox now holds the private key and can exfiltrate it. Unacceptable for any tenant that doesn't already trust the sandboxed process with the key. This is the status quo workaround.
2. Group cert/key as sibling string credentials. Zero schema changes; cheap. But puts private-key bytes through the same resolver code paths built for header substitution, which is a large blast radius for a bug. Rotation isn't atomic — three separate writes leave a window where a valid cert is paired with an old key. Rejected on security grounds.
3. In-driver issuance (Vault PKI, ACME, cert-manager CSR). Attractive on paper: match cert TTL to sandbox TTL, mint on demand. In practice it duplicates issuance infrastructure every real deployment already runs, forces every driver to become an issuance client on top of a storage client, and adds lease-renewal failure modes. External rotation via cert-manager / Vault pipelines / SPIRE covers all real use cases with simpler code and a smaller blast radius.
4. Sidecar proxy per sandbox. Terminate mTLS in a sidecar that the sandbox talks to over plaintext localhost, with the sidecar holding the cert. Duplicates functionality the L7 supervisor already provides, doubles the number of processes per sandbox, and doesn't integrate with existing provider bindings.
5. New standalone credential type outside the driver system. Bypasses all the workspace scoping, audit, and lifecycle machinery already in the credential drivers. Non-starter — cert credentials need the same provenance and rotation controls as bearer tokens, plus more.

The proposed design reuses the existing credential-driver contract, the existing endpoint binding, and the existing supervisor TLS setup, adding one new resolver hook and one new credential shape.

Agent Investigation

Explored the codebase to confirm no existing feature covers this case and to identify concrete extension points.

No existing mTLS-to-upstream support. All existing mtls / client_cert mentions in the tree are control-plane (CLI ↔ gateway, gateway ↔ driver, cluster PKI) — none is sandbox-to-upstream.

• Credential contract is string-only: proto/credential_driver.proto:55 (StoreCredentialRequest.value: string), proto/credential_driver.proto:115 (ResolvedCredential.value: string).
• Only placeholder scheme is PLACEHOLDER_PREFIX = "openshell:resolve:env:" at crates/openshell-core/src/secrets.rs:10.
• Accepted auth_style values in crates/openshell-providers/src/profiles.rs — basic | bearer | header | path | query. No cert-shaped variant.
• Upstream TLS is a single shared Arc<ClientConfig> built with .with_no_client_auth() in crates/openshell-supervisor-network/src/l7/tls.rs (inside build_upstream_client_config). There is no destination-keyed hook.
• Driver capabilities in GetCredentialDriverCapabilitiesResponse (proto/credential_driver.proto:36) expose supports_list and supports_expires_at but nothing for certificate storage or key isolation.
• Docs explicitly acknowledge the gap: docs/security/best-practices.mdx:124-126 recommends tls: skip for mTLS upstreams and notes that it disables credential injection and L7 inspection.

Extension points already present.

• StaticCredentialEndpointBinding at proto/openshell.proto:1780 already scopes credentials to (host, port, path) tuples — natural home for client_identity_ref.
• Both kubernetes-secrets and vault drivers are already workspace-aware via SHA-256 name/path derivation, so cert credentials inherit workspace isolation.
• Replacing the hard-coded .with_no_client_auth() with a rustls::client::ResolvesClientCert that consults the endpoint binding is a single-site change; endpoints without a binding fall through to the current behavior.
• Kubernetes already types kubernetes.io/tls Secrets with paired tls.crt / tls.key keys, mapping 1:1 onto a cert bundle. Vault KV can hold the same shape written by external rotation tooling. Both fit the passive-observer model without driver-side issuance.

Related upstream issues.

• Credentialed Cloudflare-fronted upstreams have no working TLS mode: unconditional termination (#533) collides with Cloudflare rejecting the re-originated handshake #2455 — Credentialed Cloudflare-fronted upstreams have no working TLS mode. Different problem (fingerprint-preserving passthrough vs. client-cert presentation), small infrastructure overlap in per-endpoint outbound TLS config. Should not block this.
• Tracking: generalize brokered credential delivery beyond inference.local and Microsoft S2S #1755 — Tracking: generalize brokered credential delivery. Mentions mTLS/DPoP only as "provider-scoped enhancements, not universal acceptance criteria." No concrete design.
• Support SPIFFE-backed provider token grants on non-Kubernetes drivers (Docker, Podman, VM) #2708, bug(supervisor): SPIFFE-enabled sandboxes crash because identity mount namespace is never prepared #2571, feat: Add SPIFFE workload identity provider for sandbox attestation-based credentials #1665 — SPIFFE support. Currently JWT-SVID as OAuth2 client_assertion, still bearer tokens on the wire, not X.509-SVID mTLS. Would compose naturally with this proposal (external issuer, driver reads).

Motivating case. Reported downstream by an operator whose enterprise inference API requires an x509 client-certificate chain; every sandbox depending on that provider fails to reach the upstream because there is no path to present the client identity from the proxy.

Checklist

• I've reviewed existing issues and the architecture docs
• This is a design proposal, not a "please build this" request