feat(sandbox,podman): trust corporate CA for https:// proxies and intercepted TLS

[feloy]

Summary

Adds support for https:// corporate egress proxies and for trusting an operator-provided corporate CA bundle, so sandboxes can reach a forward proxy that terminates TLS with a private CA and can operate behind a TLS-intercepting proxy (mitmproxy, squid ssl-bump) that re-signs tunneled server certificates. The feature is delivered through the hardened, driver-supplied argument model — never environment variables — and is fail-closed.

Related Issue

Refs #1792

Changes

• Shared URL validator (openshell-core): parse_upstream_proxy_url now accepts https:// (recording the scheme via a new UpstreamProxyAddr.secure), keeping the explicit-port requirement; adds the PROXY_CA_MOUNT_PATH container mount constant.
• Supervisor (openshell-supervisor-network):
  • ProxyEndpoint gains a TLS client config; for an https:// proxy the connection to the proxy is wrapped in TLS (verifying the proxy certificate) before the CONNECT handshake.
  • PrefixedStream now wraps a Plain/Tls UpstreamStream enum so the relay paths consume either transport transparently.
  • The corporate CA bundle is read fail-closed (unreadable or certificate-free file is fatal) and reuses the existing build_upstream_client_config; tls_connect_upstream was already generic, so l7/tls.rs is unchanged.
  • run.rs folds the corporate CA into the sandbox combined trust bundle (write_ca_files) and the L7 upstream verification store, so intercepted upstream handshakes succeed and sandbox workloads trust the re-signed certificates.
• Config surface: new --upstream-proxy-ca-bundle supervisor argument (no env =); new Podman driver proxy_ca_bundle setting (TOML key, --sandbox-proxy-ca-bundle, OPENSHELL_SANDBOX_PROXY_CA_BUNDLE). The driver validates the pairing (a CA bundle requires a proxy URL), bind-mounts the host PEM read-only into the sandbox (a CA certificate is not secret, so a plain bind mount rather than a driver secret), and performs a create-time readability check.
• Dev tooling: the standalone gateway task (tasks/scripts/gateway.sh) passes OPENSHELL_SANDBOX_PROXY_CA_BUNDLE through to the generated Podman config, so a local gateway can be pointed at a TLS-intercepting proxy without hand-editing the regenerated TOML.
• Docs: updated docs/reference/gateway-config.mdx, the Podman driver README.md, and the architecture/sandbox.md trust-model section.

Testing

• mise run pre-commit passes
• Unit tests added/updated — URL parsing (http/https, socks rejected), TLS-wrapped CONNECT against a fake TLS proxy (trusted-CA success and untrusted-cert rejection), fail-closed CA-bundle validation, and Podman driver config validation + bind-mount/argv assertions.
• E2E tests added/updated — new serialized podman_corporate_proxy case drives an https:// proxy end to end: the fake proxy self-signs, exposes its CA, and that CA is fed back via proxy_ca_bundle. (Runs only in the Podman e2e job.)
• Manually verified against a real TLS-intercepting proxy — mitmproxy on the Podman sandbox network, reached as an https:// proxy with its CA supplied via proxy_ca_bundle. This exercises both halves in one run: the TLS-wrapped CONNECT to the proxy listener, and the upstream certificate that mitmproxy re-signs with the same CA.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[derekwaynecarr]

Rather than make a change for just podman, can you explore having parity across all compute drivers, or note why its not possible before we merge this?

[feloy]

Rather than make a change for just podman, can you explore having parity across all compute drivers, or note why its not possible before we merge this?

My idea would be to implement by pieces as small as possible. The first part (#2245) is connection to proxy with http only and Podman only. This PR is connection to proxy with https and certificate, for Podman only. The follow-up PRs would be the full stack (http+https) for each other driver. For Docker, it should be very similar, but I still didn't investigate what would be needed for others drivers.

[kevin-pedretti]

Just a quick note to say I tested this PR and it worked in my environment. This got things working with my proxy that does SSL interception.

It may be useful to separate the host path and gateway container paths so they don't need to match. For example, something like:

  [podman]
  proxy_ca_bundle = "/etc/openshell/certs/ca-bundle.pem"  # Path for gateway to validate
  proxy_ca_bundle_host_path = "/home/myuser/ca-bundle.pem"  # Path for host Podman

  # If host_path is unset, use proxy_ca_bundle (backwards compatible)

With the current setup, I had to be careful to make sure the host and container paths to ca-bundle.pem matched. Otherwise sandbox creation fails when the gateway tries to validate the (single) path specified by proxy_ca_bundle (a host path).

[feloy]

Just a quick note to say I tested this PR and it worked in my environment. This got things working with my proxy that does SSL interception.

It may be useful to separate the host path and gateway container paths so they don't need to match. For example, something like:

  [podman]
  proxy_ca_bundle = "/etc/openshell/certs/ca-bundle.pem"  # Path for gateway to validate
  proxy_ca_bundle_host_path = "/home/myuser/ca-bundle.pem"  # Path for host Podman

  # If host_path is unset, use proxy_ca_bundle (backwards compatible)

With the current setup, I had to be careful to make sure the host and container paths to ca-bundle.pem matched. Otherwise sandbox creation fails when the gateway tries to validate the (single) path specified by proxy_ca_bundle (a host path).

@kevin-pedretti thanks for testing this PR.

I don't think this change is necessary: proxy_ca_bundle is where the ca bundle is found in your host ("/home/myuser/ca-bundle.pem" in your case) , and the ca bundle will be mounted in an hardcoded path in the container ("/etc/openshell/tls/proxy/ca-bundle.pem").

[johntmyers]

gator-agent

PR Review Status

Validation: This PR is project-valid as a focused follow-up to the already-triaged corporate-proxy gap in #1792 and the earlier HTTP proxy work in #2245. Thanks @derekwaynecarr — I checked the driver-parity concern you raised. @feloy explained that this is an incremental Podman delivery with other drivers planned as follow-ups; the supervisor portion is shared, so that scope choice does not prevent code review, while maintainers retain the final scope decision. I also checked @feloy's explanation of the CA path behavior: the configured host path is mounted at a fixed container path, matching the current driver implementation.
Head SHA: a6b67a548178b5f4a65aeaf7e93adef947a53c49
Base SHA: 877ddbacb4b915fa1d5bc6f02302d9a5c13a18e1
Merge base SHA: 877ddbacb4b915fa1d5bc6f02302d9a5c13a18e1
Patch ID: f7885d7f4a97aabbe6342feda4064351f7b4d5ef
Gator payload: 4
Review mode: initial
Previous reviewed SHA: none
Review budget exhausted: no
Maintainer decision required: no

Blocking findings:

• GATOR-a6b67a54-01: the corporate CA validation counts decoded PEM blocks without checking whether rustls accepts any certificate as a trust anchor, so a malformed bundle can pass startup and fail later despite the documented fail-closed behavior. See the inline finding.

Carried findings:

• None

Non-blocking suggestions:

• For an https:// proxy, consider allowing proxy_auth_file without proxy_auth_allow_insecure; the credential is sent only after the verified TLS connection, so the current plaintext-risk acknowledgement and diagnostic are misleading for this newly supported path. Relevant sites: crates/openshell-driver-podman/src/config.rs:368, crates/openshell-supervisor-network/src/upstream_proxy.rs:483, and crates/openshell-driver-podman/README.md:370.

Docs: Fern reference docs and the Podman/architecture documentation are updated.

Next state: gator:in-review

[johntmyers]

gator-agent

Warning — GATOR-a6b67a54-01

Invariant: A configured corporate CA bundle must contain at least one X.509 certificate that rustls actually accepts as a trust anchor; syntactically valid PEM framing with invalid DER must not satisfy fail-closed validation and silently fall back to other roots.

Prerequisite: A Podman gateway operator configures proxy_ca_bundle with a non-empty regular file whose PEM block decodes but whose DER is not a valid X.509 trust anchor, such as a truncated or incorrectly generated bundle.

Entry point → sink: documented [openshell.drivers.podman] proxy_ca_bundle with a supported corporate proxy → UpstreamProxyConfig::from_args accepts it, then build_upstream_client_config ignores the unusable certificate and the first private-CA proxy or intercepted-upstream TLS handshake fails.

Base → head: the base had no CA-bundle input and rejected https:// proxy URLs → this head counts every successfully base64-decoded CERTIFICATE block with rustls_pemfile::certs(...).flatten() even when RootCertStore later rejects its DER; built-in roots keep the aggregate store non-empty, so startup succeeds.

Impact: A documented present-but-invalid setting is accepted rather than failing closed. Sandboxes start, but corporate TLS egress then fails at runtime with an opaque outage; a publicly trusted proxy listener can also operate although the explicitly configured corporate bundle contributed no usable anchor.

Reproducer: In the upstream_proxy unit module, install the ring provider, write -----BEGIN CERTIFICATE-----\nAQID\n-----END CERTIFICATE-----\n to a temporary file, and construct config with HTTPS_PROXY=https://proxy.corp.com:3130 and PROXY_CA_BUNDLE=<temp path>. Head returns Ok; the expected deterministic result is an error naming PROXY_CA_BUNDLE and reporting zero usable certificates/trust anchors.

PR ownership: This PR introduces proxy_ca_bundle, documents unreadable or certificate-free bundles as fatal, and adds this insufficient count check; the misleading acceptance does not exist at base. The same helper is reread from run.rs:223, so both proxy-listener and intercepted-upstream paths share the invariant.

Requested change: Validate decoded certificates with the same rustls RootCertStore acceptance path used for TLS and reject the bundle unless at least one certificate is added successfully. Mixed bundles may remain tolerant, but add the invalid-DER reproducer so zero accepted anchors fail at supervisor startup.

[johntmyers]

gator-agent

PR Review Status

Validation: This remains project-valid as a focused follow-up to the corporate-proxy gap in #1792 and the earlier HTTP proxy work in #2245.
Head SHA: dfae55cf632c5d877b5c4c147cbb92342dd5d4da
Base SHA: 8d67250a5d17348eb96c4fa46226b06d8041f2ba
Merge base SHA: 8d67250a5d17348eb96c4fa46226b06d8041f2ba
Patch ID: 98563e7dcacbcdc6a729440625f6c16f2f9955d5
Gator payload: 4
Review mode: follow_up
Previous reviewed SHA: a6b67a548178b5f4a65aeaf7e93adef947a53c49
Review budget exhausted: no
Maintainer decision required: no

Thanks @feloy. I checked the two remediation commits on the author-only delta after the rebase. GATOR-a6b67a54-01 is resolved: the CA bundle is now validated through RootCertStore::add_parsable_certificates, zero accepted anchors fail at startup, and the invalid-DER reproducer covers the requested case. I also checked the HTTPS proxy-auth follow-up; the driver and supervisor now waive the insecure acknowledgement only for a verified TLS proxy connection, with matching tests and documentation.

Blocking findings:

• No blocking findings remain.

Carried findings:

• None. GATOR-a6b67a54-01 is resolved by this head.

Docs: Fern reference docs, Podman documentation, and architecture documentation are updated.

Next state: gator:watch-pipeline after the required E2E workflow is confirmed queued, running, or complete.

[github-actions]

Label test:e2e applied, but pull-request/2512 does not exist yet. A maintainer needs to comment /ok to test dfae55cf632c5d877b5c4c147cbb92342dd5d4da to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

[johntmyers]

/ok to test dfae55c

[johntmyers]

/ok to test 16e1fd6

[johntmyers]

gator-agent

PR Review Status

Validation: This remains project-valid as a focused follow-up to the corporate-proxy gap in #1792 and the earlier HTTP proxy work in #2245.
Head SHA: 16e1fd6d95adee12018c27a6545f4b9b5b1aaa0d
Base SHA: 8d67250a5d17348eb96c4fa46226b06d8041f2ba
Merge base SHA: 8d67250a5d17348eb96c4fa46226b06d8041f2ba
Patch ID: 90adf1e86e27a793bbe9269a32561ba48e7b4acf
Gator payload: 4
Review mode: follow_up
Previous reviewed SHA: dfae55cf632c5d877b5c4c147cbb92342dd5d4da
Review budget exhausted: no
Maintainer decision required: no

The author-only delta is a Rustfmt line wrap in read_proxy_ca_bundle; it does not change behavior or invalidate the previous resolution of GATOR-a6b67a54-01.

Blocking findings:

• No blocking findings remain.

Carried findings:

• None. GATOR-a6b67a54-01 remains resolved.

Docs: Fern reference docs, Podman documentation, and architecture documentation remain updated; this formatting-only delta does not change UX.

Next state: gator:in-review until the required current-head Branch Checks, Helm Lint, and E2E workflows are confirmed queued, running, or complete; then gator:watch-pipeline.

[johntmyers]

/ok to test 2713f54

[johntmyers]

gator-agent

PR Review Status

Validation: This remains project-valid as a focused follow-up to the corporate-proxy gap in #1792 and the earlier HTTP proxy work in #2245.
Head SHA: 2713f542e1b443e56fd95719389c264f4ee8d5fa
Base SHA: 8d67250a5d17348eb96c4fa46226b06d8041f2ba
Merge base SHA: 8d67250a5d17348eb96c4fa46226b06d8041f2ba
Patch ID: a4f17c918e57aeb76f5ee80a8ad7910ac2b10f82
Gator payload: 4
Review mode: critical_only
Previous reviewed SHA: 16e1fd6d95adee12018c27a6545f4b9b5b1aaa0d
Review budget exhausted: yes
Maintainer decision required: no

The author-only delta corrects the Kubernetes unsupported-proxy-scheme test to use socks5:// now that https:// is supported. The bounded critical-only review found no newly introduced Critical defect. The earlier GATOR-a6b67a54-01 remains resolved, and this delta introduces no qualifying scope growth.

Blocking findings:

• No blocking findings remain.

Carried findings:

• None. GATOR-a6b67a54-01 remains resolved.

Docs: Fern reference docs, Podman documentation, and architecture documentation remain updated; this test-only delta does not change UX.

Next state: gator:in-review until the required current-head Branch Checks, Helm Lint, and E2E workflows are confirmed queued, running, or complete; then gator:watch-pipeline.

[johntmyers]

/ok to test c74372e

[github-actions]

Label test:e2e applied for c74372e. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[johntmyers]

gator-agent

PR Review Status

Validation: This remains project-valid as a focused follow-up to the corporate-proxy gap in #1792 and the earlier HTTP proxy work in #2245.
Head SHA: c74372e024a807fcc7adccfb28e80b0a4e63ea4c
Base SHA: 8d67250a5d17348eb96c4fa46226b06d8041f2ba
Merge base SHA: 8d67250a5d17348eb96c4fa46226b06d8041f2ba
Patch ID: 7f4409c3d576aff81849cda256065a5f254182b4
Gator payload: 4
Review mode: critical_only
Previous reviewed SHA: 2713f542e1b443e56fd95719389c264f4ee8d5fa
Review budget exhausted: yes
Maintainer decision required: no

The author-only delta replaces the E2E proxy fixture's CA-as-listener certificate with a CA-signed leaf certificate. The bounded critical-only review found no newly introduced Critical defect. The earlier GATOR-a6b67a54-01 remains resolved, and this test-fixture correction introduces no qualifying scope growth.

Blocking findings:

• No blocking findings remain.

Carried findings:

• None. GATOR-a6b67a54-01 remains resolved.

Docs: Fern reference docs, Podman documentation, and architecture documentation remain updated; this test-only delta does not change UX.

Test dispatch: test:e2e is applied, the copy-PR mirror matches this head, and current-head Branch Checks, Helm Lint, and Branch E2E workflows are queued.

Next state: gator:watch-pipeline