diff --git a/AGENTS.md b/AGENTS.md index 9135ed2b..c920f419 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -82,7 +82,7 @@ and deserialized through a registry. | `task/`, `task_step/` | A `Task` holds its `TaskStep`s inline; `Task.run()` executes them as a DAG. `depends_on` (None means all prior steps) gates a step, independent steps run concurrently, `fail_task_on_error` makes a failure fatal or tolerated, `retry_config` rolls a failed span back through the step journal and re-dispatches it. Built-in steps live in `task_step/task_steps/` (`deploy_env`, `deploy_agent`, `prompt_agent`, the verifiers under `verifiers/`, and more). | | `store/` | Four store ABCs with local and cloud implementations: `DocumentStore` (SQLite, MongoDB), `ObjectStore` (filesystem, S3, Cloud Storage), `ImageStore` (local OCI registry, ECR), `SecretStore` (env vars or file, AWS Secrets Manager, Google Cloud Secret Manager). `VersionedEntityStore` implements the shared versioned get/put logic, `QueryBuilder` is the immutable chained query API, `store/base.py` holds the error types. A new backend must pass the conformance kits in `tst/store/`. | | `config/` | The `Config` singleton (`get_config`, `configure`, `reset_config`) in `config/runtime.py`, file discovery in `config/loader.py`, and `load_impl`, which resolves `module:Class` pointers. `agent_env.store` re-exports the config names for compatibility. | -| `providers/` | `providers/sandbox_providers/` holds the sandbox providers `local`, `modal`, `modal_vm`, `e2b`; `[sandbox] default` and `agent_default` accept a comma-separated fallback chain. `providers/env_providers/` holds the environment providers: `EnvironmentProvider` (an env's containers and state store) and `EnvironmentGatewayProvider`, which renders a docker-compose for the gateway and its MCP servers inside the sandbox; `providers/env_state/` holds env-state providers (`local_postgres` built in). | +| `providers/` | `providers/sandbox_providers/` holds the sandbox providers `local`, `modal`, `modal_vm`, `e2b`, `sail_vm` (the `sail` extra); `[sandbox] default` and `agent_default` accept a comma-separated fallback chain. `providers/env_providers/` holds the environment providers: `EnvironmentProvider` (an env's containers and state store) and `EnvironmentGatewayProvider`, which renders a docker-compose for the gateway and its MCP servers inside the sandbox; `providers/env_state/` holds env-state providers (`local_postgres` built in). | | `a2a_agent/` | The `A2AAgent` entity (`a2a_agent`), its stores and the validator steps. The protocol package provides the agent-side framework. | | `runner/` | The `[runner]` seam: `Runner.submit()` returns `(run_id, instance_id)`; `LocalRunner` is built in. | | `explorer/` | Optional local web UI: `agent-env up`, needs the `explorer` extra, binds loopback `:8234`. | diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 7eb436bb..51e2fb0c 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -14,7 +14,7 @@ marks, or contributor names to endorse or promote Scale AI, or related products. ## Python dependencies -The runtime dependencies of `agentenv-framework` and its `explorer` and `gcp` extras, at the versions resolved in `uv.lock`. Development-only dependencies are not listed. +The runtime dependencies of `agentenv-framework` and its `explorer`, `gcp` and `sail` extras, at the versions resolved in `uv.lock`. Development-only dependencies are not listed. ### a2a-sdk 0.3.26 @@ -128,6 +128,13 @@ The runtime dependencies of `agentenv-framework` and its `explorer` and `gcp` ex - Source: - [License text 12](#license-text-12) +### cloudpickle 3.1.2 + +- License: BSD-3-Clause +- Author: The cloudpickle developer team +- Source: +- [License text 70](#license-text-70) + ### colorama 0.4.6 - License: BSD @@ -660,6 +667,13 @@ The runtime dependencies of `agentenv-framework` and its `explorer` and `gcp` ex - Source: - [License text 1](#license-text-1), [License text 42](#license-text-42) +### sail 0.12.8 + +- License: Apache-2.0 +- Author: Sail +- Source: +- License text: not shipped with the package; see its source + ### shellingham 1.5.4 - License: ISC @@ -7663,3 +7677,40 @@ OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. ***************************************************************************** */ ``` + +### License text 70 + +```text +This module was extracted from the `cloud` package, developed by +PiCloud, Inc. + +Copyright (c) 2015, Cloudpickle contributors. +Copyright (c) 2012, Regents of the University of California. +Copyright (c) 2009 PiCloud, Inc. http://www.picloud.com. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + * Neither the name of the University of California, Berkeley nor the + names of its contributors may be used to endorse or promote + products derived from this software without specific prior written + permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED +TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +``` diff --git a/pyproject.toml b/pyproject.toml index b2c5253c..312ddeb6 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -64,8 +64,12 @@ gcp = [ "google-cloud-storage>=3.0", "requests>=2.31", ] +# The sail_vm sandbox provider (Sail Research Sailboxes), selected with --sandbox sail_vm. +sail = [ + "sail~=0.12.8", # Sailbox exec/fs/listener/egress APIs validated against the 0.12 line +] dev = [ - "agentenv-framework[explorer,gcp]", # so the explorer and GCP backend tests run in CI + "agentenv-framework[explorer,gcp,sail]", # so the explorer, GCP backend and Sail provider tests run in CI "griffelib==2.3.0", # the plugin API check; exact, since what it reports as a break changes between releases "moto>=5.0.0", "psycopg2-binary>=2.9.0", # the gateway server module's driver; outside its container only the tests import it diff --git a/src/agent_env/providers/sandbox_providers/sail_vm/README.md b/src/agent_env/providers/sandbox_providers/sail_vm/README.md new file mode 100644 index 00000000..fc007ce9 --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/sail_vm/README.md @@ -0,0 +1,238 @@ +# `sail_vm` sandbox provider + +Runs agent-env sandboxes on [Sail Research Sailboxes](https://docs.sailresearch.com/sailboxes). These are +Linux VMs booted from Sail's `devbox` image, which ships Docker and Compose v2 and runs as root. Like +`modal_vm` and `e2b`, it is a VM provider: agent-env's docker-in-VM flows run on it unchanged. That covers +the gateway's docker-compose, an agent's `docker run`, image loading and artifact collection. + +## Set up + +### 1. Install the extra + +The Sail SDK is an optional extra: + +```bash +pip install 'agentenv-framework[sail]' # or: uv add 'agentenv-framework[sail]' +``` + +Without it, selecting `sail_vm` fails with a `ConfigError` that names this extra. Nothing else needs it. + +### 2. Get a Sail API key + +Create a key in the [Sail dashboard](https://app.sailresearch.com). For local work, export it: + +```bash +export SAIL_API_KEY=sk_... +``` + +For a shared deployment, put it in your secret store instead, for example as `sail_api_key`, and reference +it with `secret:sail_api_key`. Never put the key itself in `config.toml`. + +### 3. Configure + +A complete `.agentenv/config.toml` that runs every environment and agent on Sailboxes, with the local stores: + +```toml +[sandbox] +default = "sail_vm" # environments and sandboxes +agent_default = "sail_vm" # agents + +[sandbox.providers.sail_vm.config] +api_key = "env:SAIL_API_KEY" # or "secret:sail_api_key" + +# The model endpoint agents call. Sail injects its key into their requests (see below), so it must be HTTPS. +[model] +base_url = "https://litellm.example.com" +api_key = "env:LITELLM_API_KEY" +``` + +To keep the local default and use Sail per run instead, add only the `[sandbox.providers.sail_vm.config]` +table and pass `--sandbox sail_vm`. `agent-env config show` prints the file in effect and masks the key. + +### 4. Run something + +The bundled `hello` task deploys a Sailbox, loads a file into it and checks it, with no model needed: + +```console +$ agent-env run hello --sandbox sail_vm +[tasks/hello.json] step 1/3 box (deploy_sandbox) +[tasks/hello.json] step 1/3 box done in 2.4s +[tasks/hello.json] step 2/3 load (load_artifact) +[tasks/hello.json] step 3/3 hello (verify_sandbox) +[tasks/hello.json] passed in 3.1s + +Tasks: + tasks/hello.json v1: passed (hello: 1), 3.1s + +Tore down 1 sandbox. +``` + +`agent-env -v run …` also logs each Sailbox as it starts +(`Sail VM sandbox started: sailbox_id=sb_… app=agent-env size=s …`). + +## Configuration reference + +All keys go under `[sandbox.providers.sail_vm.config]`. Unknown keys are refused. + +| Key | Default | Meaning | +|---|---|---| +| `api_key` | required | The Sail API key, as an `env:` or `secret:` reference. | +| `app` | `"agent-env"` | The Sail App every Sailbox belongs to; Sail groups and bills by App. | +| `min_size` | `"s"` | The smallest Sailbox size to pick: `s`, `m` or `l`. | +| `auto_sleep` | `false` | Let Sail sleep an idle Sailbox; the first request after waking waits a few seconds. | +| `auto_sleep_min_idle_seconds` | unset | 1–3600 seconds of idleness before Sail may sleep a Sailbox; turns `auto_sleep` on. | +| `runtime_threads` | the SDK's own | The size of the SDK's network thread pool (1–256). | +| `inject_model_key` | `true` | Keep an agent's model key out of its Sailbox (see "The agent's model key"). | + +A process uses one Sail API key: the SDK reads it from `SAIL_API_KEY` when it builds its process-wide +client. The provider sets that variable only for that one build and then restores it. Workloads never +see the key. + +## Examples + +**A shared deployment.** The key comes from the secret store, and costs are grouped under their own App: + +```toml +[sandbox] +default = "sail_vm" +agent_default = "sail_vm" +attribution = { team = "env-pod", project_id = "env:PROJECT_ID?unassigned" } + +[sandbox.providers.sail_vm.config] +api_key = "secret:sail_api_key" +app = "agent-env-prod" +min_size = "m" + +[model] +base_url = "https://litellm.example.com" +api_key = "secret:litellm_api_key" +``` + +**A per-run model key.** A run's override key is injected the same way as the configured one, so short-lived +per-run keys never reach a Sailbox either: + +```bash +agent-env task run --id my-task --agent-sandbox sail_vm --env-sandbox sail_vm \ + --litellm-api-key "$RUN_SCOPED_KEY" --judge-litellm-api-key "$JUDGE_SCOPED_KEY" +``` + +**A sandbox with restricted egress.** In a task, `deploy_sandbox` and `deploy_agent` take a +`network_policy`; Sail enforces it for the VM and its containers: + +```json +{"id": "box", "type": "deploy_sandbox", "sandbox_name": "box", "sandbox_mode": "vm", "sandbox_type": "sail_vm", + "network_policy": {"mode": "allowlist", "allow_hosts": ["pypi.org", "*.github.com"], "allow_cidrs": ["10.0.0.0/8"]}} +``` + +**Long, mostly idle runs.** Let Sail sleep a Sailbox after 10 idle minutes; it wakes on traffic or a command: + +```toml +[sandbox.providers.sail_vm.config] +api_key = "secret:sail_api_key" +auto_sleep_min_idle_seconds = 600 +``` + +**An internal, non-HTTPS model endpoint.** Injection needs HTTPS, so pass the key into the Sailbox as other +providers do: + +```toml +[sandbox.providers.sail_vm.config] +api_key = "secret:sail_api_key" +inject_model_key = false +``` + +**A fallback chain.** Try Sail first and fall back to E2B when a Sailbox can't be created in time: + +```toml +[sandbox] +default = "sail_vm,e2b" +``` + +**From Python.** Build the configured provider and create a VM directly: + +```python +import asyncio + +from agent_env.providers.sandbox_providers.sandbox_provider import build_sandbox_provider + + +async def main() -> None: + provider = build_sandbox_provider("sail_vm") + sandbox = await provider.create_vm(cpu=1, memory=2048, exposed_ports=[8080], timeout=900) + try: + print(await sandbox.exec_with_output("docker", "info", "--format", "{{.ServerVersion}}")) + print(sandbox.tunnel_urls[8080]) # https://sb--8080.sail.box + finally: + await sandbox.terminate() + + +asyncio.run(main()) +``` + +## Resources and lifetime + +- **Size:** a Sailbox's size fixes its vCPU (`s`, `m`, `l` = 1, 4, 8). The provider picks the smallest + size, no smaller than `min_size`, that covers the requested CPU. +- **Memory and disk:** these are ceilings, not reservations, since Sail bills observed usage. Requests + are rounded up to whole GiB, into the size's range: memory 2–64, 8–128 or 16–256 GiB, and disk 8–128, + 32–512 or 64–1024 GiB. +- **Refused requests:** a request no size can meet is refused before anything is created. +- **Lifetime:** the sandbox's `timeout` is the Sailbox's hard maximum lifetime. + +## Networking + +- **Ports:** each exposed port gets a public URL, `https://sb--.sail.box`, which fills + `tunnel_urls`. It serves HTTP and WebSocket, with no platform authentication, like Modal's tunnels. +- **Egress:** allow-all, or an allowlist of hostnames, `*.domain` wildcards, IPv4 addresses and IPv4 + CIDRs, up to Sail's 128 entries. IPv6 entries are refused. The provider adds the hosts of signed + download URLs to an allowlist before it loads images or objects. +- **Reconnect:** `get_sandbox` restores the ports and the applied egress policy. A policy it can't + represent leaves `network_policy` unknown, and image loading then fails closed. + +## The agent's model key + +With `inject_model_key` on, an agent's `LITELLM_API_KEY` never enters its Sailbox: + +1. The key is stored as a Sail secret named `AGENTENV_LITELLM_`, one per distinct key. +2. The Sailbox is created with a saved egress policy. On requests to `LITELLM_BASE_URL`'s host, the policy + sets the `authorization: Bearer …` and `x-api-key` headers from that secret. Sail adds the key as each + request leaves the box. +3. Inside the box, every command and file agent-env sends carries the placeholder + `sail-injected-model-key` in place of the key. +4. A `docker` shim on the box gives every container the VM's CA bundle (`SSL_CERT_FILE`, + `REQUESTS_CA_BUNDLE`, `NODE_EXTRA_CA_CERTS`, `CURL_CA_BUNDLE`). Sail terminates TLS for the model host + with its own CA, so containers need it to trust those requests. + +Requirements and behaviour: + +- **Endpoint:** the model endpoint must be HTTPS and reachable from Sail. +- **Key length:** keys shorter than 16 characters are refused, because scrubbing replaces the key + wherever it appears. +- **Other keys are refused:** a Sailbox refuses any command or file that would carry a model key Sail + doesn't inject for it. One example is an agent deployed into a sandbox it didn't create. +- **Turning it off:** set `inject_model_key = false` to pass keys in as other providers do. +- **Cleanup:** terminate deletes the Sailbox's policy. The secret is kept, so a concurrent launch with the + same key never loses it. To remove secrets for retired keys, sweep `AGENTENV_LITELLM_*` with + `sail secret list`. + +## Attribution and cost + +Sailboxes have no labels. The provider names each one `ae--`, which you can +find with the SDK's `Sailbox.list(search=...)`. It also logs one `agent_env.sail_vm_sandbox_started` event per box, +with `sailbox_id`, the App and the full attribution. To attribute cost, join Sail's per-Sailbox spend +(`GET /sailboxes/spend`) on `sailbox_id`. + +## Things to know + +- **Automatic checkpoints:** Sail checkpoints every Sailbox's disk for host-failure recovery, and this + can't be turned off. Anything a workload's container env holds lands there, apart from the injected + model key. With the S3 object store, keep `share_credentials` off for Sail runs. +- **Docker-in-Docker:** containers an agent starts with its own Docker-in-Docker don't get the CA bundle. +- **No `SAIL_MODE`:** the provider talks to Sail's production endpoints. + +## Testing + +`tst/integration/providers/sandbox_providers/sail_vm_sandbox_smoke_test.py` runs against a real Sail +account when `[sandbox.providers.sail_vm.config]` resolves. Otherwise it skips with +`agentenv-capability-missing: remote_sandbox`. `gateway_test.py` and `task_steps_test.py` include a +`sail_vm` case. diff --git a/src/agent_env/providers/sandbox_providers/sail_vm/__init__.py b/src/agent_env/providers/sandbox_providers/sail_vm/__init__.py new file mode 100644 index 00000000..480242d3 --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/sail_vm/__init__.py @@ -0,0 +1 @@ +"""Sail Research Sailbox VM sandbox provider (the ``sail`` extra); see README.md.""" diff --git a/src/agent_env/providers/sandbox_providers/sail_vm/_sdk.py b/src/agent_env/providers/sandbox_providers/sail_vm/_sdk.py new file mode 100644 index 00000000..e76ccc69 --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/sail_vm/_sdk.py @@ -0,0 +1,64 @@ +"""The Sail SDK (the ``sail`` extra), authenticated with the provider's configured key. + +The Python SDK takes its key (and its thread-pool size) only from the environment, read once when it +builds its process-wide client. Both are set for that one build and restored, so subprocesses never +inherit them and a process holds one Sail key. +""" + +from __future__ import annotations + +import hashlib +import os +import threading +from types import ModuleType +from typing import Any + +import sail + +from agent_env.config.errors import ConfigError + +API_KEY_ENV = "SAIL_API_KEY" +RUNTIME_THREADS_ENV = "SAIL_RUNTIME_THREADS" + +_lock = threading.Lock() +_installed_key: str | None = None +_apps: dict[str, Any] = {} + + +def _fingerprint(api_key: str) -> str: + return hashlib.sha256(api_key.encode()).hexdigest() + + +def connect(api_key: str, app_name: str, *, runtime_threads: int | None = None, sdk: ModuleType | Any | None = None) -> tuple[Any, Any]: + """The SDK module and the Sail App ``app_name`` (minted if missing), authenticating on first use. + + Blocking: call it through ``asyncio.to_thread``. + """ + global _installed_key + if sdk is None: + sdk = sail + fingerprint = _fingerprint(api_key) + with _lock: + if _installed_key is None: + overrides = {API_KEY_ENV: api_key} + if runtime_threads is not None: + overrides[RUNTIME_THREADS_ENV] = str(runtime_threads) + previous = {name: os.environ.get(name) for name in overrides} + os.environ.update(overrides) + try: + sdk.reset_transports() + _apps[app_name] = sdk.App.find(name=app_name, mint_if_missing=True) + finally: + for name, value in previous.items(): + if value is None: + os.environ.pop(name, None) + else: + os.environ[name] = value + _installed_key = fingerprint + elif _installed_key != fingerprint: + raise ConfigError( + "a process can use one Sail API key: another [sandbox.providers.sail] key is already in use" + ) + elif app_name not in _apps: + _apps[app_name] = sdk.App.find(name=app_name, mint_if_missing=True) + return sdk, _apps[app_name] diff --git a/src/agent_env/providers/sandbox_providers/sail_vm/model_key.py b/src/agent_env/providers/sandbox_providers/sail_vm/model_key.py new file mode 100644 index 00000000..e86cc282 --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/sail_vm/model_key.py @@ -0,0 +1,208 @@ +"""Keeping the model key out of a Sailbox: Sail adds it to the agent's model requests as they leave. + +The key is stored as a Sail secret named from its SHA-256, so one key is one secret and different keys never +share one. The secret is never deleted, so no launch can remove one another, concurrent or in another +process, still needs; a Sailbox's saved policy, which only names it, is deleted with the Sailbox. A saved egress policy on the Sailbox sets it as the auth headers of requests to the model endpoint's +host. Inside the Sailbox the key is replaced by a placeholder, and containers start trusting the CA Sail +terminates those requests' TLS with. +""" + +from __future__ import annotations + +import hashlib +import json +import re +import shlex +import uuid +from dataclasses import dataclass, field +from typing import Any, Mapping +from urllib.parse import urlparse + +SECRET_PREFIX = "AGENTENV_LITELLM_" +POLICY_PREFIX = "agentenv-" +PLACEHOLDER = "sail-injected-model-key" +KEY_ENV = "LITELLM_API_KEY" +BASE_URL_ENV = "LITELLM_BASE_URL" + +#: The Sailbox's CA bundle, which includes the CA Sail intercepts TLS with, and where containers see it. +VM_CA_BUNDLE = "/etc/ssl/certs/ca-certificates.crt" +CONTAINER_CA_BUNDLE = "/etc/ssl/certs/sailbox-ca-bundle.crt" +CONTAINER_TRUST_ENV = { + "SSL_CERT_FILE": CONTAINER_CA_BUNDLE, + "REQUESTS_CA_BUNDLE": CONTAINER_CA_BUNDLE, + "NODE_EXTRA_CA_CERTS": CONTAINER_CA_BUNDLE, + "CURL_CA_BUNDLE": CONTAINER_CA_BUNDLE, +} +#: Ahead of /usr/bin on the Sailbox's PATH, so every ``docker run`` / ``create`` passes through it. +DOCKER_SHIM_PATH = "/usr/local/bin/docker" + +_SECRET_REF = re.compile(r"\$\{secrets\.([A-Za-z0-9_]+)\}") +# A model-key env var assignment and the whole shell word assigned: quoted, escaped and bare parts alike. +_KEY_NAMES = ("LITELLM_API_KEY", "ANTHROPIC_API_KEY") +# A shell word: quoted, escaped and bare parts, concatenated. +_SHELL_WORD = re.compile(r"(?:'[^']*'|\"(?:[^\"\\]|\\.)*\"|\\.|[^\s'\"\\;|&<>()])+") +# An assigned value, as written, that is wholly one variable reference a shell expands: $NAME, ${NAME}, "$NAME", "${NAME}". +_REFERENCE = re.compile(r'^"?\$(?:[A-Za-z_][A-Za-z0-9_]*|\{[A-Za-z_][A-Za-z0-9_]*\})"?$') +# In text that isn't a shell script: KEY=value, or a YAML / JSON KEY: value. +_LITERAL_ASSIGNMENT = re.compile( + r"\b(?:LITELLM_API_KEY|ANTHROPIC_API_KEY)['\"]?(?:=|\s*:\s*)['\"]?([^\s'\",}#]*)" +) +#: Shorter keys are refused: scrubbing replaces every occurrence, which for a short one hits unrelated text. +MIN_KEY_LENGTH = 16 + +# By secret name: the key, so a handle reconnected here can still scrub its agent's key, and the launches +# and Sailboxes in this process that need it, so the key is forgotten once none does. Process memory only. +_injected_keys: dict[str, str] = {} +_holders: dict[str, set[str]] = {} + + +def secret_name(key: str) -> str: + return SECRET_PREFIX + hashlib.sha256(key.encode()).hexdigest()[:32].upper() + + +def carries_model_key(text: str, *, shell: bool) -> bool: + """Whether ``text`` assigns a model-key env var a value other than the placeholder. In a ``shell`` script + each word is decoded as the shell would (quotes and escapes, in the name too), and a value that is wholly + one unquoted or double-quoted variable reference isn't one; in other text, every value is literal.""" + if not shell: + return any(value and value != PLACEHOLDER for value in _LITERAL_ASSIGNMENT.findall(text)) + for word in _SHELL_WORD.findall(text): + try: + decoded = "".join(shlex.split(word)) + except ValueError: + return True + name, assigned, value = decoded.partition("=") + if not assigned or name not in _KEY_NAMES or not value or value == PLACEHOLDER: + continue + if not _REFERENCE.match(_written_value(word)): + return True + return False + + +def _written_value(word: str) -> str: + """The part of a shell word after its first ``=`` outside single quotes and escapes, as written.""" + quote = None + index = 0 + while index < len(word): + char = word[index] + if quote == "'": + if char == "'": + quote = None + elif char == "=": + return word[index + 1:] + elif char == "\\": + index += 1 + elif char == '"': + quote = None if quote == '"' else '"' + elif char == "'": + quote = "'" + index += 1 + return "" + + +def docker_shim() -> str: + """A ``docker`` wrapper that gives every container it runs or creates the Sailbox's CA bundle.""" + trust = " ".join(f"-e {name}={value}" for name, value in CONTAINER_TRUST_ENV.items()) + flags = f"-v {VM_CA_BUNDLE}:{CONTAINER_CA_BUNDLE}:ro {trust}" + return f"""#!/bin/sh +case "$1 $2" in + "container run"*|"container create"*) sub="$1 $2"; shift 2; exec /usr/bin/docker $sub {flags} "$@" ;; +esac +case "$1" in + run|create) sub="$1"; shift; exec /usr/bin/docker "$sub" {flags} "$@" ;; +esac +exec /usr/bin/docker "$@" +""" + + +@dataclass +class ModelKeyInjection: + """The model endpoint host, the Sail secret holding the key, and the saved policy applying it. + ``key`` is None on a reconnected Sailbox unless the configured key is the one injected.""" + + host: str + secret: str + key: str | None = field(default=None, repr=False) + policy_id: str | None = None + #: Saved policies a swap may or may not have applied (Sail couldn't say), deleted once it can. + unsettled_policy_ids: list[str] = field(default_factory=list) + #: Every saved policy made for one Sailbox is named under this prefix, so teardown from any handle, in + #: any process, finds them all; a reconnected handle takes it from the applied policy's name. + policy_prefix: str = field(default_factory=lambda: f"{POLICY_PREFIX}{uuid.uuid4().hex[:12]}") + + @classmethod + def for_env(cls, env: Mapping[str, str]) -> ModelKeyInjection | None: + """The injection for an agent ``env`` carrying a model key, or None when it carries none.""" + key = env.get(KEY_ENV) + if not key or key == PLACEHOLDER: + return None + base_url = env.get(BASE_URL_ENV) or "" + parsed = urlparse(base_url) + if parsed.scheme != "https" or not parsed.hostname: + raise ValueError( + f"Sail injects the model key only into HTTPS requests, but {BASE_URL_ENV} is {base_url!r}; " + "use an https endpoint or set inject_model_key = false in [sandbox.providers.sail_vm.config]" + ) + if len(key) < MIN_KEY_LENGTH: + raise ValueError( + f"Sail model-key injection needs a {KEY_ENV} of at least {MIN_KEY_LENGTH} characters, to scrub it " + "safely; set inject_model_key = false in [sandbox.providers.sail_vm.config] to pass it in" + ) + return cls(host=parsed.hostname, secret=secret_name(key), key=key) + + @classmethod + def from_document(cls, document: Any, policy_id: str | None) -> ModelKeyInjection | None: + """The injection a saved policy document applies, or None when its rules aren't one.""" + rules = document.get("rules") if isinstance(document, dict) else None + if not isinstance(rules, dict) or len(rules) != 1: + return None + (host,) = rules + names = set(_SECRET_REF.findall(json.dumps(rules))) + if len(names) != 1: + return None + (name,) = names + injection = cls(host=host, secret=name, policy_id=policy_id) + return injection if name.startswith(SECRET_PREFIX) and rules == injection.rules() else None + + def rules(self) -> dict[str, Any]: + """Sail egress rules setting the key as both auth headers model endpoints read.""" + ref = f"${{secrets.{self.secret}}}" + return {self.host: [{"request": {"set": {"headers": {"authorization": f"Bearer {ref}", "x-api-key": ref}}}}]} + + def matches(self, other: ModelKeyInjection | None) -> bool: + return other is not None and (other.host, other.secret) == (self.host, self.secret) + + def hold(self, holder: str) -> None: + """Record ``holder`` (a launch or a Sailbox id) as needing this injection's secret.""" + if self.key is not None: + _injected_keys[self.secret] = self.key + _holders.setdefault(self.secret, set()).add(holder) + + def release(self, holder: str) -> None: + """Drop ``holder``, forgetting the key once no launch or Sailbox in this process needs it.""" + holders = _holders.get(self.secret, set()) + holders.discard(holder) + if not holders: + _holders.pop(self.secret, None) + _injected_keys.pop(self.secret, None) + + def recover_key(self, candidate: str | None) -> None: + """Recover the key this injection's secret was named from: one this process injected, else + ``candidate`` when it is that key.""" + self.key = _injected_keys.get(self.secret) + if self.key is None and candidate and secret_name(candidate) == self.secret: + self.key = candidate + + def _encodings(self) -> list[str]: + """The key as agent-env writes it into commands: shell-quoted, inside single quotes, and raw.""" + return [shlex.quote(self.key), self.key.replace("'", "'\\''"), self.key] + + def scrub(self, text: str) -> str: + for encoded in self._encodings() if self.key else (): + text = text.replace(encoded, PLACEHOLDER) + return text + + def scrub_bytes(self, data: bytes) -> bytes: + for encoded in self._encodings() if self.key else (): + data = data.replace(encoded.encode(), PLACEHOLDER.encode()) + return data diff --git a/src/agent_env/providers/sandbox_providers/sail_vm/provider.py b/src/agent_env/providers/sandbox_providers/sail_vm/provider.py new file mode 100644 index 00000000..27116548 --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/sail_vm/provider.py @@ -0,0 +1,423 @@ +"""Sail Research Sailbox VM sandbox provider.""" + +from __future__ import annotations + +import asyncio +import logging +import math +import re +import uuid +from typing import Any, Awaitable, Callable, ClassVar, Self + +from agent_env.attribution import PIPELINE_STEP_KEY, RUN_ID_KEY, Attribution +from agent_env.config import get_config +from agent_env.config.errors import ConfigError +from agent_env.providers.sandbox_providers.sail_vm import _sdk +from agent_env.providers.sandbox_providers.sail_vm.model_key import ModelKeyInjection +from agent_env.providers.sandbox_providers.sail_vm.sandbox import ( + MAX_ALLOWLIST_ENTRIES, + SailVmSandbox, + create_saved_policy, + egress_document, + policy_name, + release_injection, +) +from agent_env.providers.sandbox_providers.sandbox import NetworkPolicy, NetworkPolicyUnsupportedError +from agent_env.providers.sandbox_providers.sandbox_provider import ( + SANDBOX_MODE_VM, + SandboxProvider, + apply_default_attribution, +) + +logger = logging.getLogger(__name__) + +SANDBOX_STARTED_EVENT = "agent_env.sail_vm_sandbox_started" + +#: (size, vCPU, (min, max) memory GiB, (min, max) disk GiB), smallest first. Memory and disk are +#: ceilings, not reservations: Sail bills observed usage. +_SIZES: tuple[tuple[str, int, tuple[int, int], tuple[int, int]], ...] = ( + ("s", 1, (2, 64), (8, 128)), + ("m", 4, (8, 128), (32, 512)), + ("l", 8, (16, 256), (64, 1024)), +) +_SIZE_NAMES = tuple(size[0] for size in _SIZES) +_LISTENER_TIMEOUT = 60 +_LISTENER_POLL_INTERVAL = 1 +_MAX_NAME_LENGTH = 128 +_REAP_ATTEMPTS = 3 + +_reapers: set[asyncio.Task] = set() + + +def sailbox_shape(cpu: float, memory_mb: int, disk_size_gb: float, *, min_size: str = "s") -> tuple[str, int, int]: + """The smallest size at or above ``min_size`` covering the request, and its memory and disk ceilings + in GiB, each rounded up to whole GiB and into the size's range.""" + memory_gib = math.ceil(memory_mb / 1024) + disk_gib = math.ceil(disk_size_gb) + for name, vcpu, (memory_min, memory_max), (disk_min, disk_max) in _SIZES[_SIZE_NAMES.index(min_size):]: + if cpu <= vcpu and memory_gib <= memory_max and disk_gib <= disk_max: + return name, max(memory_gib, memory_min), max(disk_gib, disk_min) + raise ValueError( + f"no Sailbox size fits cpu={cpu}, memory={memory_mb}MiB, disk={disk_size_gb}GB " + "(largest is l: 8 vCPU, 256 GiB memory, 1024 GiB disk)" + ) + + +def sailbox_name(attribution: Attribution) -> str: + """``ae-`` plus the attribution values in key order, slugged: for people and ``list(search=)``.""" + slugs = [re.sub(r"[^A-Za-z0-9]+", "-", str(attribution[key])).strip("-") for key in sorted(attribution)] + return "-".join(["ae", uuid.uuid4().hex[:8], *filter(None, slugs)])[:_MAX_NAME_LENGTH].rstrip("-") + + +async def _reap(sandbox: SailVmSandbox) -> None: + for attempt in range(_REAP_ATTEMPTS): + try: + await sandbox.terminate() + logger.info("Terminated Sailbox %s, created after its caller was cancelled", sandbox.sandbox_id) + return + except Exception as exc: # noqa: BLE001 - every failure is retried, then reported + logger.warning("Terminating orphaned Sailbox %s failed (attempt %s): %s", sandbox.sandbox_id, attempt + 1, exc) + await asyncio.sleep(2 ** attempt) + logger.error( + "Orphaned Sailbox %s is still running after %s termination attempts; it stops at its max lifetime", + sandbox.sandbox_id, _REAP_ATTEMPTS, + ) + + +async def _terminate_named(sdk: Any, app: Any, name: str) -> bool: + """Terminate every live Sailbox in ``app`` called ``name`` (one a create made though it reported failing), + retrying through a brief outage; False when that couldn't be confirmed.""" + for attempt in range(_REAP_ATTEMPTS): + try: + for box in await sdk.Sailbox.list.aio(app_id=app, search=name): + if box.name == name and box.status not in ("terminated", "terminating"): + await box.terminate.aio() + logger.info("Terminated Sailbox %s, created though its create reported a failure", box.sailbox_id) + return True + except Exception as exc: # noqa: BLE001 - retried, then reported + logger.warning("Checking for a Sailbox %s left by a failed create failed (attempt %s): %s", name, attempt + 1, exc) + await asyncio.sleep(2 ** attempt) + logger.error("A Sailbox %s may be left by a failed create; it stops at its max lifetime", name) + return False + + +async def _create_or_reclaim( + create: Any, wrap: Callable[[Any], SailVmSandbox], release: Callable[[], Awaitable[None]] | None = None, +) -> Any: + """Await a Sailbox create. If the caller is cancelled first, terminate (``wrap``ped, so its model-key + policy and secret go too) the Sailbox it yields, which would otherwise keep running with no handle; if + that create then fails, ``release`` what it was given.""" + task = asyncio.ensure_future(create) + + def terminate_orphan(done: asyncio.Future) -> None: + if done.cancelled() or done.exception() is not None: + cleanup = release() if release is not None else None + else: + cleanup = _reap(wrap(done.result())) + if cleanup is None: + return + reaper = asyncio.ensure_future(cleanup) + _reapers.add(reaper) + reaper.add_done_callback(_reapers.discard) + + try: + return await asyncio.shield(task) + except asyncio.CancelledError: + task.add_done_callback(terminate_orphan) + raise + + +class SailVmSandboxProvider(SandboxProvider): + """Docker-capable Sailboxes. ``api_key`` comes from resolved provider config (a ``secret:`` reference) + and never reaches a workload. With ``inject_model_key`` (the default) neither does an agent's model key: + Sail adds it to the agent's requests to the model endpoint (see ``model_key``).""" + + EGRESS_HOSTS: ClassVar[tuple[str, ...]] = ("*.sail.box",) + + def __init__( + self, + *, + api_key: str, + app: str = "agent-env", + min_size: str = "s", + auto_sleep: bool = False, + auto_sleep_min_idle_seconds: int | None = None, + runtime_threads: int | None = None, + inject_model_key: bool = True, + sdk: Any | None = None, + ): + self._api_key = api_key + self._app_name = app + self._min_size = min_size + self._auto_sleep = auto_sleep or auto_sleep_min_idle_seconds is not None + self._auto_sleep_min_idle_seconds = auto_sleep_min_idle_seconds + self._runtime_threads = runtime_threads + self._inject_model_key = inject_model_key + self._sdk = sdk + self._app: Any | None = None + + def __repr__(self) -> str: + return f"SailVmSandboxProvider(app={self._app_name!r})" + + @classmethod + def from_config(cls, **config: Any) -> Self: + section = "[sandbox.providers.sail_vm.config]" + api_key = config.get("api_key") + if not isinstance(api_key, str) or not api_key.strip(): + raise ConfigError(f"{section} requires a non-empty 'api_key' (e.g. \"secret:sail_api_key\")") + app = config.get("app", "agent-env") + if not isinstance(app, str) or not app.strip(): + raise ConfigError(f"{section} 'app' must be a non-empty string") + if config.get("min_size", "s") not in _SIZE_NAMES: + raise ConfigError(f"{section} 'min_size' must be one of {list(_SIZE_NAMES)}") + for key in ("auto_sleep", "inject_model_key"): + if not isinstance(config.get(key, False), bool): + raise ConfigError(f"{section} '{key}' must be true or false") + for key, (low, high) in {"auto_sleep_min_idle_seconds": (1, 3600), "runtime_threads": (1, 256)}.items(): + value = config.get(key) + if value is not None and (isinstance(value, bool) or not isinstance(value, int) or not low <= value <= high): + raise ConfigError(f"{section} '{key}' must be an integer from {low} to {high}") + unknown = set(config) - { + "api_key", "app", "min_size", "auto_sleep", "auto_sleep_min_idle_seconds", "runtime_threads", "inject_model_key", + } + if unknown: + raise ConfigError(f"{section} has unknown key(s): {sorted(unknown)}") + return cls(**config) + + @classmethod + def supports_network_policy(cls, policy: NetworkPolicy) -> bool: + """Allow-all, or an allowlist of hostnames, IPv4 addresses and IPv4 CIDRs within Sail's entry limit.""" + if not policy.restricts_egress: + return True + return ( + len(policy.allow_hosts) + len(policy.allow_cidrs) <= MAX_ALLOWLIST_ENTRIES + and not any(":" in entry for entry in (*policy.allow_hosts, *policy.allow_cidrs)) + ) + + async def _connect(self) -> tuple[Any, Any]: + if self._app is None: + self._sdk, self._app = await asyncio.to_thread( + _sdk.connect, self._api_key, self._app_name, runtime_threads=self._runtime_threads, sdk=self._sdk, + ) + return self._sdk, self._app + + def _auto_sleep_setting(self, sdk: Any) -> Any: + if not self._auto_sleep: + return sdk.AutoSleep.never() + if self._auto_sleep_min_idle_seconds is not None: + return sdk.AutoSleep.not_before(self._auto_sleep_min_idle_seconds) + return sdk.AutoSleep.default() + + async def create_vm( + self, + *, + image: str | None = None, + boot_mode: str | None = None, + cpu: float = 1.0, + memory: int = 8192, + disk_size_gb: float = 10, + timeout: int = 3600 * 2, + exposed_ports: list[int] | None = None, + setup_for_gateway: bool = True, + attribution: Attribution | None = None, + network_policy: NetworkPolicy | None = None, + ) -> SailVmSandbox: + """Create a Sailbox from the devbox image; ``timeout`` is its hard maximum lifetime.""" + if image is not None: + raise ValueError("the Sail provider boots its own Docker-capable image; image overrides are unsupported") + del boot_mode + return await self._create( + cpu=cpu, memory=memory, disk_size_gb=disk_size_gb, timeout=timeout, exposed_ports=exposed_ports, + setup_for_gateway=setup_for_gateway, attribution=attribution, network_policy=network_policy, injection=None, + ) + + async def _create( + self, + *, + cpu: float, + memory: int, + disk_size_gb: float, + timeout: int, + exposed_ports: list[int] | None, + setup_for_gateway: bool, + attribution: Attribution | None, + network_policy: NetworkPolicy | None, + injection: ModelKeyInjection | None, + ) -> SailVmSandbox: + size, memory_gib, disk_gib = sailbox_shape(cpu, memory, disk_size_gb, min_size=self._min_size) + effective_policy = self.effective_network_policy(network_policy) + if injection is not None: + effective_policy = effective_policy.with_hosts([injection.host]) + if not self.supports_network_policy(effective_policy): + raise NetworkPolicyUnsupportedError( + f"Sail enforces allow-all or up to {MAX_ALLOWLIST_ENTRIES} hostname/IPv4 allowlist entries, " + f"not {effective_policy.to_dict()}" + ) + resolved_attribution = { + key: str(value) for key, value in apply_default_attribution(dict(attribution or {})).items() if value is not None + } + ports = list(dict.fromkeys(exposed_ports or [])) + sdk, app = await self._connect() + egress: Any = egress_document(effective_policy) + + launch = f"launch-{uuid.uuid4().hex}" + + def wrap(raw: Any) -> SailVmSandbox: + sandbox = SailVmSandbox( + raw, sdk=sdk, tunnel_urls={}, network_policy=effective_policy, injection=injection, + refuse_model_keys=self._inject_model_key, + ) + if injection is not None: + injection.release(launch) + return sandbox + + name = sailbox_name(resolved_attribution) + + async def release() -> None: + """Undo a create that failed or was abandoned: terminate any Sailbox Sail made under ``name`` (its + response may have been lost), then release the injection's policy.""" + terminated = await _terminate_named(sdk, app, name) + if injection is not None: + if terminated: + await release_injection(sdk, injection, launch) + else: + injection.release(launch) + logger.error("Egress policy %s stays with the unconfirmed Sailbox %s; delete it by name", injection.policy_id, name) + + creating = False + try: + if injection is not None: + injection.hold(launch) + await sdk.Secret.set.aio(injection.secret, injection.key) + egress = await create_saved_policy(sdk, policy_name(injection), egress_document(effective_policy, injection)) + injection.policy_id = egress.id + creating = True + raw = await _create_or_reclaim(sdk.Sailbox.create.aio( + app=app, + image=sdk.Image.devbox("amd64"), + name=name, + size=size, + memory_limit_gib=memory_gib, + disk_limit_gib=disk_gib, + max_lifetime_seconds=timeout, + ingress_ports=ports, + auto_sleep=self._auto_sleep_setting(sdk), + egress_policy=egress, + ), wrap, release) + except BaseException as exc: + # A create cancelled in flight is _create_or_reclaim's to clean up once it settles. + if not (creating and isinstance(exc, asyncio.CancelledError)): + await release() + raise + sandbox = wrap(raw) + try: + if raw.status in ("failed", "create_failed"): + raise RuntimeError(f"Sailbox {raw.sailbox_id} failed to start: {raw.error_message}") + sandbox.tunnel_urls = await self._tunnel_urls(raw, ports) + sandbox.mode = SANDBOX_MODE_VM + logger.info( + "Sail VM sandbox started: sailbox_id=%s app=%s size=%s memory=%sGiB disk=%sGiB model_key_injected=%s attribution=%s", + raw.sailbox_id, self._app_name, size, memory_gib, disk_gib, injection is not None, resolved_attribution, + extra={ + "event": SANDBOX_STARTED_EVENT, + "sail_sailbox_id": raw.sailbox_id, + "sail_app_name": self._app_name, + "sail_attribution": resolved_attribution, + PIPELINE_STEP_KEY: resolved_attribution.get(PIPELINE_STEP_KEY), + RUN_ID_KEY: resolved_attribution.get(RUN_ID_KEY), + "size": size, + "memory_limit_gib": memory_gib, + "disk_limit_gib": disk_gib, + }, + ) + if setup_for_gateway: + await sandbox.setup_vm_for_gateway(ports) + if injection is not None: + await sandbox.install_container_trust() + return sandbox + except BaseException: + try: + await sandbox.terminate() + except Exception as cleanup_error: # noqa: BLE001 - cleanup must not mask the create failure + logger.warning("Failed to terminate Sailbox %s after setup failure: %s", raw.sailbox_id, cleanup_error) + raise + + @staticmethod + async def _tunnel_urls(raw: Any, ports: list[int]) -> dict[int, str]: + """Each exposed port's public URL, polling until Sail has routed all of them.""" + loop = asyncio.get_running_loop() + deadline = loop.time() + _LISTENER_TIMEOUT + while True: + urls = { + listener.guest_port: listener.endpoint.url + for listener in await raw.listeners.aio() + if listener.endpoint is not None and getattr(listener.endpoint, "url", None) + } + missing = [port for port in ports if port not in urls] + if not missing: + return {port: urls[port] for port in ports} + if loop.time() >= deadline: + raise RuntimeError(f"Sailbox {raw.sailbox_id} has no public URL for port(s) {missing} after {_LISTENER_TIMEOUT}s") + await asyncio.sleep(_LISTENER_POLL_INTERVAL) + + async def create_sandbox( + self, + *, + image_name: str, + port: int, + env: dict[str, str], + cpu: float = 1.0, + memory: int = 8192, + disk_size_gb: float = 10, + timeout: int = 3600 * 2, + attribution: Attribution | None = None, + network_policy: NetworkPolicy | None = None, + ) -> SailVmSandbox: + """A bare VM the caller loads and starts ``image_name`` in, as on the other VM providers. A model key in + ``env`` is injected by Sail rather than passed in (unless ``inject_model_key`` is off).""" + del image_name + injection = ModelKeyInjection.for_env(env) if self._inject_model_key else None + return await self._create( + cpu=cpu, memory=memory, disk_size_gb=disk_size_gb, timeout=timeout, exposed_ports=[port], + setup_for_gateway=True, attribution=attribution, network_policy=network_policy, injection=injection, + ) + + async def create_container(self, **kwargs: Any) -> SailVmSandbox: + """The inherited login-pull-run, then the registry credentials removed from the VM disk, which Sail + checkpoints for host-failure recovery.""" + sandbox = await super().create_container(**kwargs) + try: + await sandbox.exec_script("rm -f /root/.docker/config.json") + except BaseException: + await sandbox.terminate() + raise + return sandbox + + async def get_sandbox(self, sandbox_id: str) -> SailVmSandbox: + """Reconnect, restoring ports, the applied egress policy and any model-key injection. The key itself + is recovered only when it is the configured ``[model]`` key, so the reconnected handle scrubs it too.""" + sdk, _ = await self._connect() + raw = await sdk.Sailbox.get.aio(sandbox_id) + tunnel_urls = { + listener.guest_port: listener.endpoint.url + for listener in await raw.listeners.aio() + if listener.endpoint is not None and getattr(listener.endpoint, "url", None) + } + applied = getattr(raw, "egress_policy", None) + injection = ModelKeyInjection.from_document(getattr(applied, "document", None), getattr(applied, "policy_id", None)) + if injection is not None: + try: + injection.recover_key(get_config().get_litellm_api_key()) + except Exception: # noqa: BLE001 - no configured key to recover; the handle just can't scrub it + pass + sandbox = SailVmSandbox( + raw, sdk=sdk, tunnel_urls=tunnel_urls, network_policy=None, injection=injection, + refuse_model_keys=self._inject_model_key, + ) + sandbox.network_policy = sandbox.adopt_applied_policy(applied) + if sandbox.network_policy is None: + logger.warning( + "Sailbox %s has an egress policy agent-env can't represent (%r); image loading will fail closed", + sandbox_id, applied, + ) + return sandbox diff --git a/src/agent_env/providers/sandbox_providers/sail_vm/sandbox.py b/src/agent_env/providers/sandbox_providers/sail_vm/sandbox.py new file mode 100644 index 00000000..d709981e --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/sail_vm/sandbox.py @@ -0,0 +1,485 @@ +"""A Sailbox (a Sail Research Linux VM with Docker) presented through the :class:`VmSandbox` contract.""" + +from __future__ import annotations + +import asyncio +import contextlib +import logging +import shlex +import time +import uuid +import weakref +from ipaddress import ip_network +from typing import Any, Optional +from urllib.parse import urlparse + +from agent_env.config import get_config +from agent_env.providers.sandbox_providers.sail_vm.model_key import ( + DOCKER_SHIM_PATH, + POLICY_PREFIX, + ModelKeyInjection, + carries_model_key, + docker_shim, +) +from agent_env.providers.sandbox_providers.sandbox import CURL_RETRY_FLAGS, NetworkMode, NetworkPolicy, VmSandbox +from agent_env.providers.sandbox_providers.sandbox_provider import SANDBOX_MODE_VM + +logger = logging.getLogger(__name__) + +#: Sail's limit on entries in one egress allowlist. +MAX_ALLOWLIST_ENTRIES = 128 +_CLEANUP_ATTEMPTS = 3 +_STREAM_CHUNK_BYTES = 8 * 1024 * 1024 +_SHELLS = frozenset({"bash", "sh"}) + +# One lock per Sailbox and event loop, shared by every handle to it while an update is in flight. +_policy_locks: weakref.WeakValueDictionary[tuple[asyncio.AbstractEventLoop, str], asyncio.Lock] = ( + weakref.WeakValueDictionary() +) + + +def egress_document(policy: NetworkPolicy, injection: ModelKeyInjection | None = None) -> dict[str, Any]: + """``policy`` as a Sail egress-policy document (``{}`` allows everything), with ``injection``'s rules.""" + document: dict[str, Any] = {} + if policy.mode is NetworkMode.ALLOWLIST: + document["allowlist"] = [*policy.allow_hosts, *policy.allow_cidrs] + if injection is not None: + document["rules"] = injection.rules() + return document + + +def policy_from_document(document: Any, injection: ModelKeyInjection | None = None) -> NetworkPolicy | None: + """The agent-env policy a Sail egress document applies, or None when it can't be represented. Rules are + representable only as ``injection``'s, which add headers and leave egress alone.""" + if not isinstance(document, dict): + return None + document = dict(document) + rules = document.pop("rules", None) + if rules is not None and (injection is None or rules != injection.rules()): + return None + if not document: + return NetworkPolicy() + entries = document.get("allowlist") + if set(document) != {"allowlist"} or not isinstance(entries, list) or not all(isinstance(e, str) and e for e in entries): + return None + hosts, cidrs = [], [] + for entry in entries: + try: + ip_network(entry, strict=False) + except ValueError: + hosts.append(entry) + else: + cidrs.append(entry) + return NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=tuple(hosts), allow_cidrs=tuple(cidrs)) + + +def _allows_host(policy: NetworkPolicy, host: str) -> bool: + """Whether an allowlist entry already admits ``host``: an exact match, or a ``*.domain`` it sits under.""" + return any(entry == host or (entry.startswith("*.") and host.endswith(entry[1:])) for entry in policy.allow_hosts) + + +class ModelKeyRefusedError(RuntimeError): + """A command or file would have carried a model key Sail doesn't inject for this Sailbox into it.""" + + +def policy_name(injection: ModelKeyInjection) -> str: + return f"{injection.policy_prefix}-{uuid.uuid4().hex[:12]}" + + +async def create_saved_policy(sdk: Any, name: str, document: dict[str, Any]) -> Any: + """Create a saved egress policy. If the create fails or is cancelled, the policy Sail may still have made + is found by ``name`` and deleted, so a lost response can't leave one naming a secret behind.""" + try: + return await sdk.EgressPolicy.create.aio(name, document) + except BaseException: + await delete_policy_named(sdk, name) + raise + + +async def delete_policy_named(sdk: Any, name: str, *, family: bool = False) -> None: + """Delete the saved policy called ``name`` (with ``family``, every one named ``name-…``), if Sail made + any, retrying through a brief outage; one Sail stays unreachable for is logged by name so it can be + swept (``sail egress-policy list``).""" + for attempt in range(_CLEANUP_ATTEMPTS): + try: + for summary in await sdk.EgressPolicy.list.aio(search=name): + if summary.name == name or (family and summary.name.startswith(f"{name}-")): + await _delete_policy_by_id(sdk, summary.id) + return + except Exception as exc: # noqa: BLE001 - retried, then reported; never masks the caller's outcome + logger.warning("Looking up egress policy %s for cleanup failed (attempt %s): %s", name, attempt + 1, exc) + await asyncio.sleep(2 ** attempt) + logger.error("Egress policy %s may be left in Sail, naming a model-key secret; delete it by name", name) + + +async def _delete_policy_by_id(sdk: Any, policy_id: str) -> None: + try: + await (await sdk.EgressPolicy.get.aio(policy_id)).delete.aio() + except sdk.NotFoundError: + pass + + +async def delete_saved_policy(sdk: Any, policy_id: str | None) -> None: + """Delete a saved egress policy, best effort: a leftover one names a secret but holds no value.""" + if policy_id is None: + return + try: + await _delete_policy_by_id(sdk, policy_id) + except Exception as exc: # noqa: BLE001 - reported, never masks the caller's outcome + logger.warning("Could not delete egress policy %s: %s", policy_id, exc) + + +async def release_injection(sdk: Any, injection: ModelKeyInjection, holder: str) -> None: + """Delete ``holder``'s saved policy; the secret it names stays (see ``model_key``).""" + injection.release(holder) + await delete_saved_policy(sdk, injection.policy_id) + + +class _BytesReader: + def __init__(self, value: bytes): + self._value = value + + async def read(self) -> bytes: + return self._value + + +class _CompletedProcess: + """An exec that failed before it produced output, in the process shape ``VmSandbox`` reads.""" + + def __init__(self, stdout: bytes, stderr: bytes, exit_code: int): + self.stdout = _BytesReader(stdout) + self.stderr = _BytesReader(stderr) + self._exit_code = exit_code + + async def wait(self) -> int: + return self._exit_code + + +class _Stream: + """One output stream, pumped from the moment the exec starts so no byte is dropped: iterate it to + stream (``collect_artifacts`` does), or ``read()`` it whole. The queue is bounded, so a slow reader + pauses the command rather than growing memory; a stream nobody claims is drained by ``wait()``. A + transient failure just ends the stream, leaving ``wait()`` to report exit -1 for ``exec_script`` to + retry; any other is raised to the reader.""" + + _MAX_CHUNKS = 64 + + def __init__(self, chunks, transient: tuple[type[BaseException], ...]): + self._transient = transient + self._queue: asyncio.Queue[bytes | None] = asyncio.Queue(maxsize=self._MAX_CHUNKS) + self.claimed = False + self.error: BaseException | None = None + self.pump = asyncio.ensure_future(self._pump(chunks)) + + async def _pump(self, chunks) -> None: + try: + async for chunk in chunks: + await self._queue.put(chunk) + except Exception as exc: # noqa: BLE001 - re-raised to the reader and by wait() + self.error = exc + finally: + await self._queue.put(None) + + async def _chunks(self): + while (chunk := await self._queue.get()) is not None: + yield chunk + if self.error is not None and not isinstance(self.error, self._transient): + raise self.error + + def __aiter__(self): + self.claimed = True + return self._chunks() + + async def read(self) -> bytes: + return b"".join([chunk async for chunk in self]) + + async def drain(self) -> None: + if not self.claimed: + self.claimed = True + async for _ in self._chunks(): + pass + + +class _SailProcess: + """A running Sail exec; a lost host or transport while it runs is exit -1, which ``exec_script`` retries.""" + + def __init__(self, process: Any, sdk: Any): + self._process = process + self._transient = (sdk.SailboxHostLostError, sdk.TransportError) + self.stdout = _Stream(process.stdout_bytes, self._transient) + self.stderr = _Stream(process.stderr_bytes, self._transient) + + async def wait(self) -> int: + await asyncio.gather(self.stdout.drain(), self.stderr.drain(), return_exceptions=True) + await asyncio.gather(self.stdout.pump, self.stderr.pump) + try: + exit_code = int((await self._process.wait()).exit_code) + except self._transient: + return -1 + for error in (self.stdout.error, self.stderr.error): + if isinstance(error, self._transient): + return -1 + if error is not None: + raise error + return exit_code + + +class SailVmSandbox(VmSandbox): + """A Sailbox from the Docker-capable devbox image; commands run as root. With a model-key ``injection``, + the key's value never enters the Sailbox: every command and file is scrubbed of it. With + ``refuse_model_keys``, one that still carries a model key (one Sail doesn't inject here) is refused.""" + + type = "sail_vm" + _DOCKER_PROBE_TIMEOUT = 10 + + def __init__( + self, sailbox: Any, *, sdk: Any, tunnel_urls: dict[int, str], network_policy: NetworkPolicy | None, + injection: ModelKeyInjection | None = None, refuse_model_keys: bool = False, + ): + self._sailbox = sailbox + self._sdk = sdk + self._injection = injection + self._refuse_model_keys = refuse_model_keys + if injection is not None: + injection.hold(sailbox.sailbox_id) + self.sandbox_id = sailbox.sailbox_id + self.tunnel_urls = tunnel_urls + self.vnc_url = None + self.mode = SANDBOX_MODE_VM + self.network_policy = network_policy + + async def terminate(self) -> None: + """Terminate the Sailbox, then delete its model-key policy: the one applied, which another handle may + have replaced.""" + if self._injection is None: + await self._terminate_sailbox() + return + async with self._policy_lock(): + try: + self.adopt_applied_policy((await self._sdk.Sailbox.get.aio(self.sandbox_id)).egress_policy) + except Exception as exc: # noqa: BLE001 - fall back to the policy this handle last applied + logger.info("Could not read Sailbox %s's applied policy before terminate: %s", self.sandbox_id, exc) + await self._terminate_sailbox() + await delete_policy_named(self._sdk, self._injection.policy_prefix, family=True) + for policy_id in self._injection.unsettled_policy_ids: + await delete_saved_policy(self._sdk, policy_id) + await release_injection(self._sdk, self._injection, self.sandbox_id) + + async def _terminate_sailbox(self) -> None: + try: + await self._sailbox.terminate.aio() + except self._sdk.NotFoundError: + logger.info("Sailbox %s was already gone at terminate", self.sandbox_id) + + def _policy_lock(self) -> asyncio.Lock: + """The lock every handle to this Sailbox in this event loop holds to change or tear down its policy.""" + return _policy_locks.setdefault((asyncio.get_running_loop(), self.sandbox_id), asyncio.Lock()) + + async def install_container_trust(self) -> None: + """Have every container started on this Sailbox trust the CA Sail injects the model key behind.""" + await self._sailbox.fs.write.aio(DOCKER_SHIM_PATH, docker_shim(), mode=0o755) + + async def exec(self, *command: str) -> _SailProcess | _CompletedProcess: + """Run argv to completion. A leading ``sudo`` is dropped (commands already run as root, and the + guest's hostname doesn't resolve, so sudo warns on every call); a lost host or transport maps to + exit -1, which ``exec_script`` retries.""" + return await self._run(*command) + + async def exec_with_output(self, *args: str) -> tuple[int, str, str]: + """``(exit_code, stdout, stderr)``; a command whose output stream was cut (exit -1) returns no stdout, + so a caller that skips the exit code can't mistake partial output for the whole.""" + process = await self.exec(*args) + stdout, stderr = await asyncio.gather(process.stdout.read(), process.stderr.read()) + exit_code = await process.wait() + return exit_code, "" if exit_code == -1 else stdout.decode(), stderr.decode() + + async def _run(self, *command: str, timeout: Optional[int] = None) -> _SailProcess | _CompletedProcess: + argv = list(command[1:] if command[:1] == ("sudo",) else command) + if self._injection is not None: + argv = [self._injection.scrub(arg) for arg in argv] + if self._refuse_model_keys and any( + carries_model_key(arg, shell=index > 0 and argv[index - 1] == "-c" and argv[0] in _SHELLS) + for index, arg in enumerate(argv) + ): + raise self._refusal() + try: + process = await self._sailbox.exec.aio( + argv, timeout=timeout, output_mode="pipe", idempotency_key=uuid.uuid4().hex, + ) + except (self._sdk.SailboxHostLostError, self._sdk.TransportError) as exc: + return _CompletedProcess(b"", f"{type(exc).__name__}: {exc}".encode(), -1) + return _SailProcess(process, self._sdk) + + async def _exec_with_output(self, *command: str, timeout: int) -> tuple[int, str, str]: + process = await self._run(*command, timeout=timeout) + stdout, stderr = await asyncio.gather(process.stdout.read(), process.stderr.read()) + return await process.wait(), stdout.decode(errors="replace"), stderr.decode(errors="replace") + + async def wait_for_vm(self) -> None: + """Poll ``docker info`` until the daemon answers, starting dockerd once if it is not running.""" + deadline = time.monotonic() + self._VM_READY_TIMEOUT + started = False + attempts = 0 + detail = "" + while True: + attempts += 1 + exit_code, _, stderr = await self._exec_with_output( + "docker", "info", "--format", "{{.ServerVersion}}", timeout=self._DOCKER_PROBE_TIMEOUT, + ) + if exit_code == 0: + logger.info("Docker ready in Sailbox %s after %s probe(s)", self.sandbox_id, attempts) + return + detail = stderr.strip() + if time.monotonic() >= deadline: + break + if not started: + await self.exec_script("pgrep -x dockerd > /dev/null || (nohup dockerd > /var/log/dockerd.log 2>&1 &)") + started = True + await asyncio.sleep(self._VM_READY_POLL_INTERVAL) + raise RuntimeError( + f"Docker not ready in Sailbox {self.sandbox_id} after {self._VM_READY_TIMEOUT}s ({attempts} probes): {detail[-500:]}" + ) + + async def setup_vm_for_gateway(self, exposed_ports: Optional[list[int]] = None) -> None: + """Wait for Docker and require Compose v2. Sail routes exposed ports itself, so no firewall rules.""" + await self.wait_for_vm() + exit_code, stdout, stderr = await self.exec_with_output("docker", "compose", "version") + if exit_code != 0: + raise RuntimeError(f"Sailbox {self.sandbox_id} has no Docker Compose v2: {(stderr or stdout).strip()}") + + async def _write_unsigned_object(self, object_store: Any, object_url: str, vm_path: str) -> None: + """Stream an object the store can't sign straight onto the VM host through Sail's filesystem API, + a chunk at a time, instead of base64 over exec.""" + with contextlib.closing(await asyncio.to_thread(object_store.open, object_url)) as source: + async with await self._sailbox.fs.write_stream.aio(vm_path) as writer: + while chunk := await asyncio.to_thread(source.read, _STREAM_CHUNK_BYTES): + await writer.write(chunk) + + async def _write_bytes_to_vm_path(self, data: bytes, vm_path: str) -> None: + if self._injection is not None: + data = self._injection.scrub_bytes(data) + if self._refuse_model_keys and carries_model_key(data.decode(errors="replace"), shell=False): + raise self._refusal() + await self._sailbox.fs.write.aio(vm_path, data) + + def _refusal(self) -> ModelKeyRefusedError: + return ModelKeyRefusedError( + f"Refusing to send a model key into Sailbox {self.sandbox_id}: Sail injects model keys only for an agent " + "deployed on its own Sail sandbox, so this one would land on the Sailbox's disk. Deploy the agent on its " + "own sandbox, or set inject_model_key = false in [sandbox.providers.sail_vm.config] to pass keys in." + ) + + async def apply_network_policy(self, policy: NetworkPolicy) -> None: + """Replace the Sailbox's egress policy; applies to new connections. A model-key injection needs a + saved policy (only those can name a secret), so a new one replaces the old, which is deleted.""" + async with self._policy_lock(): + await self._apply_policy(policy) + + async def _apply_policy(self, policy: NetworkPolicy) -> None: + """``apply_network_policy`` for a caller already holding ``_policy_lock``.""" + if self._injection is None: + await self._sailbox.set_egress_policy.aio(egress_document(policy)) + else: + saved = await create_saved_policy(self._sdk, policy_name(self._injection), egress_document(policy, self._injection)) + previous = self._injection.policy_id + try: + await self._sailbox.set_egress_policy.aio(saved) + except BaseException as exc: + known, applied = await self._applied_policy_id() + if not known: + self._injection.unsettled_policy_ids.append(saved.id) + raise + if applied != saved.id: + await delete_saved_policy(self._sdk, saved.id) + raise + logger.info("Sailbox %s took policy %s though setting it reported a failure", self.sandbox_id, saved.id) + await self._settle(saved.id, previous) + if not isinstance(exc, Exception): + raise + else: + await self._settle(saved.id, previous) + self.network_policy = policy + + async def _applied_policy_id(self) -> tuple[bool, str | None]: + """Whether Sail could say which saved policy is applied, and its id.""" + try: + return True, getattr((await self._sdk.Sailbox.get.aio(self.sandbox_id)).egress_policy, "policy_id", None) + except Exception as exc: # noqa: BLE001 - an unknown answer is kept unknown, never guessed + logger.info("Could not read Sailbox %s's applied policy: %s", self.sandbox_id, exc) + return False, None + + async def _settle(self, applied: str, *superseded: str | None) -> None: + """Track ``applied`` as this Sailbox's policy and delete the others it replaced, settled ones included.""" + stale = [*superseded, *self._injection.unsettled_policy_ids] + self._injection.policy_id, self._injection.unsettled_policy_ids = applied, [] + for policy_id in dict.fromkeys(stale): + if policy_id != applied: + await delete_saved_policy(self._sdk, policy_id) + + def adopt_applied_policy(self, applied: Any) -> NetworkPolicy | None: + """The agent-env policy Sail reports applied (``Sailbox.egress_policy``), or None when it can't be + represented: a saved policy counts only when it carries this Sailbox's model-key injection.""" + if applied is None: + return None + document = getattr(applied, "document", None) + if getattr(applied, "policy_id", None) is not None: + saved = ModelKeyInjection.from_document(document, applied.policy_id) + if self._injection is None or not self._injection.matches(saved): + return None + self._injection.policy_id = applied.policy_id + name = getattr(applied, "name", None) + if isinstance(name, str) and name.startswith(POLICY_PREFIX) and "-" in name[len(POLICY_PREFIX):]: + self._injection.policy_prefix = name.rsplit("-", 1)[0] + return policy_from_document(document, self._injection) + + def _known_policy(self, purpose: str) -> NetworkPolicy: + if self.network_policy is None: + raise RuntimeError( + f"Cannot {purpose} in Sailbox {self.sandbox_id}: its applied egress policy is unknown, " + "so the signed download hosts cannot be added safely" + ) + return self.network_policy + + async def _allow_download_hosts(self, urls: list[str | None], purpose: str) -> None: + """Add the hosts of signed download ``urls`` to the Sailbox's applied egress policy. Hosts the cached + policy already admits need nothing (it only ever lags the applied one); a new host re-reads the + applied policy under a per-Sailbox lock, so concurrent downloads don't drop each other's hosts. + Refuses before exceeding Sail's limit.""" + hosts = {parsed.hostname for url in urls if url and (parsed := urlparse(url)).hostname} + cached = self._known_policy(purpose) + if not cached.restricts_egress or all(_allows_host(cached, host) for host in hosts): + return + async with self._policy_lock(): + self.network_policy = self.adopt_applied_policy((await self._sdk.Sailbox.get.aio(self.sandbox_id)).egress_policy) + policy = self._known_policy(purpose) + if not policy.restricts_egress: + return + missing = sorted(host for host in hosts if not _allows_host(policy, host)) + if not missing: + return + if len(policy.allow_hosts) + len(policy.allow_cidrs) + len(missing) > MAX_ALLOWLIST_ENTRIES: + raise RuntimeError( + f"Cannot {purpose} in Sailbox {self.sandbox_id}: adding {missing} would exceed " + f"Sail's {MAX_ALLOWLIST_ENTRIES}-entry egress allowlist" + ) + await self._apply_policy(policy.with_hosts(missing)) + + async def load_docker_images(self, artifacts: list) -> None: + """Load images, first adding their signed-download hosts to a restrictive policy.""" + if not artifacts: + return + self._known_policy("load Docker images") + signed_urls = await self._signed_image_urls(artifacts) + await self._allow_download_hosts(signed_urls, "load Docker images") + await self._load_docker_images(artifacts, signed_urls) + + async def _download_object_to_vm(self, object_url: str, vm_path: str) -> None: + """Download through a signed URL whose host a restrictive policy now allows, or stream the bytes + through Sail's filesystem API when the store cannot sign one.""" + object_store = get_config().get_object_store() + signed = await asyncio.to_thread(object_store.signed_get_url, object_url) + if signed is None: + await self._write_unsigned_object(object_store, object_url, vm_path) + return + await self._allow_download_hosts([signed], "download an object") + await self.exec_script(f"curl -fsSL {CURL_RETRY_FLAGS} {shlex.quote(signed)} -o {shlex.quote(vm_path)}") diff --git a/src/agent_env/providers/sandbox_providers/sandbox_provider.py b/src/agent_env/providers/sandbox_providers/sandbox_provider.py index a6aa7747..68009580 100644 --- a/src/agent_env/providers/sandbox_providers/sandbox_provider.py +++ b/src/agent_env/providers/sandbox_providers/sandbox_provider.py @@ -123,7 +123,7 @@ async def create_container( await self._start_container(sandbox, image_name=image_name, port=port, env=env) sandbox.mode = SANDBOX_MODE_CONTAINER return sandbox - except Exception: + except BaseException: try: await sandbox.terminate() except Exception: @@ -222,6 +222,7 @@ def effective_network_policy(cls, policy: Optional[NetworkPolicy]) -> NetworkPol "modal": "agent_env.providers.sandbox_providers.modal_sandbox:ModalSandboxProvider", "modal_vm": "agent_env.providers.sandbox_providers.modal_vm_sandbox:ModalVmSandboxProvider", "e2b": "agent_env.providers.sandbox_providers.e2b:E2BSandboxProvider", + "sail_vm": "agent_env.providers.sandbox_providers.sail_vm.provider:SailVmSandboxProvider", "local": "agent_env.providers.sandbox_providers.local_sandbox:LocalSandboxProvider", } diff --git a/tst/integration/env/gateway/gateway_test.py b/tst/integration/env/gateway/gateway_test.py index 8d9cedbc..c4d609f0 100644 --- a/tst/integration/env/gateway/gateway_test.py +++ b/tst/integration/env/gateway/gateway_test.py @@ -446,11 +446,17 @@ def agentenv_website_env() -> WebsiteEnv: id="modal_vm", marks=skip_without_remote_sandbox("modal_vm"), ), + pytest.param( + "sail_vm", + id="sail_vm", + marks=skip_without_remote_sandbox("sail_vm"), + ), ]) def sandbox_provider(request): from agent_env.providers import ( ModalSandboxProvider, ModalVmSandboxProvider, + build_sandbox_provider, reset_env_sandbox_provider, reset_sandbox_provider, set_env_sandbox_provider, @@ -463,6 +469,9 @@ def sandbox_provider(request): elif request.param == "modal_vm": set_sandbox_provider(ModalVmSandboxProvider()) set_env_sandbox_provider(ModalVmSandboxProvider()) + elif request.param == "sail_vm": + set_sandbox_provider(build_sandbox_provider("sail_vm")) + set_env_sandbox_provider(build_sandbox_provider("sail_vm")) try: yield request.param finally: diff --git a/tst/integration/env/task_step/task_steps_test.py b/tst/integration/env/task_step/task_steps_test.py index 340a9208..c351f045 100644 --- a/tst/integration/env/task_step/task_steps_test.py +++ b/tst/integration/env/task_step/task_steps_test.py @@ -274,11 +274,17 @@ def echo_agent() -> A2AAgent: id="modal_vm", marks=skip_without_remote_sandbox("modal_vm"), ), + pytest.param( + "sail_vm", + id="sail_vm", + marks=skip_without_remote_sandbox("sail_vm"), + ), ]) def sandbox_provider(request): from agent_env.providers import ( ModalSandboxProvider, ModalVmSandboxProvider, + build_sandbox_provider, reset_agent_sandbox_provider, reset_env_sandbox_provider, reset_sandbox_provider, @@ -295,6 +301,10 @@ def sandbox_provider(request): set_sandbox_provider(ModalVmSandboxProvider()) set_env_sandbox_provider(ModalVmSandboxProvider()) set_agent_sandbox_provider(ModalVmSandboxProvider()) + elif request.param == "sail_vm": + set_sandbox_provider(build_sandbox_provider("sail_vm")) + set_env_sandbox_provider(build_sandbox_provider("sail_vm")) + set_agent_sandbox_provider(build_sandbox_provider("sail_vm")) try: yield request.param finally: diff --git a/tst/integration/providers/sandbox_providers/sail_vm_sandbox_smoke_test.py b/tst/integration/providers/sandbox_providers/sail_vm_sandbox_smoke_test.py new file mode 100644 index 00000000..c0906a96 --- /dev/null +++ b/tst/integration/providers/sandbox_providers/sail_vm_sandbox_smoke_test.py @@ -0,0 +1,121 @@ +"""Sail Sailbox smoke test. + +Runs when the resolved config can build the ``sail_vm`` sandbox provider (``[sandbox.providers.sail_vm.config]`` +with ``api_key = "secret:sail_api_key"``), the ``remote_sandbox`` capability, and skips with the declared +reason otherwise. The test never reads or prints the API key. +""" + +from __future__ import annotations + +import hashlib +import json +import secrets + +import httpx +import pytest +import pytest_asyncio + +from agent_env.providers.sandbox_providers.sail_vm.model_key import PLACEHOLDER, secret_name +from agent_env.providers.sandbox_providers.sandbox import NetworkMode, NetworkPolicy +from agent_env.providers.sandbox_providers.sandbox_provider import build_sandbox_provider +from tst.util.capabilities import skip_without_remote_sandbox + +_PORT = 8080 + +pytestmark = [ + pytest.mark.integration, + pytest.mark.int_test_slow, + pytest.mark.asyncio, + skip_without_remote_sandbox("sail_vm"), +] + + +@pytest.fixture(scope="module") +def sail_provider(): + return build_sandbox_provider("sail_vm") + + +@pytest_asyncio.fixture(scope="module") +async def sail_sandbox(sail_provider): + sandbox = await sail_provider.create_vm(cpu=1.0, memory=2048, exposed_ports=[_PORT], timeout=900) + try: + yield sandbox + finally: + await sandbox.terminate() + + +async def test_sailbox_runs_docker_serves_its_port_and_reconnects(sail_provider, sail_sandbox): + assert (sail_sandbox.type, sail_sandbox.mode) == ("sail_vm", "vm") + + await sail_sandbox.exec_script( + f"docker run -d --name web -p {_PORT}:80 public.ecr.aws/nginx/nginx:alpine > /dev/null" + ) + url = sail_sandbox.tunnel_urls[_PORT] + assert url.startswith("https://") and url.endswith(".sail.box") + async with httpx.AsyncClient(timeout=30) as client: + for _ in range(30): + response = await client.get(url) + if response.status_code == 200: + break + assert response.status_code == 200 + assert "nginx" in response.text + + data = bytes(range(256)) * 4096 + await sail_sandbox.write_host_file(data, "/tmp/agent-env-smoke/blob.bin") + exit_code, digest, stderr = await sail_sandbox.exec_with_output("sha256sum", "/tmp/agent-env-smoke/blob.bin") + assert exit_code == 0, stderr + assert digest.split()[0] == hashlib.sha256(data).hexdigest() + + reconnected = await sail_provider.get_sandbox(sail_sandbox.sandbox_id) + assert reconnected.sandbox_id == sail_sandbox.sandbox_id + assert reconnected.tunnel_urls == sail_sandbox.tunnel_urls + assert reconnected.network_policy == NetworkPolicy() + + +async def test_an_allowlist_is_enforced_for_containers_in_the_sailbox(sail_provider): + policy = NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",)) + sandbox = await sail_provider.create_vm(cpu=1.0, memory=2048, exposed_ports=[], timeout=600, network_policy=policy) + try: + assert "*.sail.box" in sandbox.network_policy.allow_hosts + exit_code, out, stderr = await sandbox.exec_with_output( + "bash", "-c", + "curl -s -m 10 -o /dev/null -w '%{http_code}' https://pypi.org/simple/; echo; " + "curl -s -m 10 -o /dev/null https://example.com && echo example-reachable || echo example-blocked", + ) + assert exit_code == 0, stderr + assert out.split() == ["200", "example-blocked"] + reconnected = await sail_provider.get_sandbox(sandbox.sandbox_id) + assert reconnected.network_policy == sandbox.network_policy + finally: + await sandbox.terminate() + + +_ECHO_HOST = "httpbin.org" +_CLIENT = ( + "import json, os, urllib.request; " + f"r = urllib.request.Request('https://{_ECHO_HOST}/headers', headers={{'Authorization': 'Bearer ' + os.environ['LITELLM_API_KEY']}}); " + "print(json.loads(urllib.request.urlopen(r, timeout=20).read())['headers']['Authorization'])" +) + + +async def test_an_agents_model_key_is_injected_by_sail_and_never_enters_the_sailbox(sail_provider): + """A throwaway key against a header-echo host stands in for the model endpoint. The provider leaves the + key's secret in Sail; this test deletes its own.""" + key = f"sk-agentenv-smoke-{secrets.token_hex(16)}" + env = {"LITELLM_API_KEY": key, "LITELLM_BASE_URL": f"https://{_ECHO_HOST}/v1"} + sandbox = await sail_provider.create_sandbox(image_name="unused", port=_PORT, env=env, cpu=1.0, memory=2048, timeout=900) + try: + flags = " ".join(f"-e {name}='{value}'" for name, value in env.items()) + await sandbox.exec_script(f"docker run -d --name agent-api {flags} public.ecr.aws/docker/library/python:3.12-slim sleep 600 > /dev/null") + exit_code, sent, stderr = await sandbox.exec_with_output("docker", "exec", "agent-api", "python", "-c", _CLIENT) + assert exit_code == 0, stderr + assert sent.strip() == f"Bearer {key}" + + _, inside, _ = await sandbox.exec_with_output("docker", "exec", "agent-api", "printenv", "LITELLM_API_KEY") + assert inside.strip() == PLACEHOLDER + _, config, _ = await sandbox.exec_with_output("docker", "inspect", "agent-api") + assert key not in config and PLACEHOLDER in json.dumps(json.loads(config)[0]["Config"]["Env"]) + finally: + await sandbox.terminate() + secret = await sandbox._sdk.Secret.get.aio(secret_name(key)) + await secret.delete.aio() diff --git a/tst/unit/config/test_explain_and_sources.py b/tst/unit/config/test_explain_and_sources.py index d2c33924..2e894796 100644 --- a/tst/unit/config/test_explain_and_sources.py +++ b/tst/unit/config/test_explain_and_sources.py @@ -160,8 +160,9 @@ def test_sources_says_how_the_file_was_found(config_file, tmp_path, monkeypatch) assert "walking up" in next(s for s in sources() if s.kind == KIND_FILE).detail -# Read for identity, never for configuration, so it is not a layer. -_NOT_CONFIGURATION = {"USER"} +# Read for identity (USER), or set from [sandbox.providers.sail_vm.config] for the Sail SDK to read +# (SAIL_*), never configuration agent-env takes, so none is a layer. +_NOT_CONFIGURATION = {"USER", "SAIL_API_KEY", "SAIL_RUNTIME_THREADS"} def _env_vars_read_by(root): diff --git a/tst/unit/providers/env_providers/env_gateway_provider_test.py b/tst/unit/providers/env_providers/env_gateway_provider_test.py index 3cf1b74d..233cbc7e 100644 --- a/tst/unit/providers/env_providers/env_gateway_provider_test.py +++ b/tst/unit/providers/env_providers/env_gateway_provider_test.py @@ -548,6 +548,28 @@ async def test_modal_vm_provider_routes_to_vm_path_not_containers(): gp._deploy_via_containers.assert_not_awaited() +@pytest.mark.asyncio +async def test_sail_vm_provider_routes_to_vm_path_not_containers(): + """A Sailbox is a Docker-capable VM: the gateway deploys onto it with docker-compose.""" + from agent_env.env.gateway import GatewayMode + from agent_env.providers.sandbox_providers.sail_vm.provider import SailVmSandboxProvider + + gp = EnvironmentGatewayProvider() + gp._deploy_via_vm = AsyncMock(return_value="VM_RESULT") + gp._deploy_via_containers = AsyncMock(return_value="CONTAINER_RESULT") + + result = await gp.create_gateway( + sandbox_provider=SailVmSandboxProvider(api_key="sail-test-key"), + mcp_servers=[MCPServerConfig(image="mcp-a", environment_name="a")], + mcp_server_images=[MagicMock(image_name="mcp-a")], + gateway_mode=GatewayMode.PERFORMANCE, + ttl_seconds=60, + disk_size_gb=10, + ) + assert result == "VM_RESULT" + gp._deploy_via_containers.assert_not_awaited() + + @pytest.mark.asyncio async def test_build_local_store_no_services(): """No services at all (and website_configs left unset, as the container path calls it) must diff --git a/tst/unit/providers/sandbox_providers/egress_hosts_test.py b/tst/unit/providers/sandbox_providers/egress_hosts_test.py index 6221ce85..7024b69d 100644 --- a/tst/unit/providers/sandbox_providers/egress_hosts_test.py +++ b/tst/unit/providers/sandbox_providers/egress_hosts_test.py @@ -95,4 +95,4 @@ def test_the_builtin_floor_is_pinned(builtin_registry): """Every restricted sandbox gets these, so an empty allowlist is not total denial. Pinned rather than counted: a new entry should be argued for in review. Platforms registered from config add theirs on top.""" - assert set(all_sandbox_egress_hosts()) == {"*.modal.host", "*.w.modal.host", "*.e2b.app"} + assert set(all_sandbox_egress_hosts()) == {"*.modal.host", "*.w.modal.host", "*.e2b.app", "*.sail.box"} diff --git a/tst/unit/providers/sandbox_providers/network_policy_stamping_test.py b/tst/unit/providers/sandbox_providers/network_policy_stamping_test.py index f18454ec..0b801a53 100644 --- a/tst/unit/providers/sandbox_providers/network_policy_stamping_test.py +++ b/tst/unit/providers/sandbox_providers/network_policy_stamping_test.py @@ -8,6 +8,7 @@ test_every_builtin_backend_is_covered until someone drives it here. """ +from types import SimpleNamespace from unittest.mock import AsyncMock, MagicMock, patch import pytest @@ -16,6 +17,8 @@ from agent_env.providers.sandbox_providers.local_sandbox import LocalSandboxProvider from agent_env.providers.sandbox_providers.modal_sandbox import ModalSandboxProvider from agent_env.providers.sandbox_providers.modal_vm_sandbox import ModalVmSandboxProvider +from agent_env.providers.sandbox_providers.sail_vm import _sdk as sail_sdk +from agent_env.providers.sandbox_providers.sail_vm.provider import SailVmSandboxProvider from agent_env.providers.sandbox_providers.sandbox import Sandbox from agent_env.providers.sandbox_providers.sandbox_provider import _BUILTIN_SANDBOX_PROVIDERS @@ -73,10 +76,21 @@ async def _e2b() -> Sandbox: return await provider.create_vm(exposed_ports=[], setup_for_gateway=False) +async def _sail() -> Sandbox: + sailbox = MagicMock(sailbox_id="sb_test", status="running") + sailbox.listeners.aio = AsyncMock(return_value=[]) + sdk = MagicMock() + sdk.Sailbox.create.aio = AsyncMock(return_value=sailbox) + with patch.object(sail_sdk, "connect", return_value=(sdk, SimpleNamespace(id="app_test"))): + provider = SailVmSandboxProvider(api_key="test-key", sdk=sdk) + return await provider.create_vm(exposed_ports=[], setup_for_gateway=False) + + FACTORIES = { "modal": _modal, "modal_vm": _modal_vm, "e2b": _e2b, + "sail_vm": _sail, "local": _local, } diff --git a/tst/unit/providers/sandbox_providers/sail_vm_model_key_test.py b/tst/unit/providers/sandbox_providers/sail_vm_model_key_test.py new file mode 100644 index 00000000..db731bae --- /dev/null +++ b/tst/unit/providers/sandbox_providers/sail_vm_model_key_test.py @@ -0,0 +1,889 @@ +"""Model-key injection on Sailboxes: the key stays at Sail, the Sailbox only ever holds a placeholder.""" + +import asyncio +import logging +import shlex +import subprocess +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from agent_env.providers.sandbox_providers.sail_vm import _sdk +from agent_env.providers.sandbox_providers.sail_vm import model_key +from agent_env.providers.sandbox_providers.sail_vm.model_key import ( + DOCKER_SHIM_PATH, + PLACEHOLDER, + ModelKeyInjection, + carries_model_key, + docker_shim, + secret_name, +) +from agent_env.providers.sandbox_providers.sail_vm.provider import SailVmSandboxProvider +from agent_env.providers.sandbox_providers.sail_vm.sandbox import ( + ModelKeyRefusedError, + SailVmSandbox, + create_saved_policy, + delete_policy_named, +) +from agent_env.providers.sandbox_providers.sandbox import NetworkMode, NetworkPolicy + +_KEY = "sk-live-model-key-0123456789" +_ENV = {"LITELLM_API_KEY": _KEY, "LITELLM_BASE_URL": "https://llm.example.com/v1", "A2A_PORT": "8000"} + + +class _NotFound(Exception): + pass + + +class _HostLost(Exception): + pass + + +def _fake_sdk(sailbox): + saved = [] + + async def create_policy(name, document): + policy = MagicMock(id=f"ep_{len(saved) + 1}", document=document) + policy.name = name + policy.delete.aio = AsyncMock() + saved.append(policy) + return policy + + secret = MagicMock() + secret.delete.aio = AsyncMock() + sdk = SimpleNamespace( + App=SimpleNamespace(find=MagicMock(return_value=SimpleNamespace(id="app_1"))), + Sailbox=SimpleNamespace( + create=SimpleNamespace(aio=AsyncMock(return_value=sailbox)), + get=SimpleNamespace(aio=AsyncMock(return_value=sailbox)), + list=SimpleNamespace(aio=AsyncMock(return_value=[])), + ), + Image=SimpleNamespace(devbox=MagicMock(return_value="devbox-amd64")), + AutoSleep=SimpleNamespace(never=lambda: "never", default=lambda: "default", not_before=lambda s: s), + Secret=SimpleNamespace(set=SimpleNamespace(aio=AsyncMock()), get=SimpleNamespace(aio=AsyncMock(return_value=secret))), + EgressPolicy=SimpleNamespace( + create=SimpleNamespace(aio=AsyncMock(side_effect=create_policy)), + get=SimpleNamespace(aio=AsyncMock(side_effect=lambda policy_id: next(p for p in saved if p.id == policy_id))), + list=SimpleNamespace(aio=AsyncMock(side_effect=lambda search: [p for p in saved if search in p.name])), + ), + reset_transports=MagicMock(), + NotFoundError=_NotFound, + SailboxHostLostError=_HostLost, + TransportError=_HostLost, + ) + return sdk, saved, secret + + +def _sailbox(ports=(8000,)): + sailbox = MagicMock(sailbox_id="sb_1", status="running") + sailbox.listeners.aio = AsyncMock(return_value=[ + SimpleNamespace(guest_port=p, endpoint=SimpleNamespace(url=f"https://sb-1-{p}.sail.box")) for p in ports + ]) + sailbox.terminate.aio = AsyncMock() + sailbox.fs.write.aio = AsyncMock() + sailbox.set_egress_policy.aio = AsyncMock() + return sailbox + + +@pytest.fixture(autouse=True) +def fresh_key_state(monkeypatch): + monkeypatch.setattr(_sdk, "_installed_key", None) + monkeypatch.setattr(_sdk, "_apps", {}) + monkeypatch.setattr(model_key, "_injected_keys", {}) + monkeypatch.setattr(model_key, "_holders", {}) + + +@pytest.fixture(autouse=True) +def no_vm_setup(monkeypatch): + monkeypatch.setattr(SailVmSandbox, "setup_vm_for_gateway", AsyncMock()) + + +def _process(stdout=b""): + async def chunks(data): + yield data + + process = MagicMock() + process.stdout_bytes = chunks(stdout) + process.stderr_bytes = chunks(b"") + process.wait = AsyncMock(return_value=SimpleNamespace(exit_code=0)) + return process + + +def test_secrets_are_named_from_the_key_so_keys_never_share_one(): + assert secret_name(_KEY) == secret_name(_KEY) + assert secret_name(_KEY) != secret_name(_KEY + "x") + assert secret_name(_KEY).startswith("AGENTENV_LITELLM_") and _KEY not in secret_name(_KEY) + + +def test_an_env_without_a_model_key_injects_nothing(): + assert ModelKeyInjection.for_env({"LITELLM_BASE_URL": "https://llm.example.com"}) is None + assert ModelKeyInjection.for_env({"LITELLM_API_KEY": PLACEHOLDER}) is None + + +@pytest.mark.parametrize("base_url", ["http://llm.internal:4000", "", "llm.example.com"]) +def test_injection_needs_an_https_model_endpoint(base_url): + with pytest.raises(ValueError, match="only into HTTPS requests"): + ModelKeyInjection.for_env({"LITELLM_API_KEY": _KEY, "LITELLM_BASE_URL": base_url}) + + +def test_rules_set_both_auth_headers_from_the_secret_and_never_hold_the_key(): + injection = ModelKeyInjection.for_env(_ENV) + ref = f"${{secrets.{secret_name(_KEY)}}}" + assert injection.rules() == { + "llm.example.com": [{"request": {"set": {"headers": {"authorization": f"Bearer {ref}", "x-api-key": ref}}}}] + } + assert _KEY not in repr(injection) and _KEY not in str(injection.rules()) + + +def test_an_injection_is_recovered_from_its_saved_policy_only(): + injection = ModelKeyInjection.for_env(_ENV) + recovered = ModelKeyInjection.from_document({"allowlist": ["x.example"], "rules": injection.rules()}, "ep_1") + assert injection.matches(recovered) and recovered.policy_id == "ep_1" and recovered.key is None + foreign = {"rules": {"api.github.com": [{"request": {"set": {"headers": {"authorization": "Bearer ${secrets.GITHUB_TOKEN}"}}}}]}} + assert ModelKeyInjection.from_document(foreign, "ep_2") is None + assert ModelKeyInjection.from_document({"allowlist": []}, "ep_3") is None + + +def test_a_key_injected_by_this_process_is_recovered_even_when_it_is_not_the_configured_one(): + injected = ModelKeyInjection.for_env(_ENV) + injected.hold("sb_1") + recovered = ModelKeyInjection.from_document({"rules": injected.rules()}, "ep_1") + recovered.recover_key("the-configured-model-key") + assert recovered.key == _KEY + + +def test_in_another_process_only_the_configured_key_is_recovered(monkeypatch): + rules = ModelKeyInjection.for_env(_ENV).rules() + monkeypatch.setattr(model_key, "_injected_keys", {}) + recovered = ModelKeyInjection.from_document({"rules": rules}, "ep_1") + recovered.recover_key("some-other-key") + assert recovered.key is None + recovered.recover_key(_KEY) + assert recovered.scrub(f"-e LITELLM_API_KEY='{_KEY}'") == f"-e LITELLM_API_KEY='{PLACEHOLDER}'" + + +@pytest.mark.parametrize( + ("args", "expected_prefix"), + [ + (["run", "-d", "--name", "agent-api", "img"], ["run", "-v"]), + (["create", "--name", "agent-api", "img"], ["create", "-v"]), + (["container", "run", "img"], ["container", "run", "-v"]), + (["exec", "agent-api", "ls"], ["exec", "agent-api", "ls"]), + (["load"], ["load"]), + ], +) +def test_the_docker_shim_adds_the_ca_bundle_only_to_containers_it_starts(tmp_path, args, expected_prefix): + shim = tmp_path / "docker" + shim.write_text(docker_shim().replace("/usr/bin/docker", "printf '%s\\n'")) + shim.chmod(0o755) + argv = subprocess.run([str(shim), *args], capture_output=True, text=True, check=True).stdout.split("\n")[:-1] + assert argv[: len(expected_prefix)] == expected_prefix + trusts = f"{model_key.VM_CA_BUNDLE}:{model_key.CONTAINER_CA_BUNDLE}:ro" in argv + assert trusts is (expected_prefix[-1] == "-v") + if trusts: + assert "NODE_EXTRA_CA_CERTS=/etc/ssl/certs/sailbox-ca-bundle.crt" in argv + assert argv[-len(args) + (2 if args[0] == "container" else 1):] == args[(2 if args[0] == "container" else 1):] + + +@pytest.mark.asyncio +async def test_an_agent_sailbox_gets_its_key_through_a_secret_and_a_saved_policy(caplog): + caplog.set_level(logging.DEBUG) + sailbox = _sailbox() + sdk, saved, _ = _fake_sdk(sailbox) + + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + + sdk.Secret.set.aio.assert_awaited_once_with(secret_name(_KEY), _KEY) + (policy,) = saved + assert policy.document == {"rules": ModelKeyInjection.for_env(_ENV).rules()} + kwargs = sdk.Sailbox.create.aio.await_args.kwargs + assert kwargs["egress_policy"] is policy + assert _KEY not in repr(kwargs) + sailbox.fs.write.aio.assert_awaited_once_with(DOCKER_SHIM_PATH, docker_shim(), mode=0o755) + assert sandbox._injection.policy_id == policy.id + assert _KEY not in caplog.text + + +@pytest.mark.asyncio +async def test_a_restricted_agent_sailbox_allows_the_model_endpoint_it_injects_into(): + sdk, saved, _ = _fake_sdk(_sailbox()) + restricted = NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",)) + + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + image_name="agent:1", port=8000, env=_ENV, network_policy=restricted, + ) + + assert "llm.example.com" in saved[0].document["allowlist"] + assert "llm.example.com" in sandbox.network_policy.allow_hosts + + +@pytest.mark.asyncio +async def test_injection_can_be_turned_off_to_pass_the_key_in(): + sdk, saved, _ = _fake_sdk(_sailbox()) + sandbox = await SailVmSandboxProvider(api_key="sail-key", inject_model_key=False, sdk=sdk).create_sandbox( + image_name="agent:1", port=8000, env={**_ENV, "LITELLM_BASE_URL": "http://llm.internal:4000"}, + ) + sdk.Secret.set.aio.assert_not_awaited() + assert saved == [] and sandbox._injection is None + assert sdk.Sailbox.create.aio.await_args.kwargs["egress_policy"] == {} + + +@pytest.mark.asyncio +async def test_a_plain_endpoint_is_refused_before_anything_is_created(): + sdk, saved, _ = _fake_sdk(_sailbox()) + with pytest.raises(ValueError, match="only into HTTPS requests"): + await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + image_name="agent:1", port=8000, env={**_ENV, "LITELLM_BASE_URL": "http://llm.internal:4000"}, + ) + sdk.Secret.set.aio.assert_not_awaited() + sdk.Sailbox.create.aio.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_a_failed_create_deletes_its_policy_but_never_the_secret(): + sdk, saved, secret = _fake_sdk(_sailbox()) + sdk.Sailbox.create.aio = AsyncMock(side_effect=RuntimeError("no capacity")) + with pytest.raises(RuntimeError, match="no capacity"): + await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + saved[0].delete.aio.assert_awaited_once() + secret.delete.aio.assert_not_awaited() + assert secret_name(_KEY) not in model_key._injected_keys + + +@pytest.mark.asyncio +async def test_a_create_cancelled_while_saving_the_policy_forgets_the_key(): + sdk, _, secret = _fake_sdk(_sailbox()) + started = asyncio.Event() + + async def hang(name, document): + started.set() + await asyncio.Event().wait() + + sdk.EgressPolicy.create.aio = AsyncMock(side_effect=hang) + task = asyncio.ensure_future( + SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + ) + await started.wait() + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + sdk.Sailbox.create.aio.assert_not_awaited() + secret.delete.aio.assert_not_awaited() + assert secret_name(_KEY) not in model_key._injected_keys + + +@pytest.mark.asyncio +async def test_a_create_abandoned_in_flight_that_then_fails_deletes_its_policy(): + sdk, saved, secret = _fake_sdk(_sailbox()) + started, fail = asyncio.Event(), asyncio.Event() + + async def create(**_kwargs): + started.set() + await fail.wait() + raise RuntimeError("no capacity") + + sdk.Sailbox.create.aio = create + task = asyncio.ensure_future( + SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + ) + await started.wait() + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + saved[0].delete.aio.assert_not_awaited() + fail.set() + for _ in range(10): + await asyncio.sleep(0) + saved[0].delete.aio.assert_awaited_once() + secret.delete.aio.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_gateway_and_plain_vms_never_inject(): + sdk, saved, _ = _fake_sdk(_sailbox(ports=())) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_vm(exposed_ports=[]) + assert saved == [] and sandbox._injection is None + + +@pytest.mark.asyncio +async def test_commands_and_files_carry_the_placeholder_never_the_key(): + sailbox = _sailbox() + sailbox.exec.aio = AsyncMock(return_value=_process()) + sdk, _, _ = _fake_sdk(sailbox) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + + await sandbox.exec_script(f"docker run -d -e LITELLM_API_KEY='{_KEY}' -e ANTHROPIC_API_KEY={_KEY} agent:1") + await sandbox.write_host_file(f"key: {_KEY}\n".encode(), "/opt/agent/config.yaml") + + sent = [arg for call in sailbox.exec.aio.await_args_list for arg in call.args[0]] + (script,) = [arg for arg in sent if "docker run" in arg] + assert script.count(PLACEHOLDER) == 2 + assert not any(_KEY in arg for arg in sent) + assert sailbox.fs.write.aio.await_args.args == ("/opt/agent/config.yaml", f"key: {PLACEHOLDER}\n".encode()) + + +@pytest.mark.asyncio +async def test_terminate_deletes_the_policy_and_keeps_the_secret(): + sailbox = _sailbox() + sdk, saved, secret = _fake_sdk(sailbox) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + + await sandbox.terminate() + + sailbox.terminate.aio.assert_awaited_once() + saved[0].delete.aio.assert_awaited() + secret.delete.aio.assert_not_awaited() + assert secret_name(_KEY) not in model_key._injected_keys + + +@pytest.mark.asyncio +async def test_widening_an_injected_sailbox_replaces_its_saved_policy(): + sailbox = _sailbox() + sdk, saved, _ = _fake_sdk(sailbox) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + image_name="agent:1", port=8000, env=_ENV, + network_policy=NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",)), + ) + + await sandbox.apply_network_policy(sandbox.network_policy.with_hosts(["bucket.example"])) + + first, second = saved + assert "bucket.example" in second.document["allowlist"] and second.document["rules"] == first.document["rules"] + sailbox.set_egress_policy.aio.assert_awaited_once_with(second) + first.delete.aio.assert_awaited_once() + assert sandbox._injection.policy_id == second.id + + +@pytest.mark.asyncio +async def test_a_failed_policy_swap_deletes_the_replacement_and_keeps_the_old_one(): + sailbox = _sailbox() + sdk, saved, _ = _fake_sdk(sailbox) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + image_name="agent:1", port=8000, env=_ENV, + network_policy=NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",)), + ) + sailbox.set_egress_policy.aio = AsyncMock(side_effect=RuntimeError("api down")) + + with pytest.raises(RuntimeError, match="api down"): + await sandbox.apply_network_policy(sandbox.network_policy.with_hosts(["bucket.example"])) + + first, second = saved + second.delete.aio.assert_awaited_once() + first.delete.aio.assert_not_awaited() + assert sandbox._injection.policy_id == first.id + + +@pytest.mark.asyncio +async def test_reconnect_in_another_process_scrubs_the_configured_key(monkeypatch): + injection = ModelKeyInjection.for_env(_ENV) + sailbox = _sailbox() + sailbox.egress_policy = SimpleNamespace(policy_id="ep_9", document={"allowlist": ["llm.example.com"], "rules": injection.rules()}) + sailbox.exec.aio = AsyncMock(return_value=_process()) + sdk, _, _ = _fake_sdk(sailbox) + monkeypatch.setattr(model_key, "_injected_keys", {}) + monkeypatch.setattr( + "agent_env.providers.sandbox_providers.sail_vm.provider.get_config", lambda: MagicMock(get_litellm_api_key=lambda: _KEY) + ) + + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).get_sandbox("sb_1") + + assert sandbox.network_policy == NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("llm.example.com",)) + assert sandbox._injection.policy_id == "ep_9" + await sandbox.exec_with_output("docker", "exec", "-e", f"LITELLM_API_KEY={_KEY}", "agent-api", "pytest") + assert sailbox.exec.aio.await_args.args[0][3] == f"LITELLM_API_KEY={PLACEHOLDER}" + + +@pytest.mark.asyncio +async def test_reconnect_treats_someone_elses_saved_policy_as_unknown(): + sailbox = _sailbox() + sailbox.egress_policy = SimpleNamespace(policy_id="ep_7", document={"allowlist": ["a.example"]}) + sdk, _, _ = _fake_sdk(sailbox) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).get_sandbox("sb_1") + assert sandbox.network_policy is None and sandbox._injection is None + + +@pytest.mark.asyncio +async def test_reconnect_in_the_deploying_process_scrubs_the_agents_own_key(monkeypatch): + agent_key = "sk-agent-override-key-42" + sailbox = _sailbox() + sdk, _, _ = _fake_sdk(sailbox) + provider = SailVmSandboxProvider(api_key="sail-key", sdk=sdk) + created = await provider.create_sandbox(image_name="agent:1", port=8000, env={**_ENV, "LITELLM_API_KEY": agent_key}) + (policy,) = [c.args[1] for c in sdk.EgressPolicy.create.aio.await_args_list] + sailbox.egress_policy = SimpleNamespace(policy_id=created._injection.policy_id, document=policy) + sailbox.exec.aio = AsyncMock(return_value=_process()) + monkeypatch.setattr( + "agent_env.providers.sandbox_providers.sail_vm.provider.get_config", lambda: MagicMock(get_litellm_api_key=lambda: _KEY) + ) + + reconnected = await provider.get_sandbox("sb_1") + await reconnected.exec_with_output("docker", "exec", "-e", f"LITELLM_API_KEY={agent_key}", "agent-api", "pytest") + + assert sailbox.exec.aio.await_args.args[0][3] == f"LITELLM_API_KEY={PLACEHOLDER}" + + +@pytest.mark.parametrize( + ("text", "shell", "carries"), + [ + (f"docker run -e LITELLM_API_KEY='{_KEY}' img", True, True), + (f"docker exec -e ANTHROPIC_API_KEY={_KEY} c pytest", True, True), + (f"- LITELLM_API_KEY={_KEY}", False, True), + ("docker run -e LITELLM_API_KEY='$test-secret' img", True, True), + ("docker run -e LITELLM_API_KEY=''\\''sk-secret' img", True, True), + ("docker run -e LITELLM_API_KEY=$test-secret img", True, True), + ('docker run -e LITELLM_API_KEY="$LITELLM_API_KEY-x" img', True, True), + ("LITELLM_API_KEY=$test-secret", False, True), + ("LITELLM_API_KEY=$LITELLM_API_KEY", False, True), + (f"-e LITELLM_API_KEY='{PLACEHOLDER}'", True, False), + (f"-e LITELLM_API_KEY={PLACEHOLDER}", False, False), + ('-e LITELLM_API_KEY="$LITELLM_API_KEY"', True, False), + ("-e LITELLM_API_KEY=${LITELLM_API_KEY}", True, False), + ("-e LITELLM_API_KEY=$LITELLM_API_KEY", True, False), + ("echo LITELLM_API_KEY=", True, False), + ("docker run -e OTHER=1 img", True, False), + ], +) +def test_a_model_key_assignment_is_detected_as_the_shell_reads_it(text, shell, carries): + assert carries_model_key(text, shell=shell) is carries + + +def test_the_agents_own_escaping_of_a_key_is_seen_through(): + key = "'sk-starts-with-a-quote" + escaped = key.replace("'", "'\\''") + assert carries_model_key(f"docker run -e LITELLM_API_KEY='{escaped}' img", shell=True) + + +@pytest.mark.asyncio +async def test_a_dollar_value_in_a_plain_argument_is_a_literal_key_and_refused(): + sailbox = _sailbox(ports=()) + sailbox.exec.aio = AsyncMock(return_value=_process()) + sdk, _, _ = _fake_sdk(sailbox) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_vm(exposed_ports=[]) + with pytest.raises(ModelKeyRefusedError): + await sandbox.exec_with_output("docker", "exec", "-e", "LITELLM_API_KEY=$looks-like-a-var", "agent-api", "true") + await sandbox.exec_with_output("bash", "-c", 'docker exec -e LITELLM_API_KEY="$LITELLM_API_KEY" agent-api true') + + +@pytest.mark.asyncio +async def test_a_sailbox_without_injection_refuses_a_model_key_rather_than_take_it(): + sailbox = _sailbox(ports=()) + sailbox.exec.aio = AsyncMock(return_value=_process()) + sdk, _, _ = _fake_sdk(sailbox) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_vm(exposed_ports=[]) + sailbox.exec.aio.reset_mock() + + with pytest.raises(ModelKeyRefusedError, match="Deploy the agent on its own sandbox"): + await sandbox.exec_script(f"docker run -d --name agent-api -e LITELLM_API_KEY='{_KEY}' agent:1") + with pytest.raises(ModelKeyRefusedError): + await sandbox.write_host_file(f"LITELLM_API_KEY={_KEY}\n".encode(), "/opt/agent/.env") + assert not any(_KEY in arg for call in sailbox.exec.aio.await_args_list for arg in call.args[0]) + sailbox.fs.write.aio.assert_not_awaited() + + await sandbox.exec_script('docker run -e LITELLM_API_KEY="$LITELLM_API_KEY" agent:1') + + +@pytest.mark.asyncio +async def test_with_injection_off_keys_pass_through_as_before(): + sailbox = _sailbox(ports=()) + sailbox.exec.aio = AsyncMock(return_value=_process()) + sdk, _, _ = _fake_sdk(sailbox) + sandbox = await SailVmSandboxProvider(api_key="sail-key", inject_model_key=False, sdk=sdk).create_vm(exposed_ports=[]) + await sandbox.exec_script(f"docker run -e LITELLM_API_KEY='{_KEY}' agent:1") + assert _KEY in sailbox.exec.aio.await_args.args[0][2] + + +@pytest.mark.asyncio +async def test_a_reconnected_handle_that_cannot_recover_the_key_refuses_it(monkeypatch): + sailbox = _sailbox() + sailbox.egress_policy = SimpleNamespace(policy_id="ep_9", document={"rules": ModelKeyInjection.for_env(_ENV).rules()}) + sailbox.exec.aio = AsyncMock(return_value=_process()) + sdk, _, _ = _fake_sdk(sailbox) + monkeypatch.setattr(model_key, "_injected_keys", {}) + monkeypatch.setattr( + "agent_env.providers.sandbox_providers.sail_vm.provider.get_config", lambda: MagicMock(get_litellm_api_key=lambda: "other") + ) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).get_sandbox("sb_1") + with pytest.raises(ModelKeyRefusedError): + await sandbox.exec_with_output("docker", "exec", "-e", f"LITELLM_API_KEY={_KEY}", "agent-api", "pytest") + sailbox.exec.aio.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_terminate_deletes_the_policy_actually_applied_even_if_another_handle_replaced_it(): + sailbox = _sailbox() + sdk, saved, _ = _fake_sdk(sailbox) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + replacement = await sdk.EgressPolicy.create.aio("agentenv-replacement", saved[0].document) + sailbox.egress_policy = SimpleNamespace(policy_id=replacement.id, document=replacement.document) + + await sandbox.terminate() + + replacement.delete.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_a_policy_create_whose_response_is_lost_is_found_by_name_and_deleted(): + sdk, saved, _ = _fake_sdk(_sailbox()) + created = sdk.EgressPolicy.create.aio.side_effect + + async def create_then_lose(name, document): + await created(name, document) + raise TimeoutError("response lost") + + sdk.EgressPolicy.create.aio = AsyncMock(side_effect=create_then_lose) + with pytest.raises(TimeoutError): + await create_saved_policy(sdk, "agentenv-lost", {"rules": {}}) + saved[0].delete.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_the_key_is_remembered_until_the_last_box_here_that_needs_it_is_gone(): + first, second = _sailbox(), _sailbox() + second.sailbox_id = "sb_2" + sdk, _, secret = _fake_sdk(first) + sdk.Sailbox.create.aio = AsyncMock(side_effect=[first, second]) + provider = SailVmSandboxProvider(api_key="sail-key", sdk=sdk) + one = await provider.create_sandbox(image_name="agent:1", port=8000, env=_ENV) + two = await provider.create_sandbox(image_name="agent:1", port=8000, env=_ENV) + + await one.terminate() + assert model_key._injected_keys[secret_name(_KEY)] == _KEY + await two.terminate() + assert secret_name(_KEY) not in model_key._injected_keys + secret.delete.aio.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_a_failed_launch_never_touches_the_secret_a_concurrent_launch_with_the_same_key_needs(): + sailbox = _sailbox() + sdk, _, secret = _fake_sdk(sailbox) + first_waiting, go = asyncio.Event(), asyncio.Event() + calls = 0 + + async def create(**_kwargs): + nonlocal calls + calls += 1 + if calls == 1: + first_waiting.set() + await go.wait() + return sailbox + raise RuntimeError("no capacity") + + sdk.Sailbox.create.aio = create + provider = SailVmSandboxProvider(api_key="sail-key", sdk=sdk) + pending = asyncio.ensure_future(provider.create_sandbox(image_name="agent:1", port=8000, env=_ENV)) + await asyncio.wait_for(first_waiting.wait(), timeout=5) + with pytest.raises(RuntimeError, match="no capacity"): + await provider.create_sandbox(image_name="agent:1", port=8000, env=_ENV) + assert model_key._injected_keys[secret_name(_KEY)] == _KEY + go.set() + sandbox = await asyncio.wait_for(pending, timeout=5) + await sandbox.terminate() + secret.delete.aio.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_terminate_waits_for_a_policy_swap_in_flight_on_another_handle(): + sailbox = _sailbox() + sdk, saved, _ = _fake_sdk(sailbox) + provider = SailVmSandboxProvider(api_key="sail-key", sdk=sdk) + sandbox = await provider.create_sandbox( + image_name="agent:1", port=8000, env=_ENV, + network_policy=NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",)), + ) + sailbox.egress_policy = SimpleNamespace(policy_id=saved[0].id, document=saved[0].document) + other = SailVmSandbox(sailbox, sdk=sdk, tunnel_urls={}, network_policy=sandbox.network_policy, + injection=ModelKeyInjection.from_document(saved[0].document, saved[0].id)) + swapping, release = asyncio.Event(), asyncio.Event() + + async def slow_set(policy): + swapping.set() + await release.wait() + sailbox.egress_policy = SimpleNamespace(policy_id=policy.id, document=policy.document) + + sailbox.set_egress_policy.aio = AsyncMock(side_effect=slow_set) + sdk.Sailbox.get.aio = AsyncMock(side_effect=lambda _id: sailbox) + swap = asyncio.ensure_future(other._allow_download_hosts(["https://bucket.example/a"], "download")) + await asyncio.wait_for(swapping.wait(), timeout=5) + teardown = asyncio.ensure_future(sandbox.terminate()) + await asyncio.sleep(0.01) + sailbox.terminate.aio.assert_not_awaited() + release.set() + await asyncio.wait_for(asyncio.gather(swap, teardown), timeout=5) + + replacement = saved[-1] + replacement.delete.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_lost_response_cleanup_retries_through_a_brief_outage(monkeypatch, caplog): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.sandbox.asyncio.sleep", AsyncMock()) + sdk, saved, _ = _fake_sdk(_sailbox()) + await sdk.EgressPolicy.create.aio("agentenv-lost", {"rules": {}}) + listing = sdk.EgressPolicy.list.aio.side_effect + sdk.EgressPolicy.list.aio = AsyncMock(side_effect=[RuntimeError("down"), listing(search="agentenv-lost")]) + + await delete_policy_named(sdk, "agentenv-lost") + + saved[0].delete.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_lost_response_cleanup_that_never_reaches_sail_names_the_policy_to_sweep(monkeypatch, caplog): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.sandbox.asyncio.sleep", AsyncMock()) + sdk, _, _ = _fake_sdk(_sailbox()) + sdk.EgressPolicy.list.aio = AsyncMock(side_effect=RuntimeError("down")) + + await delete_policy_named(sdk, "agentenv-lost") + + assert sdk.EgressPolicy.list.aio.await_count == 3 + assert "Egress policy agentenv-lost may be left in Sail" in caplog.text + + +@pytest.mark.asyncio +async def test_lost_response_cleanup_retries_a_failed_delete_too(monkeypatch): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.sandbox.asyncio.sleep", AsyncMock()) + sdk, saved, _ = _fake_sdk(_sailbox()) + await sdk.EgressPolicy.create.aio("agentenv-lost", {"rules": {}}) + saved[0].delete.aio = AsyncMock(side_effect=[RuntimeError("down"), None]) + + await delete_policy_named(sdk, "agentenv-lost") + + assert saved[0].delete.aio.await_count == 2 + + +@pytest.mark.asyncio +async def test_a_direct_policy_update_waits_for_terminate_on_another_handle(): + sailbox = _sailbox() + sdk, saved, _ = _fake_sdk(sailbox) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + image_name="agent:1", port=8000, env=_ENV, + network_policy=NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",)), + ) + reading, release = asyncio.Event(), asyncio.Event() + + async def slow_get(_id): + reading.set() + await release.wait() + return SimpleNamespace(egress_policy=SimpleNamespace(policy_id=saved[0].id, document=saved[0].document)) + + sdk.Sailbox.get.aio = AsyncMock(side_effect=slow_get) + other = SailVmSandbox(sailbox, sdk=sdk, tunnel_urls={}, network_policy=sandbox.network_policy, + injection=ModelKeyInjection.from_document(saved[0].document, saved[0].id)) + teardown = asyncio.ensure_future(sandbox.terminate()) + await asyncio.wait_for(reading.wait(), timeout=5) + update = asyncio.ensure_future(other.apply_network_policy(sandbox.network_policy.with_hosts(["late.example"]))) + await asyncio.sleep(0.01) + assert len(saved) == 1 + release.set() + await asyncio.wait_for(asyncio.gather(teardown, update), timeout=5) + sailbox.terminate.aio.assert_awaited_once() + assert sailbox.set_egress_policy.aio.await_count == 1 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("quoting", ["agent", "shlex"]) +async def test_a_key_with_an_apostrophe_is_scrubbed_in_its_shell_encoded_form(quoting): + + + key = "sk-a'b-quoted-key" + sailbox = _sailbox() + sailbox.exec.aio = AsyncMock(return_value=_process()) + sdk, _, _ = _fake_sdk(sailbox) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + image_name="agent:1", port=8000, env={**_ENV, "LITELLM_API_KEY": key}, + ) + encoded = "'" + key.replace("'", "'\\''") + "'" if quoting == "agent" else shlex.quote(key) + + await sandbox.exec_script(f"docker run -d --name agent-api -e LITELLM_API_KEY={encoded} agent:1") + + script = sailbox.exec.aio.await_args.args[0][2] + assert "sk-a" not in script and PLACEHOLDER in script + assert shlex.split(script.split("LITELLM_API_KEY=", 1)[1])[0] == PLACEHOLDER + + +@pytest.mark.parametrize( + ("text", "shell", "carries"), + [ + ("docker run -e 'LITELLM_API_KEY'=other-secret img", True, True), + ('docker run -e "LITELLM_API_KEY=other-secret" img', True, True), + ('docker run -e "LITELLM_API_KEY=$LITELLM_API_KEY" img', True, False), + ("LITELLM_API_KEY: other-secret", False, True), + ('{"LITELLM_API_KEY": "sk-x"}', False, True), + (f'{{"LITELLM_API_KEY": "{PLACEHOLDER}"}}', False, False), + ("os.environ['LITELLM_API_KEY']", False, False), + ], +) +def test_a_quoted_name_and_yaml_or_json_forms_are_detected(text, shell, carries): + assert carries_model_key(text, shell=shell) is carries + + +def test_a_key_too_short_to_scrub_safely_is_refused(): + with pytest.raises(ValueError, match="at least 16 characters"): + ModelKeyInjection.for_env({**_ENV, "LITELLM_API_KEY": "docker"}) + + +@pytest.mark.asyncio +async def test_a_swap_whose_response_was_lost_but_applied_keeps_the_replacement_and_deletes_the_old(): + sailbox = _sailbox() + sdk, saved, _ = _fake_sdk(sailbox) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + image_name="agent:1", port=8000, env=_ENV, + network_policy=NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",)), + ) + + async def applied_then_lost(policy): + sailbox.egress_policy = SimpleNamespace(policy_id=policy.id, document=policy.document) + raise TimeoutError("response lost") + + sailbox.set_egress_policy.aio = AsyncMock(side_effect=applied_then_lost) + sdk.Sailbox.get.aio = AsyncMock(side_effect=lambda _id: sailbox) + + await sandbox.apply_network_policy(sandbox.network_policy.with_hosts(["bucket.example"])) + + first, second = saved + first.delete.aio.assert_awaited_once() + second.delete.aio.assert_not_awaited() + assert sandbox._injection.policy_id == second.id + + +@pytest.mark.parametrize( + ("text", "carries"), + [ + ("docker run -e LITELLM_API_K\\EY=other-secret img", True), + ("docker run -e LITELLM_API_KEY=other-secret=$HOME img", True), + ('docker run -e "LITELLM_API_KEY=$X-x" img', True), + ("docker run -e 'LITELLM_API_KEY=$X' img", True), + ('docker run -e "LITELLM_API_KEY=$LITELLM_API_KEY" img', False), + ], +) +def test_names_are_decoded_and_only_a_whole_reference_passes(text, carries): + assert carries_model_key(text, shell=True) is carries + + +async def _restricted_injected_sandbox(): + sailbox = _sailbox() + sdk, saved, _ = _fake_sdk(sailbox) + sandbox = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox( + image_name="agent:1", port=8000, env=_ENV, + network_policy=NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",)), + ) + return sandbox, sailbox, sdk, saved + + +@pytest.mark.asyncio +async def test_a_swap_whose_outcome_sail_cannot_report_keeps_both_policies_until_terminate(): + sandbox, sailbox, sdk, saved = await _restricted_injected_sandbox() + sailbox.set_egress_policy.aio = AsyncMock(side_effect=TimeoutError("response lost")) + sdk.Sailbox.get.aio = AsyncMock(side_effect=RuntimeError("api down")) + + with pytest.raises(TimeoutError): + await sandbox.apply_network_policy(sandbox.network_policy.with_hosts(["bucket.example"])) + + first, second = saved + first.delete.aio.assert_not_awaited() + second.delete.aio.assert_not_awaited() + assert sandbox._injection.unsettled_policy_ids == [second.id] + await sandbox.terminate() + first.delete.aio.assert_awaited() + second.delete.aio.assert_awaited() + + +@pytest.mark.asyncio +async def test_a_cancelled_swap_that_applied_still_settles_and_re_raises_the_cancellation(): + sandbox, sailbox, sdk, saved = await _restricted_injected_sandbox() + + async def applied_then_cancelled(policy): + sailbox.egress_policy = SimpleNamespace(policy_id=policy.id, document=policy.document) + raise asyncio.CancelledError + + sailbox.set_egress_policy.aio = AsyncMock(side_effect=applied_then_cancelled) + sdk.Sailbox.get.aio = AsyncMock(side_effect=lambda _id: sailbox) + + with pytest.raises(asyncio.CancelledError): + await sandbox.apply_network_policy(sandbox.network_policy.with_hosts(["bucket.example"])) + + first, second = saved + first.delete.aio.assert_awaited_once() + assert sandbox._injection.policy_id == second.id + + +@pytest.mark.asyncio +async def test_teardown_through_a_fresh_handle_deletes_every_policy_made_for_the_box(): + sandbox, sailbox, sdk, saved = await _restricted_injected_sandbox() + sailbox.set_egress_policy.aio = AsyncMock(side_effect=TimeoutError("response lost")) + sdk.Sailbox.get.aio = AsyncMock(side_effect=RuntimeError("api down")) + with pytest.raises(TimeoutError): + await sandbox.apply_network_policy(sandbox.network_policy.with_hosts(["bucket.example"])) + first, second = saved + assert first.name.rsplit("-", 1)[0] == second.name.rsplit("-", 1)[0] == sandbox._injection.policy_prefix + + sailbox.egress_policy = SimpleNamespace(policy_id=first.id, name=first.name, document=first.document) + sdk.Sailbox.get.aio = AsyncMock(side_effect=lambda _id: sailbox) + fresh = await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).get_sandbox("sb_1") + await fresh.terminate() + + first.delete.aio.assert_awaited() + second.delete.aio.assert_awaited() + + +@pytest.mark.asyncio +async def test_a_create_whose_response_was_lost_terminates_the_box_sail_made_then_deletes_its_policy(): + sailbox = _sailbox() + sdk, saved, _ = _fake_sdk(sailbox) + made = MagicMock(sailbox_id="sb_lost", status="running") + made.terminate.aio = AsyncMock() + gone = MagicMock(sailbox_id="sb_old", status="terminated") + gone.terminate.aio = AsyncMock() + + async def create_then_lose(**kwargs): + made.name = gone.name = kwargs["name"] + raise TimeoutError("response lost") + + sdk.Sailbox.create.aio = AsyncMock(side_effect=create_then_lose) + sdk.Sailbox.list.aio = AsyncMock(side_effect=lambda **kw: [made, gone] if kw["search"] == made.name else []) + + with pytest.raises(TimeoutError): + await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + + made.terminate.aio.assert_awaited_once() + gone.terminate.aio.assert_not_awaited() + saved[0].delete.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_a_lost_create_is_reclaimed_through_a_brief_outage_before_its_policy_goes(monkeypatch): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.provider.asyncio.sleep", AsyncMock()) + sdk, saved, _ = _fake_sdk(_sailbox()) + made = MagicMock(sailbox_id="sb_lost", status="running") + made.terminate.aio = AsyncMock(side_effect=[RuntimeError("api down"), None]) + + async def create_then_lose(**kwargs): + made.name = kwargs["name"] + raise TimeoutError("response lost") + + sdk.Sailbox.create.aio = AsyncMock(side_effect=create_then_lose) + sdk.Sailbox.list.aio = AsyncMock(side_effect=lambda **kw: [made]) + + with pytest.raises(TimeoutError): + await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + + assert made.terminate.aio.await_count == 2 + saved[0].delete.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_an_unconfirmed_lost_create_keeps_its_policy_rather_than_fail_deleting_it(monkeypatch, caplog): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.provider.asyncio.sleep", AsyncMock()) + sdk, saved, _ = _fake_sdk(_sailbox()) + sdk.Sailbox.create.aio = AsyncMock(side_effect=TimeoutError("response lost")) + sdk.Sailbox.list.aio = AsyncMock(side_effect=RuntimeError("api down")) + + with pytest.raises(TimeoutError): + await SailVmSandboxProvider(api_key="sail-key", sdk=sdk).create_sandbox(image_name="agent:1", port=8000, env=_ENV) + + assert sdk.Sailbox.list.aio.await_count == 3 + saved[0].delete.aio.assert_not_awaited() + assert "stays with the unconfirmed Sailbox" in caplog.text + assert secret_name(_KEY) not in model_key._injected_keys diff --git a/tst/unit/providers/sandbox_providers/sail_vm_sandbox_provider_test.py b/tst/unit/providers/sandbox_providers/sail_vm_sandbox_provider_test.py new file mode 100644 index 00000000..bce5ff88 --- /dev/null +++ b/tst/unit/providers/sandbox_providers/sail_vm_sandbox_provider_test.py @@ -0,0 +1,409 @@ +"""Unit tests for the Sail Sailbox provider; the SDK is a fake, never imported.""" + +import asyncio +import logging +import os +import subprocess +import sys +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from agent_env.config.errors import ConfigError +from agent_env.providers.sandbox_providers.sail_vm import _sdk +from agent_env.providers.sandbox_providers.sail_vm import provider as provider_module +from agent_env.providers.sandbox_providers.sail_vm.provider import ( + SANDBOX_STARTED_EVENT, + SailVmSandboxProvider, + sailbox_name, + sailbox_shape, +) +from agent_env.providers.sandbox_providers.sail_vm.sandbox import SailVmSandbox +from agent_env.providers.sandbox_providers.sandbox import NetworkMode, NetworkPolicy, NetworkPolicyUnsupportedError + + +class _SdkError(Exception): + pass + + +def _listener(port, url=True): + endpoint = SimpleNamespace(url=f"https://sb-1-{port}.sail.box") if url else None + return SimpleNamespace(guest_port=port, endpoint=endpoint) + + +def _sailbox(ports=(), status="running"): + sailbox = MagicMock(sailbox_id="sb_1", status=status, error_message="no capacity") + sailbox.listeners.aio = AsyncMock(return_value=[_listener(port) for port in ports]) + sailbox.terminate.aio = AsyncMock() + sailbox.egress_policy = SimpleNamespace(policy_id=None, document={}) + return sailbox + + +def _fake_sdk(sailbox): + sdk = SimpleNamespace( + App=SimpleNamespace(find=MagicMock(return_value=SimpleNamespace(id="app_1"))), + Sailbox=SimpleNamespace( + create=SimpleNamespace(aio=AsyncMock(return_value=sailbox)), + get=SimpleNamespace(aio=AsyncMock(return_value=sailbox)), + list=SimpleNamespace(aio=AsyncMock(return_value=[])), + ), + Image=SimpleNamespace(devbox=MagicMock(return_value="devbox-amd64")), + AutoSleep=SimpleNamespace( + never=lambda: "never", default=lambda: "default", not_before=lambda seconds: f"not_before:{seconds}" + ), + reset_transports=MagicMock(), + NotFoundError=_SdkError, + SailboxHostLostError=_SdkError, + TransportError=_SdkError, + ) + return sdk + + +@pytest.fixture(autouse=True) +def fresh_key_state(monkeypatch): + monkeypatch.setattr(_sdk, "_installed_key", None) + monkeypatch.setattr(_sdk, "_apps", {}) + monkeypatch.delenv(_sdk.API_KEY_ENV, raising=False) + monkeypatch.delenv(_sdk.RUNTIME_THREADS_ENV, raising=False) + + +@pytest.fixture +def setup(monkeypatch): + setup = AsyncMock() + monkeypatch.setattr(SailVmSandbox, "setup_vm_for_gateway", setup) + return setup + + +def _provider(sdk, **config): + return SailVmSandboxProvider(api_key="sail-secret", sdk=sdk, **config) + + +@pytest.mark.parametrize( + ("config", "message"), + [ + ({}, "requires a non-empty 'api_key'"), + ({"api_key": " "}, "requires a non-empty 'api_key'"), + ({"api_key": "k", "app": ""}, "'app' must be a non-empty string"), + ({"api_key": "k", "min_size": "xl"}, "'min_size' must be one of"), + ({"api_key": "k", "auto_sleep": "yes"}, "'auto_sleep' must be true or false"), + ({"api_key": "k", "auto_sleep_min_idle_seconds": 0}, "from 1 to 3600"), + ({"api_key": "k", "auto_sleep_min_idle_seconds": True}, "from 1 to 3600"), + ({"api_key": "k", "runtime_threads": 257}, "from 1 to 256"), + ({"api_key": "k", "region": "us"}, "unknown key"), + ], +) +def test_from_config_rejects_invalid_config(config, message): + with pytest.raises(ConfigError, match=message): + SailVmSandboxProvider.from_config(**config) + + +def test_importing_core_or_the_provider_package_never_loads_the_sail_sdk(): + script = ( + "import sys, agent_env, agent_env.providers, agent_env.providers.sandbox_providers.sail_vm; " + "print('sail' in sys.modules)" + ) + out = subprocess.run([sys.executable, "-c", script], capture_output=True, text=True, check=True).stdout.strip() + assert out == "False" + + +def test_construction_never_sets_the_key(): + SailVmSandboxProvider.from_config(api_key="sail-secret") + assert _sdk.API_KEY_ENV not in os.environ + + +@pytest.mark.parametrize( + ("cpu", "memory", "disk", "min_size", "shape"), + [ + (1.0, 8192, 10, "s", ("s", 8, 10)), + (0.5, 1024, 1, "s", ("s", 2, 8)), + (2.0, 4096, 10, "s", ("m", 8, 32)), + (1.0, 100 * 1024, 10, "s", ("m", 100, 32)), + (1.0, 2048, 600, "s", ("l", 16, 600)), + (1.0, 1500, 10.2, "m", ("m", 8, 32)), + ], +) +def test_shape_is_the_smallest_covering_size_with_ceilings_rounded_up(cpu, memory, disk, min_size, shape): + assert sailbox_shape(cpu, memory, disk, min_size=min_size) == shape + + +@pytest.mark.parametrize(("cpu", "memory", "disk"), [(9, 8192, 10), (1, 300 * 1024, 10), (1, 8192, 2000)]) +def test_a_request_no_size_fits_is_refused(cpu, memory, disk): + with pytest.raises(ValueError, match="no Sailbox size fits"): + sailbox_shape(cpu, memory, disk) + + +def test_name_carries_slugged_attribution_in_key_order_within_128_chars(): + name = sailbox_name({"run_id": "inst-9f3c", "project_id": "p/123", "team": "env pod"}) + prefix, random, *rest = name.split("-", 2) + assert (prefix, len(random)) == ("ae", 8) + assert name.endswith("p-123-inst-9f3c-env-pod") + assert len(sailbox_name({"k": "x" * 300})) == 128 + + +@pytest.mark.asyncio +async def test_create_vm_sends_the_shape_lifetime_ports_and_policy(setup): + sailbox = _sailbox(ports=[8080, 9000]) + sdk = _fake_sdk(sailbox) + + sandbox = await _provider(sdk).create_vm( + cpu=2.0, memory=4096, disk_size_gb=20, timeout=900, exposed_ports=[8080, 9000, 8080], + attribution={"run_id": "inst-1"}, + ) + + kwargs = sdk.Sailbox.create.aio.await_args.kwargs + assert kwargs["app"].id == "app_1" + assert kwargs["image"] == "devbox-amd64" + sdk.Image.devbox.assert_called_once_with("amd64") + assert kwargs["name"].startswith("ae-") and kwargs["name"].endswith("-inst-1") + assert {k: kwargs[k] for k in ("size", "memory_limit_gib", "disk_limit_gib", "max_lifetime_seconds")} == { + "size": "m", "memory_limit_gib": 8, "disk_limit_gib": 32, "max_lifetime_seconds": 900, + } + assert kwargs["ingress_ports"] == [8080, 9000] + assert kwargs["auto_sleep"] == "never" + assert kwargs["egress_policy"] == {} + assert "api_key" not in kwargs and "env" not in kwargs + assert isinstance(sandbox, SailVmSandbox) + assert (sandbox.type, sandbox.mode, sandbox.sandbox_id) == ("sail_vm", "vm", "sb_1") + assert sandbox.tunnel_urls == {8080: "https://sb-1-8080.sail.box", 9000: "https://sb-1-9000.sail.box"} + assert sandbox.network_policy == NetworkPolicy() + setup.assert_awaited_once_with([8080, 9000]) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("config", "expected"), + [({}, "never"), ({"auto_sleep": True}, "default"), ({"auto_sleep_min_idle_seconds": 600}, "not_before:600")], +) +async def test_auto_sleep_is_off_unless_configured(setup, config, expected): + sdk = _fake_sdk(_sailbox()) + await _provider(sdk, **config).create_vm(exposed_ports=[]) + assert sdk.Sailbox.create.aio.await_args.kwargs["auto_sleep"] == expected + + +@pytest.mark.asyncio +async def test_allowlist_becomes_a_sail_allowlist_with_the_platform_floor(setup): + sdk = _fake_sdk(_sailbox()) + policy = NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",), allow_cidrs=("10.0.0.0/8",)) + + sandbox = await _provider(sdk).create_vm(exposed_ports=[], network_policy=policy) + + entries = sdk.Sailbox.create.aio.await_args.kwargs["egress_policy"]["allowlist"] + assert entries[0] == "pypi.org" and entries[-1] == "10.0.0.0/8" + assert {"*.sail.box", "*.e2b.app"} <= set(entries) + assert sandbox.network_policy.allow_cidrs == ("10.0.0.0/8",) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "policy", + [ + NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_cidrs=("2001:db8::/32",)), + NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=tuple(f"h{i}.example" for i in range(130))), + ], +) +async def test_an_unenforceable_policy_is_refused_before_provisioning(policy): + sdk = _fake_sdk(_sailbox()) + assert SailVmSandboxProvider.supports_network_policy(policy) is False + with pytest.raises(NetworkPolicyUnsupportedError): + await _provider(sdk).create_vm(exposed_ports=[], network_policy=policy) + sdk.Sailbox.create.aio.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_a_failed_sailbox_is_terminated_and_reported(): + sailbox = _sailbox(status="failed") + with pytest.raises(RuntimeError, match="failed to start: no capacity"): + await _provider(_fake_sdk(sailbox)).create_vm(exposed_ports=[]) + sailbox.terminate.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_a_setup_failure_terminates_the_sailbox(setup): + sailbox = _sailbox() + setup.side_effect = RuntimeError("docker never came up") + with pytest.raises(RuntimeError, match="docker never came up"): + await _provider(_fake_sdk(sailbox)).create_vm(exposed_ports=[]) + sailbox.terminate.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_tunnel_urls_wait_until_every_port_is_routed(setup, monkeypatch): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.provider._LISTENER_POLL_INTERVAL", 0) + sailbox = _sailbox() + sailbox.listeners.aio = AsyncMock(side_effect=[[_listener(8080, url=False)], [_listener(8080)]]) + sandbox = await _provider(_fake_sdk(sailbox)).create_vm(exposed_ports=[8080]) + assert sandbox.tunnel_urls == {8080: "https://sb-1-8080.sail.box"} + + +@pytest.mark.asyncio +async def test_a_port_that_never_routes_fails_the_create(setup, monkeypatch): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.provider._LISTENER_POLL_INTERVAL", 0) + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.provider._LISTENER_TIMEOUT", 0) + sailbox = _sailbox() + with pytest.raises(RuntimeError, match=r"no public URL for port\(s\) \[8080\]"): + await _provider(_fake_sdk(sailbox)).create_vm(exposed_ports=[8080]) + sailbox.terminate.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_a_cancelled_create_terminates_the_sailbox_it_produces(): + sailbox = _sailbox() + sdk = _fake_sdk(sailbox) + created = asyncio.Event() + + async def slow_create(**_kwargs): + await created.wait() + return sailbox + + sdk.Sailbox.create.aio = slow_create + caller = asyncio.ensure_future(_provider(sdk).create_vm(exposed_ports=[])) + await asyncio.sleep(0.01) + caller.cancel() + with pytest.raises(asyncio.CancelledError): + await caller + created.set() + for _ in range(5): + await asyncio.sleep(0) + sailbox.terminate.aio.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_a_failing_orphan_termination_is_retried_then_reported(monkeypatch, caplog): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.provider.asyncio.sleep", AsyncMock()) + sandbox = MagicMock(sandbox_id="sb_1") + sandbox.terminate = AsyncMock(side_effect=RuntimeError("api down")) + + await provider_module._reap(sandbox) + + assert sandbox.terminate.await_count == provider_module._REAP_ATTEMPTS + assert "Orphaned Sailbox sb_1 is still running" in caplog.text + + +@pytest.mark.asyncio +async def test_create_logs_the_attribution_join_event_without_the_key(setup, caplog): + caplog.set_level(logging.INFO, logger="agent_env.providers.sandbox_providers.sail_vm.provider") + await _provider(_fake_sdk(_sailbox())).create_vm(exposed_ports=[], attribution={"run_id": "inst-1", "team": "t"}) + + (record,) = [r for r in caplog.records if getattr(r, "event", None) == SANDBOX_STARTED_EVENT] + assert record.sail_sailbox_id == "sb_1" + assert record.sail_app_name == "agent-env" + assert record.sail_attribution == {"run_id": "inst-1", "team": "t"} + assert record.run_id == "inst-1" + assert "sail-secret" not in caplog.text + + +@pytest.mark.asyncio +async def test_create_sandbox_is_a_bare_vm_that_ignores_image_and_env(setup): + sdk = _fake_sdk(_sailbox(ports=[8080])) + await _provider(sdk).create_sandbox(image_name="registry.example/agent:1", port=8080, env={"TOKEN": "workload-token-value"}) + kwargs = sdk.Sailbox.create.aio.await_args.kwargs + assert kwargs["ingress_ports"] == [8080] + assert "workload-token-value" not in repr(kwargs) + + +@pytest.mark.asyncio +async def test_image_overrides_are_refused(): + with pytest.raises(ValueError, match="image overrides are unsupported"): + await _provider(_fake_sdk(_sailbox())).create_vm(image="ubuntu:22.04") + + +@pytest.mark.asyncio +async def test_create_container_removes_the_registry_login_from_the_vm(monkeypatch): + sandbox = MagicMock(spec=SailVmSandbox) + sandbox.exec_script = AsyncMock() + monkeypatch.setattr( + "agent_env.providers.sandbox_providers.sandbox_provider.SandboxProvider.create_container", + AsyncMock(return_value=sandbox), + ) + result = await _provider(_fake_sdk(_sailbox())).create_container(image_name="r/i:1", port=80, env={}) + assert result is sandbox + sandbox.exec_script.assert_awaited_once_with("rm -f /root/.docker/config.json") + + +@pytest.mark.asyncio +async def test_get_sandbox_restores_ports_and_the_applied_policy(): + sailbox = _sailbox(ports=[8080]) + sailbox.egress_policy = SimpleNamespace(policy_id=None, document={"allowlist": ["pypi.org", "10.0.0.0/8"]}) + sdk = _fake_sdk(sailbox) + + sandbox = await _provider(sdk).get_sandbox("sb_1") + + sdk.Sailbox.get.aio.assert_awaited_once_with("sb_1") + assert sandbox.tunnel_urls == {8080: "https://sb-1-8080.sail.box"} + assert sandbox.network_policy == NetworkPolicy( + mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",), allow_cidrs=("10.0.0.0/8",) + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "applied", + [ + None, + SimpleNamespace(policy_id="ep_1", document={}), + SimpleNamespace(policy_id=None, document={"allowlist": ["a.example"], "rules": []}), + ], +) +async def test_get_sandbox_leaves_an_unrepresentable_policy_unknown(applied): + sailbox = _sailbox() + sailbox.egress_policy = applied + sandbox = await _provider(_fake_sdk(sailbox)).get_sandbox("sb_1") + assert sandbox.network_policy is None + + +def test_the_key_is_set_only_while_the_sdk_builds_its_client(): + sdk = _fake_sdk(_sailbox()) + seen = {} + + def find(**kwargs): + seen["key"] = os.environ.get(_sdk.API_KEY_ENV) + seen["threads"] = os.environ.get(_sdk.RUNTIME_THREADS_ENV) + return SimpleNamespace(id="app_1") + + sdk.App.find = MagicMock(side_effect=find) + _sdk.connect("sail-secret", "agent-env", runtime_threads=16, sdk=sdk) + + assert seen == {"key": "sail-secret", "threads": "16"} + assert _sdk.API_KEY_ENV not in os.environ + assert _sdk.RUNTIME_THREADS_ENV not in os.environ + sdk.reset_transports.assert_called_once() + + +def test_operator_settings_are_overridden_for_the_build_and_then_restored(monkeypatch): + monkeypatch.setenv(_sdk.API_KEY_ENV, "operator-key") + monkeypatch.setenv(_sdk.RUNTIME_THREADS_ENV, "4") + sdk = _fake_sdk(_sailbox()) + sdk.App.find = MagicMock(side_effect=lambda **_: (os.environ[_sdk.API_KEY_ENV], os.environ[_sdk.RUNTIME_THREADS_ENV])) + + _, app = _sdk.connect("sail-secret", "agent-env", runtime_threads=16, sdk=sdk) + + assert app == ("sail-secret", "16") + assert (os.environ[_sdk.API_KEY_ENV], os.environ[_sdk.RUNTIME_THREADS_ENV]) == ("operator-key", "4") + + +def test_the_key_is_installed_once_per_process_and_apps_are_cached(): + sdk = _fake_sdk(_sailbox()) + for _ in range(3): + _sdk.connect("sail-secret", "agent-env", sdk=sdk) + _sdk.connect("sail-secret", "other-app", sdk=sdk) + + sdk.reset_transports.assert_called_once() + assert [c.kwargs["name"] for c in sdk.App.find.call_args_list] == ["agent-env", "other-app"] + + +def test_a_second_key_in_the_same_process_is_refused(): + sdk = _fake_sdk(_sailbox()) + _sdk.connect("sail-secret", "agent-env", sdk=sdk) + with pytest.raises(ConfigError, match="one Sail API key"): + _sdk.connect("another-key", "agent-env", sdk=sdk) + + +def test_a_rejected_key_can_be_retried(): + sdk = _fake_sdk(_sailbox()) + sdk.App.find = MagicMock(side_effect=[PermissionError("Invalid API key"), SimpleNamespace(id="app_1")]) + with pytest.raises(PermissionError): + _sdk.connect("sail-secret", "agent-env", sdk=sdk) + assert _sdk.API_KEY_ENV not in os.environ + _sdk.connect("sail-secret", "agent-env", sdk=sdk) + assert sdk.reset_transports.call_count == 2 diff --git a/tst/unit/providers/sandbox_providers/sail_vm_sandbox_test.py b/tst/unit/providers/sandbox_providers/sail_vm_sandbox_test.py new file mode 100644 index 00000000..e60027a7 --- /dev/null +++ b/tst/unit/providers/sandbox_providers/sail_vm_sandbox_test.py @@ -0,0 +1,358 @@ +"""Unit tests for the Sailbox adapter over a fake SDK Sailbox.""" + +import asyncio +import io +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from agent_env.providers.sandbox_providers.sail_vm.sandbox import SailVmSandbox, egress_document, policy_from_document +from agent_env.providers.sandbox_providers.sandbox import NetworkMode, NetworkPolicy + + +class _HostLost(Exception): + pass + + +class _NotFound(Exception): + pass + + +_SDK = SimpleNamespace(SailboxHostLostError=_HostLost, TransportError=_HostLost, NotFoundError=_NotFound) +_ALLOWLIST = NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",), allow_cidrs=("10.0.0.0/8",)) + + +async def _chunks(*parts: bytes): + for part in parts: + yield part + + +def _process(stdout=b"", stderr=b"", exit_code=0): + process = MagicMock() + process.stdout_bytes = _chunks(stdout[:3], stdout[3:]) + process.stderr_bytes = _chunks(stderr) + process.wait = AsyncMock(return_value=SimpleNamespace(exit_code=exit_code)) + return process + + +def _sandbox(*outcomes, policy=NetworkPolicy()): + sailbox = MagicMock(sailbox_id="sb_1") + sailbox.exec.aio = AsyncMock(side_effect=list(outcomes)) + sailbox.terminate.aio = AsyncMock() + sailbox.fs.write.aio = AsyncMock() + applied = {"document": egress_document(policy) if policy is not None else {"rules": []}} + + async def set_policy(document): + await asyncio.sleep(0) + applied["document"] = document + + sailbox.set_egress_policy.aio = AsyncMock(side_effect=set_policy) + sdk = SimpleNamespace(**vars(_SDK), Sailbox=SimpleNamespace(get=SimpleNamespace(aio=AsyncMock( + side_effect=lambda _id: SimpleNamespace(egress_policy=SimpleNamespace(policy_id=None, document=applied["document"])) + )))) + return SailVmSandbox(sailbox, sdk=sdk, tunnel_urls={}, network_policy=policy), sailbox + + +@pytest.mark.asyncio +async def test_exec_returns_exact_bytes_and_exit_code_and_drops_a_leading_sudo(): + binary = bytes(range(256)) + sandbox, sailbox = _sandbox(_process(stdout=binary, stderr=b"warn\n", exit_code=7)) + + process = await sandbox.exec("sudo", "bash", "-c", "echo 'a b'", "sudo") + + assert (await process.stdout.read(), await process.stderr.read(), await process.wait()) == (binary, b"warn\n", 7) + args, kwargs = sailbox.exec.aio.await_args + assert args == (["bash", "-c", "echo 'a b'", "sudo"],) + assert kwargs["output_mode"] == "pipe" + assert kwargs["timeout"] is None + assert len(kwargs["idempotency_key"]) == 32 + + +@pytest.mark.asyncio +async def test_stdout_streams_chunk_by_chunk_for_large_reads(): + sandbox, _ = _sandbox(_process(stdout=b"abcdefgh")) + process = await sandbox.exec("cat", "big") + assert [chunk async for chunk in process.stdout] == [b"abc", b"defgh"] + assert await process.wait() == 0 + + +@pytest.mark.asyncio +async def test_a_host_lost_while_the_command_runs_is_exit_minus_one(): + process = _process(stdout=b"partial") + process.wait = AsyncMock(side_effect=_HostLost("migrated")) + sandbox, _ = _sandbox(process) + assert (await sandbox.exec_with_output("true"))[0] == -1 + + +@pytest.mark.asyncio +async def test_output_nobody_reads_is_drained_by_wait_rather_than_blocking_the_command(): + process = MagicMock() + + async def many_chunks(): + for _ in range(1000): + yield b"x" * 1024 + + process.stdout_bytes = many_chunks() + process.stderr_bytes = _chunks(b"") + process.wait = AsyncMock(return_value=SimpleNamespace(exit_code=0)) + sandbox, _ = _sandbox(process) + + assert await asyncio.wait_for((await sandbox.exec("yes")).wait(), timeout=5) == 0 + + +async def _failing_chunks(error): + yield b"partial" + raise error + + +@pytest.mark.asyncio +async def test_a_broken_stream_is_raised_not_reported_as_success(): + process = _process() + process.stdout_bytes = _failing_chunks(ValueError("stream corrupted")) + sandbox, _ = _sandbox(process) + running = await sandbox.exec("cat", "f") + with pytest.raises(ValueError, match="stream corrupted"): + await running.stdout.read() + with pytest.raises(ValueError, match="stream corrupted"): + await running.wait() + + +@pytest.mark.asyncio +async def test_a_stream_cut_by_a_migration_ends_and_waits_to_exit_minus_one(): + process = _process() + process.stdout_bytes = _failing_chunks(_HostLost("moved")) + sandbox, _ = _sandbox(process) + running = await sandbox.exec("cat", "f") + assert await running.stdout.read() == b"partial" + assert await running.wait() == -1 + + +@pytest.mark.asyncio +async def test_exec_with_output_never_returns_the_partial_stdout_of_a_cut_stream(): + process = _process(stderr=b"moved\n") + process.stdout_bytes = _failing_chunks(_HostLost("moved")) + sandbox, _ = _sandbox(process) + assert await sandbox.exec_with_output("cat", "trajectory.jsonl") == (-1, "", "moved\n") + + +@pytest.mark.asyncio +async def test_exec_script_retries_a_command_whose_output_stream_was_cut(monkeypatch): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sandbox.asyncio.sleep", AsyncMock()) + cut = _process() + cut.stdout_bytes = _failing_chunks(_HostLost("moved")) + sandbox, sailbox = _sandbox(cut, _process(stdout=b"loaded\n")) + + assert await sandbox.exec_script("docker load < image.tar", max_retries=1) == "loaded\n" + assert sailbox.exec.aio.await_count == 2 + + +@pytest.mark.asyncio +async def test_each_exec_gets_its_own_idempotency_key(): + sandbox, sailbox = _sandbox(_process(), _process()) + await sandbox.exec("true") + await sandbox.exec("true") + keys = [call.kwargs["idempotency_key"] for call in sailbox.exec.aio.await_args_list] + assert keys[0] != keys[1] + + +@pytest.mark.asyncio +async def test_a_lost_host_is_exit_minus_one_which_exec_script_retries(monkeypatch): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sandbox.asyncio.sleep", AsyncMock()) + sandbox, sailbox = _sandbox(_HostLost("host gone"), _process(stdout=b"ok\n")) + + assert await sandbox.exec_script("echo ok", max_retries=1) == "ok\n" + assert sailbox.exec.aio.await_count == 2 + + +@pytest.mark.asyncio +async def test_other_sdk_errors_propagate(): + sandbox, _ = _sandbox(PermissionError("Invalid API key")) + with pytest.raises(PermissionError): + await sandbox.exec("true") + + +@pytest.mark.asyncio +async def test_wait_for_vm_starts_dockerd_once_when_it_is_not_running(monkeypatch): + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.sandbox.asyncio.sleep", AsyncMock()) + sandbox, sailbox = _sandbox( + _process(stderr=b"Cannot connect to the Docker daemon", exit_code=1), + _process(), + _process(stderr=b"still starting", exit_code=1), + _process(stdout=b"20.10.24\n"), + ) + + await sandbox.wait_for_vm() + + commands = [call.args[0] for call in sailbox.exec.aio.await_args_list] + assert commands[0][:2] == ["docker", "info"] + assert commands[1][:2] == ["bash", "-c"] and "nohup dockerd" in commands[1][2] + assert [c[:2] for c in commands[2:]] == [["docker", "info"], ["docker", "info"]] + assert sailbox.exec.aio.await_args_list[0].kwargs["timeout"] == SailVmSandbox._DOCKER_PROBE_TIMEOUT + + +@pytest.mark.asyncio +async def test_wait_for_vm_reports_the_last_error_when_docker_never_answers(monkeypatch): + monkeypatch.setattr(SailVmSandbox, "_VM_READY_TIMEOUT", 0) + sandbox, _ = _sandbox(_process(stderr=b"daemon down", exit_code=1)) + with pytest.raises(RuntimeError, match="Docker not ready in Sailbox sb_1.*daemon down"): + await sandbox.wait_for_vm() + + +@pytest.mark.asyncio +async def test_setup_requires_compose_v2(monkeypatch): + monkeypatch.setattr(SailVmSandbox, "wait_for_vm", AsyncMock()) + sandbox, _ = _sandbox(_process(stderr=b"docker: 'compose' is not a docker command", exit_code=1)) + with pytest.raises(RuntimeError, match="no Docker Compose v2"): + await sandbox.setup_vm_for_gateway([8080]) + + +@pytest.mark.asyncio +async def test_host_files_are_written_with_the_native_filesystem_api(): + sandbox, sailbox = _sandbox(_process()) + await sandbox.write_host_file(b"\x00\x01" * 100_000, "/opt/data/blob.bin") + sailbox.fs.write.aio.assert_awaited_once_with("/opt/data/blob.bin", b"\x00\x01" * 100_000) + + +@pytest.mark.asyncio +async def test_terminate_tolerates_an_already_deleted_sailbox(): + sandbox, sailbox = _sandbox() + sailbox.terminate.aio.side_effect = _NotFound("sailbox not found") + await sandbox.terminate() + + +@pytest.mark.asyncio +async def test_image_loading_fails_closed_when_the_policy_is_unknown(): + sandbox, _ = _sandbox(policy=None) + with pytest.raises(RuntimeError, match="applied egress policy is unknown"): + await sandbox.load_docker_images([object()]) + + +@pytest.mark.asyncio +async def test_image_loading_widens_an_allowlist_with_the_signed_download_hosts(monkeypatch): + sandbox, sailbox = _sandbox(policy=_ALLOWLIST) + monkeypatch.setattr( + SailVmSandbox, "_signed_image_urls", + AsyncMock(return_value=["https://bucket.s3.amazonaws.com/a?sig=1", None]), + ) + load = AsyncMock() + monkeypatch.setattr(SailVmSandbox, "_load_docker_images", load) + + await sandbox.load_docker_images(["a", "b"]) + + sailbox.set_egress_policy.aio.assert_awaited_once_with( + {"allowlist": ["pypi.org", "bucket.s3.amazonaws.com", "10.0.0.0/8"]} + ) + assert sandbox.network_policy.allow_hosts == ("pypi.org", "bucket.s3.amazonaws.com") + load.assert_awaited_once_with(["a", "b"], ["https://bucket.s3.amazonaws.com/a?sig=1", None]) + + +@pytest.mark.asyncio +async def test_image_loading_leaves_an_allow_all_policy_alone(monkeypatch): + sandbox, sailbox = _sandbox() + monkeypatch.setattr(SailVmSandbox, "_signed_image_urls", AsyncMock(return_value=["https://x.example/a"])) + monkeypatch.setattr(SailVmSandbox, "_load_docker_images", AsyncMock()) + await sandbox.load_docker_images(["a"]) + sailbox.set_egress_policy.aio.assert_not_awaited() + sandbox._sdk.Sailbox.get.aio.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_widening_past_sails_allowlist_limit_is_refused(monkeypatch): + full = NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=tuple(f"h{i}.example" for i in range(128))) + sandbox, sailbox = _sandbox(policy=full) + monkeypatch.setattr(SailVmSandbox, "_signed_image_urls", AsyncMock(return_value=["https://bucket.example/a"])) + with pytest.raises(RuntimeError, match="exceed Sail's 128-entry egress allowlist"): + await sandbox.load_docker_images(["a"]) + sailbox.set_egress_policy.aio.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_a_host_under_an_allowed_wildcard_needs_no_new_entry(monkeypatch): + full = NetworkPolicy( + mode=NetworkMode.ALLOWLIST, allow_hosts=("*.s3.amazonaws.com", *(f"h{i}.example" for i in range(127))) + ) + sandbox, sailbox = _sandbox(policy=full) + monkeypatch.setattr(SailVmSandbox, "_signed_image_urls", AsyncMock(return_value=["https://bucket.s3.amazonaws.com/a"])) + monkeypatch.setattr(SailVmSandbox, "_load_docker_images", AsyncMock()) + await sandbox.load_docker_images(["a"]) + sailbox.set_egress_policy.aio.assert_not_awaited() + sandbox._sdk.Sailbox.get.aio.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_concurrent_downloads_through_separate_handles_keep_each_others_hosts(): + sandbox, sailbox = _sandbox(policy=_ALLOWLIST) + other_handle = SailVmSandbox(sailbox, sdk=sandbox._sdk, tunnel_urls={}, network_policy=_ALLOWLIST) + + await asyncio.gather( + sandbox._allow_download_hosts(["https://a.example/x"], "download"), + other_handle._allow_download_hosts(["https://b.example/y"], "download"), + ) + + assert set(sailbox.set_egress_policy.aio.await_args.args[0]["allowlist"]) >= {"a.example", "b.example"} + + +@pytest.mark.asyncio +async def test_a_host_already_allowed_is_not_reapplied(monkeypatch): + sandbox, sailbox = _sandbox(policy=_ALLOWLIST) + monkeypatch.setattr(SailVmSandbox, "_signed_image_urls", AsyncMock(return_value=["https://pypi.org/a"])) + monkeypatch.setattr(SailVmSandbox, "_load_docker_images", AsyncMock()) + await sandbox.load_docker_images(["a"]) + sailbox.set_egress_policy.aio.assert_not_awaited() + + +def _object_store(monkeypatch, signed): + store = MagicMock( + signed_get_url=MagicMock(return_value=signed), + open=MagicMock(side_effect=lambda _url: io.BytesIO(b"\x00payload")), + ) + monkeypatch.setattr( + "agent_env.providers.sandbox_providers.sail_vm.sandbox.get_config", lambda: MagicMock(get_object_store=lambda: store) + ) + return store + + +@pytest.mark.asyncio +async def test_a_signed_object_download_allows_its_host_first(monkeypatch): + _object_store(monkeypatch, "https://bucket.s3.amazonaws.com/f?sig=1") + sandbox, sailbox = _sandbox(_process(), policy=_ALLOWLIST) + + await sandbox.load_s3_file("s3://bucket/f", "/tmp/f") + + sailbox.set_egress_policy.aio.assert_awaited_once_with( + {"allowlist": ["pypi.org", "bucket.s3.amazonaws.com", "10.0.0.0/8"]} + ) + script = sailbox.exec.aio.await_args.args[0][2] + assert script.startswith("curl -fsSL") and "'https://bucket.s3.amazonaws.com/f?sig=1'" in script + + +@pytest.mark.asyncio +async def test_an_unsignable_object_is_streamed_through_the_filesystem_api(monkeypatch): + _object_store(monkeypatch, None) + monkeypatch.setattr("agent_env.providers.sandbox_providers.sail_vm.sandbox._STREAM_CHUNK_BYTES", 4) + sandbox, sailbox = _sandbox(policy=None) + writer = MagicMock(write=AsyncMock()) + stream = MagicMock(__aenter__=AsyncMock(return_value=writer), __aexit__=AsyncMock(return_value=False)) + sailbox.fs.write_stream.aio = AsyncMock(return_value=stream) + + await sandbox.load_s3_file("file:///store/f", "/tmp/f") + + sailbox.fs.write_stream.aio.assert_awaited_once_with("/tmp/f") + assert b"".join(call.args[0] for call in writer.write.await_args_list) == b"\x00payload" + assert [len(call.args[0]) for call in writer.write.await_args_list] == [4, 4] + stream.__aexit__.assert_awaited_once() + sailbox.exec.aio.assert_not_awaited() + + +@pytest.mark.parametrize("policy", [NetworkPolicy(), _ALLOWLIST, NetworkPolicy(mode=NetworkMode.ALLOWLIST)]) +def test_egress_documents_round_trip(policy): + assert policy_from_document(egress_document(policy)) == policy + + +@pytest.mark.parametrize( + "document", + [None, {"no_network": True}, {"allowlist": ["a.example"], "blocked": ["b.example"]}, {"allowlist": "a.example"}], +) +def test_documents_agent_env_cannot_represent_are_unknown(document): + assert policy_from_document(document) is None diff --git a/tst/unit/providers/sandbox_providers/sandbox_provider_test.py b/tst/unit/providers/sandbox_providers/sandbox_provider_test.py index 2457527a..3265e2ec 100644 --- a/tst/unit/providers/sandbox_providers/sandbox_provider_test.py +++ b/tst/unit/providers/sandbox_providers/sandbox_provider_test.py @@ -119,6 +119,45 @@ async def create_sandbox(self, **kwargs): assert result.mode == "container" +@pytest.mark.asyncio +async def test_a_create_container_cancelled_during_the_pull_terminates_the_vm(): + pulling = asyncio.Event() + + class _HangingVm(VmSandbox): + type = "fake-vm" + + def __init__(self): + self.sandbox_id = "vm-fake" + self.tunnel_urls = {} + self.vnc_url = None + self.mode = "vm" + self.terminated = False + + async def terminate(self) -> None: + self.terminated = True + + async def exec_script(self, script: str) -> str: + pulling.set() + await asyncio.Event().wait() + return "" + + vm = _HangingVm() + + class _VmStyleProvider(SandboxProvider): + async def create_vm(self, **kwargs): + return vm + + async def create_sandbox(self, **kwargs): + raise NotImplementedError + + task = asyncio.ensure_future(_VmStyleProvider().create_container(image_name="nginx:latest", port=8080, env={})) + await pulling.wait() + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + assert vm.terminated + + @pytest.mark.asyncio async def test_create_container_mints_the_registry_login_off_the_event_loop(): """A remote registry's login is a network round trip, an IAM token exchange for one.""" diff --git a/tst/unit/providers/sandbox_providers/test_config_sandbox_providers.py b/tst/unit/providers/sandbox_providers/test_config_sandbox_providers.py index 8244561f..5d233b8a 100644 --- a/tst/unit/providers/sandbox_providers/test_config_sandbox_providers.py +++ b/tst/unit/providers/sandbox_providers/test_config_sandbox_providers.py @@ -1,6 +1,7 @@ """Unit tests for config.toml-declared custom sandbox providers + the deploy-time type guard.""" import asyncio +import os import textwrap import pytest @@ -10,6 +11,7 @@ from agent_env.config import reset_config from agent_env.providers.sandbox_providers.e2b.provider import E2BSandboxProvider from agent_env.providers.sandbox_providers.local_sandbox import LocalSandboxProvider +from agent_env.providers.sandbox_providers.sail_vm.provider import SailVmSandboxProvider from agent_env.providers.sandbox_providers.sandbox import Sandbox from agent_env.providers.sandbox_providers.sandbox_provider import ( SandboxProvider, @@ -213,6 +215,38 @@ def test_e2b_missing_base_template_is_a_config_error(monkeypatch, tmp_path): build_sandbox_provider("e2b") +def test_sail_builtin_receives_interpolated_key_without_touching_the_sdk(monkeypatch, tmp_path): + cfg = _write_config(tmp_path, """ + [sandbox.providers.sail_vm.config] + api_key = "env:SAIL_TEST_API_KEY" + app = "agent-env-test" + auto_sleep_min_idle_seconds = 600 + """) + monkeypatch.setenv("AGENT_ENV_CONFIG", str(cfg)) + monkeypatch.setenv("SAIL_TEST_API_KEY", "resolved-sail-key") + monkeypatch.delenv("SAIL_API_KEY", raising=False) + + provider = build_sandbox_provider("sail_vm") + + assert isinstance(provider, SailVmSandboxProvider) + assert provider._api_key == "resolved-sail-key" + assert provider._app_name == "agent-env-test" + assert provider._auto_sleep_min_idle_seconds == 600 + assert "SAIL_API_KEY" not in os.environ + assert "resolved-sail-key" not in repr(provider) + + +def test_sail_missing_api_key_is_a_config_error(monkeypatch, tmp_path): + cfg = _write_config(tmp_path, """ + [sandbox.providers.sail_vm.config] + app = "agent-env-test" + """) + monkeypatch.setenv("AGENT_ENV_CONFIG", str(cfg)) + + with pytest.raises(ConfigError, match="requires a non-empty 'api_key'"): + build_sandbox_provider("sail_vm") + + def test_builtin_config_reaches_chain_members(monkeypatch, tmp_path): cfg = _write_config(tmp_path, """ [sandbox.providers.modal.config] diff --git a/tst/unit/test_capabilities.py b/tst/unit/test_capabilities.py index 81c5816b..3aecc9d8 100644 --- a/tst/unit/test_capabilities.py +++ b/tst/unit/test_capabilities.py @@ -165,3 +165,18 @@ def test_e2b_present_when_the_provider_builds(monkeypatch): def test_e2b_other_failures_read_as_present_rather_than_absent(monkeypatch): _fake_e2b_builder(monkeypatch, RuntimeError("api key rejected")) assert capabilities.remote_sandbox_is_available("e2b") is True + + +@pytest.mark.parametrize( + ("outcome", "available"), + [(ConfigError("[sandbox.providers.sail_vm.config] requires a non-empty 'api_key'"), False), (object(), True), (RuntimeError("boom"), True)], +) +def test_sail_availability_follows_the_provider_build(monkeypatch, outcome, available): + def build(spec): + assert spec == "sail_vm" + if isinstance(outcome, Exception): + raise outcome + return outcome + + monkeypatch.setattr("agent_env.providers.sandbox_providers.sandbox_provider.build_sandbox_provider", build) + assert capabilities.remote_sandbox_is_available("sail_vm") is available diff --git a/tst/unit/test_optional_extras_stay_optional.py b/tst/unit/test_optional_extras_stay_optional.py index 0bc91f80..b40e59ba 100644 --- a/tst/unit/test_optional_extras_stay_optional.py +++ b/tst/unit/test_optional_extras_stay_optional.py @@ -24,9 +24,12 @@ def find_spec(self, fullname, path=None, target=None): sys.meta_path.insert(0, Refuse()) """ -# Not core: the explorer is the explorer extra's own package, the gateway runs only in its -# container image, and the code runner is a script that reads its arguments on import. -_NOT_CORE = ("agent_env.explorer", "agent_env.env.gateway", "agent_env.task_step.task_steps.run_code_runner") +# Not core: the explorer and the sail_vm provider are their extras' own packages, the gateway runs +# only in its container image, and the code runner is a script that reads its arguments on import. +_NOT_CORE = ( + "agent_env.explorer", "agent_env.providers.sandbox_providers.sail_vm", "agent_env.env.gateway", + "agent_env.task_step.task_steps.run_code_runner", +) _IMPORT_CORE = _BLOCKING + """ import ast, pkgutil @@ -123,3 +126,9 @@ def test_core_and_every_store_module_import_without_the_extra(extra): def test_a_gcp_impl_without_the_extra_names_the_extra_to_install(impl, base): message = _run(_LOAD_IMPL, _only_in("gcp"), impl, base) assert "pip install 'agentenv-framework[gcp]'" in message, message + + +def test_the_sail_vm_provider_without_the_extra_names_the_extra_to_install(): + impl = "agent_env.providers.sandbox_providers.sail_vm.provider:SailVmSandboxProvider" + message = _run(_LOAD_IMPL, _only_in("sail"), impl, "agent_env.providers.sandbox_providers.sandbox_provider:SandboxProvider") + assert "pip install 'agentenv-framework[sail]'" in message, message diff --git a/tst/util/capabilities.py b/tst/util/capabilities.py index e276b661..ef86c8f6 100644 --- a/tst/util/capabilities.py +++ b/tst/util/capabilities.py @@ -19,7 +19,7 @@ #: Only the agent-driving paths need a model endpoint. MODEL_ENDPOINT = "model_endpoint_configured" -#: The remote sandbox providers (``modal``, ``modal_vm``, ``e2b``) need credentials the resolved +#: The remote sandbox providers (``modal``, ``modal_vm``, ``e2b``, ``sail_vm``) need credentials the resolved #: config may not carry; the local default never does. REMOTE_SANDBOX = "remote_sandbox" @@ -75,21 +75,21 @@ def skip_without_model_endpoint() -> pytest.MarkDecorator: def remote_sandbox_is_available(provider: str) -> bool: - """Whether the resolved config can build the ``modal`` / ``modal_vm`` / ``e2b`` sandbox + """Whether the resolved config can build the ``modal`` / ``modal_vm`` / ``e2b`` / ``sail_vm`` sandbox provider, credentials included. - Modal: any failure to resolve the credentials answers False. E2B: building the provider - resolves ``[sandbox.providers.e2b.config]`` and its ``secret:`` references; only a + Modal: any failure to resolve the credentials answers False. E2B and Sail: building the provider + resolves ``[sandbox.providers..config]`` and its ``secret:`` references; only a ``ConfigError`` (absent or incomplete config) answers False, any other failure answers True and lets the test fail on the real problem, as ``model_endpoint_is_configured`` does.""" - if provider not in ("modal", "modal_vm", "e2b"): + if provider not in ("modal", "modal_vm", "e2b", "sail_vm"): raise ValueError(f"unknown remote sandbox provider {provider!r}") - if provider == "e2b": + if provider in ("e2b", "sail_vm"): from agent_env.config.errors import ConfigError from agent_env.providers.sandbox_providers.sandbox_provider import build_sandbox_provider try: - build_sandbox_provider("e2b") + build_sandbox_provider(provider) except ConfigError: return False except Exception: diff --git a/uv.lock b/uv.lock index 39317153..babbcf6f 100644 --- a/uv.lock +++ b/uv.lock @@ -83,6 +83,7 @@ dev = [ { name = "pytest-timeout" }, { name = "pytest-xdist" }, { name = "requests" }, + { name = "sail" }, { name = "uvicorn" }, ] explorer = [ @@ -96,11 +97,14 @@ gcp = [ { name = "google-cloud-storage" }, { name = "requests" }, ] +sail = [ + { name = "sail" }, +] [package.metadata] requires-dist = [ { name = "a2a-sdk", specifier = "==0.3.26" }, - { name = "agentenv-framework", extras = ["explorer", "gcp"], marker = "extra == 'dev'" }, + { name = "agentenv-framework", extras = ["explorer", "gcp", "sail"], marker = "extra == 'dev'" }, { name = "agentenv-framework-protocol", editable = "packages/agentenv-protocol" }, { name = "boto3", specifier = ">=1.35.0" }, { name = "certifi", specifier = ">=2024.2.2" }, @@ -131,11 +135,12 @@ requires-dist = [ { name = "python-multipart", specifier = ">=0.0.13" }, { name = "pyyaml", specifier = ">=6.0" }, { name = "requests", marker = "extra == 'gcp'", specifier = ">=2.31" }, + { name = "sail", marker = "extra == 'sail'", specifier = "~=0.12.8" }, { name = "starlette", specifier = ">=0.40" }, { name = "uvicorn", specifier = ">=0.47" }, { name = "uvicorn", marker = "extra == 'explorer'", specifier = ">=0.29" }, ] -provides-extras = ["explorer", "gcp", "dev"] +provides-extras = ["explorer", "gcp", "sail", "dev"] [[package]] name = "agentenv-framework-protocol" @@ -623,6 +628,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/58/50/6c0d534c5f134586a8e1ba4e330569e32f057e33372ae556463212fb4cd3/click-8.5.0-py3-none-any.whl", hash = "sha256:255bc9599cf7748b4b1a446ccc735421bd08a2ae529a8b88597d3de5664ee360", size = 125251, upload-time = "2026-08-26T13:33:12.928Z" }, ] +[[package]] +name = "cloudpickle" +version = "3.1.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/27/fb/576f067976d320f5f0114a8d9fa1215425441bb35627b1993e5afd8111e5/cloudpickle-3.1.2.tar.gz", hash = "sha256:7fda9eb655c9c230dab534f1983763de5835249750e85fbcef43aaa30a9a2414", size = 22330, upload-time = "2025-11-03T09:25:26.604Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/88/39/799be3f2f0f38cc727ee3b4f1445fe6d5e4133064ec2e4115069418a5bb6/cloudpickle-3.1.2-py3-none-any.whl", hash = "sha256:9acb47f6afd73f60dc1df93bb801b472f05ff42fa6c84167d25cb206be1fbf4a", size = 22228, upload-time = "2025-11-03T09:25:25.534Z" }, +] + [[package]] name = "colorama" version = "0.4.6" @@ -2886,6 +2900,22 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/46/5f/4c174edad94f82de888ac00a5ddd8d07b35609b6c94f0bdf4d74af57703e/s3transfer-0.19.0-py3-none-any.whl", hash = "sha256:777cc2415536f1debadb5c2ef7779275d0fc0fe0e042411cdd6caebeb2685262", size = 90101, upload-time = "2026-06-16T19:44:50.439Z" }, ] +[[package]] +name = "sail" +version = "0.12.8" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cloudpickle" }, + { name = "protobuf" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/51/63/8bb772e93ee9bdeab0a68a37b88b567015590468c9474b878e6f3c6fc7db/sail-0.12.8-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:43d12688928065e5dde17e72e7fea65bf49733df0a6f9819bb4760ef91d4b667", size = 15361157, upload-time = "2026-10-06T17:37:52.987Z" }, + { url = "https://files.pythonhosted.org/packages/4c/ee/b138d5fe3be97e3b6285214a76bdf2881eeb31afbab11f2f69618cd2a9ea/sail-0.12.8-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:aac5b20e428b52da2dac65534f06a0af267c67e7dc821bcc160ec7f1557c4696", size = 14649143, upload-time = "2026-10-06T17:37:56.09Z" }, + { url = "https://files.pythonhosted.org/packages/06/d1/8a2c5baf5a3ee9edee66ea65aebe1ee5237401b8b13bf8a3737622d9f094/sail-0.12.8-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8a8dbd1f3659941caf1567859ed7285cedb2b8d5bf64c18395fe6daec635ff91", size = 16525128, upload-time = "2026-10-06T17:37:58.789Z" }, + { url = "https://files.pythonhosted.org/packages/b0/d8/11778c650c67d479f0bea86082a6edbca44b66a6520807798a3509892559/sail-0.12.8-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:9589e149a8cf1f6847042b658b17e3764b1e1366ee5b44c47a83578d496c8022", size = 16246879, upload-time = "2026-10-06T17:38:01.375Z" }, + { url = "https://files.pythonhosted.org/packages/1e/5d/6070a931a2ff9247ad8c548be7b74929262e8dee2fe613bf460dac313a97/sail-0.12.8-cp39-abi3-win_amd64.whl", hash = "sha256:c9602dcc17c69b2d783433d62d207ea30e009c4470627554abc1a5ce3a533d82", size = 13017169, upload-time = "2026-10-06T17:38:04.133Z" }, +] + [[package]] name = "setuptools" version = "84.0.0"