diff --git a/README.md b/README.md index 07d04bd2..fb84d2fd 100644 --- a/README.md +++ b/README.md @@ -7,1692 +7,23 @@ agent-env is a Python SDK and CLI for building, deploying and running agentic environments and the tasks that grade agents inside them. Environments are containerized servers that speak the open `agentenv-protocol`; agent-env builds them into versioned images, deploys them behind a gateway, points an agent at them, and scores what the agent did. -## Contents +**Documentation: [www.agentenvframework.com/docs](https://www.agentenvframework.com/docs)** covers environments, artifacts, agents, tasks, the registry and plugins. -1. [What agent-env is](#what-agent-env-is) -2. [Install](#install) -3. [Quickstart: a local environment you can call, then a graded run](#quickstart-a-local-environment-you-can-call-then-a-graded-run) -4. [Build an environment](#build-an-environment) -5. [Register, deploy, connect and load an environment](#register-deploy-connect-and-load-an-environment) -6. [Define a task](#define-a-task) -7. [Agents](#agents) -8. [Reward and verification](#reward-and-verification) -9. [Run tasks: locally, then at scale](#run-tasks-locally-then-at-scale) -10. [Read results, evaluate and iterate](#read-results-evaluate-and-iterate) -11. [Configuration reference](#configuration-reference) -12. [Run on Google Cloud](#run-on-google-cloud) -13. [CLI reference](#cli-reference) -14. [Extend agent-env](#extend-agent-env) -15. [Contribute, release, license](#contribute-release-license) - -## What agent-env is - -agent-env gives you five primitives: an **environment** an agent acts in, an **artifact** that seeds or captures state, a **task** that strings deploy, prompt and grading steps into a DAG, an **agent** that speaks the A2A protocol, and an **eval** that groups tasks. Every primitive is a versioned document in a store you choose; the default store is a local SQLite file, so a laptop with Docker is a complete installation. - -### The environment model - -- **Environment card.** Every environment server publishes `GET /.well-known/agent-env.json`: its `name`, `protocolVersion` (`1.0`), the JSON-RPC data-plane URL, its `capabilities` (`tools`, `operations`, `extensions`) and any `children_environments`. The name comes from the `ENVIRONMENT_NAME` variable (agent-env sets it to the env's registered name), then `@environment_card(name=...)`, then the class name. -- **MCP tools.** `@tool` methods become real MCP tools served over streamable HTTP at `/mcp`. Tools are what the agent calls. -- **Data plane.** `POST /agentenv` is a JSON-RPC endpoint with three operations: `data/reset`, `data/add` and `data/get`. Seeding, loading and exporting world state go through it, so a run can start from a known state and end with an exportable one. This replaces a gym-style `reset()`/`step()` API: the agent acts through tools, and the harness controls state through the data plane. -- **Universe data.** Seed files are registered as versioned artifacts (`EnvironmentArtifact`, bundled into an `EnvironmentUniverseArtifact`) and loaded into a running instance with a reset followed by an add. -- **Snapshots.** A `snapshot_env` step exports the state of a running instance as a new universe artifact. Baking a snapshot into a pre-loaded image is limited to multi-server environments not on a container-mode sandbox, with the built-in local Postgres state backend and an object store that can presign uploads. -- **Gateway.** Every deploy places a gateway in front of the environment. It composes the card (children under `/svc/mcp-/...`), proxies MCP and the data plane, and adds control endpoints: `/state`, `/step`, per-role tool gating (`/tools/disable`, `/tools/enable`), a virtual clock, triggers and an NDJSON `/trajectory` of tool calls. - -### The reward loop in one sentence - -A task deploys an environment and an agent, prompts the agent, runs one or more verifiers over the response and the captured trajectory, and optionally snapshots the resulting environment state; the verifiers write per-criterion results and a score into the run context. The chain is drawn once in [The canonical chain](#the-canonical-chain-the-one-run-diagram). - -### Two packages and the seams - -| Package | What it is | Who installs it | -|---|---|---| -| `agentenv-protocol` (`packages/agentenv-protocol/`) | The wire contract plus the server and agent SDKs: `AgentEnvEnvironment`, the card, data-plane and extension decorators, and the A2A (agent-to-agent protocol) agent framework behind the `agent` extra. Depends only on `pydantic`, `starlette` and `httpx`; `mcp` is imported lazily. | Environment images and agent images. An environment image does not need `agentenv-framework`. | -| `agentenv-framework` (this repository; import package `agent_env`, command `agent-env`) | The runtime and CLI: stores, image builds, the gateway, sandbox providers, task steps, verifiers, evals, the local explorer. | The machine that builds, deploys and runs. | - -Every backend in `agent-env` is a seam: a TOML table naming `impl = "module.path:ClassName"` plus a `config` table. Stores (document, object, image, secret), sandbox providers, state providers, the runner, task steps, artifacts, environment kinds, CLI groups and explorer routes all plug in this way, and a bad pointer fails loudly with `ConfigError` the first time the seam is used. An installed package can also register its types through entry points, with no config at all. Conformance suites exist for the four store seams only. Details are in [Extend agent-env](#extend-agent-env); server and agent authoring depth is in [`packages/agentenv-protocol/README.md`](packages/agentenv-protocol/README.md). - -## Install - -### Prerequisites and trust model - -- **Python 3.11 or newer** for `agentenv-framework` (the protocol package alone accepts 3.10). Verified on CPython 3.11.9, 3.11.14 and 3.12.12; the public CI runs 3.12. -- **uv** for the one-command install below. `pip` works too; see the alternative in [Install the packages](#install-the-packages). -- **Docker** with a running daemon, needed at the point of use: the `local` sandbox, every `put` that builds an image, and the first-run bootstrap of `agent-env up`. The unit tests need neither Docker nor network. -- **A model endpoint** for anything that prompts or judges an agent: any OpenAI-compatible base URL and key, or provider-prefixed model names routed natively. See [Point at a model](#point-at-a-model). -- **Trust model.** The `local` sandbox runs environments and agents as ordinary containers on your Docker daemon, one compose stack per deploy, with host ports published on the machine. Treat it as a development convenience, not a security boundary. The explorer served by `agent-env up` binds `127.0.0.1` only, has no `host` setting, and rejects foreign `Host` headers with HTTP 421. Configuration values hold references (`env:NAME`, `secret:KEY`), not secrets; the run context written to disk strips API keys. agent-env does not copy your AWS credentials into agent or environment containers unless the S3 object store opts in with `share_credentials` (see [Stores and secrets](#stores-and-secrets)). Every HTTP client agent-env makes verifies TLS; there is no setting or variable that turns that off. - -### Install the packages - -Neither package is on PyPI yet. Install both from a clone. The distribution is named `agentenv-framework`; the import package is `agent_env` and the command is `agent-env`. The repository is a uv workspace, so one command installs `agentenv-framework` and `agentenv-protocol` as editable packages with their dependencies from PyPI: - -```bash -git clone agent-env && cd agent-env -uv sync --extra dev -``` - -`uv sync` creates `.venv` (it picked CPython 3.12.12 on the verification host) and installs about 120 packages from PyPI, roughly 250 MB, or 280 MB with `--extra dev`. Plain `uv sync` installs the runtime dependencies only: no `pytest`, no explorer. Add `--extra dev` for the test suite or `--extra explorer` for `agent-env up`. Run the CLI as `.venv/bin/agent-env` or activate the environment. - -The committed `uv.lock` is kept current: the release bump writes both packages' new versions into it, and CI fails a pull request that leaves it stale, so `uv sync --locked --extra dev` installs exactly what it records. - -With plain `pip`, install the protocol package first, because no public index has it and `pip install -e '.[dev]'` on its own fails with `No matching distribution found for agentenv-protocol>=0.1.0`: - -```bash -python3.11 -m venv .venv -.venv/bin/pip install -e ./packages/agentenv-protocol -.venv/bin/pip install -e '.[dev]' -``` - -`make install` runs those two installs (about 50 s, 410 MB); it is what [CONTRIBUTING.md](CONTRIBUTING.md) and CI use. - -### Install the command on its own - -To use `agent-env` rather than develop it, install it as a uv tool from an index that has both packages, then add plugins to it with `agent-env plugin add` (see [Manage plugins](#manage-plugins)): - -```bash -uv tool install agentenv-framework -agent-env --version -``` - -Upgrade it with `uv tool upgrade agentenv-framework`, which keeps the plugins. Running `uv tool install agentenv-framework` again replaces the tool with what that one command names and drops them. An exact `==` pin leaves `uv tool upgrade` nothing to move to, so pin only when you mean to stay. - -### Optional extras and bundled cloud SDKs - -| Extra | Contents | Needed for | -|---|---|---| -| `agentenv-framework[explorer]` | `fastapi`, `uvicorn` | `agent-env up` (refuses to start without it) | -| `agentenv-framework[gcp]` | `google-api-core`, `google-auth`, `google-cloud-secret-manager`, `google-cloud-storage`, `requests` | `GcsObjectStore`, `GcpSecretManagerSecretStore`, `FirestoreMongoDocumentStore`, `GoogleAccessTokenCredentials` | -| `agentenv-framework[dev]` | `explorer` and `gcp` plus `moto`, `psycopg2-binary`, `pytest` and its plugins (`pytest-asyncio`, `pytest-dependency`, `pytest-socket`, `pytest-timeout`, `pytest-xdist`) | running the test suite | -| `agentenv-protocol[agent]` | `a2a-sdk[http-server]`, `uvicorn`, `regex` | authoring and serving your own A2A agent | - -The runtime dependencies of `agentenv-framework` are `agentenv-protocol`, `boto3`, `click`, `httpx`, `litellm` (the 1.96 line), `mcp` (below 2.0), `a2a-sdk` (pinned to 0.3.26), `pydantic`, `pymongo`, `pyyaml`, `modal` and `e2b` (the 2.46 line). The cloud SDKs install every time and stay inert until a config table or a `--sandbox` flag selects them; the default sandbox and stores are local. `google-cloud-storage` and `google-cloud-secret-manager`, and what they pull in, come only with the `gcp` extra; the extra also names `google-api-core`, `google-auth` and `requests`, which core already installs, because the stores import them. - -An extra belongs to `agentenv-framework` itself, so `agent-env plugin add` does not install one: name it where you install agent-env, as in `uv tool install 'agentenv-framework[gcp]'`, `pipx install 'agentenv-framework[gcp]'` or `pip install 'agentenv-framework[gcp]'`. Re-running `uv tool install` drops the plugins it does not name. A store impl whose extra is missing fails to load with an error that names the extra. There is no `asyncpg`; `psycopg2-binary` comes only with the `dev` extra. - -### Verify and platform notes - -```bash -agent-env --help -``` - -``` -Usage: agent-env [OPTIONS] COMMAND [ARGS]... - Agent environment CLI. -Options: - -v, --verbose Enable verbose logging (DEBUG level) - --help Show this message and exit. -Commands: - a2a-agent A2A agent commands. - artifact Artifact commands. - config Inspect the resolved agent-env configuration. - env Environment commands. - eval Eval commands. - plugin Inspect, add and remove installed plugins. - run Run a bundle's tasks and evals, or list the installed bundles. - task Task commands. - up Start the local agent-env stack (stores, runner, explorer)... -``` - -Installed plugins can add groups and root options; they appear in this listing. `agent-env --version` prints the installed version. - -The second check is `agent-env config show`: it prints which config file won, or `(none)`, and what every section resolved to (see [Inspect the resolved configuration](#inspect-the-resolved-configuration)). Running `--help`, `config show`, `config explain`, `config sources`, `config debug`, `plugin list`, `plugin show` or `plugin check` writes nothing to disk, though a plugin's own import code may. The first command that writes a local store (for example `task create` or `run hello`) creates your per-user state root, `~/.local/state/agent-env/` (see [Config discovery and precedence](#config-discovery-and-precedence)). - -The third check runs `hello`, the bundle agent-env ships. It needs `bash` and the usual shell tools, and no Docker, model or configuration. - -```bash -agent-env run # lists the installed bundles and their folders; hello is agent-env's own -agent-env run hello -``` - -``` -artifacts/greeting: v1 (new) -tasks/hello.json: v1 (new) -... -Tasks: - tasks/hello.json v1: passed (hello: 1), 0.1s, instance @local/agentenv-framework/hello/hello-5aii6zzz -``` - -It writes to the per-user local stores, and removes the sandbox it deployed, a work folder under `~/.agent-env-sandboxes/`, when the run ends; [Run a bundle folder](#run-a-bundle-folder) says what it does and how to run an edited copy. - -**Apple Silicon.** Every image-building `put` (`env mcp-server|gateway|service-db|website|website-browser put` and `a2a-agent put`) builds for `linux/amd64` by default, which is the right target for remote VM sandboxes but runs emulated under the `local` sandbox. Pass `--platform linux/arm64` on every `put` you intend to deploy locally, including the two bootstrap environments and the agent image. `agent-env up` has no platform flag and builds its bootstrap images for `linux/amd64`; register those two environments by hand instead (see [Start the local stack](#start-the-local-stack)). Local sandbox work directories live under `~/.agent-env-sandboxes/`; if you override that with `AGENT_ENV_LOCAL_SANDBOX_DIR`, pick a path your Docker Desktop file sharing includes. - -## Quickstart: a local environment you can call, then a graded run - -Steps 2 to 5 run cold on a laptop with Docker: a bundled example environment is built, deployed behind the gateway and called over MCP, then torn down. Step 6 runs a graded task fully locally too, with the bundled echo agent from step 3; it needs the model endpoint from step 1, a task file you write yourself and one workaround, all stated inline. Run every command from one project directory; the local stores live in its `.agentenv/`, and every id a run references must be in the same store. The commands below pass `--platform linux/arm64` because they were verified on Apple Silicon; the flag defaults to `linux/amd64`, so omit it on x86-64 hosts. - -### Point at a model - -Export an OpenAI-compatible endpoint and key. `deploy_agent` injects both into every agent container it starts, and the LLM judge reads them in-process: - -```bash -export LITELLM_BASE_URL=https:///v1 -export LITELLM_API_KEY= -``` - -No endpoint is defaulted. With neither set, a run stops at `deploy_agent` with `ConfigError: No model API key configured`; with only the key set, with `ConfigError: No model endpoint configured`. The check runs when the agent container is created, before any prompt, so even the model-free echo agent from step 3 needs both values to exist. The judge model is defaulted: with nothing configured a `rubrics_verifier` uses `claude-sonnet-4-6`, and its direct judge fails without an endpoint with `ConfigError: No model endpoint configured for 'claude-sonnet-4-6'`. If your endpoint does not serve that name, set `default_model` on the step or `[model.roles] judge` in `.agentenv/config.toml`. The agent's model comes from `--agent-model ` on `task run`, `model` on the `prompt_agent` step, or `[model.roles] agent`; the `[model]` table and the full precedence list are in [Model configuration](#model-configuration-owns-precedence). Steps 2 to 5 do not call a model. - -### Start the local stack - -Copy the all-local example config into the project directory, then check what every command run from here will read: - -```bash -mkdir -p .agentenv -cp /.agentenv/config.example.toml .agentenv/config.toml -agent-env config show -``` - -The first two lines name the file that won and how it was found (`discovered by walking up from the working directory`). Each block after that names a section, its resolved backend and the layer that supplied it: `document: LocalSqliteDocumentStore ... from [stores.document]`, `agents: default_a2a_agent_id=a2a-default from [agents]`, `explorer: port=8234 from [explorer]`. The file is optional: without it every command falls back to the same local defaults and `config show` reports `config: (none)` with every store `from built-in default`. `agent-env config debug` lists the paths discovery checked. Both commands are read-only and create nothing; see [Inspect the resolved configuration](#inspect-the-resolved-configuration). - -Every local deploy resolves two bootstrap environments by id: the service database `default-db` and the gateway `default`. Register them once: every project you run from shares one local store. The first `service-db` build takes minutes: - -```bash -agent-env env service-db put --id default-db --platform linux/arm64 -agent-env env gateway put --id default --platform linux/arm64 -``` - -The first `put` is the first store write: it creates `document_store/documents.db` and `object_store/` under `~/.local/state/agent-env/`, your per-user state root (see [Config discovery and precedence](#config-discovery-and-precedence)), and writes nothing into the project. Images push to a local OCI registry on `127.0.0.1:5000`. The first push starts the `agentenv-registry` container (`registry:2`, `--restart unless-stopped`) that hosts it; nothing is started if a registry already answers there. - -Optional: `agent-env up` registers the same two environments (as `linux/amd64` images; it has no platform flag) and serves the explorer API at `http://127.0.0.1:8234`. It needs the `explorer` extra and the config file copied above, and it runs in the foreground until Ctrl-C, so start it in a second terminal: - -```bash -agent-env up -``` - -`up` prints the resolved backends (all local with this profile), a `bootstrap` block, and `explorer http://127.0.0.1:8234`. The first run builds four images and takes a few minutes; later runs report `already registered` and are ready in about a second. `up` runs the explorer as a host process and starts no environment containers. Nothing in this quickstart calls the explorer; its API, port setting and `--no-bootstrap` are in [The local explorer and runner](#the-local-explorer-and-runner). - -### Register an agent (the bundled echo agent) - -Tasks (see [Define a task](#define-a-task)) deploy an agent by id; the default id is `a2a-default` (`[agents] default_a2a_agent_id` in `.agentenv/config.toml` changes it). An A2A agent is a container that agent-env prompts over the agent-to-agent protocol and configures through the `urn:agentenv:*` extensions; see [Agents](#agents). The repository ships no model-backed agent. It ships the deterministic agent its own test suites deploy, at `tst/data/a2a_agent/` (`Dockerfile`, `agent.py`): it answers `Echo: `, records a trajectory, and calls no tool and no model. Its Dockerfile installs `agentenv-protocol[agent]` from a copy of the protocol package in the build context, so assemble a context and register it under the default id: - -```bash -mkdir echo_agent -cp /tst/data/a2a_agent/{Dockerfile,agent.py} echo_agent/ -cp -R /packages/agentenv-protocol echo_agent/agentenv-protocol -agent-env a2a-agent put --id a2a-default --dockerfile echo_agent/Dockerfile --context echo_agent --skip-validation --platform linux/arm64 -``` - -``` -Building Docker image... -Creating DockerImageArtifact... -Created artifact: id=a2a-agent-a2a-default version=1 -Registering A2A agent... -Created A2A agent: id=a2a-default version=1 image=a2a-agent-a2a-default:1 -``` - -Copy the whole package directory (it needs its `pyproject.toml`), leaving out any `.venv`, `dist` or `__pycache__` inside it. The image is `python:3.12-slim` plus the protocol package; the build took about 20 seconds with cached layers. `--skip-validation` is required on a fresh store (see [Validate an agent](#validate-an-agent)). `agent-env a2a-agent get --id a2a-default` prints the stored document: the image artifact `a2a-agent-a2a-default`, empty `default_env_vars`, and no agent card, because the card is read from the running container. A run with the echo agent proves the pipeline, not an agent: it makes no tool calls, so a rubric that requires one scores 0. To bring your own agent image, see [Register and deploy an A2A agent](#register-and-deploy-an-a2a-agent). - -### Build and register the example environment - -The repository ships an example server at `tst/data/agentenv_mcp/` (`server.py`, `Dockerfile`, `seed.json`): an in-memory item store with two tools, `items_add_item` and `list_items`, plus the data plane. Its Dockerfile copies the protocol package from the build context, so assemble a context first: - -```bash -mkdir items_env -cp -R /packages/agentenv-protocol/src/agentenv_protocol items_env/ -cp /tst/data/agentenv_mcp/{server.py,Dockerfile,seed.json} items_env/ -agent-env env mcp-server put --id --dockerfile items_env/Dockerfile --context items_env --platform linux/arm64 -``` - -``` -Derived environment_name='items' from the environment card. -Building MCP server Docker image... -Creating DockerImageArtifact... -Created artifact: id=mcp-server- version=1 -Creating MCPServerEnv... -Created MCPServerEnv: id= version=1 environment_name=items env_provider_type=gateway -``` - -Passing `--context /tst/data/agentenv_mcp` directly fails at `COPY agentenv_protocol`; that directory is not a self-contained build context. `put` only registers the environment; `--validate` also runs the release gate, which needs a registered `a2a-default` agent and a model endpoint; see [Build, register and the release gate](#build-register-and-the-release-gate). Repeating `put` on the same id appends a new version. The image lands in the local registry as `localhost:5000/mcp-server-:v1`; a `registry:2` container named `agentenv-registry` is started on first push if none is listening. - -### Deploy it and call a tool - -```bash -agent-env env deploy --id --ttl-seconds 1800 -``` - -``` -Sandbox backend: config default -Found env: id= version=1 type=mcp_server -Deploying (ttl=1800s, gateway_mode=performance, disk=10.0GB)... -Deployed! -Instance ID: -Env MCP Url: http://localhost:/mcp -Env Gateway Url: http://localhost: -Env DB Web Url: http://localhost:/ -Env DB MCP Url: http://localhost:/mcp -Expires At (UTC): -``` - -A local deploy took about 30 seconds; `Sandbox backend: config default` says the provider came from `[sandbox] default` in the config file. Keep the instance id: the local sandbox records the TTL but does not enforce it, so you tear down by hand (see [Tear down](#tear-down)); what the deploy started, the deploy flags and how to read the record back are in [Deploy and the instance record](#deploy-and-the-instance-record). Call a tool through the gateway from any MCP client that speaks streamable HTTP (the `mcp` client library is a dependency of agent-env, so no extra install is needed): - -```python -import asyncio -from mcp import ClientSession -from mcp.client.streamable_http import streamablehttp_client - -async def main(): - async with streamablehttp_client("") as (r, w, _): - async with ClientSession(r, w) as s: - await s.initialize() - tools = await s.list_tools() # items_add_item, list_items - res = await s.call_tool("items_add_item", {"item": "from-mcp-client", "times": 2}) # {"count": 2} - -asyncio.run(main()) -``` - -IDE MCP clients take the same URL in their config (`{"mcpServers": {"agent-env": {"url": ""}}}`); no auth header is needed locally. `curl -sS /.well-known/agent-env.json` returns the composed card with `items` under `children_environments`; raw `curl` against `/mcp` must send `Accept: application/json, text/event-stream` or it gets HTTP 406. Loading `seed.json` into the instance and the gateway control endpoints are in [Register, deploy, connect and load an environment](#register-deploy-connect-and-load-an-environment). - -### Create, run and read the example task (gated) - -The repository does not yet ship an example task file, so write this one as `task.json`, substituting your environment id: - -```json -[ - {"id": "deploy-env", "type": "deploy_env", "env_id": "", "ttl_seconds": 1800}, - {"id": "deploy-agent", "type": "deploy_agent", "env_ids": [""], "a2a_agent_id": "a2a-default", "ttl_seconds": 1800}, - {"id": "prompt", "type": "prompt_agent", "prompt_id": "p1", "timeout_seconds": 600, - "prompt": "Using the tools available to you, add one item named 'readme' to the item store, then list all stored items and reply with the exact list of items returned by the tool."}, - {"id": "verify", "type": "rubrics_verifier", "prompt_id": "p1", "verifier_id": "rubric", "use_agent_judge": false, - "output_format": "rubric_binary", "score_aggregator": "all_pass", - "criteria": [{"id": "item_added_and_listed", "weight": 1, - "criterion": "The agent called a tool to add an item named 'readme' and its final response reports the stored items, including 'readme'."}]}, - {"id": "snapshot", "type": "snapshot_env", "env_id": "", "fail_task_on_error": false} -] -``` - -One line is a workaround, which is why this step is gated: `use_agent_judge: false` grades with a direct model call instead of deploying a third sandbox for a judge agent, which here would be the echo agent registered as `a2a-default`; that judge call is the only model call in this chain. The trajectory needs no field: `prompt_agent` stores it below `prompt_agent_trajectories/prompt_id=/` in the configured object store. Then create and run: - -```bash -agent-env task create task.json --id --project-id