feat(evaluations)!: take tools as a list of EvalTool, with inline definitions

[donei003]

This change lets a developer define a tool in code. The developer does not need to create it in LaunchDarkly first. It also changes the public API of run() and init_evaluations(). This is a breaking change.

Breaking changes

• tools is now a list of EvalTool. It was a key -> callable map.
• project_key is no longer an argument of run(). Set it in init_evaluations(project_key=...) or with LD_PROJECT_KEY.
• The init_evaluations argument api_token is now api_key. There is no alias. The environment variable is still LD_API_TOKEN.

Changes

• New exported EvalTool. It is a frozen dataclass with key, implementation, schema and description. schema is required. {} is valid. description defaults to an empty string.
• source, version and project_key are not constructor arguments. A constructed EvalTool is always inline. Only evals.tools.get() makes a library tool.
• await evals.tools.get(key, implementation=...) reads a library tool, pins its version, and records its project. The read runs in a worker thread and does not block the event loop.
• run() sends no request to ai-tools. A library tool is read once by tools.get().
• validate_tools runs before any network I/O. It rejects a blank or uppercase key, a schema that is not a JSON object, a schema that is not JSON-serializable (NaN and Infinity included), a non-callable implementation, a repeated key (compared without case), a NativeTool on an inline tool, a library tool with no project, and a library tool from a different project.
• The evaluation-create body sends {key, schema, description, source: "inline"} for an inline tool and {key, version, source: "library"} for a library tool. tools is omitted when the list is empty.
• tools=[] replaces the tools of an AI Config variation. tools=None uses the variation tools, and each one needs an implementation.
• Handlers receive the same {key: callable} map for both kinds of tool.
• The client README keeps its lazy-initialization section. The run() examples no longer pass project_key.

Verification

• uv run pytest: 1421 passed, 11 skipped.
• uv run ruff check . and uv run ruff format --check .: clean.
• uv run mypy packages/*/src: no issues in 52 source files.

I did not test against a real gonfalon proxy or ai-evaluator. The source field and the inline body are asserted only against the recording fake transport.

Dependencies

• Requires the gonfalon proxy and API change (gonfalon#72707) to be deployed first.
• The cross-language spec is ai-sdks-monorepo TESTING.md section 8.

🤖 Generated with Claude Code

---

Note

Overview
This is a breaking refactor of the evaluations harness API for tools and project scoping.

init_evaluations now requires project_key (or LD_PROJECT_KEY) and renames api_token → api_key (env var stays LD_API_TOKEN). run() no longer accepts project_key.

Tools move from a dict[str, callable] to a list[EvalTool]. New exported EvalTool supports inline definitions (key, implementation, required JSON schema, optional description) without a LaunchDarkly AI library entry. await evals.tools.get(key, implementation=...) fetches library tools, pins version, and runs the GET in a worker thread. run() does not call the tool API; library tools must be resolved via tools.get() first.

Validation runs before any network I/O (keys, schemas, duplicates, cross-project library tools, NativeTool only on library tools). Evaluation-create payloads send source: "inline" with schema/description or source: "library" with version. tools=[] replaces an AI Config variation’s tools; omitting tools still requires implementations for variation-attached tools.

Docs and tests are updated throughout; LDApiClient uses api_key internally.

^{Reviewed by Cursor Bugbot for commit 4b8a819. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 2 potential issues.

3 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

[devin-ai-integration]

🔴 Library tool lookup blocks event loop

When async callers invoke ToolsClient.get, its synchronous GET blocks their event loop. Previously run moved tool reads to a worker thread, so slow lookups now stall unrelated requests.

Learn more

The management API client uses a synchronous urllib transport with a 30-second timeout and retry delays LDApiClient. The previous tool-resolution call ran inside asyncio.to_thread in run(). With the new public ToolsClient.get, an application calling it from an async request or async startup blocks the event-loop thread while the GET waits and retries. No other coroutine on that loop can progress until the lookup returns.

Example: An async request calls evals.tools.get("lookup_order", implementation=lookup_order) while the management API takes 30 seconds to respond. Every other request on the same loop waits those 30 seconds, even if it does not use evaluations.

Recommended fix: Offer an awaitable tool lookup that delegates the synchronous request to asyncio.to_thread, or implement an asynchronous management transport. Keep the version pinning and immediate error behavior when callers await the lookup.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[aknight-ld]

[nit] is this decorator redundant given the one beneath?

[jeffdupont]

Reviewed with the 1.0 freeze in mind, against the spec in launchdarkly/ai-sdks-monorepo#24 (my review there covers the spec side). Tests pass at a9c40bc: make test 1417 passed, 11 skipped, exit 0, and make typecheck and make lint are clean. Merged onto current main (fee904a, 7 commits past your base), it merges cleanly and still passes: 1421 passed, typecheck clean.

The behaviour matches #24 everywhere I checked. run() issues no /ai-tools request, tools.get() pins the version and records the project, the create-body shapes are right, the cross-project guard works, and bad lists fail with zero requests. There's no JS counterpart. That's expected, since the GA plan defers evals-from-code in JS past 1.0.

Four things I'd like settled before GA, because each one is a public signature:

1. The root name Tool. It's new in __all__, and in JS the root Tool is already the AI Config tool definition (index.ts:78). So both SDKs would export a root Tool with different meanings, and Python couldn't later add the AI Config Tool (types.py:79) under its JS name. I'd rename it EvalTool. Separately, my monorepo #33 moves the evaluations names to launchdarkly_ai_server.experimental.evaluations, so adding a root name now means one more to move later.
2. Tool is mutable, so "constructed means inline" doesn't hold. Reproduced at a9c40bc: t = Tool(key="search_docs", implementation=f); t.source = "library"; t.version = 99 passes validate_tools([t], "proj") and sends {"key": "search_docs", "version": 99, "source": "library"}. Its project_key is None, and the guard at tools.py:150-153 skips that case. Making the class frozen=True after 1.0 would break anyone who assigns to it, so now is the time. Inline below.
3. tools.get() blocks the event loop. Devin raised this. The docstring now says not to call it inside a running loop, but run() is async, so that's exactly where callers will be, including the README example. Making it async later is a breaking change. I'd make it awaitable now.
4. schema is optional here but required in the spec. Tool(key="a", implementation=f) validates and sends "schema": {}. §8.4 says a missing schema throws. Requiring it later is breaking, while relaxing it later is additive, so I'd drop the default.

Smaller notes:

• This breaks existing callers: run(project_key=...) is gone, init_evaluations(api_token=...) is now api_key= with no alias, and tools went from a map to a list. All of that is fine before 1.0. But the PR title has no !, and squash merges here use the PR title and body as the commit, so release-please won't flag the break in the changelog. feat(evaluations)!: ... would. The title also still says "inline tool definitions", which undersells the change.
• The description still describes the earlier InlineTool-in-a-map design: the reviewer decision, the test names, the 1318-test count. It becomes the squash commit body, so it's worth refreshing.
• packages/client/README.md: the hunk at line 165 replaces the lazy-initialization section and the init_client / get_client / shutdown / inspect_config table instead of adding beside them. That looks accidental. The same README still passes project_key= to run() at lines 64 and 100, which now raises TypeError, and line 81 still says "project_key is supplied per run rather than during initialization".
• Interaction with #127 (eval rows defined in code): both PRs change run()'s signature, and merging the two heads conflicts in __init__.py, api.py, module.py and runner.py (git merge-tree). #127 keeps project_key on run() and names its type InlineDatasetRow, while this PR drops InlineTool in favour of "constructed means inline". I'd agree one naming convention for both before either merges.
• Release order: inline entries and source depend on gonfalon#72707, which is approved but not merged. I haven't checked how the current API treats source on a library entry, so I can't say whether library-only runs keep working if this ships first.
• The lowercase check also runs on tools.get(), so a library tool the API accepted as Search_Docs can't be used from code. More on that on #24.
• dataclasses.replace(library_tool, implementation=g) returns an inline tool carrying the library schema (source, version and project_key reset because they're init=False), and that skips the project guard. It's rare, but worth a line in the docstring.
• On aknight's types.py:60 nit: the extra @dataclass leaves AIConfig.__dataclass_params__.frozen as False. Instances still refuse assignment and stay hashable (I checked), so deleting the line is safe.

[jeffdupont]

Adding Tool here freezes it at 1.0. In JS, the root Tool is the AI Config tool definition (index.ts:78, types.ts:88). Python has the same type unexported at types.py:79. If this ships as is, the same root name means two different things across the SDKs, and Python can't adopt the JS name later. EvalTool would avoid both problems. If monorepo #33 lands, this moves to experimental.evaluations anyway.

[jeffdupont]

init=False keeps these out of the constructor, but a caller can still assign to them. t.source = "library"; t.version = 99 on a constructed tool passes validate_tools and goes out as a library entry (reproduced). I'd make the class @dataclass(frozen=True) and set these three in _library with object.__setattr__. That also stops schema being reassigned after validation, though the dict itself can still be mutated in place.

[jeffdupont]

A library tool always has a project_key when it comes from tools.get(), so None here means the tool was built some other way. Skipping the check in that case is what lets a forged library tool through. I'd treat source == "library" with no project_key as an error, and drop the project_key is not None condition on the module side too, since run() always passes one.

[jeffdupont]

Callers are inside an event loop when they set up a run, because run() is awaited, and this blocks the loop for the 30 s timeout plus up to 3 retries. async def get(...) that does await asyncio.to_thread(self._api.get, path) keeps the same validation-first behaviour and matches what run() already does for every other management call. That's easy to change now and breaking after 1.0.

[jeffdupont]

§8.4 in monorepo #24 says schema is required on an inline tool, with {} legal but absence not. With this default, Tool(key="a", implementation=f) is accepted and sends "schema": {}. _library always passes schema, so making it required (no default) only affects inline construction. description keeps its default.

[jeffdupont]

This hunk replaces the "lazy initialization" section and the init_client / get_client / shutdown / inspect_config table that were here on main. They aren't anywhere else in the file now. I think the new section was meant to go next to them, not in their place. Separately, lines 64 and 100 still pass project_key= to run(), and line 81 says it's supplied per run.