Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions clients/cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -301,6 +301,8 @@ Exit codes are unchanged: a `tools/call` that returns `isError:true` is still wr
all — at one tool and keeps calling it, so seeing how a server behaves under
concurrent or sustained traffic does not mean rewriting that setup in a
separate script ([#2439](https://github.com/modelcontextprotocol/inspector/issues/2439)).
`mcpdo` offers the same mode against an open connection — see
[`tools/call --load`](../mcpdo/README.md#load-testing-a-tool-toolscall---load).

```bash
# 20 calls in flight for 30 seconds
Expand Down
102 changes: 101 additions & 1 deletion clients/cli/__tests__/load-test.test.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { describe, it, expect } from "vitest";
import { describe, it, expect, vi } from "vitest";
import { runCli } from "./helpers/cli-runner.js";
import {
expectCliSuccess,
Expand Down Expand Up @@ -394,6 +394,106 @@ describe("load-test engine", () => {
expect(summary.elapsedMs).toBe(1500);
});

it("starts no further call once its signal aborts", async () => {
const abort = new AbortController();
let calls = 0;
const call = async (): Promise<LoadCallOutcome> => {
if (++calls === 3) abort.abort();
return "ok";
};
// A rate, so the waits between those calls end on their timers.
const summary = await runLoad(
call,
{ concurrency: 1, requests: 100, rate: 1000 },
fakeClock(),
abort.signal,
);
expect(summary.requests).toBe(3);
});

it("cuts a rate-limit wait short when its signal aborts", async () => {
const abort = new AbortController();
// A timer that never fires: only the abort can end the wait.
const clock: LoadClock = {
now: () => 1000,
sleep: () => new Promise<void>(() => {}),
};
const run = runLoad(
async () => "ok",
{ concurrency: 1, requests: 5, rate: 1 },
clock,
abort.signal,
);
await new Promise((r) => setImmediate(r));
abort.abort();
expect((await run).requests).toBe(1);
});

it("does not run a claimed slot once another worker aborted", async () => {
const abort = new AbortController();
let calls = 0;
const call = async (): Promise<LoadCallOutcome> => {
calls++;
abort.abort();
return "ok";
};
await runLoad(
call,
{ concurrency: 2, requests: 10 },
fakeClock(),
abort.signal,
);
expect(calls).toBe(1);
});

it("drops the real clock's rate timer when the run aborts", async () => {
vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] });
try {
const abort = new AbortController();
// Default (real) clock: the second slot is 1000s away.
const run = runLoad(
async () => "ok",
{ concurrency: 1, requests: 2, rate: 0.001 },
undefined,
abort.signal,
);
await vi.waitFor(() => expect(vi.getTimerCount()).toBe(1));
abort.abort();
expect((await run).requests).toBe(1);
expect(vi.getTimerCount()).toBe(0);
} finally {
vi.useRealTimers();
}
});

it("lets the event loop run while calls fail instantly", async () => {
const abort = new AbortController();
// Only a macrotask can abort: if the run never yields, this never fires
// before the run has spent its whole request count.
setImmediate(() => abort.abort());
const limit = 200_000;
const summary = await runLoad(
async () => {
throw new Error("rejected up front");
},
{ concurrency: 4, requests: limit },
undefined,
abort.signal,
);
expect(summary.requests).toBeGreaterThan(0);
expect(summary.requests).toBeLessThan(limit);
});

it("starts nothing when its signal has already aborted", async () => {
const summary = await runLoad(
async () => "ok",
{ concurrency: 2, requests: 5 },
fakeClock(),
AbortSignal.abort(),
);
expect(summary.requests).toBe(0);
});

it("reports an empty run without dividing by zero", async () => {
const summary = await runLoad(
async () => "ok",
Expand Down
57 changes: 57 additions & 0 deletions clients/mcpdo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ mcpdo auth/clear --all --yes
mcpdo tools/list
mcpdo tools/call echo message:=hi
mcpdo tools/call echo '{"message":"hi"}'
mcpdo tools/call echo message:=hi --load --load-requests 500 # load test; see below
mcpdo @test-stdio resources/list
mcpdo logging/tail # long-lived; Ctrl-C to stop
mcpdo connections/list
Expand Down Expand Up @@ -221,6 +222,62 @@ mcpdo connect my-server --config path/to/mcp.json --elicit off
mcpdo connect https://example.com/mcp --elicit url
```

## Load testing a tool (`tools/call --load`)

`tools/call --load` is the one-shot CLI's
[load mode](../cli/README.md#load-testing-a-tool---load), run against an open
connection ([#2744](https://github.com/modelcontextprotocol/inspector/issues/2744)).
It calls one tool repeatedly and concurrently and prints a single summary in
place of a result: `ok`, `toolErrors`, `errors`, `errorRate`, `throughput`,
latency `min`/`mean`/`p50`/`p90`/`p99`/`max`, and the most frequent error
messages. Both CLIs run the same engine, so the flags, their defaults and the
summary fields mean the same thing in each.

```bash
# 20 calls in flight for 30 seconds
mcpdo tools/call search query:=mcp --load --load-concurrency 20 --load-duration 30

# exactly 500 calls, started at no more than 50 per second, as JSON
mcpdo @prod tools/call search query:=mcp --load --load-requests 500 --load-rate 50 --format json
```

| Flag | Meaning |
| ------------------------ | ------------------------------------------------------------------------------------------------------------ |
| `--load` | Load-test the tool instead of calling it once. |
| `--load-concurrency <n>` | Most calls in flight at once (positive integer; default `10`). |
| `--load-duration <sec>` | Stop starting calls after this many seconds; in-flight calls finish. Default `10` without `--load-requests`. |
| `--load-requests <n>` | Stop after starting this many calls. With `--load-duration` too, whichever comes first ends the run. |
| `--load-rate <n>` | Start at most this many calls per second, spaced evenly (default: unlimited). |

Each `--load-*` flag requires `--load`, and `--load` cannot be combined with
`--app-info`. `--format json` prints the summary object itself, with no
`{ result }` envelope, as for every other mcpdo command. The exit code is `0`
whenever the run completes, however many calls failed — read `errorRate`.

How a run behaves on a shared daemon connection:

- **The run happens inside the daemon**, on the connection's live client — its
transport, headers and OAuth tokens — and the `mcpdo` process makes one round
trip for the whole run. The IPC call has no client-side deadline, so a long
`--load-duration` is not cut off; the connection's own request timeout still
applies to each call.
- **Later commands on the same connection wait for it.** The daemon runs one
`rpc` command per connection at a time (so that an elicitation can be routed
to the call that raised it), and the load run is one such command. A
`tools/list` issued mid-run starts when the run ends. Other connections, and
an already-running `logging/tail` or `resources/subscribe` stream, are
unaffected.
- **Ctrl-C ends the run**: no further calls start and those in flight are
cancelled, so the connection is free for the next command at once. No summary
is printed, since the caller has gone.
- **A dropped connection ends the run early** with the summary of what
completed, rather than spending the rest of the window counting instant
failures. The next command revives the connection as usual.
- **Elicitations are cancelled.** Many calls are in flight at once and none of
them can be answered mid-run, so a server that elicits during a load run sees
each request cancelled. Use `connect --elicit off` to keep the server from
asking at all.

## Relation to one-shot CLI

| | One-shot | Connection (`mcpdo`) |
Expand Down
Loading
Loading