Skip to content

perf.md

Latest commit

 

History

History
198 lines (154 loc) · 9.73 KB

perf.md

File metadata and controls

198 lines (154 loc) · 9.73 KB
Raw

Performance gate, load test & profiling (Step 4.6)

rag_gateway.perf + tests/perf/ + eval/gateway_load_v0/ + ragctl perf make the gateway's latency a measured, enforced contract.

Overview

The gateway is the Phase 2 pipeline over swappable backends. With the default noop backends there is no real I/O, so per-request wall time is the gateway's own overhead (FastAPI routing, middleware, query understanding, routing/fusion, retrieval-cache lookup, Pydantic (de)serialisation). Step 4.6 measures that overhead and gates it:

  • The gate — server-side gateway.request_duration_ms p99 ≤ 30 ms per hot route, in-process, no infrastructure. This is distinct from the end-to-end with real backends budget (≤ 250 ms SaaS / ≤ 500 ms dev) — see budgets.
  • The load test — an in-process concurrent harness (deterministic, CI-friendly) plus a Locust file for real over-the-wire load.
  • The profiler — a cProfile breakdown of the async pipeline by own CPU time.

See ADR-0025 and architecture/latency.md.

Usage

The gate

task perf            # uv run pytest tests/perf/ -v -m perf
# or:
uv run pytest tests/perf/ -v -m perf

It builds the noop-backed app, drives warmup + measured requests at /v1/retrieve, /v1/query, and /healthz, and asserts each route's server-side p99 ≤ budget. Tunable via env vars (no source edit):

Env var Default Meaning
RAG_PERF_REQUESTS 300 Measured requests (round-robin across routes).
RAG_PERF_WARMUP 30 Warmup requests fired + discarded first.
RAG_PERF_P99_BUDGET_MS 30 Server-side p99 budget per route.
RAG_PERF_ATTEMPTS 2 Accept the first clean run; fail only if all exceed budget.

The gate is perf-marked, excluded from the cross-OS unit sweep (-m "not perf") and enforced by a dedicated single-runner CI job (perf-gate) where timing is stable.

ragctl perf — one-command local probe

ragctl perf                                  # per-route p50/p95/p99 + verdict
ragctl perf --requests 500 --profile         # add the hottest pipeline frames
ragctl perf --concurrency 8                   # stress one event loop (p99 = queueing)
ragctl perf --budget 50                       # override the budget (ms)

It always exits 0 (the verdict is informational; the authoritative gate is tests/perf/).

The load + profile harness

task load-test       # uv run python -m eval.gateway_load_v0.harness
uv run python -m eval.gateway_load_v0.harness --requests 4000 --concurrency 64
uv run python -m eval.gateway_load_v0.harness --no-profile
uv run python -m eval.gateway_load_v0.harness --check    # exit 1 if over budget

Writes eval/gateway_load_v0/report.json + report.md (per-route latency table + the hottest frames). Default concurrency is 1 (the overhead snapshot matching the gate); --concurrency N stresses a single event loop and the report flags that the p99 then reflects queueing, not per-request overhead.

Locust — real over-the-wire load

Locust is not a project dependency — install it ad hoc:

uv pip install locust
uv run uvicorn rag_gateway.app:app --port 8000        # in another shell
locust -f eval/gateway_load_v0/locustfile.py --host http://localhost:8000 \
    --headless -u 50 -r 10 -t 30s

Drives the hot read paths in a 3:2:1 (retrieve:query:healthz) mix.

Published budgets

tests/contract/budgets.py is the registry performance.md references — the per-SPI p99 table (dev + SaaS profiles) plus the gateway budgets:

Budget Value Enforced by
Gateway overhead p99 (noop backends) 30 ms tests/perf/ (this step)
Gateway end-to-end p99 (real backends, cache miss) 500 ms dev / 250 ms SaaS documented target (backend-bound)
Per-SPI methods (Embedder.embed, VectorRetrievalBackend.retrieve_ids, …) see table a backend's own perf harness

The overhead number's source of truth is rag_gateway.perf.DEFAULT_GATEWAY_OVERHEAD_P99_MS; budgets.py mirrors it and a drift test (tests/contract/test_budgets.py) keeps them equal.

API surface

rag_gateway.perf:

Symbol Purpose
async measure_gateway_overhead(*, app=None, scenarios=None, requests, concurrency=1, warmup, budget_ms) -> LatencyReport Drive the app in-process; summarise per-route server + client latency.
profile_gateway(*, app=None, scenarios=None, requests, warmup, top_n) -> list[ProfileEntry] cProfile a sequential batch (warmup outside the profiler); hottest frames by own time.
default_scenarios(...) -> list[Scenario] The default probe mix (/v1/retrieve, /v1/query, /healthz).
LatencyReport routes, requests, concurrency, budget_ms, profile; .within_budget(), .worst_server_p99_ms(), .route(name), .as_dict().
RouteLatency server + client p50/p95/p99/max/mean for one route; .within_budget.
ProfileEntry one profile row (function, ncalls, tottime_ms, cumtime_ms).
Scenario one request shape (name, method, path, body, route_label).
DEFAULT_GATEWAY_OVERHEAD_P99_MS / DEFAULT_REQUESTS / DEFAULT_WARMUP Defaults.

Internals

  • Server-side metric. After warmup the app's MetricsCollector is reset, then the measured load runs; the gate reads the gateway.request_duration_ms histogram (route + method labels) the Step 3.11 middleware records. Percentiles are nearest-rank.
  • Sequential vs concurrent. concurrency=1 measures pure per-request overhead; higher values drive one event loop where requests queue on the GIL (latency-under-load, not overhead).
  • Profiling. Warmup runs outside the profiler so one-time import path-scanning (posix.stat / importlib) doesn't mask steady-state frames; entries sort by own time (tottime) so the asyncio loop doesn't dominate the top.

Extension points

  • Add a probed route: pass a custom scenarios=[Scenario(...)] to measure_gateway_overhead / profile_gateway (the harness + ragctl use the default mix).
  • Profile a richer pipeline: pass an app=build_app(retrieval_router=<seeded router>) so retrieval returns chunks and rerank/pack run (the default empty corpus short-circuits them).
  • Real-backend budgets: a backend implementation consults tests/contract/budgets.py (spi_budget(spi, method)) from its own perf harness.

Chaos under load (Step 7.1)

eval/gateway_chaos_v0/ adds the resilience gate alongside the latency one: it drives the in-process gateway under concurrent load while injecting backend faults and asserts graceful degradation — the circuit breaker opens on the failing backend, the fallback ladder keeps serving, and no request 5xx-es.

task chaos-test                                              # python -m eval.gateway_chaos_v0.harness --check
uv run python -m eval.gateway_chaos_v0.harness --all-fail    # worst case → graceful empty answers
pytest -m perf tests/perf/test_chaos_under_load.py            # the gate
  • FaultSpec (failure probability + injected latency, seeded) + Chaos{Vector, Keyword,Graph}RetrievalBackend — SPI wrappers (like the breaker wrappers) that inject faults on the read path, wired behind real breakers in the harness.
  • The asserted property is timing-independent (no 5xx, 100% success, breaker opens), so it's a deterministic CI gate; the p99 budget is the harness --check.
  • The harness reuses the Phase-4 breakers + fallback — it validates them, it builds no new resilience.

The 1000-QPS / p99 < 500 ms acceptance is a cluster run (the Locust v1 suite in eval/gateway_load_v0/locustfile.py), documented in guides/load-testing.md — not a CI gate. See ADR-0043.

Chaos kill-matrix (Step 7.2)

eval/gateway_chaos_v0/kill_matrix.py + tests/perf/test_chaos_kill_matrix.py extend the 7.1 chaos gate from the three retrieval backends to the full hot-path backend set, proving the GA bar: no single backend failure can 5xx the gateway.

task chaos-kill                                              # kill_matrix --check
uv run python -m eval.gateway_chaos_v0.kill_matrix           # writes kill_matrix.{json,md}
uv run python -m eval.gateway_chaos_v0.kill_matrix --target reranker
pytest -m perf tests/perf/test_chaos_kill_matrix.py          # the gate

It kills each of vector / keyword / graph / embedder / retrieval_cache / reranker / llm in turn (100 % unavailable) behind the real breakers + fallback, drives /v1/query, and asserts no 5xx, 100 % success, the on-path retrieval breaker opens, and the expected degraded shape. A seeded keyword corpus + the store's real hydrate make rerank/pack/generate run, so every kill exercises a real degrade path.

Two gaps were hardened (chaos found them): the gateway now degrades-open on a retrieval-cache outage (treats it as a miss — gateway.cache.degraded, on /v1/query + /v1/retrieve) and on a reranker outage (falls back to retrieval-only — gateway.rerank.degraded, honouring the RerankPipeline's documented "caller decides" contract). The LLM (answer-generation guard) and embedder (understanding's per-component catch) already degraded.

The graph backend is only consulted when the caller supplies graph seeds, so a standard query never calls it — its kill is survivable by construction (no breaker-open required).

Cluster-level chaos (LitmusChaos pod-delete + backend network faults) is the runbook in infra/chaos/ + guides/chaos-engineering.md — not a CI gate. See ADR-0044.