Skip to content

status-api.md

Latest commit

 

History

History
191 lines (153 loc) · 7.83 KB

status-api.md

File metadata and controls

191 lines (153 loc) · 7.83 KB
Raw

Status & Metrics API — rag-gateway (Step 3.11)

Operator-facing surface for watching a running gateway: health, metrics, a structured-log tail, and connector status — over REST (poll), SSE (log stream), and WebSocket (live health/metrics push). Backed by two in-process primitives in rag-observability (MetricsCollector, LogTail). No external store, no credentials required in the default gateway.

See also: ADR-0020 (transport choice), architecture/status-metrics-gui.md (design), guides/operator-console-live.md (quickstart), and reference/admin-ui.md (the console pages that consume this).

Overview

Method Path Purpose
GET /v1/status/health Overall + per-component liveness roll-up.
GET /v1/status/metrics Counter / gauge / histogram snapshot + process info.
GET /v1/status/logs Recent structured log records (filterable).
GET /v1/status/logs/stream SSE tail of new log records.
WS /v1/status/ws WebSocket push of health + metrics snapshots.
GET /v1/connectors/status Registered connectors + sync state.

All are added by make_status_router() and included in build_app(). CORS is installed by build_app(enable_cors=True) (default) so a browser console on a different origin can call them; origins come from RAG_GATEWAY_CORS_ORIGINS (comma-separated; default http://localhost:3100,http://127.0.0.1:3100).

Identity. Health and metrics carry no tenant rows and are unauthenticated like /healthz. The logs tail is operator-global and only filterable by tenant. Browser EventSource/WebSocket cannot set headers, so the stream and socket take tenant (and filters) as query params.

Usage

Health — GET /v1/status/health

curl -s localhost:8000/v1/status/health | jq
{
  "status": "up",
  "service": "rag-gateway",
  "version": "0.8.0",
  "uptime_s": 42.7,
  "started_at": "2026-06-04T15:00:00+00:00",
  "checked_at": "2026-06-04T15:00:42.700+00:00",
  "components": [
    {"name": "gateway", "kind": "gateway", "status": "up", "detail": "uptime 43s"},
    {"name": "retrieval router", "kind": "component", "status": "up", "detail": "RetrievalRouter"},
    {"name": "POST /v1/query", "kind": "surface", "status": "up", "detail": "12 reqs · 0% errors"}
  ]
}
  • status is the worst component status (up < degraded < down).
  • kind partitions components: gateway (the process), component (wired SPIs read off app.state), surface (per-route health derived from real traffic — up with no 5xx, degraded under 50% error ratio, down at or above 50%).

Metrics — GET /v1/status/metrics

curl -s localhost:8000/v1/status/metrics | jq '.histograms[0]'
{
  "name": "gateway.request_duration_ms",
  "labels": {"route": "/v1/query", "method": "POST"},
  "count": 12, "sum": 1820.4, "min": 88.1, "max": 342.0,
  "mean": 151.7, "p50": 121.0, "p95": 184.0, "p99": 342.0
}

Recorded by a request-timing middleware for every non-/v1/status HTTP request:

Metric Type Labels
gateway.requests_total counter route, method, status (2xx/4xx/5xx)
gateway.request_duration_ms histogram route, method
gateway.requests_in_flight gauge —
gateway.errors_total counter route, method

Aggregates (count/sum/min/max) are exact; p50/p95/p99 are nearest-rank over a bounded per-series reservoir (default 2048 samples). The status surface is excluded from its own metrics so the long-lived SSE stream does not dominate the latency histogram.

Logs — GET /v1/status/logs

Query params: limit (1–2000, default 200), level (minimum severity), event (event-prefix match), tenant (exact). Returns the most recent matching records in chronological order:

curl -s "localhost:8000/v1/status/logs?level=WARN&limit=50" | jq '.records[-1]'
{"seq": 318, "ts": "2026-06-04T15:00:41.992+00:00", "level": "WARN",
 "module": "webhooks.dispatch", "msg": "delivery retry scheduled",
 "event": "webhook.delivery_retry", "tenant_id": "acme",
 "fields": {"attempt": 2, "http_status": 503}}}

Log stream — GET /v1/status/logs/stream (SSE)

curl -N "localhost:8000/v1/status/logs/stream?level=INFO&backlog=20"
: open
data: {"seq": 300, "ts": "...", "level": "INFO", "event": "gateway.query_complete", ...}
data: {"seq": 301, ...}
: keep-alive

Opens with a : open comment (immediate flush), replays up to backlog recent records (0–1000, default 100), then emits each new record as a data: frame. Query params: level, event, tenant, interval_ms (250–10000, default 1000), backlog. Comments (: lines) are keep-alives the browser ignores.

Live push — WS /v1/status/ws (WebSocket)

const ws = new WebSocket("ws://localhost:8000/v1/status/ws?channels=health,metrics&interval_ms=2000");
ws.onmessage = (e) => {
  const frame = JSON.parse(e.data); // {kind:"snapshot", ts, health?, metrics?}
};

Sends the first frame immediately on connect, then one per interval. Query params: interval_ms (250–10000, default 1000), channels (comma list of health / metrics; default both).

Connector status — GET /v1/connectors/status

{"connectors": [], "total": 0}

Reads an injectable ConnectorStatusStore (in-memory, empty by default — the default gateway has no connector runtime yet). Tenant-scoped when the request carries identity. Populated entries look like:

{"id": "conn_fs_001", "tenant_id": "acme", "name": "Support KB",
 "type": "filesystem", "status": "healthy", "last_run": "2026-06-04T14:46:00+00:00",
 "documents_ingested": 12480, "watermark": "cursor:abc", "detail": "Filesystem"}

Internals

The read-side lives in rag-observability:

  • MetricsCollector (rag_observability.metrics) — thread-safe counter/gauge/histogram registry. incr / set_gauge / adjust_gauge / observe; snapshot() returns a sorted, JSON-ready MetricsSnapshot. get_metrics_collector() returns a process-wide default; build_app gives each app its own collector for test isolation.
  • LogTail + RingBufferLogHandler (rag_observability.logbuffer) — a bounded ring buffer (default 2000) with a monotonic seq per record, and a logging.Handler that mirrors rag.* records into it (matching the _JsonFormatter field extraction). attach_log_tail() attaches the handler idempotently. Streamers call since(cursor) to poll for new records.

The gateway wires both in build_app: it stamps app.state.started_at, app.state.metrics, app.state.log_tail, app.state.connector_status_store; installs the metrics middleware and CORS; and includes make_status_router(). Wire models (HealthResponse, MetricsResponse, LogsResponse, ConnectorsStatusResponse, …) are defined in rag_gateway.status.

Extension points

  • Custom metrics. Call get_metrics_collector().incr(...) / .observe(...) from anywhere; they appear in /v1/status/metrics and the WS push automatically.
  • Connector runtime. Implement ConnectorStatusStore (or reuse InMemoryConnectorStatusStore) and pass it to build_app(connector_status_store=…); a durable ingest runtime records ConnectorStatusViews into it and the endpoint begins serving live status with no surface change.
  • Deeper health. build_health() reads wiring + per-route error rates today; add probe results to the components list there (Phase 4 reliability work).
  • CORS origins. Set RAG_GATEWAY_CORS_ORIGINS (comma-separated; * is honoured and forces allow_credentials=false), or pass build_app(enable_cors=False) to install your own middleware.