Skip to content

gateway.md

Latest commit

 

History

History
349 lines (269 loc) · 15.9 KB

gateway.md

File metadata and controls

349 lines (269 loc) · 15.9 KB
Raw

rag-gateway — Gateway service (Step 3.1)

rag-gateway 0.4 ships the 🥈 Curl-able RAG milestone: a FastAPI service exposing the full Phase 2 retrieval pipeline as REST endpoints with OpenAPI 3.1 auto-generated docs.

Overview

Route Purpose
GET /healthz Public liveness probe.
GET /readyz Public readiness probe — wiring completes in the app factory, so ready means fully constructed and routable.
GET /v1/info Public service descriptor (version + endpoint list).
POST /v1/ingest/document Multipart document ingest (Step 1.10).
POST /v1/query Full RAG: understanding → router → optional rerank / pack / LLM generate.
POST /v1/retrieve Retrieval-only subset (no rerank / pack / generate).
GET /v1/corpora List corpora visible to the calling tenant.
GET /v1/corpora/{id} Describe one corpus.
POST /v1/corpora Create a corpus for the calling tenant (admin surface, Step 3.10).
PATCH /v1/corpora/{id} Update a corpus's mutable fields (admin surface).
DELETE /v1/corpora/{id} Delete a corpus owned by the calling tenant (admin surface, Step 3.10).
GET /openapi.json Auto-generated OpenAPI 3.1 spec.
GET /docs Swagger UI.
GET /redoc ReDoc UI.

Quick start

# Start the gateway against the dev / noop wiring.
uv run uvicorn rag_gateway.app:app --host 0.0.0.0 --port 8000

# Query it.
curl -s http://localhost:8000/v1/query \
  -H 'content-type: application/json' \
  -d '{
    "tenant_id": "demo",
    "principal_id": "curl",
    "query": "what is RAG?",
    "top_k": 5
  }' | jq

Full walkthrough — including ragctl ingest then curl /v1/query against a seeded corpus — in docs/guides/curl-quickstart.md.

POST /v1/query

End-to-end RAG query. Composes QueryUnderstandingPipeline, RetrievalRouter, RerankPipeline, ContextPacker, and (optionally) the wired LLM into a single request.

Request body — QueryRequest

Field Type Default Description
tenant_id string required Tenant scope. Every downstream SPI sees this tenant.
principal_id string required Acting principal id. Becomes part of the RequestContext.
query string (min_length=1) required The user's question or search input.
corpus_ids string[] [] Restrict retrieval to these corpora. Empty = every visible corpus.
top_k int (1..100) 10 Maximum chunks returned from retrieval.
filters object {} Free-form FilterExpr-equivalent dict; translated by the active backend.
request_id string auto Caller-supplied request id; auto-generated if absent. Echoed in X-Request-Id response header.
rerank bool true Run the cross-encoder reranker (Step 2.7).
pack bool true Apply ContextPacker budget-fit + reorder (Step 2.8).
pack_budget int (64..128_000) 2048 Token budget for the packer.
generate bool false Call the wired LLM to produce an answer. Default off so the gateway runs creds-free.
generate_max_tokens int (1..32_000) 1024 Cap on output tokens for the LLM call.
generate_temperature float (0..2) 0.0 LLM sampling temperature.

Response — QueryResponse

Field Type Notes
request_id string The id echoed in X-Request-Id.
query string Echoed input.
decision RoutingDecision Which backends ran + per-call weights.
chunks Chunk[] Hydrated reranked chunks. Empty when rerank=false.
chunk_refs ChunkRef[] Router output. Always populated when retrieval ran.
citations Citation[] Per-chunk excerpts ordered by position-in-context.
packed PackedContext | null Present iff pack=true.
answer Answer | null Present iff generate=true and the LLM call succeeded. See Answer generation.
timings StageTimings Wall-time per stage (ms); null for stages that didn't run.
trace TraceContext OTel trace info (propagated from traceparent if present).

Error responses

HTTP Code Trigger
400 value_error Pydantic validation, invalid top_k, etc.
401 auth_error Missing / invalid auth (admin routes; the body-driven routes ignore auth).
403 acl_denied PolicyEngine denied a chunk read or egress.
422 (FastAPI default) Request body validation failure with field paths.
429 rate_limit_error Per-tenant rate limit exceeded.
502 retrieval_error Downstream retrieval backend failure.
500 various Unhandled error; rare and indicates a bug.

4xx / 5xx bodies use the GatewayError envelope:

{
  "error": {
    "code": "retrieval_error",
    "message": "vector store unavailable",
    "request_id": "abc-123",
    "trace_id": "0123456789abcdef...",
    "context": {}
  }
}

Answer generation

When generate=true, the gateway:

  1. Consults PolicyEngine with the egress_text decision over the packed chunks. A deny outcome short-circuits and returns answer=null (the chunks + citations are still in the response); a transform outcome substitutes the transformed chunks before generation.
  2. Builds a numbered context block — each packed chunk gets a 1-based index that matches the citation positions.
  3. Calls LLM.complete with a fixed system prompt instructing the model to cite by [1] / [2] / etc.
  4. Returns an Answer with the model output text + citation list (one citation per chunk that was in the prompt) + token-usage metadata.

LLM failures degrade silently: the response still carries chunks + citations + packed context, just with answer=null. The structured-log event gateway.answer.failed records the failure for operator dashboards.

POST /v1/retrieve

Retrieval-only. Same identifying triple + corpus_ids / top_k / filters as /v1/query; no rerank / pack / generate fields. Returns chunks: ChunkRef[] directly — useful for callers bringing their own reranker.

GET /v1/corpora

List corpora visible to the calling tenant. Tenant is read from X-Tenant-Id header in dev mode or extracted from the bearer token in production.

curl -s http://localhost:8000/v1/corpora \
  -H 'x-tenant-id: demo' \
  -H 'x-principal-id: curl' | jq

Response:

{
  "corpora": [
    {"id": "docs", "tenant_id": "demo", "display_name": "Docs", ...},
    {"id": "wiki", "tenant_id": "demo", "display_name": "Wiki", ...}
  ],
  "total": 2
}

GET /v1/corpora/{corpus_id}

Describe one corpus. Cross-tenant lookups return 404, not 403 — this is deliberate. The response shape doesn't distinguish "not found" from "not visible" so callers can't probe for other tenants' corpora.

POST /v1/corpora

Create a corpus owned by the calling tenant (the admin write surface behind the console's Corpora page, Step 3.10). The body carries only display fields and embedding metadata — id is minted server-side (corpus_…) and tenant_id is taken from the authenticated request, so a caller can neither choose an id nor create into another tenant. document_count starts at 0 and is maintained by the ingest path. Returns 201 with the created Corpus.

curl -s -X POST http://localhost:8000/v1/corpora \
  -H 'content-type: application/json' \
  -H 'x-tenant-id: demo' \
  -H 'x-principal-id: curl' \
  -d '{
    "display_name": "Support KB",
    "description": "Customer support tickets",
    "embedding_model": "text-embedding-3-small",
    "embedding_dimension": 1536
  }' | jq
Field Type Required Notes
display_name string ✓ Human-readable name (min length 1).
description string Defaults to "".
embedding_model string | null Informational — documents the model the corpus is embedded with.
embedding_dimension int | null Embedding vector dimension (≥ 1).
metadata object Free-form tenant metadata.

A blank display_name is rejected with 422; a request without tenant / principal credentials is rejected with 401.

PATCH /v1/corpora/{corpus_id}

Update a corpus's mutable fields — display_name, description, metadata. PATCH semantics: only the fields present in the body change; omitted fields are left untouched. embedding_model / embedding_dimension are immutable (they define the corpus's vector space — changing them would invalidate every embedding already indexed under it), as are id, tenant_id, and document_count. Returns 200 with the updated Corpus. Cross-tenant or unknown ids return 404 — the same invisibility rule as GET, so a caller can neither mutate nor probe for another tenant's corpora.

curl -s -X PATCH http://localhost:8000/v1/corpora/corpus_8f… \
  -H 'content-type: application/json' \
  -H 'x-tenant-id: demo' \
  -H 'x-principal-id: curl' \
  -d '{"display_name": "Support KB (EMEA)"}' | jq

DELETE /v1/corpora/{corpus_id}

Delete a corpus owned by the calling tenant. Returns 204 on success. Cross-tenant or unknown ids return 404 — the same invisibility rule as GET, so a caller can neither delete nor probe for another tenant's corpora.

curl -s -X DELETE http://localhost:8000/v1/corpora/corpus_8f… \
  -H 'x-tenant-id: demo' \
  -H 'x-principal-id: curl' -i

OpenAPI 3.1

The spec is auto-generated by FastAPI at /openapi.json. Routes carry tags, summary, description (from the route docstring), and responses tables for the documented error codes.

Interactive docs:

  • GET /docs — Swagger UI
  • GET /redoc — ReDoc

The spec validates against OpenAPI 3.1 as of FastAPI 0.115+.

Authentication

The gateway middleware (apps/gateway/src/rag_gateway/middleware.py) reads:

Source Mode Notes
Authorization: Bearer <token> Production Resolves to Principal via the wired Auth SPI.
X-Tenant-Id + X-Principal-Id headers Dev / demo Builds a synthetic Principal with no real verification.
Request body tenant_id + principal_id /v1/query + /v1/retrieve These routes accept identity in the body so they're curl-friendly; production deployments should still require Authorization and reject mismatched bodies.

/v1/corpora reads from headers only (not body). In production, wire a real Auth impl that rejects requests without Authorization:

from rag_gateway.app import build_app
from my_app.auth import ProductionAuth

app = build_app(
    auth=ProductionAuth(...),
    default_tenant_id=None,  # disable dev fallback
)

OTel traces

The gateway middleware parses the W3C traceparent header and propagates it into the RequestContext.trace. Each pipeline stage opens its own canonical OTel span:

Span Owner
gateway.query The route handler (wraps the per-stage spans).
gateway.retrieve The retrieve route handler.
query.understand QueryUnderstandingPipeline
retrieve.route RetrievalRouter.execute
rerank RerankPipeline
pack ContextPacker

The gateway.query span carries rag.tenant_id + rag.gateway.{query_chars,rerank,pack,generate,results_n,citations_n,answer_generated,elapsed_ms}.

Structured logging

gateway.query_complete is emitted once per /v1/query call:

{
  "event_kind": "gateway.query_complete",
  "tenant_id": "demo",
  "request_id": "abc-123",
  "shape": "semantic",
  "chunks_n": 5,
  "citations_n": 5,
  "answer_generated": true,
  "elapsed_ms": 42.1
}

Other events: gateway.answer.failed (LLM call failed under generate=true), gateway.answer.policy_denied (PolicyEngine denied the egress check).

ragctl query smoke command

For local smoke testing without a live HTTP listener:

ragctl query "what is RAG?" \
  --tenant demo \
  --top-k 5 \
  --no-rerank --no-pack

Drives the in-process gateway via TestClient against build_app() with the noop wiring. See ragctl.md for the full flag reference.

Production wiring

build_app(...) accepts every pipeline as a keyword argument so production deployments swap noops for real backends:

from rag_gateway.app import build_app
from rag_query.pipeline import QueryUnderstandingPipeline
from rag_retrieval import HybridRetriever, RetrievalRouter, SemanticEmbeddingCache
# ... real backend imports ...

app = build_app(
    understanding=QueryUnderstandingPipeline(
        rewriter=OpenAIQueryRewriter(api_key=...),
        decomposer=LLMQueryDecomposer(...),
        hyde=HyDEGenerator(llm=anthropic_llm, embedder=openai_embedder, embedding_cache=SemanticEmbeddingCache(redis_embedding_cache)),
    ),
    retrieval_router=RetrievalRouter(
        hybrid=HybridRetriever(
            vector_backend=PgVectorStore(...),
            keyword_backend=ElasticsearchKeywordStore(...),
            graph_backend=Neo4jGraphStore(...),
        ),
        embedder=openai_embedder,
    ),
    reranker=production_reranker,
    packer=ContextPacker(),
    llm=anthropic_llm,
    corpus_store=PostgresCorpusStore(...),
    auth=ProductionAuth(...),
    policy_engine=ProductionPolicyEngine(...),
    default_tenant_id=None,
)

See architecture/gateway.md for the design rationale and the dependency injection pattern.

Config-driven serving

rag_gateway.serve:create_app (v1.0.1) is the production serve entrypoint for containers and Helm — a uvicorn factory that picks its wiring from the environment:

RAG_CONFIG_PATH App served
set to a readable rag.yaml build_app_from_config(load(path)) — config-driven backends, corpora, tenants, governance
unset / empty build_app() — noop wiring; every route serves with zero infrastructure
set but missing / invalid fails fast with ConfigError — a typo'd mount crashes the pod loudly instead of silently serving noop wiring in production 
# The gateway image's default CMD — no arguments needed:
uvicorn rag_gateway.serve:create_app --factory --host 0.0.0.0 --port 8000

# Bare container — serves the noop-wired app on :8000:
docker run -p 8000:8000 ghcr.io/officialcodework/agentcontextos/rag-gateway:1.0.1

# Config-driven — mount a rag.yaml and point RAG_CONFIG_PATH at it:
docker run -p 8000:8000 \
  -v "$PWD/rag.yaml:/app/config/rag.yaml:ro" \
  -e RAG_CONFIG_PATH=/app/config/rag.yaml \
  ghcr.io/officialcodework/agentcontextos/rag-gateway:1.0.1

Internals — the Helm flow

The chart wires this end-to-end: it renders .Values.config.ragYaml (or a minimal valid default) into a ConfigMap, mounts it at /app/config/rag.yaml, and sets RAG_CONFIG_PATH to match. ${VAR} / ${VAR:-default} references inside the file are interpolated from the container environment at load time, so secrets stay in Secret-injected env vars (e.g. ${RAG_POSTGRES_PASSWORD}) and never land in the ConfigMap. RagConfig is strict (extra='forbid'): an unknown key fails validation and the pod CrashLoops with the offending field path in the error — by design, so a malformed config is impossible to miss.

Probes: GET /healthz is liveness, GET /readyz is readiness. All wiring happens inside the app factory before uvicorn binds the socket, so a ready response means the app is fully constructed and routable.

Related