Skip to content

experiments.md

Latest commit

 

History

History
181 lines (151 loc) · 9.63 KB

experiments.md

File metadata and controls

181 lines (151 loc) · 9.63 KB
Raw

A/B testing & shadow mode (Step 5.7)

Compare two configs on live traffic — measure a candidate variant and decide with statistics whether it beats the control. See ADR-0032. Slice 5.7a delivers the analyzer + sample collector + dashboard; slice 5.7b adds shadow mode — an observe-only candidate fan-out that feeds the collector from live traffic; slice 5.7c adds A/B routing — deterministically serving the candidate to a fraction of users (the first slice that can change a response).

Overview

  • The analyzer (rag_config.eval.analyze_ab_experiment) is a pure, stdlib-only function: given two sample lists it returns lift + a confidence interval + a significance verdict.
  • The collector (rag_observability.ABExperimentTracker) holds bounded per-(experiment, variant) metric windows. It only holds samples (so rag_observability stays free of a rag_config dependency); the gateway composes it with the analyzer.
  • The surface is GET /v1/status/experiments — one comparison per running experiment.

Usage

from rag_config.eval import analyze_ab_experiment

r = analyze_ab_experiment(
    samples_a=[0.51, 0.49, 0.50, ...],   # control outcomes
    samples_b=[0.78, 0.81, 0.79, ...],   # candidate outcomes
    experiment="routing_v2", metric="satisfaction",
    confidence=0.95, min_samples=30,
)
r.status        # "analyzed" | "insufficient_data"
r.lift          # (mean_b - mean_a) / mean_a   → +0.55
r.diff          # mean_b - mean_a
r.ci_lower, r.ci_upper   # confidence interval on diff
r.p_value
r.significant   # whether the CI excludes 0

ABAnalysisResult (in rag_core.eval) carries experiment / metric / variant_a / variant_b / status / n_a / n_b / mean_a / mean_b / diff / lift / ci_lower / ci_upper / p_value / confidence / significant.

from rag_observability import ABExperimentTracker

tracker = ABExperimentTracker(window_size=500, control="control", candidate="candidate")
tracker.observe("routing_v2", "control", 0.50)     # fed per query (5.7b/c)
tracker.observe("routing_v2", "candidate", 0.80)
tracker.samples("routing_v2", "candidate")          # the window (a copy)

HTTP

GET /v1/status/experiments → { "experiments": [ABAnalysisResult, …], "total": N }. Each entry compares the control against the candidate variant of one experiment. Empty when experiments are disabled or none have been observed.

Console (Step 5.7d)

The admin Live Status page (apps/admin-ui, /status) renders an A/B experiments card over this endpoint: one row per running experiment showing the lift (relative, colour-coded), a verdict badge (candidate wins / candidate worse / no change / collecting), the control→candidate means, the confidence interval, and per-arm sample counts. Live-wired with a seed fallback (the demo badge) when no gateway is configured. See admin-ui.md.

CLI

ragctl experiments --lift 0.25     # feed demo control/candidate samples, print lift + CI
ragctl shadow --lift 0.25          # drive the real ShadowRunner end-to-end, print the A/B verdict
ragctl ab --rate 0.5 --lift 0.25   # drive the real ABRouter — assign, serve, record, analyze

Configuration (cfg.experiments)

Field Default Meaning
enabled false Opt-in — A/B routing (5.7c) can change responses.
window_size 500 Per-(experiment, variant) rolling sample cap.
min_samples 30 Below this on either side → insufficient_data.
confidence 0.95 Confidence level for the interval / significance.
control_variant / candidate_variant control / candidate The two compared labels.
shadow_enabled false Turn on the shadow fan-out (requires enabled).
shadow_sample_rate 0.1 Fraction [0, 1] of live queries shadowed (deterministic per request_id).
shadow_experiment shadow Experiment id the shadow samples land under.
shadow_candidate.{vector,keyword,graph}_weight 1.0 The shadow candidate retriever's RRF fusion weights.
routing_enabled false Turn on A/B routing — serves the candidate (requires enabled).
routing_sample_rate 0.1 Fraction [0, 1] of requests served the candidate (deterministic per request_id); the rest get control.
routing_experiment ab Experiment id the served samples land under.
routing_candidate.{vector,keyword,graph}_weight 1.0 The A/B-routing candidate retriever's RRF fusion weights.

Shadow mode (Step 5.7b)

Shadow mode runs a candidate retriever alongside the served (control) path on a sample of live queries — observe-only, never affecting the response — and feeds both variants' outcome into the tracker, so the dashboard can compare them on real traffic before anyone serves the candidate.

  • Never alters the request. The candidate retrieval is scheduled as a FastAPI BackgroundTask, so it runs after the response is sent; the served path's latency is untouched. It is skipped on a retrieval-cache hit (the control did not retrieve, so there is nothing comparable to measure).
  • Degrade-open. A candidate failure logs experiment.shadow_failed and is swallowed — the response was already sent, so shadowing can never break a query.
  • Deterministic sampling. Whether a query is shadowed is a pure hash of its request_id against shadow_sample_rate, so the decision is reproducible (and testable) with no RNG in the hot path.
  • The outcome metric is the mean retrieval score of the result set (outcome_metric, higher = better) — the same backend-only signal the drift monitors use, needing no served answer (the candidate is never shown). Richer metrics (latency, recall proxies) are a future extension.
  • The candidate is any SupportsRoute. The config-driven build constructs a second RetrievalRouter whose RRF fusion weights come from shadow_candidate, so an operator can shadow-test an alternative balance (e.g. vector-heavy) against the live config; production injects a candidate over the real backends via build_app(shadow_runner=…).

ShadowRunner (in rag_gateway.experiments) ties these together; /v1/query and /v1/retrieve schedule it when one is wired and the request is in the sample. See architecture/ab-shadow-mode.md.

A/B routing (Step 5.7c)

A/B routing deterministically assigns each query to a variant and, for the routing_sample_rate fraction assigned to the candidate, serves the candidate retrieval config to the user — the first slice that changes which response a user gets, so it is gated separately (routing_enabled) from shadow.

  • Serves, not observes. Only the assigned arm runs: the control arm takes the normal served path; the candidate arm is served ABRouter.candidate.route(...) inline. A routed query costs the same as a normal one (one retrieval).
  • Deterministic assignment. The variant is hash(request_id) < routing_sample_rate ? candidate : control — reproducible, sticky per request, testable. rate ≤ 0 routes everyone to control (a no-op); rate ≥ 1 routes everyone to the candidate.
  • Variant-partitioned cache. The retrieval-cache key includes the assigned variant, so the control and candidate arms never share a cached result.
  • Records the served arm. Each fresh retrieval records its served outcome (mean retrieval score) under its arm; over many requests both windows fill and GET /v1/status/experiments analyses them — now from served traffic.
  • Tags the response. QueryResponse.experiment / RetrieveResponse.experiment carry an ExperimentAssignment (experiment / variant / is_candidate) when routing assigned the request; None otherwise.
  • Not degrade-open. The candidate is the served path, so a candidate failure surfaces as a normal retrieval error rather than silently swapping to control.
r = client.post("/v1/query", json={"tenant_id": "t1", "principal_id": "p1", "query": "…"})
r.json()["experiment"]   # {"experiment": "ab", "variant": "candidate", "is_candidate": true} | null

ABRouter (in rag_gateway.experiments) decides + records + tags; the /v1/query and /v1/retrieve handlers own the served-path wiring. See architecture/ab-routing.md.

Internals

  • Normal-approximation Welch test. With min_samples ≥ 30 the CLT makes a normal approximation to an unequal-variance two-sample test sound, so the analyzer uses statistics.NormalDist (stdlib) for the quantile + CDF — no numpy/scipy. significant is whether the CI on the mean difference excludes 0.
  • Decoupled by design. The tracker holds samples; the analyzer is pure; the gateway endpoint composes them. rag_observability never imports rag_config.
  • Opt-in + inert. build_app leaves experiment_tracker + shadow_runner + ab_router unset; build_app_from_config builds the tracker when enabled, the shadow runner only when enabled and shadow_enabled, and the A/B router only when enabled and routing_enabled.

Extension points

  • Feed it — shadow mode (5.7b) calls tracker.observe(...) from a background task; A/B routing (5.7c) feeds it from the served path via ABRouter.record.
  • Swap the test — replace the normal approximation in analyze_ab_experiment with an exact t-test (scipy) or a sequential test without changing the result shape or the endpoint.
  • More metrics — observe under (experiment, variant) for several metrics and analyze each.