Skip to content

drift.md

Latest commit

 

History

History
118 lines (91 loc) · 4.82 KB

drift.md

File metadata and controls

118 lines (91 loc) · 4.82 KB
Raw

Drift monitors — reference (Step 5.5)

rag-drift detects when production retrieval degrades over time by comparing a reference window against a current window for five signals, and alerts (structured event + webhook) when a monitor crosses its threshold.

Overview

Piece Where
PSI primitive (pure) rag_drift.population_stability_index
One monitor rag_drift.DriftMonitor
The five monitors + alerting rag_drift.DriftMonitorRegistry
Domain types rag_core.types (DriftSnapshot / DriftReport / DriftMetric / DriftMethod / DriftStatus)
Event rag_observability.events (DriftEvent, drift.detected)
Webhook WebhookEventType.drift_detected (Step 3.9)
Config cfg.drift (rag_config.DriftConfig)
Gateway routes GET /v1/status/drift, POST /v1/status/drift/{metric}/rebaseline
CLI ragctl drift

The five monitors

Metric Method Signal observed Drift when
query_distribution PSI query length (tokens) PSI ≥ psi_threshold
embedding_psi PSI query-embedding L2 norm (HyDE) PSI ≥ psi_threshold
retrieval_score PSI top retrieval score PSI ≥ psi_threshold
citation_clickthrough mean-drop 1.0 per citation click, else 0.0 drop ≥ mean_drop_threshold
faithfulness mean-drop guard grounded-claim fraction drop ≥ mean_drop_threshold

Each monitor is ok, drifted, or insufficient_data (fewer than min_samples in either window).

PSI rule of thumb: < 0.1 no shift, 0.1–0.25 moderate, > 0.25 significant; the default alert floor is 0.2.

Usage

GET /v1/status/drift

Infra-scoped (no tenant). Computing the report also emits drift.detected for any monitor that has newly crossed its threshold.

curl -s localhost:8000/v1/status/drift
# {"monitors":[{"metric":"query_distribution","method":"psi","status":"drifted",
#   "statistic":0.41,"threshold":0.2,"reference_n":500,"current_n":500,...}, ...],
#  "drifted_n":1,"generated_at":"..."}

POST /v1/status/drift/{metric}/rebaseline

The operator "accept the new normal" — promotes a monitor's current window to its reference and clears the current window. 404 when drift is disabled, 400 on an unknown metric.

curl -s -X POST localhost:8000/v1/status/drift/query_distribution/rebaseline

ragctl drift

ragctl drift --shift 6          # seed a baseline, feed a shifted current, report
ragctl drift --samples 40       # smaller windows

Programmatic

from rag_drift import DriftMonitorRegistry
from rag_drift.registry import DriftConfig
from rag_core.types import DriftMetric

reg = DriftMonitorRegistry(config=DriftConfig(psi_threshold=0.2))
reg.seed_reference(DriftMetric.query_distribution, baseline_lengths)
reg.observe(DriftMetric.query_distribution, 12.0)
report = await reg.evaluate()   # emits drift.detected on a transition into drift

Configuration

drift:
  enabled: true             # serve GET /v1/status/drift + feed monitors from the live paths
  window_size: 500          # per-monitor reference + current window cap (bounded FIFO)
  min_samples: 30           # below this a monitor reports insufficient_data
  psi_bins: 10              # PSI histogram buckets
  psi_threshold: 0.2        # PSI drift floor (query distribution / embedding / retrieval score)
  mean_drop_threshold: 0.1  # mean-drop trigger (clickthrough / faithfulness)
  alert_tenant: ""          # tenant whose webhook subscriptions get drift.detected (empty = log only)

Internals

  • PSI bins the combined value range into psi_bins equal-width buckets and sums (cur_frac − ref_frac)·ln(cur_frac/ref_frac); empty bins are epsilon-floored.
  • Detection happens on evaluate() (the dashboard poll), not the hot path. A metric emits drift.detected once per transition into drift; a recovery is silent (re-alerts if it drifts again) — breaker.opened semantics.
  • Alerting is the PII-free drift.detected structured event plus a drift.detected webhook when alert_tenant + a dispatcher are configured.
  • Observe-only: feeding a monitor is an O(1) deque append; drift never rejects or mutates a request.

Extension points

  • New signal — add a DriftMetric member, its method in rag_drift.registry._MONITOR_SPECS, and an observe call at its source.
  • Per-dimension embedding PSI — average PSI across embedding dimensions instead of the norm scalar.
  • Per-tenant drift — key the registry by tenant (currently infra-scoped).

See also