Skip to content

cost.md

Latest commit

 

History

History
81 lines (65 loc) · 3.79 KB

cost.md

File metadata and controls

81 lines (65 loc) · 3.79 KB
Raw

Cost anomaly (Step 5.6c)

Per-tenant detection of spend spikes — the read-side sibling of the drift monitors, but per-tenant and spike-shaped. See ADR-0031 and architecture/cost-anomaly.md.

Overview

The quota enforcer tracks a tenant's cumulative monthly cost; this tracks the rate. rag_observability.cost.CostTracker keeps a bounded rolling window of recent per-request token costs per tenant and flags when the recent mean jumps well above its own rolling baseline. It is observe-only (never rejects or mutates a request) and pull-based (the verdict is computed on demand by GET /v1/status/cost).

Usage

from rag_observability import CostTracker

tracker = CostTracker(
    window_size=200,       # per-tenant rolling sample cap (bounded FIFO)
    recent_window=20,      # newest N samples = the "recent" mean
    min_samples=30,        # below this → insufficient_data
    z_threshold=3.0,       # std-devs above baseline to alert
    ratio_threshold=0.5,   # also require a +50% jump
    micro_dollars_per_1k_tokens=30_000,  # $0.03 / 1k tokens; 0 = token-only
)

tracker.observe("acme", tokens=420)      # fed from the gateway, O(1)
snap = tracker.snapshot("acme")          # CostSnapshot
snap.status                              # "ok" | "elevated" | "insufficient_data"
snap.ratio                               # recent_mean / baseline_mean

CostSnapshot fields: tenant_id, status, samples_n, recent_n, recent_mean_tokens, baseline_mean_tokens, baseline_std_tokens, ratio, z_score, recent_cost_micros, baseline_cost_micros.

HTTP

GET /v1/status/cost?tenant_id=acme → CostStatusResponse (mirrors CostSnapshot). Tenant resolves from the tenant_id query param, else the X-Tenant-Id header, else "default". Returns insufficient_data when cost tracking is disabled or the tenant has seen too few requests. The admin console's Live Status page renders this as a Cost anomaly card.

Configuration (cfg.cost)

Field Default Meaning
enabled true Wire the tracker (inert in plain build_app).
window_size 200 Per-tenant rolling sample cap.
recent_window 20 Newest N samples forming the recent mean.
min_samples 30 Below this many observations → insufficient_data.
z_threshold 3.0 Std-devs above baseline that trip an anomaly.
ratio_threshold 0.5 Fractional jump (recent/baseline − 1) also required.
dollars_per_1k_tokens 0.0 Price for the dollar figures (0 = token-only).

Internals

  • Scale-free detection. Cost = tokens × a constant price, so the verdict is identical in tokens or dollars; detection runs on the token series (always available) and the snapshot prices the means for display.
  • Two gates. elevated needs recent > baseline by ratio_threshold and z_threshold σ — except the z-gate relaxes when the baseline std is 0 (a flat baseline), so a genuine spike there still trips while noise does not.
  • Fed from one choke point. The gateway's record_request_usage (every answer route's post-LLM hook) calls observe before the quota block, so it runs even when quotas are disabled.
  • No new event. Detection is pull-only; a cost.anomaly_detected push alert + webhook is deferred.

Extension points

  • Swap the statistic (EWMA, robust median/MAD) by editing CostTracker.snapshot — the CostSnapshot shape and the endpoint are stable.
  • A push alert: emit a transition-edge event from snapshot (mirror drift.detected) and register it in rag_observability.events.
  • Per-model pricing: replace the single micro_dollars_per_1k_tokens constant with a model→price map and observe (model, tokens).