Skip to content

pilot.md

Latest commit

 

History

History
107 lines (83 loc) · 5.11 KB

pilot.md

File metadata and controls

107 lines (83 loc) · 5.11 KB
Raw

ragctl pilot — design-partner pilot tooling (Step 7.4)

Onboard a design-partner pilot and report its weekly KPIs. The command group is the tooling half of the Step 7.4 pilot program; the process is the operator runbook and the ADR.

Overview

Command Does
ragctl pilot onboard Print the per-tenant rag.yaml block + the onboarding checklist for a pilot
ragctl pilot report Assemble the weekly KPI dashboard + a PASS/FAIL verdict from the platform's own signal components

Both are self-contained — no servers, no infrastructure — mirroring the seed-then-report shape of ragctl feedback / ragctl drift.

Usage

pilot onboard

ragctl pilot onboard --tenant acme-support --vertical customer-support

Renders a ready-to-paste tenants: block (namespace + acl_labels + a redact PII policy + a QPS quota), the governance flags to enable for the security review (acl / pii / injection / audit / feedback / drift), and the onboarding checklist. For --vertical customer-support it also points at the kit corpus (docs/pilots/customer-support/corpus/).

Option Default Meaning
--tenant / -t required Pilot tenant id (used for namespace + the report)
--vertical / -v customer-support Pilot vertical (sets the ACL label + kit pointer)

pilot report

ragctl pilot report --tenant acme-support --week 3
pilot KPI report — tenant=acme-support week=3
────────────────────────────────────────────────────────────────
  volume:        60 signals (explicit=43 implicit=17)
  satisfaction:  mean=+0.733  +52/-8  clicks=17   [ok]
  drift:         0/5 monitors drifting   [ok]
  cost:          status=ok ratio=1.00 tokens=820   [ok]
  quality:       see `ragctl eval` (recall, faithfulness, cite-precision)
  latency:       see `ragctl perf` (gateway p99 vs 500 ms target)
────────────────────────────────────────────────────────────────
  verdict:       PASS — all online signals green
Option Default Meaning
--tenant / -t acme-support Pilot tenant id
--week / -w 1 Pilot week number (label only)
--samples / -n 60 Representative signal volume to seed
--drift-shift 0.0 Shift the current window to simulate degradation (0 = stable)

Pass a non-zero --drift-shift (e.g. 6) to see the drift monitors flip and the verdict become FAIL — the same edge the production drift monitor catches.

KPI → signal mapping

pilot report reads the platform's own signal components — the same ones behind GET /v1/status/{feedback,drift,cost}:

KPI line Component Status endpoint
satisfaction rag_feedback.aggregate_feedback → FeedbackStats GET /v1/status/feedback
drift rag_drift.DriftMonitorRegistry.evaluate → DriftReport GET /v1/status/drift
cost rag_observability.CostTracker.snapshot → CostSnapshot GET /v1/status/cost
quality cross-referenced ragctl eval (per-pilot golden set)
latency cross-referenced ragctl perf / GET /v1/status/metrics

The verdict gates on the three online signals the command computes directly (satisfaction ≥ +0.5, zero drifting monitors, cost not elevated); quality and latency carry their own gates (ragctl eval, ragctl perf).

Internals

  • Seed-then-report, no infrastructure. report seeds a deterministic representative stream (a positive-skewed feedback mix, a stable-or-shifted drift window, a flat cost series) into real platform components and reads their public aggregations. This is the same pattern as ragctl feedback / drift / compliance demo — a demonstration over representative data, since the real partner traffic is external (see ADR-0046).
  • No new package / core type / governed SPI call / dist change. pilot is a pure ragctl reader; the schema/openapi/policy-coverage gates need no new entry.

Extension points

  • Drive a live tenant. Swap the seeded streams for a build_app_from_config
    • TestClient workload (or a real deployment's GET /v1/status/*) to report a live pilot rather than a representative one.
  • Add a KPI. Any new platform signal with a pull-based snapshot (a new GET /v1/status/*) can become a report line + a verdict gate.

See also