Skip to content

feedback.md

Latest commit

 

History

History
126 lines (100 loc) · 4.99 KB

feedback.md

File metadata and controls

126 lines (100 loc) · 4.99 KB
Raw

Online feedback — reference (Step 5.4)

rag-feedback captures explicit feedback and implicit signals about a query, PII-redacts free-text comments, persists them tenant-scoped, and aggregates them into a per-tenant dashboard.

Overview

Piece Where
Recorder (normalise → redact → store → emit) rag_feedback.FeedbackRecorder
Aggregator (pure) rag_feedback.aggregate_feedback
Score / kind helpers rag_feedback.score_for_signal / kind_for_signal
Domain types rag_core.types (FeedbackRecord / FeedbackStats / FeedbackKind / FeedbackSignal)
Store SPI + noop rag_core.spi.FeedbackStore / rag_core.spi.noop.NoopFeedbackStore
Wire types rag_core.gateway_types (FeedbackRequest / FeedbackAck)
Events rag_observability.events (FeedbackEvent, feedback.recorded, feedback.record_degraded)
Config cfg.feedback (rag_config.FeedbackConfig)
Gateway routes POST /v1/feedback, GET /v1/status/feedback
CLI ragctl feedback

Signals

A submission carries one FeedbackSignal. kind (explicit / implicit) is inferred from it when omitted, and a [-1, 1] score is derived:

Signal Kind Score
thumb_up explicit +1
thumb_down explicit −1
rating (1–5) explicit (rating−3)/2 → −1…+1
comment explicit 0
citation_click implicit +1
answer_copied implicit +1
accepted implicit +1
regenerated implicit −1
dwell implicit 0 (raw value in value)

Usage

POST /v1/feedback

Body identity (tenant_id / principal_id) like /v1/query; request_id is the query being rated.

# explicit thumbs up
curl -s localhost:8000/v1/feedback -H 'content-type: application/json' -d '{
  "tenant_id": "acme", "principal_id": "alice", "request_id": "q-123", "signal": "thumb_up"
}'

# rating + comment (the comment is PII-redacted before storage)
curl -s localhost:8000/v1/feedback -d '{
  "tenant_id":"acme","principal_id":"alice","request_id":"q-123",
  "signal":"rating","rating":5,"comment":"great, but reach me at a@b.com"
}'

# implicit signal — a user clicked a citation
curl -s localhost:8000/v1/feedback -d '{
  "tenant_id":"acme","principal_id":"alice","request_id":"q-123",
  "signal":"citation_click","target_chunk_id":"c-7"
}'

The response is a FeedbackAck — {feedback_id, request_id, kind, signal, score, stored, comment_redacted}. stored=false (with feedback_id=null) when feedback is disabled on the gateway or recording degraded; the submission is acknowledged either way (recording is degrade-open). Invalid signal / out-of-range rating → 422.

GET /v1/status/feedback — the dashboard

Per-tenant aggregate. Tenant resolves from the tenant_id query param, else the X-Tenant-Id header, else "default". Returns a FeedbackStats:

curl -s 'localhost:8000/v1/status/feedback?tenant_id=acme'
# {"tenant_id":"acme","total":12,"explicit_n":7,"implicit_n":5,
#  "positive_n":9,"negative_n":2,"neutral_n":1,"comment_n":4,
#  "citation_click_n":3,"regenerated_n":2,"mean_score":0.58,
#  "by_signal":{"thumb_up":5,"rating":2,...}}

ragctl feedback

ragctl feedback q-42 --tenant acme      # record explicit + implicit, print the aggregate
ragctl feedback q-42 --no-redact        # store the comment without redaction

Configuration

feedback:
  enabled: true             # POST /v1/feedback live; GET /v1/status/feedback served
  redact_comments: true     # run the wired PIIDetector over comments before storage
  max_comment_chars: 2000   # comments truncated to this length
  max_records: 10000        # in-memory bound on the default store

Internals

  • Recorder. record() normalises the score, redacts the comment (when a detector is wired and redact_comments), builds a FeedbackRecord, stores it, and emits feedback.recorded — all degrade-open (a store / redaction failure logs feedback.record_degraded and returns None).
  • Privacy. The default gateway wires a pass-through NoopPIIDetector; production injects a real one. comment_redacted records whether a redactor ran. The event never carries the comment text.
  • Aggregation. aggregate_feedback(tenant_id, records) is pure and backend-agnostic; the dashboard calls it over FeedbackStore.list(...).

Extension points

  • Real PII detector — inject a Presidio-backed PIIDetector into FeedbackRecorder (the gateway wires NoopPIIDetector by default).
  • Durable store — implement FeedbackStore (e.g. Postgres) for cross-worker aggregation; the noop in-memory store is the default.
  • New signal — add a FeedbackSignal member + its polarity in rag_feedback.recorder._SIGNAL_POLARITY.

See also