Skip to content

guard.md

Latest commit

 

History

History
194 lines (150 loc) · 7.57 KB

guard.md

File metadata and controls

194 lines (150 loc) · 7.57 KB
Raw

rag-guard reference (Step 4.3)

Public API for the hallucination guard. See architecture/hallucination-guard.md for the design rationale and ADR-0022 for the decision record.

Overview

The guard verifies a generated answer against the retrieved context one claim at a time. It splits the answer into claims, scores each claim's faithfulness against the evidence via a pluggable NLI scorer, and — per a per-tenant threshold — annotates, redacts, or blocks unsupported claims, emitting the guard.claim_blocked event when any claim fails.

Exports

Symbol Module Kind
HallucinationGuard rag_guard.guard orchestrator
GuardConfig rag_guard.guard frozen dataclass
LexicalNLIScorer rag_guard.scorers dependency-free NLIScorer impl
SentenceClaimExtractor / ClaimExtractor rag_guard.claims claim splitter + Protocol
split_into_claims rag_guard.claims pure function
REDACTION_MARKER / BLOCK_NOTICE rag_guard.guard action output constants
NLIScorer rag_core.spi.nli scorer SPI ABC
NoopNLIScorer rag_core.spi.noop always-entails impl
NLILabel / NLIScore rag_core.types NLI verdict types
GuardVerdict / GuardAction rag_core.types StrEnums
ClaimVerdict / GuardResult rag_core.types frozen Pydantic models

HallucinationGuard, GuardConfig, and LexicalNLIScorer are re-exported from rag_guard.

Usage

from rag_guard import HallucinationGuard, GuardConfig
from rag_guard.scorers import LexicalNLIScorer
from rag_core.types import GuardAction

guard = HallucinationGuard(
    scorer=LexicalNLIScorer(),
    config=GuardConfig(threshold=0.5, action=GuardAction.ANNOTATE),
)

result, answer = await guard.check(
    ctx,
    answer="Paris is the capital of France. The moon is made of cheese.",
    evidence=["Paris is the capital of France, on the Seine."],
)
# result.verdict == GuardVerdict.FLAGGED, result.blocked_n == 1
# answer unchanged (ANNOTATE); REDACT would strip the moon claim, BLOCK withhold all

check returns (GuardResult, final_answer) — the decision record plus the answer text after the configured action (mirrors FallbackChain.run's (FallbackResult, refs) shape; the answer text is not embedded in the result).

Types

NLILabel

entailment · neutral · contradiction — the NLI relation between the evidence (premise) and a claim (hypothesis).

NLIScore

Field Type Meaning
label NLILabel arg-max relation
entailment float P(premise entails hypothesis) in [0, 1] — the number thresholded on
contradiction float P(premise refutes hypothesis); 0.0 for scorers that don't separate it

GuardVerdict

pass · flagged · blocked — overall assessment. flagged = unsupported claims found but the answer was left intact (annotate); blocked = the answer was modified or withheld (redact / block).

GuardAction

annotate (observe only) · redact (drop unsupported claims) · block (withhold the whole answer). annotate is the behaviour-neutral default.

ClaimVerdict

Field Type Meaning
claim str the claim text (response-side only — never logged)
label NLILabel best NLI label across evidence
entailment float best entailment across evidence
supported bool entailment >= threshold

GuardResult

Frozen decision record. Carries the per-claim verdicts but not the final answer text (returned alongside).

Field Type Meaning
verdict GuardVerdict overall assessment
action GuardAction the configured action
threshold float resolved per-tenant entailment floor
claims_n int claims extracted from the answer
blocked_n int claims that fell below the threshold
claims tuple[ClaimVerdict, ...] per-claim verdicts
reason str advisory free-text — never parse in production

GuardConfig

Frozen dataclass; all fields optional. (This is the runtime config — the enabled flag lives on the rag.yaml guard block, handled by the gateway, mirroring FallbackConfig.)

Field Default Meaning
threshold 0.5 default entailment floor
per_tenant_thresholds {} per-tenant overrides keyed by tenant id
action GuardAction.ANNOTATE annotate / redact / block
min_claim_chars 12 drop fragments shorter than this
max_claims 50 hard cap on claims scored per answer

NLIScorer SPI

class NLIScorer(HealthCheckMixin, abc.ABC):
    async def score(self, ctx, premise, hypothesis) -> NLIScore: ...      # mandatory
    async def score_batch(self, ctx, pairs) -> list[NLIScore]: ...        # default loops over score

Mirrors Reranker: one atomic method plus a batch entry point whose default loops — override score_batch to amortise model encoding. The guard owns the evidence strategy (it scores each claim against each evidence passage and keeps the best entailment), so a backend only answers the narrow NLI question.

Shipped impls: NoopNLIScorer (always entailment=1.0, the behaviour-neutral default) and LexicalNLIScorer (token-overlap heuristic — the fraction of a claim's content tokens present in the evidence; dependency-free, deterministic).

Configuration (rag.yaml)

guard:
  enabled: true            # opt-in; disabled by default
  threshold: 0.5           # entailment floor in [0, 1]
  action: annotate         # annotate | redact | block
  per_tenant_thresholds:
    acme: 0.7
  min_claim_chars: 12
  max_claims: 50

When enabled, the gateway checks every generated answer on /v1/query (generate=true), /v1/chat/completions, and the MCP query tool. The default scorer is LexicalNLIScorer; inject a real NLI model via build_app(hallucination_guard=...).

Gateway response surface

  • REST /v1/query — answer.guard carries the GuardResult; answer.text already reflects the action.
  • OpenAI /v1/chat/completions — rag.guard carries the verdict; a blocked verdict sets the choice's finish_reason to content_filter. In streaming mode the verdict rides the terminal chunk (advisory — deltas already sent).

Observability

  • Span guard.check — attributes rag.tenant_id, rag.guard.{threshold,action,scorer,claims_n,blocked_n,verdict,min_entailment,degraded,elapsed_ms}.
  • Event guard.claim_blocked (typed GuardEvent) — emitted once per check when blocked_n > 0. Carries counts, threshold, min_entailment, verdict, action, scorer — never the answer or claim text.

ragctl guard

ragctl guard "Paris is in France. The moon is cheese." \
    -c "Paris is the capital of France." \
    --threshold 0.5 --action annotate

Drives HallucinationGuard + LexicalNLIScorer over the provided context; prints the per-claim verdicts and the (possibly redacted / blocked) answer. No model download, no external services. Flags: --context/-c (repeatable), --threshold/-T, --action/-a, --tenant/-t.

Extension points

  • Real NLI model — implement NLIScorer.score (override score_batch to batch-encode) and inject via build_app(hallucination_guard=HallucinationGuard(scorer=...)).
  • Atomic-claim extraction — implement ClaimExtractor.extract (e.g. an LLM-based decomposer) and pass extractor= to HallucinationGuard.