Skip to content

eval-harness.md

Latest commit

 

History

History
215 lines (171 loc) · 9.82 KB

eval-harness.md

File metadata and controls

215 lines (171 loc) · 9.82 KB
Raw

Eval harness — reference (Steps 5.2–5.3)

The offline golden-set eval harness scores retrieval and answer quality against a labelled golden set, fully offline and deterministically. It completes the Step 0.8 eval skeleton (eval-skeleton.md) by wiring real retrieval, adding nDCG and a dependency-free faithfulness metric, shipping a 500-query / 5-domain golden set, and rendering an HTML report. Step 5.3 adds the CI eval gate on top: a committed baseline, a floor + regression-delta comparison, and a PR-comment diff table (CI eval gate).

Overview

Piece Where
Metric functions (pure) rag_config.eval
HTML report renderer rag_config.eval.render_html_report
Golden-set schema + report types rag_core.eval
Golden set (committed data) tests/eval/golden/<domain>.jsonl (500 queries)
Synthetic corpus + embedder eval/golden_set_v0/corpus.py
Golden-set generator eval/golden_set_v0/generate.py
Harness runner eval/golden_set_v0/harness.py
CLI ragctl eval run / ragctl eval show
Thresholds tests/eval/thresholds.yaml
Gate runner (Step 5.3) eval/golden_set_v0/gate.py
Committed baseline (Step 5.3) tests/eval/baselines/main.json
Gate comparison + comment (Step 5.3) rag_config.eval.compare_to_baseline / render_gate_comment
Workflow (Step 5.3) .github/workflows/eval-gate.yml

Metrics

All five are computed per sample and macro-averaged overall, per domain, and per difficulty.

Metric Function Meaning
Recall@k recall_at_k(expected, retrieved, k) Fraction of expected source_uris in the top-k.
MRR mrr(expected, retrieved) 1 / rank of the first relevant hit.
nDCG@k ndcg_at_k(expected, retrieved, k) Rank-discounted gain (binary relevance), normalised to the ideal ordering.
Faithfulness lexical_faithfulness(answer, contexts) Mean per-claim token coverage of the answer against the retrieved contexts. Dependency-free; RAGAS optional.
Citation Precision citation_precision(expected, retrieved, k) Fraction of the top-k that are relevant (precision@k).

Edge-case conventions match the existing metrics: empty expected → 1.0 (vacuously perfect) for recall/MRR/nDCG; empty retrieved → 0.0; lexical_faithfulness returns 1.0 for an answer with no content tokens and 0.0 when contexts carry no usable tokens.

Usage

Run the harness

# Full 500-query run → writes report.json / report.md / report.html next to the harness
uv run python -m eval.golden_set_v0.harness

# One domain, a subset, or a different cut-off
uv run python -m eval.golden_set_v0.harness --domain governance_security
uv run python -m eval.golden_set_v0.harness --queries 50 --no-write
uv run python -m eval.golden_set_v0.harness --k 5

# Gate: exit 1 if recall@10 / mrr / faithfulness fall below their thresholds.yaml floor
uv run python -m eval.golden_set_v0.harness --check

run_harness(*, k=10, domain=None, max_queries=None, golden_dir=..., faithfulness_top_k=5) returns a fully-populated EvalReport for programmatic use.

Regenerate the golden set

The golden set is committed data. After editing the concepts in corpus.py:

uv run python -m eval.golden_set_v0.generate           # rewrite tests/eval/golden/
uv run python -m eval.golden_set_v0.generate --check   # verify committed == generated

ragctl eval

The lightweight CLI runs over any golden JSONL file or directory (using run_eval, whose default retriever is the stub — wire a real retriever or use the harness above for real numbers):

ragctl eval run tests/eval/golden/ --top-k 10 --output report.json --html report.html
ragctl eval show report.json --verbose

eval run prints recall / mrr / nDCG / faithfulness / citation precision and a per-domain breakdown, writes the JSON report, and (with --html) a self-contained HTML report. eval show renders a saved report, including the per-domain table.

CI eval gate (Step 5.3)

The gate runs the harness, compares the fresh metrics against the committed baseline (tests/eval/baselines/main.json), enforces the thresholds.yaml floors and per-metric regression deltas, and renders a PR-comment diff table. It exits non-zero when a gated metric breaches its floor or regresses beyond tolerance.

# Gate the current tree (exit 1 on regression / floor breach)
uv run python -m eval.golden_set_v0.gate

# Write the Markdown PR comment that CI posts
uv run python -m eval.golden_set_v0.gate --comment-out gate-comment.md

# Refresh the baseline after an intended, reviewed metric change (maintainer)
uv run python -m eval.golden_set_v0.gate --update-baseline

The two checks, both read from thresholds.yaml hard_gates:

Check Rule Purpose
Floor current ≥ min Absolute quality bar; holds on the first run. Shared with the harness --check.
Regression baseline − current ≤ max_regression_delta Fails even above the floor, so quality can't erode run-over-run.

Only recall@10 / mrr / faithfulness gate (the metrics this harness measures); nDCG and citation precision are shown but never gate. The acl_violation_rate / pii_egress_rate hard gates are enforced by the red-team suites, not here.

run_gate(...) returns (EvalReport, EvalComparison, EvalBaseline). EvalBaseline is the compact, stable projection of an EvalReport (headline means + by_domain / by_difficulty, no run_id / created_at / samples), so re-capturing unchanged metrics yields a byte-identical file. The .github/workflows/eval-gate.yml workflow runs on every PR, posts/updates one sticky comment (marker <!-- agentcontextos-eval-gate -->), and should be a required status check ("Eval gate") in branch protection.

Regression bisector (Step 5.6d)

When the gate goes red, the bisector finds which commit caused it.

# Binary-search HEAD's history for the first commit that dropped recall@10 below 0.85
uv run python -m eval.golden_set_v0.bisect \
    --good <known-good-ref> --bad HEAD --metric recall_at_k_mean --threshold 0.85

It lists the commits in <good>..<bad> (oldest→newest) and binary-searches them — only O(log n) commits are actually run — printing each probe and the first regressing commit (exit 1) or "no regression" (exit 0). --metric is any EvalReport mean (recall_at_k_mean / mrr_mean / ndcg_at_k_mean / faithfulness_mean / citation_precision_mean); --threshold is the floor.

Each candidate runs in a throwaway git worktree — your working tree is never touched — and runs that commit's in-repo code: the worktree's packages/*/src are prepended to PYTHONPATH ahead of the installed env, so the bisector varies the platform code while third-party deps come from the current environment (the harness is deterministic + offline, so this is stable). A commit whose harness can't run (e.g. predates Step 5.2) scores None and counts as regressed, so the search never silently skips a broken commit.

The pure search core is rag_config.eval.bisect_commits(commits, probe) → rag_core.eval.BisectResult (with per-commit BisectSteps); the git/worktree orchestration (commits_between, score_commit_via_worktree, run_bisect) is in eval/golden_set_v0/bisect.py. run_bisect accepts injectable commits + score_fn so the search is unit-tested without git. Assumes a monotonic range (once regressed, stays regressed) — the standard git-bisect contract.

EvalReport shape

class EvalReport:
    run_id: str
    created_at: datetime
    golden_set_path: str
    sample_count: int
    recall_at_k_mean: float
    mrr_mean: float
    ndcg_at_k_mean: float                       # Step 5.2
    faithfulness_mean: float | None
    citation_precision_mean: float
    by_domain: dict[str, GroupEvalMetrics]      # Step 5.2
    by_difficulty: dict[str, GroupEvalMetrics]  # Step 5.2
    samples: list[EvalMetrics]
    metadata: dict[str, Any]                     # k, retriever, faithfulness mode

GroupEvalMetrics carries sample_count + the five macro-averages for one domain or difficulty group.

Internals

  • Corpus. 5 domains × 20 concepts × 3 passages = 300 documents. Each passage's chunk_id is its source_uri, so a retrieved ChunkRef maps straight back to the golden set with no side table.
  • Retrieval. The harness drives the real HybridRetriever with both the dense (HashingEmbedder → NoopVectorStore cosine) and sparse (NoopKeywordStore token overlap) paths, fused via RRF. Queries are not corpus-scoped, so cross-domain distractors make the metrics non-trivial.
  • Determinism. No network, no LLM, no randomness — the same numbers on every platform and every run, which is what lets Step 5.3 use report.json as a regression baseline.

Extension points

  • Swap the faithfulness scorer — pass --ragas to ragctl eval run (requires pip install 'rag-config[eval]'), or call RagasAdapter directly. A future cross-encoder NLI scorer can implement the same idea behind rag_core.spi.nli.NLIScorer.
  • Grow the golden set — add concepts to corpus.py and rerun generate.py; the drift test keeps the committed files honest.
  • Real-backend eval — point a future harness variant at live pgvector/Qdrant instead of the noop stores (out of scope for the CI baseline).

See also