Batch hallucination-rate evaluation and reporting over a dataset

[zhenzhudingzhen528-cpu]

Summary

• JSONL and CSV input
• overall and grouped rates with explicit denominators
• span histograms and typed counts
• top-N flagged examples
• JSON and Markdown output
• network-free unit tests

Changed paths:

• scripts/report_hallucination_rates.py
• tests/test_report_hallucination_rates_pytest.py

Related issue

Closes #56.

Type of change

• Bug fix
• Feature
• Documentation
• Tests
• Refactor or maintenance

Testing

• python -m pytest tests/test_report_hallucination_rates_pytest.py -q — 23 passed
• ruff format --check
• ruff check
• git diff --check
• Other: current Windows full suite — 161 passed, 8 failed
• Other: recorded pre-change Windows baseline — 138 passed, 8 failed; recorded cause: existing Unix-only resource import in scripts/benchmark_detectors.py on Windows
• Exact failing test node IDs were not recorded, so failure-set equality was not verified.

Checklist

• I kept the PR focused on one change.
• I added or updated tests/docs when needed.
• I checked that no secrets, API keys, or credentials are included.

Rights & sign-off (required)

• I certify that I have the right to submit this code and that it may be distributed under the repository's MIT license. (attestation Only pyproject now, added ruff support, github workflows #1, mit-contribution-rights-v2)

[adaamko]

Thanks — this is a careful implementation and it hits the #56 spec. I ran the acceptance command end-to-end against lettucedect-v2-mmbert-base + the taxonomy head on a 10-row file: every count reconciles (overall, per-group denominators, typed category/subcategory counts, confidence bins), the tests are network-free, and the full suite is green on top of current main (201 passed).

One blocker before merge:

1. Progress indicator is never visible (scripts/report_hallucination_rates.py, score_records logging at INFO around line 275, and main()): main() never configures logging, so at the default WARNING level a real run prints nothing while scoring — issue step 3 asks for a progress indicator. Simplest fix: tqdm around the record loop (already a dependency; see scripts/evaluate_span_model.py), or logging.basicConfig(level=logging.INFO) in main(). Adjust the caplog assertion accordingly.

Non-blocking nits, take or leave:

• CSV errors on a missing --group-by column while JSONL silently buckets every row under <missing> — I'd make those consistent.
• _atomic_write and the NaN/bool/>1 confidence sanitisation guard against values the detectors cannot produce; the script could be a fair bit shorter without them.
• The generator-.throw lambda in test_main_surfaces_detector_configuration_value_error is hard to read; a small def that raises would do.
• Please tick the "no secrets" checkbox in the description (I checked the diff; there are none).

With the progress fix in, this merges and closes #56.