Skip to content

fix(evaluators): raise ValueError on empty input instead of ZeroDivisionError - #12572

Draft
mturac wants to merge 1 commit into
deepset-ai:mainfrom
mturac:fix/issue-12543
Draft

fix(evaluators): raise ValueError on empty input instead of ZeroDivisionError#12572
mturac wants to merge 1 commit into
deepset-ai:mainfrom
mturac:fix/issue-12543

Conversation

@mturac

@mturac mturac commented Sep 2, 2026

Copy link
Copy Markdown

fixes #12543

DocumentMAPEvaluator, DocumentMRREvaluator, DocumentRecallEvaluator, and AnswerExactMatchEvaluator crash with ZeroDivisionError when called with empty input lists. DocumentNDCGEvaluator allready handles this correctly with early ValueError.

added same guard to all four evaluators for consistancy — early ValueError in run() when either input list is empty.

added 3 tests per evaluator (12 total) to verify ValueError on empty input. existing evaluator tests still pass.

Four of five evaluators (MAP, MRR, Recall, AnswerExactMatch) crash
with ZeroDivisionError when passed empty input lists.  Mirror the
ValueError guard already present in DocumentNDCGEvaluator so all
five raise a clear error instead of an arithmetic crash.

Closes deepset-ai#12543
@mturac
mturac requested a review from a team as a code owner September 2, 2026 08:34
@mturac
mturac requested review from anakin87 and removed request for a team September 2, 2026 08:34
@vercel

vercel Bot commented Sep 2, 2026

Copy link
Copy Markdown

@mturac is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@HaystackBot

Copy link
Copy Markdown
Contributor

Hi @mturac, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

@HaystackBot
HaystackBot removed the request for review from anakin87 September 2, 2026 09:51
@HaystackBot HaystackBot added the cla-pending PR is in draft until the contributor signs the CLA label Sep 2, 2026
@HaystackBot
HaystackBot marked this pull request as draft September 2, 2026 09:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-pending PR is in draft until the contributor signs the CLA topic:tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 of 5 Document*Evaluators crash with ZeroDivisionError on empty input

3 participants