Skip to content

fix: support unhashable metadata comparison values in recall and NDCG evaluators - #12480

Open
CoralGarden52 wants to merge 1 commit into
deepset-ai:mainfrom
CoralGarden52:fix/support-unhashable-document-comparison
Open

fix: support unhashable metadata comparison values in recall and NDCG evaluators#12480
CoralGarden52 wants to merge 1 commit into
deepset-ai:mainfrom
CoralGarden52:fix/support-unhashable-document-comparison

Conversation

@CoralGarden52

Copy link
Copy Markdown
Contributor

Related Issues

No existing issue was found for this behavior.

Proposed Changes

Document.meta supports JSON-serializable values, including lists and dictionaries.
However, when document_comparison_field points to such a value:

  • DocumentRecallEvaluator raises TypeError because it stores comparison values in a set.
  • DocumentNDCGEvaluator raises TypeError because it uses comparison values as dictionary keys.

This is inconsistent with DocumentMAPEvaluator, which already supports unhashable comparison values.

This change adds a shared internal helper that converts nested list, tuple, and dictionary values into hashable representations while preserving their comparison semantics:

  • list and tuple order is preserved;
  • dictionary key insertion order is ignored;
  • nested containers are supported;
  • composite values are compared as complete values, not as element intersections.

Existing deduplication behavior is preserved:

  • Recall counts each unique comparison value once.
  • NDCG keeps the highest relevance score for duplicate ground-truth values.
  • Duplicate retrieved values are credited only once.

How did you test it?

  • Added regression tests for list and dictionary metadata values.
  • Added tests for nested values and list ordering.
  • Added tests for duplicate values and NDCG relevance-score handling.
  • Target evaluator tests: 110 passed.
  • Evaluator tests excluding the optional SAS evaluator: 175 passed, 4 skipped.
  • Ran Ruff, mypy, Reno lint, and pre-commit successfully.
  • SAS model-backed integration tests were intentionally not run because they require model weights.

Notes for the reviewer

The implementation follows the duplicate-value semantics introduced by
#12255 and
#11959.

This is an internal implementation change with no public API changes.

This PR was fully generated with an AI assistant. The changes have been reviewed and the relevant tests have been run.

@CoralGarden52
CoralGarden52 requested a review from a team as a code owner August 27, 2026 02:09
@CoralGarden52
CoralGarden52 requested review from bogdankostic and removed request for a team August 27, 2026 02:09
@vercel

vercel Bot commented Aug 27, 2026

Copy link
Copy Markdown

Someone is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Aug 27, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants