Skip to content

Allow separate models for multimodal retrieval judges - #103

Merged
oliverholworthy merged 2 commits into
NVIDIA-NeMo:mainfrom
oliverholworthy:codex/per-stage-retrieval-judges
Oct 20, 2026
Merged

oliverholworthy merged 2 commits into
NVIDIA-NeMo:mainfrom
oliverholworthy:codex/per-stage-retrieval-judges

Conversation

@oliverholworthy

@oliverholworthy oliverholworthy commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Allow different models for summary quality, query metadata, self-sufficiency, relevance, and localization judgments. Unspecified stages continue using the existing judge configuration.

For example, add a text-only self-sufficiency judge to an existing configuration:

judges:
  self_sufficiency:
    model: your-text-judge
    endpoint: https://text.example.com/v1
    credential_env: JUDGE_API_KEY

Each override supplies its own model settings. Routing, batching, token checks, caching, and exported provenance consistently use the resolved model. Automatic table-of-contents confirmation uses the fallback judge, which is also included in validation and provenance when active.

Validation: UV_LINK_MODE=symlink make all — 917 passed, 2 optional Nemotron-consumer tests skipped. Includes 17 new routing, cache, budget, and provenance tests. No live-model benchmark.

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
oliverholworthy requested review from a team and shan-nvidia as code owners October 19, 2026 23:01
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
oliverholworthy merged commit ef6b6da into NVIDIA-NeMo:main Oct 20, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants