Skip to content

Check active multimodal models before generation - #104

Open
oliverholworthy wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
oliverholworthy:codex/retrieval-model-preflight
Open

oliverholworthy wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
oliverholworthy:codex/retrieval-model-preflight

Conversation

@oliverholworthy

@oliverholworthy oliverholworthy commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Run Data Designer's native check_models() before multimodal generation, so a bad credential, unavailable model, or unreachable endpoint for a later judge fails before earlier stages incur generation costs.

This runs automatically for each distinct active generator/judge configuration and any hosted summary embedding model. Shared deployments are checked once; generation batches skip repeat probes. Failed checks leave a new run unfrozen, and completed resumes still verify artifacts without credentials or model calls.

The checks make real text/embedding requests using the configured inference options. They verify basic provider readiness, not image support or judge schemas. Data Designer's DATA_DESIGNER_SKIP_MODEL_HEALTH_CHECKS=1 opt-out is respected.

Validation: UV_LINK_MODE=symlink make all — 942 passed, 2 optional Nemotron-consumer tests skipped; lint, plugin validation, generated-file checks, and docs passed. Regression coverage exercises the native readiness path with mocked model transport, including provider errors, routing, deduplication, embeddings, and resume integrity. No live model requests were used in validation.

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
oliverholworthy requested review from a team and shan-nvidia as code owners October 20, 2026 14:48
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy oliverholworthy changed the title Validate multimodal model settings before generation Check multimodal model credentials before generation Oct 20, 2026
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy oliverholworthy changed the title Check multimodal model credentials before generation Check active multimodal models before generation Oct 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant