Repository navigation
FEAT GUI: Expose built-in dataset provider metadata #2754
Description
Is your feature request related to a problem? Please describe.
The explorer should help users discover built-in datasets that are not loaded yet. Provider names alone do not explain modalities, size, variants, sources, or whether credentials and gated access are required.
Work item 9 of 17 in #2744. Technical prerequisites: the dataset identity/summary contract and catalog from items 1 and 3. This starts the provider-loading milestone after item 8; keep not ready yet until promoted.
Describe the solution you'd like
Add metadata-only descriptors for registered built-in dataset providers and integrate them with the existing dataset catalog contract.
- Distinguish in-memory datasets from available providers and expose a loaded/available filter without confusing name collisions or partial/variant loads.
- Include available description, source URL, license/provenance, modality, harm categories, size information, and supported variants/options. Missing information remains unknown.
- Expose only documented, supported non-secret load options. Use registered provider identities; do not accept arbitrary Python paths, code, or unvalidated kwargs from clients.
- Describe credential and external gated-access prerequisites without returning credential values. A configured credential is not proof that its account has dataset access.
- Keep enumeration side-effect free: no dataset materialization, downloads, model calls, or credential-dependent fetches merely to display cards.
Acceptance criteria:
- Loaded-only, provider-only, and combined views distinguish provenance and availability correctly.
- Known provider metadata is displayed without confusing an estimate or coarse size category with an exact loaded count.
- Missing metadata or an unsupported load option does not break the whole catalog.
- Gated/public examples can be described without real credentials or remote fetches.
- Existing name-list clients remain compatible and metadata/read-only tests are included.
Describe alternatives you've considered, if relevant
Avoid a hardcoded frontend dataset list, a second provider registry, fetching every dataset for metadata, or a generic code-execution/loader endpoint. This issue does not load datasets; item 10 adds that operation.
Additional context
Start with SeedDatasetProvider, SeedDatasetMetadata, provider metadata, and the dataset backend service/models. Preserve the framework rule that downstream consumers read seeds from memory after loading. Coordinate count provenance with #2735 and follow dataset/Python/model/test instructions.
Relationship to scenario dataset generation
Coordinate with the scenario dataset plan and the still-open #3143. That PR proposes shared catalog counts/provider availability, aggregate stored origins, and scenario source metadata. Reuse its finalized projections when available instead of creating duplicate DTO fields or a second discovery path. This issue still supplies the richer built-in-provider descriptors, supported non-secret options, and access prerequisites that the explorer needs.
Stored origin and provider availability are different facts. Current seed details already expose SeedRecord.origin; a dataset can contain multiple origins or unknown provenance. Do not label a dataset generated/local/remote solely from its name or registered provider. TargetObjectiveProvider from #2904 is intentionally excluded from automatic registration/discovery; configured-provider support in #3116 does not make it a normal unloaded built-in card. Do not enumerate or call generation targets to fill metadata, or infer that a stored generated dataset can be regenerated from its provenance alone.
Generator configuration UI and richer generic generation capabilities remain optional follow-ups. These links do not promote this item or change the existing sequence.
Activity