Repository navigation
FEAT GUI: Configure a scenario from selected dataset examples #2759
Description
Is your feature request related to a problem? Please describe.
Selecting several dataset examples should lead to a configurable scenario, not concatenate unrelated prompts into chat or immediately call a target. Users need to see the exact selection and execution configuration before launching.
Work item 15 of 17 in #2744. Technical prerequisites: the table/selection IDs, the general scorer registry API from #2755/#2990, the exact-selection contract in #2757, and the scenario/default-selection policy in #2760. Keep not ready yet until promoted.
Describe the solution you'd like
Add explicit multi-select to the explorer and a Configure scenario action. Preserve selected IDs across pages, make selected-versus-visible counts clear, and allow reviewing/removing/clearing the selection. Do not turn a page selection into an implicit select-all-matching query.
For one example the existing chat import actions remain available. For multiple examples, use the scenario path rather than concatenation or a queue of chat drafts.
Open a Scanner configuration flow using existing reusable form/estimate/launch pieces. Prefill exactly the selected examples, prompt sending, no additional converters, and a compatible configured scorer default. Allow target and scorer selection and collect missing required objectives explicitly. Show setup guidance when no compatible scorer or target is configured.
The scorer picker must list configured instances from the catalog added in #2990, not present every constructible /api/scorers/types entry as ready to use. Submit the selected scorer_registry_name through #2757. Show useful descriptions, result-family information, relevant tags, the compatible configured default, and compatibility/prerequisite guidance.
#2990 returns a registry name, full ScorerIdentifier, and description; it does not expose registry tags or choose a compatible objective default. Add the minimal shared typed metadata projection needed by this picker, backed by registry entries and the shared scenario/scorer policy from #2760. Do not infer compatibility from the identifier alone, hardcode scorer lists in the browser, duplicate the registry, or leak credentials through added metadata.
Explain empty/unusable registries, missing judge-target dependencies, stale references after reinitialization, and missing/incompatible defaults. Offer setup guidance or an explicit valid choice rather than silently substituting a scorer or treating a raw refusal detector as an objective-success scorer.
Preview the selected population and effective configuration before explicit launch confirmation. On launch, reuse the normal scenario queue/progress/history. Preserve explorer state when cancelling or returning. No new persistent named dataset or generated scenario source file is created by opening the form.
Acceptance criteria:
- Selections survive pagination and can be reviewed/cleared without selecting hidden unrequested examples.
- Configure scenario performs no target call and uses exact IDs, not a reconstructed dataset filter.
- Default settings mean prompt sending only, no extra converters, and an explicit compatible scorer.
- The picker distinguishes configured instances from buildable classes and shows tags, default selection, compatibility, and prerequisites from shared typed backend/core metadata.
- Missing objectives, invalid/stale selections or scorer names, incompatible presets/defaults, missing dependencies, and empty configuration have clear validation and setup guidance.
- Preview, estimate, confirmation, and launch agree on the exact selected examples and effective configured scorer; duplicate launch is prevented.
- Cancel/Back/navigation preserve the user's browse state, and success goes to existing scenario progress/history.
- Component/routing and any added metadata-projection tests cover configured versus buildable choices, defaults/tags, missing dependencies, stale references, and the full flow using mocked APIs and no privileged datasets.
Describe alternatives you've considered, if relevant
Do not build another launch page with duplicated state, a general scorer-builder GUI, batch chat behavior, or a direct launch from a selection checkbox. Advanced technique/converter controls are item 16; this item should deliver the useful default flow first.
Additional context
This item retains the scorer-picker metadata and setup-guidance requirements from the original #2755. The general API is complete in #2990; the dataset-scenario consumer is not. Coordinate the projection with #2760 so the UI and scenario execution use one compatibility/default policy.
Start with the Datasets table, frontend\src\components\Scenarios\ScenarioDetail.tsx and its extracted form/estimate/launch boundaries, shared parameter controls, and scenariosApi. Coordinate with #2653 instead of growing a second monolithic configuration component. Follow frontend style/test instructions.
Relationship to scenario dataset generation
This is the largest UI overlap with the scenario dataset plan. Its still-open PR 2, #3143, adds ScenarioDatasetTable, useScenarioDatasets, shared ScenarioDatasetRequest/settings, and scenario-owned ANY/SUBSET/FIXED dataset options. Coordinate with that work; once landed, reuse its finalized form/model pieces where applicable instead of adding another named-dataset picker, limit contract, or browser-owned capability policy. Do not assume the open PR is already shipped.
A scenario source table selects named populations, limits, and fetch policies. This issue instead begins with exact example IDs chosen across explorer pages. Keep those IDs authoritative and reviewable; do not expose source/cap controls that silently replace or resample the selection. Multi-page selection, the configured-scorer picker, missing-objective handling, confirmation, and return-to-explorer state remain this issue's work through #2757/#2760.
Merged generation support (#2904/#3116) means stored generated objectives can participate like other seeds. It does not add generator target/instructions/count controls to this form. Opening or estimating the configuration must not generate data, and the attack target and judging scorer are not implicit generation targets. GUI generation and destructive refresh are separately approved optional follow-ups. This coordination note leaves the current prerequisites and delivery order intact.
Activity