Skip to content

FEAT GUI: Configure a scenario from selected dataset examples #2759

Open

Description

@romanlutz

Is your feature request related to a problem? Please describe.

Selecting several dataset examples should lead to a configurable scenario, not concatenate unrelated prompts into chat or immediately call a target. Users need to see the exact selection and execution configuration before launching.

Work item 15 of 17 in #2744. Technical prerequisites: the table/selection IDs, the general scorer registry API from #2755/#2990, the exact-selection contract in #2757, and the scenario/default-selection policy in #2760. Keep not ready yet until promoted.

Describe the solution you'd like

Add explicit multi-select to the explorer and a Configure scenario action. Preserve selected IDs across pages, make selected-versus-visible counts clear, and allow reviewing/removing/clearing the selection. Do not turn a page selection into an implicit select-all-matching query.

For one example the existing chat import actions remain available. For multiple examples, use the scenario path rather than concatenation or a queue of chat drafts.

Open a Scanner configuration flow using existing reusable form/estimate/launch pieces. Prefill exactly the selected examples, prompt sending, no additional converters, and a compatible configured scorer default. Allow target and scorer selection and collect missing required objectives explicitly. Show setup guidance when no compatible scorer or target is configured.

The scorer picker must list configured instances from the catalog added in #2990, not present every constructible /api/scorers/types entry as ready to use. Submit the selected scorer_registry_name through #2757. Show useful descriptions, result-family information, relevant tags, the compatible configured default, and compatibility/prerequisite guidance.

#2990 returns a registry name, full ScorerIdentifier, and description; it does not expose registry tags or choose a compatible objective default. Add the minimal shared typed metadata projection needed by this picker, backed by registry entries and the shared scenario/scorer policy from #2760. Do not infer compatibility from the identifier alone, hardcode scorer lists in the browser, duplicate the registry, or leak credentials through added metadata.

Explain empty/unusable registries, missing judge-target dependencies, stale references after reinitialization, and missing/incompatible defaults. Offer setup guidance or an explicit valid choice rather than silently substituting a scorer or treating a raw refusal detector as an objective-success scorer.

Preview the selected population and effective configuration before explicit launch confirmation. On launch, reuse the normal scenario queue/progress/history. Preserve explorer state when cancelling or returning. No new persistent named dataset or generated scenario source file is created by opening the form.

Acceptance criteria:

  • Selections survive pagination and can be reviewed/cleared without selecting hidden unrequested examples.
  • Configure scenario performs no target call and uses exact IDs, not a reconstructed dataset filter.
  • Default settings mean prompt sending only, no extra converters, and an explicit compatible scorer.
  • The picker distinguishes configured instances from buildable classes and shows tags, default selection, compatibility, and prerequisites from shared typed backend/core metadata.
  • Missing objectives, invalid/stale selections or scorer names, incompatible presets/defaults, missing dependencies, and empty configuration have clear validation and setup guidance.
  • Preview, estimate, confirmation, and launch agree on the exact selected examples and effective configured scorer; duplicate launch is prevented.
  • Cancel/Back/navigation preserve the user's browse state, and success goes to existing scenario progress/history.
  • Component/routing and any added metadata-projection tests cover configured versus buildable choices, defaults/tags, missing dependencies, stale references, and the full flow using mocked APIs and no privileged datasets.

Describe alternatives you've considered, if relevant

Do not build another launch page with duplicated state, a general scorer-builder GUI, batch chat behavior, or a direct launch from a selection checkbox. Advanced technique/converter controls are item 16; this item should deliver the useful default flow first.

Additional context

This item retains the scorer-picker metadata and setup-guidance requirements from the original #2755. The general API is complete in #2990; the dataset-scenario consumer is not. Coordinate the projection with #2760 so the UI and scenario execution use one compatibility/default policy.

Start with the Datasets table, frontend\src\components\Scenarios\ScenarioDetail.tsx and its extracted form/estimate/launch boundaries, shared parameter controls, and scenariosApi. Coordinate with #2653 instead of growing a second monolithic configuration component. Follow frontend style/test instructions.

Relationship to scenario dataset generation

This is the largest UI overlap with the scenario dataset plan. Its still-open PR 2, #3143, adds ScenarioDatasetTable, useScenarioDatasets, shared ScenarioDatasetRequest/settings, and scenario-owned ANY/SUBSET/FIXED dataset options. Coordinate with that work; once landed, reuse its finalized form/model pieces where applicable instead of adding another named-dataset picker, limit contract, or browser-owned capability policy. Do not assume the open PR is already shipped.

A scenario source table selects named populations, limits, and fetch policies. This issue instead begins with exact example IDs chosen across explorer pages. Keep those IDs authoritative and reviewable; do not expose source/cap controls that silently replace or resample the selection. Multi-page selection, the configured-scorer picker, missing-objective handling, confirmation, and return-to-explorer state remain this issue's work through #2757/#2760.

Merged generation support (#2904/#3116) means stored generated objectives can participate like other seeds. It does not add generator target/instructions/count controls to this form. Opening or estimating the configuration must not generate data, and the attack target and judging scorer are not implicit generation targets. GUI generation and destructive refresh are separately approved optional follow-ups. This coordination note leaves the current prerequisites and delivery order intact.

Metadata

Metadata

Assignees

No one assigned

    Labels

    GUIUmbrella label for all feedback submitted via the Co-PyRIT GUIfeature-requestnot ready yetThis issue needs more definition or is blocked by a pending change.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions