Skip to content

FEAT GUI: Accept exact seed selections in scenario requests #2757

Open

Description

@romanlutz

Is your feature request related to a problem? Please describe.

A user selecting a few examples in the explorer must run exactly those examples. Current scenario requests expose dataset names, caps, and a small filter set, not an explicit selected-example contract. Reconstructing a selection from broad filters could run different data.

Work item 13 of 17 in #2744. Technical prerequisites: item 2's stable seed/group IDs and the general scorer registry API from #2755, completed in #2990. Release sequencing: keep not ready yet until both #2756 (the main explorer/provider-loading track) and #2755 (the parallel scorer API) are complete and the maintainer promotes this item. Completing #2755 early does not release this item ahead of the remaining explorer work.

Describe the solution you'd like

Extend the shared estimate/run models and configuration resolver with a bounded, typed explicit-selection input resolved from existing memory.

  • Select complete logical examples using the shipped FEAT GUI: Add paginated seed browsing API #2748/FEAT: paginated seed browsing #2989 identity: retain each API-provided selection_key with its example_id (group ID, or persisted ID for an ungrouped seed). A group ID can have members in different datasets; resolve only the selected dataset population rather than widening membership. Resolve canonical content on the backend, not from arbitrary browser-supplied file paths, copied previews, or a reconstructed dataset-name filter.
  • Preserve group membership, roles, order, source seed IDs, and existing provenance. Reuse DatasetAttackConfiguration's explicit seed/seed-group input where appropriate.
  • Browsing detail is SeedRecord inspection data, not serialized executable components. Preserve raw stored values, IDs/hashes, nullable roles/sequences, parameters, typed objective conditions and provenance. Simulated configuration remains JSON in value. Resolve/validate supported execution inputs at the existing scenario setup boundary; read-only preview/estimate must not load legacy template files, render templates, generate conversations, or silently drop unavailable members.
  • Make explicit selection mutually exclusive with conflicting dataset-name/filter/sampling overrides. Do not silently apply a scenario's default cap or random sample to an explicit selection.
  • Carry the selected configured scorer's stable registry name (scorer_registry_name) and explicit objective inputs needed by the later scenario adapter. Do not accept serialized component instances or full ScorerIdentifier objects as scenario configuration.
  • Resolve the effective scorer consistently for preview, estimate, and launch through shared registry/scenario policy. FEAT: Support custom scenarios over selected dataset examples #2760 owns compatible default selection and objective-result semantics; the request resolver must not duplicate scorer-specific compatibility rules in DTOs.
  • A missing, stale after registry reinitialization, or incompatible requested scorer must produce an actionable error and require refresh or an explicit new choice. Do not silently substitute a different scorer or turn an invalid explicit choice into a default.
  • Use the same effective selection/configuration for preview, estimate, and launch. Detect missing, changed, or invalid examples and ask the user to refresh rather than silently dropping rows or expanding to a whole dataset.
  • Preserve selected-input provenance using existing scenario/result mechanisms; do not create a new persistent dataset per selection.

Acceptance criteria:

  • Selecting N dataset-scoped examples yields those same N logical examples in resolution, with linked media, raw values, typed objective conditions, IDs/hashes and provenance intact; cross-dataset group IDs cannot widen the selection.
  • Invalid/duplicate/missing references and conflicting selection modes have defined, tested behavior.
  • Default dataset caps and filters cannot silently change an explicit selection.
  • Preview, estimate, and launch resolve the same explicit configured scorer and exact selected population. The typed contract supports the later default-selection integration in FEAT: Support custom scenarios over selected dataset examples #2760; that scenario implementation is not a prerequisite for completing this request-contract item.
  • Unknown/stale scorer names, registry reinitialization, incompatible choices, and missing prerequisites have focused tests and explicit errors without silent substitution.
  • Estimate resolution is read-only and does not fetch providers, persist seeds, or invoke models.
  • Existing named-dataset request behavior remains compatible.
  • Contract/resolver tests cover mixed examples, stale selection, provenance, and SQLite/Azure SQL-compatible memory access.

Describe alternatives you've considered, if relevant

Do not send an entire dataset to the browser, encode selections as arbitrary Python/opaque dataset_config objects, duplicate selected seeds into a new dataset, or add memory migrations. This item defines/resolves inputs; the scenario implementation is item 14.

Additional context

#2990 supplies general scorer discovery and creation, not objective-default or scenario-selection policy. Reuse it rather than adding another catalog or construction path. Registry names are runtime references, not durable saved scorer configuration.

Start with pyrit\models\catalog\scenario.py, pyrit\backend\services\scenario_configuration_resolver.py, scenario estimate/run services, and pyrit\scenario\core\dataset_configuration.py. DATASET_FILTERS currently exposes harm_categories and data_types; exact selection needs an intentional shared contract. Follow framework/model/scenario/database/test instructions.

Relationship to scenario dataset generation

Coordinate this shared request/resolver work with the scenario dataset plan, merged source/preparation APIs in #2956/#3116, and the still-open #3143. #3143 introduces ScenarioDatasetRequest, source limits/fetch settings, and scenario-owned override capabilities. Reuse the finalized shared models/resolver when available rather than creating a parallel scenario dataset contract. Its named-source selection does not implement this issue's exact example references.

Keep exact selection a distinct, mutually exclusive mode when source/filter/sampling/fetch overrides would alter its population. Neither per-source nor total defaults may reduce it: named attack sources currently default to five per source, and inline attack inputs default to five total. null/default retains defaults; it does not mean uncapped. Include a case selecting more than five examples to protect the exact-N contract.

Do not reconstruct selected rows from named-source filters: scenario harm-category filters and browsing facets do not have identical matching semantics. Nor may if_missing generation recreate a deleted selection by name. Resolve the stored dataset-scoped IDs and validate stale/changed records; future explicit refresh must make stale selections fail clearly, not choose replacements. Preserve stored generated origin/metadata without requiring a generation target. Browsing, estimates, and resume remain free of provider calls and writes. These links add no new native dependency or early release permission.

Metadata

Metadata

Assignees

No one assigned

    Labels

    GUIUmbrella label for all feedback submitted via the Co-PyRIT GUIfeature-requestnot ready yetThis issue needs more definition or is blocked by a pending change.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions