Repository navigation
FEAT GUI: Accept exact seed selections in scenario requests #2757
Description
Is your feature request related to a problem? Please describe.
A user selecting a few examples in the explorer must run exactly those examples. Current scenario requests expose dataset names, caps, and a small filter set, not an explicit selected-example contract. Reconstructing a selection from broad filters could run different data.
Work item 13 of 17 in #2744. Technical prerequisites: item 2's stable seed/group IDs and the general scorer registry API from #2755, completed in #2990. Release sequencing: keep not ready yet until both #2756 (the main explorer/provider-loading track) and #2755 (the parallel scorer API) are complete and the maintainer promotes this item. Completing #2755 early does not release this item ahead of the remaining explorer work.
Describe the solution you'd like
Extend the shared estimate/run models and configuration resolver with a bounded, typed explicit-selection input resolved from existing memory.
- Select complete logical examples using the shipped FEAT GUI: Add paginated seed browsing API #2748/FEAT: paginated seed browsing #2989 identity: retain each API-provided
selection_keywith itsexample_id(group ID, or persisted ID for an ungrouped seed). A group ID can have members in different datasets; resolve only the selected dataset population rather than widening membership. Resolve canonical content on the backend, not from arbitrary browser-supplied file paths, copied previews, or a reconstructed dataset-name filter. - Preserve group membership, roles, order, source seed IDs, and existing provenance. Reuse DatasetAttackConfiguration's explicit seed/seed-group input where appropriate.
- Browsing detail is
SeedRecordinspection data, not serialized executable components. Preserve raw stored values, IDs/hashes, nullable roles/sequences, parameters, typed objective conditions and provenance. Simulated configuration remains JSON invalue. Resolve/validate supported execution inputs at the existing scenario setup boundary; read-only preview/estimate must not load legacy template files, render templates, generate conversations, or silently drop unavailable members. - Make explicit selection mutually exclusive with conflicting dataset-name/filter/sampling overrides. Do not silently apply a scenario's default cap or random sample to an explicit selection.
- Carry the selected configured scorer's stable registry name (
scorer_registry_name) and explicit objective inputs needed by the later scenario adapter. Do not accept serialized component instances or fullScorerIdentifierobjects as scenario configuration. - Resolve the effective scorer consistently for preview, estimate, and launch through shared registry/scenario policy. FEAT: Support custom scenarios over selected dataset examples #2760 owns compatible default selection and objective-result semantics; the request resolver must not duplicate scorer-specific compatibility rules in DTOs.
- A missing, stale after registry reinitialization, or incompatible requested scorer must produce an actionable error and require refresh or an explicit new choice. Do not silently substitute a different scorer or turn an invalid explicit choice into a default.
- Use the same effective selection/configuration for preview, estimate, and launch. Detect missing, changed, or invalid examples and ask the user to refresh rather than silently dropping rows or expanding to a whole dataset.
- Preserve selected-input provenance using existing scenario/result mechanisms; do not create a new persistent dataset per selection.
Acceptance criteria:
- Selecting N dataset-scoped examples yields those same N logical examples in resolution, with linked media, raw values, typed objective conditions, IDs/hashes and provenance intact; cross-dataset group IDs cannot widen the selection.
- Invalid/duplicate/missing references and conflicting selection modes have defined, tested behavior.
- Default dataset caps and filters cannot silently change an explicit selection.
- Preview, estimate, and launch resolve the same explicit configured scorer and exact selected population. The typed contract supports the later default-selection integration in FEAT: Support custom scenarios over selected dataset examples #2760; that scenario implementation is not a prerequisite for completing this request-contract item.
- Unknown/stale scorer names, registry reinitialization, incompatible choices, and missing prerequisites have focused tests and explicit errors without silent substitution.
- Estimate resolution is read-only and does not fetch providers, persist seeds, or invoke models.
- Existing named-dataset request behavior remains compatible.
- Contract/resolver tests cover mixed examples, stale selection, provenance, and SQLite/Azure SQL-compatible memory access.
Describe alternatives you've considered, if relevant
Do not send an entire dataset to the browser, encode selections as arbitrary Python/opaque dataset_config objects, duplicate selected seeds into a new dataset, or add memory migrations. This item defines/resolves inputs; the scenario implementation is item 14.
Additional context
#2990 supplies general scorer discovery and creation, not objective-default or scenario-selection policy. Reuse it rather than adding another catalog or construction path. Registry names are runtime references, not durable saved scorer configuration.
Start with pyrit\models\catalog\scenario.py, pyrit\backend\services\scenario_configuration_resolver.py, scenario estimate/run services, and pyrit\scenario\core\dataset_configuration.py. DATASET_FILTERS currently exposes harm_categories and data_types; exact selection needs an intentional shared contract. Follow framework/model/scenario/database/test instructions.
Relationship to scenario dataset generation
Coordinate this shared request/resolver work with the scenario dataset plan, merged source/preparation APIs in #2956/#3116, and the still-open #3143. #3143 introduces ScenarioDatasetRequest, source limits/fetch settings, and scenario-owned override capabilities. Reuse the finalized shared models/resolver when available rather than creating a parallel scenario dataset contract. Its named-source selection does not implement this issue's exact example references.
Keep exact selection a distinct, mutually exclusive mode when source/filter/sampling/fetch overrides would alter its population. Neither per-source nor total defaults may reduce it: named attack sources currently default to five per source, and inline attack inputs default to five total. null/default retains defaults; it does not mean uncapped. Include a case selecting more than five examples to protect the exact-N contract.
Do not reconstruct selected rows from named-source filters: scenario harm-category filters and browsing facets do not have identical matching semantics. Nor may if_missing generation recreate a deleted selection by name. Resolve the stored dataset-scoped IDs and validate stale/changed records; future explicit refresh must make stale selections fail clearly, not choose replacements. Preserve stored generated origin/metadata without requiring a generation target. Browsing, estimates, and resume remain free of provider calls and writes. These links add no new native dependency or early release permission.
Activity