[BREAKING] FEAT: Add scenario dataset controls to CLI, REST, and GUI

.github/instructions/scenarios.instructions.md

@@ -169,8 +169,24 @@ Options:
 - Ingredients must remain complete. Use `max_per_dataset="all"` and no finite source
   `max_size`; cap the assembled attack groups with `max_total`, not ingredient rows.
 - Use `with_overrides()` instead of reconstructing a subclass or mutating its defaults.
+- Clients and Python use `ScenarioDatasetRequest` through the common `dataset_overrides`
+  parameter. `dataset_config` still replaces the default; overrides apply on top.
+- Resolve parameter/technique-dependent names in `_resolve_default_dataset_config()`,
+  before overrides, validation, and preparation. Do not rewrite user selections in validators.
+- Declare `DATASET_OPTIONS` (or parameter-aware `get_dataset_options()`): `ANY` only for
+  generic named populations, `SUBSET` for approved alternatives, `FIXED` for ingredients.
+  Disable limits that would truncate ingredients. Compound source edits are ambiguous;
+  preserve child shaping/caps and allow only supported global edits.
+- Estimates expose `dataset_defaults`, `effective_datasets`, and `dataset_options`.
+  These are response projections, not incoming edit state. Never read seeds for projections.
+  The catalog reads options and defaults from registry metadata, never from an estimate.
+- Resume replays the saved launch request; the stored run plan fixes the selected seed groups,
+  so caps and fetch policies cannot change a resumed run. Dataset names are checked by the
+  stored scenario identity. Resume never prepares datasets.
 - Keep custom shaping in `_build_attack_groups()` or `_build_groups_by_dataset_async()`.
   Reads use `_collect_seeds_for_dataset_async()` and must not fetch.
+- Build synthesized prompts from those filtered seeds, not a second unfiltered memory query.
+  Scenario code must not call memory seed-read methods directly; the scenario contract test enforces this.
 - `dataset_names`, `max_dataset_size`, `auto_fetch`, and `per_dataset()` are deprecated.
 
 `max_dataset_size` is an exact alias for `max_total`; it does not disable source caps.

doc/code/scenarios/0_scenarios.ipynb

@@ -66,10 +66,11 @@
     "\n",
     "3. **Default Dataset**: Pass `default_dataset_config=` to `super().__init__()` to specify the datasets your scenario uses out of the box.\n",
     "   - Returns a `DatasetAttackConfiguration` with named sources (e.g., `sources=[DatasetSource(name=\"my_dataset\")]`)\n",
-    "   - Users can override this at runtime via `--dataset-names` in the CLI or by passing a custom `dataset_config` programmatically\n",
+    "   - Users select sources with `--datasets name[:N|:all]`, or pass `dataset_overrides=ScenarioDatasetRequest(...)` in `set_params_from_args`. A custom `dataset_config` replaces the default; overrides apply on top.\n",
     "   - Named sources select at most 5 attack groups per dataset by default; `max_total` caps the union. For any limit, omitted/`None`/empty/`\"default\"` uses the default; `\"all\"` removes that limit.\n",
     "   - The deprecated `max_dataset_size` is an exact alias for `max_total`, not a way to disable source caps. To keep the old total-only selection, use `max_per_dataset=\"all\", max_total=10`; to remove both caps, set both to `\"all\"`. `None` now uses the default, not unlimited selection.\n",
     "   - New runs prepare missing registered datasets once; reads, estimates, and resume never fetch\n",
+    "   - Scenarios declare allowed source selections and limit controls. Estimates return resolved `dataset_defaults` and `effective_datasets` separately from requested edits. Name-only selection preserves existing source caps and fetch policies.\n",
     "\n",
     "4. **Constructor**: Use `@apply_defaults` decorator and call `super().__init__()` with scenario metadata:\n",
     "   - `name`: Descriptive name for your scenario\n",

doc/code/scenarios/0_scenarios.py

@@ -68,10 +68,11 @@
 #
 # 3. **Default Dataset**: Pass `default_dataset_config=` to `super().__init__()` to specify the datasets your scenario uses out of the box.
 #    - Returns a `DatasetAttackConfiguration` with named sources (e.g., `sources=[DatasetSource(name="my_dataset")]`)
-#    - Users can override this at runtime via `--dataset-names` in the CLI or by passing a custom `dataset_config` programmatically
+#    - Users select sources with `--datasets name[:N|:all]`, or pass `dataset_overrides=ScenarioDatasetRequest(...)` in `set_params_from_args`. A custom `dataset_config` replaces the default; overrides apply on top.
 #    - Named sources select at most 5 attack groups per dataset by default; `max_total` caps the union. For any limit, omitted/`None`/empty/`"default"` uses the default; `"all"` removes that limit.
 #    - The deprecated `max_dataset_size` is an exact alias for `max_total`, not a way to disable source caps. To keep the old total-only selection, use `max_per_dataset="all", max_total=10`; to remove both caps, set both to `"all"`. `None` now uses the default, not unlimited selection.
 #    - New runs prepare missing registered datasets once; reads, estimates, and resume never fetch
+#    - Scenarios declare allowed source selections and limit controls. Estimates return resolved `dataset_defaults` and `effective_datasets` separately from requested edits. Name-only selection preserves existing source caps and fetch policies.
 #
 # 4. **Constructor**: Use `@apply_defaults` decorator and call `super().__init__()` with scenario metadata:
 #    - `name`: Descriptive name for your scenario

doc/code/scenarios/1_common_scenario_parameters.ipynb

@@ -15,7 +15,7 @@
     "\n",
     "> **Two selection axes**: *Techniques* select attack techniques (*how* attacks run — e.g., prompt\n",
     "> sending, role play, TAP). *Datasets* select objectives (*what* is tested — e.g., harm categories,\n",
-    "> compliance topics). Use `--dataset-names` on the CLI to filter by content category.\n",
+    "> compliance topics). Use `--datasets` on the CLI to filter by content category.\n",
     "\n",
     "> **Running scenarios from the command line?** See the [Scanner documentation](../../scanner/0_scanner.md).\n",
     "\n",

doc/code/scenarios/1_common_scenario_parameters.py

@@ -19,7 +19,7 @@
 #
 # > **Two selection axes**: *Techniques* select attack techniques (*how* attacks run — e.g., prompt
 # > sending, role play, TAP). *Datasets* select objectives (*what* is tested — e.g., harm categories,
-# > compliance topics). Use `--dataset-names` on the CLI to filter by content category.
+# > compliance topics). Use `--datasets` on the CLI to filter by content category.
 #
 # > **Running scenarios from the command line?** See the [Scanner documentation](../../scanner/0_scanner.md).
 #

doc/code/scenarios/3_adaptive_scenarios.ipynb

@@ -804,7 +804,7 @@
     "# Use specific datasets and limit size\n",
     "pyrit_scan --scenario TextAdaptive --target openai_chat \\\n",
     "    --datasets airt_hate airt_violence \\\n",
-    "    --max-dataset-size 10\n",
+    "    --max-total 10\n",
     "```"
    ]
   }

doc/code/scenarios/3_adaptive_scenarios.py

@@ -301,5 +301,5 @@ def _technique_label(result) -> str:
 # # Use specific datasets and limit size
 # pyrit_scan --scenario TextAdaptive --target openai_chat \
 #     --datasets airt_hate airt_violence \
-#     --max-dataset-size 10
+#     --max-total 10
 # ```

doc/gui/0_gui.md

@@ -505,7 +505,29 @@ Defaults are saved in this browser, separately for each signed-in account. They
 
 A missing or changed default is shown as unavailable. Select a new default or clear it; CoPyRIT does not silently substitute a different target. If browser storage is unavailable, a warning states that the choice applies only in the current session.
 
-Scanner forms let you override either selection for one run. **Objective Target** is the target the scenario runs against. **Adversarial Target**, under **Parameters** above **Dataset override**, is the target used to generate attacks. The adversarial selector appears only for scenarios whose available techniques use the shared default; scenarios with no such use, including the explicit-target adversarial benchmark, do not show it or send an override. **Use server default** clears the per-run adversarial override. Changes to your defaults do not change existing chats or queued/running scans. Explicit adversarial targets in a scenario or technique still take priority. Scorer targets are unchanged.
+Scanner forms let you override either selection for one run. **Objective Target** is the target the scenario runs against. **Adversarial Target**, under **Parameters** above **Datasets**, is the target used to generate attacks. The adversarial selector appears only for scenarios whose available techniques use the shared default; scenarios with no such use, including the explicit-target adversarial benchmark, do not show it or send an override. **Use server default** clears the per-run adversarial override. Changes to your defaults do not change existing chats or queued/running scans. Explicit adversarial targets in a scenario or technique still take priority. Scorer targets are unchanged.
+
+### Scenario datasets
+
+The **Datasets** table shows each selected name, stored seed origins (including mixed
+origins), stored example count, source limit, and fetch policy. Unknown counts for
+provider-only datasets are not zero. Provider availability is separate from stored origin.
+**Reload dataset metadata** refreshes the page-scoped catalog without fetching seed data.
+Runtime changes and launches also invalidate this catalog.
+
+Use **Add dataset** for allowed alternatives or named memory datasets, **Remove** to
+change the selection, and **Use default sources** to restore parameter-aware defaults.
+FigStep allows one of `figstep` or `figstep_pro`; ingredient and compound sources are fixed.
+Unavailable limit/source controls are not submitted. Parameter and technique changes
+refresh resolved defaults without discarding edits; invalid selections display an error.
+
+**Max per dataset**, source limits, and **Max total** accept positive integers, `all`,
+or `default`/blank. Source caps apply before the total; `all` removes only that cap.
+Name-only selection retains existing explicit source settings. Fetch is `never` or
+`if_missing`: only a new run may prepare missing registered datasets. Preview and launch
+send the same edit-only dataset object, with harm/category filters preserved. Resume uses
+the saved request and the original seed selection; a different dataset selection is
+rejected. Resume never prepares datasets.
 
 #### Core Adversarial Default Override
 

doc/scanner/1_pyrit_scan.ipynb

@@ -1194,9 +1194,17 @@
     "on demand, so a full dataset preload is not required:\n",
     "\n",
     "```shell\n",
-    "pyrit_scan run garak.figstep --target openai_chat --dataset-names figstep_pro --max-dataset-size 1\n",
+    "pyrit_scan run garak.figstep --target openai_chat --datasets figstep_pro --max-total 1 --dataset-fetch if_missing\n",
     "```\n",
     "\n",
+    "Scanner and shell share `--datasets name[:N|:all]`, `--max-per-dataset`, `--max-total`,\n",
+    "and `--dataset-fetch never|if_missing`. Source limits apply before the combined total.\n",
+    "Omitted, empty, or `default` keeps the scenario setting; `all` removes only its own cap.\n",
+    "`list-scenarios` displays allowed sources and defaults. Ingredient/compound scenarios\n",
+    "restrict controls to preserve their shaping. `--dataset-filters` works with either form.\n",
+    "Deprecated `--dataset-names` and `--max-dataset-size` cannot be mixed with the new flags.\n",
+    "Discovery and estimates never fetch; only a new run prepares absent registered datasets.\n",
+    "\n",
     "Custom initialization scripts are loaded by the backend at startup: list them in the\n",
     "`initialization_scripts` section of the config the server loads (paths are relative to your\n",
     "working directory, but full paths avoid confusion). Once the server is running with that\n",

doc/scanner/1_pyrit_scan.py

@@ -125,9 +125,17 @@
 # on demand, so a full dataset preload is not required:
 #
 # ```shell
-# pyrit_scan run garak.figstep --target openai_chat --dataset-names figstep_pro --max-dataset-size 1
+# pyrit_scan run garak.figstep --target openai_chat --datasets figstep_pro --max-total 1 --dataset-fetch if_missing
 # ```
 #
+# Scanner and shell share `--datasets name[:N|:all]`, `--max-per-dataset`, `--max-total`,
+# and `--dataset-fetch never|if_missing`. Source limits apply before the combined total.
+# Omitted, empty, or `default` keeps the scenario setting; `all` removes only its own cap.
+# `list-scenarios` displays allowed sources and defaults. Ingredient/compound scenarios
+# restrict controls to preserve their shaping. `--dataset-filters` works with either form.
+# Deprecated `--dataset-names` and `--max-dataset-size` cannot be mixed with the new flags.
+# Discovery and estimates never fetch; only a new run prepares absent registered datasets.
+#
 # Custom initialization scripts are loaded by the backend at startup: list them in the
 # `initialization_scripts` section of the config the server loads (paths are relative to your
 # working directory, but full paths avoid confusion). Once the server is running with that

doc/scanner/adaptive.ipynb

@@ -26,8 +26,8 @@
     "pyrit_scan run adaptive.text_adaptive \\\n",
     "  --initializers target \\\n",
     "  --target openai_chat \\\n",
-    "  --dataset-names airt_hate \\\n",
-    "  --max-dataset-size 2 \\\n",
+    "  --datasets airt_hate \\\n",
+    "  --max-total 2 \\\n",
     "  --max-attempts-per-objective 2\n",
     "```"
    ]

doc/scanner/adaptive.py

@@ -25,8 +25,8 @@
 # pyrit_scan run adaptive.text_adaptive \
 #   --initializers target \
 #   --target openai_chat \
-#   --dataset-names airt_hate \
-#   --max-dataset-size 2 \
+#   --datasets airt_hate \
+#   --max-total 2 \
 #   --max-attempts-per-objective 2
 # ```
 

doc/scanner/airt.ipynb

@@ -102,8 +102,8 @@
     "  --initializers target \\\n",
     "  --target openai_chat \\\n",
     "  --techniques role_play_movie_script \\\n",
-    "  --dataset-names airt_hate \\\n",
-    "  --max-dataset-size 1\n",
+    "  --datasets airt_hate \\\n",
+    "  --max-total 1\n",
     "```\n",
     "\n",
     "**Available techniques:** ALL, DEFAULT, SINGLE_TURN, MULTI_TURN, role_play_movie_script, many_shot, tap"
@@ -255,8 +255,8 @@
     "\n",
     "Each sub-harm escalates a simulated multi-turn conversation toward the objective, then layers the\n",
     "selected converter techniques on top (natural-language rewrites that preserve emotional framing;\n",
-    "obfuscation converters like base64/morse are excluded). Datasets are bound to the sub-harms, so\n",
-    "`--dataset-names` is ignored (`--max-dataset-size` still applies).\n",
+    "obfuscation converters like base64/morse are excluded). `--datasets` can select a subset of the\n",
+    "selected sub-harms' datasets; `--max-total` must cover each selected sub-harm.\n",
     "\n",
     "**Available techniques:** ALL, DEFAULT, tone, language, persuasion, deterministic, crescendo"
    ]
@@ -413,7 +413,7 @@
     "  --initializers target \\\n",
     "  --target openai_chat \\\n",
     "  --techniques role_play_movie_script \\\n",
-    "  --max-dataset-size 1\n",
+    "  --max-total 1\n",
     "```\n",
     "\n",
     "**Available techniques:** Use `pyrit_scan run airt.cyber --list-scenario-parameters` to inspect\n",
@@ -570,8 +570,8 @@
     "pyrit_scan run airt.jailbreak \\\n",
     "  --initializers target \\\n",
     "  --target openai_chat \\\n",
-    "  --dataset-names harmbench \\\n",
-    "  --max-dataset-size 1\n",
+    "  --datasets harmbench \\\n",
+    "  --max-total 1\n",
     "```\n",
     "\n",
     "**Available technique selectors:** ALL, DEFAULT, and SINGLE_TURN currently select both\n",
@@ -730,8 +730,8 @@
     "pyrit_scan airt.multilingual \\\n",
     "  --initializers target \\\n",
     "  --target openai_chat \\\n",
-    "  --dataset-names harmbench \\\n",
-    "  --max-dataset-size 1\n",
+    "  --datasets harmbench \\\n",
+    "  --max-total 1\n",
     "```\n",
     "\n",
     "**Available techniques:** `prompt_sending` is the default. Every registry technique (`role_play_*`,\n",
@@ -870,7 +870,7 @@
     "plagiarism detection.\n",
     "\n",
     "```bash\n",
-    "pyrit_scan run airt.leakage --target openai_chat --techniques first_letter --max-dataset-size 1\n",
+    "pyrit_scan run airt.leakage --target openai_chat --techniques first_letter --max-total 1\n",
     "```\n",
     "\n",
     "**Available techniques:** ALL, SINGLE_TURN, MULTI_TURN, IP, SENSITIVE_DATA, FirstLetter, Image, RolePlay, Crescendo\n",
@@ -1049,7 +1049,7 @@
     "  --initializers target \\\n",
     "  --target openai_chat \\\n",
     "  --techniques context_compliance \\\n",
-    "  --max-dataset-size 1\n",
+    "  --max-total 1\n",
     "```\n",
     "\n",
     "**Available techniques:** ALL, DEFAULT, SINGLE_TURN, MULTI_TURN, ContextCompliance, RolePlay,\n",

doc/scanner/airt.py

@@ -43,8 +43,8 @@
 #   --initializers target \
 #   --target openai_chat \
 #   --techniques role_play_movie_script \
-#   --dataset-names airt_hate \
-#   --max-dataset-size 1
+#   --datasets airt_hate \
+#   --max-total 1
 # ```
 #
 # **Available techniques:** ALL, DEFAULT, SINGLE_TURN, MULTI_TURN, role_play_movie_script, many_shot, tap
@@ -84,8 +84,8 @@
 #
 # Each sub-harm escalates a simulated multi-turn conversation toward the objective, then layers the
 # selected converter techniques on top (natural-language rewrites that preserve emotional framing;
-# obfuscation converters like base64/morse are excluded). Datasets are bound to the sub-harms, so
-# `--dataset-names` is ignored (`--max-dataset-size` still applies).
+# obfuscation converters like base64/morse are excluded). `--datasets` can select a subset of the
+# selected sub-harms' datasets; `--max-total` must cover each selected sub-harm.
 #
 # **Available techniques:** ALL, DEFAULT, tone, language, persuasion, deterministic, crescendo
 
@@ -124,7 +124,7 @@
 #   --initializers target \
 #   --target openai_chat \
 #   --techniques role_play_movie_script \
-#   --max-dataset-size 1
+#   --max-total 1
 # ```
 #
 # **Available techniques:** Use `pyrit_scan run airt.cyber --list-scenario-parameters` to inspect
@@ -169,8 +169,8 @@
 # pyrit_scan run airt.jailbreak \
 #   --initializers target \
 #   --target openai_chat \
-#   --dataset-names harmbench \
-#   --max-dataset-size 1
+#   --datasets harmbench \
+#   --max-total 1
 # ```
 #
 # **Available technique selectors:** ALL, DEFAULT, and SINGLE_TURN currently select both
@@ -213,8 +213,8 @@
 # pyrit_scan airt.multilingual \
 #   --initializers target \
 #   --target openai_chat \
-#   --dataset-names harmbench \
-#   --max-dataset-size 1
+#   --datasets harmbench \
+#   --max-total 1
 # ```
 #
 # **Available techniques:** `prompt_sending` is the default. Every registry technique (`role_play_*`,
@@ -254,7 +254,7 @@
 # plagiarism detection.
 #
 # ```bash
-# pyrit_scan run airt.leakage --target openai_chat --techniques first_letter --max-dataset-size 1
+# pyrit_scan run airt.leakage --target openai_chat --techniques first_letter --max-total 1
 # ```
 #
 # **Available techniques:** ALL, SINGLE_TURN, MULTI_TURN, IP, SENSITIVE_DATA, FirstLetter, Image, RolePlay, Crescendo
@@ -308,7 +308,7 @@
 #   --initializers target \
 #   --target openai_chat \
 #   --techniques context_compliance \
-#   --max-dataset-size 1
+#   --max-total 1
 # ```
 #
 # **Available techniques:** ALL, DEFAULT, SINGLE_TURN, MULTI_TURN, ContextCompliance, RolePlay,

doc/scanner/benchmark.ipynb

@@ -35,7 +35,7 @@
     "  --target openai_chat \\\n",
     "  --adversarial-targets adversarial_chat \\\n",
     "  --techniques role_play_video_game \\\n",
-    "  --max-dataset-size 1\n",
+    "  --max-total 1\n",
     "```\n",
     "\n",
     "Pass multiple `--adversarial-targets` values to compare across models in a single run.\n",

doc/scanner/benchmark.py

@@ -34,7 +34,7 @@
 #   --target openai_chat \
 #   --adversarial-targets adversarial_chat \
 #   --techniques role_play_video_game \
-#   --max-dataset-size 1
+#   --max-total 1
 # ```
 #
 # Pass multiple `--adversarial-targets` values to compare across models in a single run.

doc/scanner/foundry.ipynb

@@ -73,7 +73,7 @@
     "**CLI example:**\n",
     "\n",
     "```bash\n",
-    "pyrit_scan run foundry.red_team_agent --target openai_chat --techniques base64 --max-dataset-size 1\n",
+    "pyrit_scan run foundry.red_team_agent --target openai_chat --techniques base64 --max-total 1\n",
     "```\n",
     "\n",
     "**Available techniques by difficulty:**\n",

doc/scanner/foundry.py

@@ -40,7 +40,7 @@
 # **CLI example:**
 #
 # ```bash
-# pyrit_scan run foundry.red_team_agent --target openai_chat --techniques base64 --max-dataset-size 1
+# pyrit_scan run foundry.red_team_agent --target openai_chat --techniques base64 --max-total 1
 # ```
 #
 # **Available techniques by difficulty:**