Skip to content

FIX: Show only supported scenario inputs in scanner forms - #3119

Merged
Richard Lundeen (richlundeen) merged 5 commits into
microsoft:mainfrom
richlundeen:richlundeen-supported-scenario-inputs
Oct 11, 2026
Merged

Richard Lundeen (richlundeen) merged 5 commits into
microsoft:mainfrom
richlundeen:richlundeen-supported-scenario-inputs

Conversation

@richlundeen

@richlundeen Richard Lundeen (richlundeen) commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Description

The scanner form showed common input fields for every scenario, even when a scenario explicitly removed those inputs from supported_parameters(). Users could enter values that the scenario could not accept, causing estimate or launch failures.

For example, Garak Exploitation (garak.exploitation) uses a fixed reference corpus and excludes dataset_config. However, its GUI still showed Dataset override, Max dataset size, Harm categories, and Data types. These request fields all map to the unsupported dataset_config input.

This fix makes controls, validation, previews, and estimate/launch requests follow each scenario's declared inputs. Hidden inputs and stale unsupported drafts are not submitted. The catalog exposes supported_parameter_names from all declarations because the existing typed supported_parameters list excludes opaque inputs such as dataset_config. Omitted common inputs also stay omitted through launch and resume.

Related: #3118 adds technique.execute, which also excludes dataset configuration. This PR fixes the generic GUI issue for all scenarios. The separate one-technique selection rule remains in #3118.

Expose all declared parameter names and gate controls, validation, and request aliases. Preserve omitted common inputs across launch and resume.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c198c14a-c4c7-4e5b-b068-1e62287c0d92
@romanlutz

Copy link
Copy Markdown
Contributor

Must Fix: Preserve the effective defaults for nullable retry/concurrency inputs

RunScenarioRequest now accepts max_concurrency: null and max_retries: null. Preparation treats these values as omitted, so the scenario uses its declared defaults, but _initialize_scenario_async saves the original null values. This check then prevents that run from resuming after a failure.

For example, this is now a valid launch request:

{"scenario_name": "garak.exploitation", "target_name": "my_target", "max_retries": null}

I reproduced the save/resume path with a real scenario registry and SQLite: the resolved retry count was 0, the saved retry count was null, and POST /api/scenarios/runs/{id}/resume returned HTTP 409 with "The saved launch configuration is incomplete". Omitting the field resumed successfully.

Could we save the resolved numeric values when these inputs are supported, while keeping None for unsupported inputs? A real save/resume regression test for each nullable field would catch this.

Location: scenario_run_service.py:341-347

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c198c14a-c4c7-4e5b-b068-1e62287c0d92
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c198c14a-c4c7-4e5b-b068-1e62287c0d92
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c198c14a-c4c7-4e5b-b068-1e62287c0d92
Resolve supported numeric launch fields using the same parameter declarations and argument precedence as initialization. Keep unsupported fields omitted and cover real SQLite failure/resume with nulls, overrides, and changed defaults.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c198c14a-c4c7-4e5b-b068-1e62287c0d92
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Oct 11, 2026
Merged via the queue into microsoft:main with commit 40ac8cc Oct 11, 2026
55 checks passed
@richlundeen
Richard Lundeen (richlundeen) deleted the richlundeen-supported-scenario-inputs branch October 11, 2026 23:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants