FIX: warn when selected techniques have no registered factory - #2462
Closed
fei (feiiiiii5) wants to merge 6 commits into
Closed
FIX: warn when selected techniques have no registered factory#2462fei (feiiiiii5) wants to merge 6 commits into
fei (feiiiiii5) wants to merge 6 commits into
Conversation
JsonSchemaResponseHandler with numeric_value=True validated the value with a bare float() cast, so "nan", "inf" and "-inf" strings passed as valid scores. Float-scale aggregation then clamps with max(0.0, min(1.0, x)); since every comparison against NaN is False, min(1.0, nan) returns 1.0 and a NaN judge response silently aggregates toward the maximum harm score instead of surfacing an error. Treat non-finite parsed values as invalid scoring responses via math.isfinite. Regression tests cover the five accepted spellings. Fixes microsoft#2458 Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
… them as auth-disabled fetchAuthConfig collapsed both non-2xx responses and network errors into an empty AuthConfig, which AuthProvider interpreted as "authentication disabled". When /api/auth/config failed transiently while protected APIs returned 401, the app rendered the normal shell with raw "Missing or invalid Authorization header" text and no login control or error surface (microsoft#2441). The two failure paths now throw so the existing AuthProvider catch renders its Authentication Error page. A 200 response with an empty config still means auth is disabled (local dev), unchanged. Fixes microsoft#2441 Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
resolve_technique_factories silently dropped selected techniques that have no registered factory, so a scenario selecting ["a", "b", "c"] where "b" is unregistered ran with 2 of 3 techniques and no indication the selection was partially ignored (microsoft#2461). Log a warning listing the dropped selections, matching how adaptive_scenario.py already surfaces skipped techniques. Fixes microsoft#2461 Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
Contributor
Author
|
Additional internal precedent found while auditing the scenario layer: |
Roman Lutz (romanlutz)
changed the title
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Roman Lutz (romanlutz)
enabled auto-merge
August 22, 2026 13:58
Resolve docstring conflict in response_handler.py (take upstream wording).
auto-merge was automatically disabled
August 22, 2026 15:41
Head branch was pushed to by a user without write access
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root Cause
resolve_technique_factories()inpyrit/scenario/core/matrix_atomic_attack_builder.pysilently drops selected techniques that have no registered factory: the dict comprehension filters ontechnique.value in all_factorieswith no signal. A scenario selecting["a", "b", "c"]where"b"is unregistered runs with 2 of 3 techniques and no indication that the selection was partially ignored (#2461).Fix
Collect the missing selections and emit a warning listing them, consistent with how
scenarios/adaptive/adaptive_scenario.pyalready warns when it skips techniques (skipped_incompatible+logger.warning). Behaviour is otherwise unchanged — the drop itself stays, since proceeding with whatever factories exist is the documented intent.Test
test_warns_when_selected_technique_has_no_factoryasserts the warning names the dropped technique.deepeval) missing from this machine's environment; syntax-verified locally and CI will arbitrate the full suite.Diff Scope
pyrit/scenario/core/matrix_atomic_attack_builder.py: + missing-selection warningtests/unit/scenario/core/test_matrix_atomic_attack_builder.py: + regression testDevelopment was AI-assisted with human review of all changes.
Fixes #2461