Skip to content

FIX: warn when selected attack techniques have no registered factory - #2466

Open
fei (feiiiiii5) wants to merge 1 commit into
microsoft:mainfrom
feiiiiii5:issue2461/silent-technique-drop
Open

FIX: warn when selected attack techniques have no registered factory#2466
fei (feiiiiii5) wants to merge 1 commit into
microsoft:mainfrom
feiiiiii5:issue2461/silent-technique-drop

Conversation

@feiiiiii5

Copy link
Copy Markdown
Contributor

Description

resolve_technique_factories() in pyrit/scenario/core/matrix_atomic_attack_builder.py silently dropped selected techniques whose factory is not registered — the docstring even documented the behavior as "silently dropped". When a scenario (or a user of extra_factories) selects a technique that was never registered, e.g. due to a typo or a custom technique added under a different name, the run quietly proceeds with fewer attacks and there is no signal anywhere that selections were discarded.

This PR emits a single warning naming every missing technique, then proceeds with the remaining ones as before:

WARNING  pyrit.scenario.core.matrix_atomic_attack_builder:matrix_atomic_attack_builder.py:171 Skipping 2 selected attack technique(s) with no registered factory: missing_a, missing_b. Register the technique(s) (or pass them via extra_factories) to include them in the run.

Design notes:

  • Warning, not an exception: dropping is intentional fallback behavior (per the original docstring), and callers may legitimately probe which techniques are available; this only makes it observable. Happy to switch to raising if maintainers prefer strictness.
  • One aggregated warning per call, names deduplicated but kept in selection order, so repeated selections don't spam the log.

Fixes #2461

Tests

Added three regression tests to TestResolveTechniqueFactories:

  • warning is emitted when a selected technique has no factory, and names it
  • no warning when all selected techniques resolve
  • one warning listing each distinct missing name once, in selection order (deduplicated)

All 35 tests in tests/unit/scenario/core/test_matrix_atomic_attack_builder.py pass locally (73 failed / 569 passed elsewhere in tests/unit/scenario/ is identical to the pre-change baseline on this machine — those failures are environment-only, unrelated modules). Ruff check + format pass on both touched files.

…d factory

resolve_technique_factories silently dropped techniques whose factory was
not registered, so a typo or an unregistered custom technique shrank the
run without any signal. Emit one warning naming the missing technique(s)
in selection order (deduplicated), and update the docstring accordingly.

Fixes microsoft#2461
@romanlutz Roman Lutz (romanlutz) changed the title fix(scenario): warn when selected attack techniques have no registered factory FIX: warn when selected attack techniques have no registered factory Aug 24, 2026
seen_missing.add(technique.value)

if missing:
logger.warning(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This warning is not visible in the normal pyrit_scan flow because backend output is redirected to pyrit_backend.log, and ScenarioRunSummary does not expose warnings. Please propagate this diagnostic through the run response and render it in the CLI so users know when selected techniques were skipped.

missing.append(technique.value)
seen_missing.add(technique.value)

if missing:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If every selected technique is missing, the default baseline still lets the run complete successfully without executing any selected technique. Please raise a targeted initialization error when the selection is nonempty and no factories resolve, while keeping the warning-and-continue behavior for partial misses.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

resolve_technique_factories silently drops selected techniques with no registered factory

2 participants