FEAT: Run one technique against an objective with technique.execute

[Richard Lundeen (richlundeen)]

Draft until we do some work to glue some paramaters together

Description

Adds technique.execute: run exactly one registered attack technique toward one free-text objective, without selecting a dataset. It can also continue from a stored manual conversation. This connects manual testing to an automated attack without changing the source conversation.

• Select any concrete technique in the active catalog. The default is red_teaming; a custom catalog without it uses the first technique in name order. Adds plain multi-turn crescendo to the opt-in extra catalog.
• Set harm_categories on the objective and its results. Use standard target selection, technique converters, and the existing adversarial target picker/context override.
• Set retries_on_objective_failure to allow additional attempts. Each attempt gets a fresh context and conversation; execution stops at the first success. max_retries remains the separate error/exception retry setting.
• Set prepended_conversation_id to copy stored history through the standard seed path. Roles, order, data types, converted values, and message metadata are retained; old request traces are removed. Prior assistant turns extend attacks' configured max_turns so manual history does not consume the new automated-turn budget. Blocked/error history is rejected explicitly.
• Baseline is off by default and remains opt-in. Direct and repeated execution share one technique-configuration path. Child results retain packaged technique/seed identities, and resume detects changes to source history through run metadata.

Technique selection contract

The scenario declares MAX_CONCRETE_TECHNIQUES = 1. The backend publishes this as max_concrete_techniques in the scenario catalog and enforces it after aggregate expansion and deduplication, for both estimates and runs. Other scenarios remain unrestricted by default.

CoPyRIT uses radio buttons when the limit is one. Selecting another technique replaces the previous selection; tags remain descriptive badges rather than bulk-selection buttons. Baseline is an independent checkbox and does not count toward the limit. Over-limit drafts cannot request an estimate or launch until corrected. The shared picker also respects larger declared limits.

The existing form renders the scenario parameters. Dataset inputs are removed through Execute.supported_parameters(). The independent generic GUI fix in #3119 makes forms honor existing declarations across all scenarios, including opaque common inputs. Until that fix lands, some generic dataset controls remain visible and must be left unset. The two changes stay in separate PRs: #3119 handles input support; this PR handles technique-selection cardinality. This change adds no common CLI flags or new technique selector parameter.

CLI

Start a backend using your normal configuration. It must register the objective and scorer targets; multi-turn techniques also need an adversarial target. --config-file selects the backend configuration when starting it.

uv run --no-sync pyrit_scan --config-file .\pyrit_conf.yaml start-server

Run one technique with one objective:

uv run --no-sync pyrit_scan run technique.execute `
  --config-file .\pyrit_conf.yaml `
  --initializers target `
  --target openai_chat `
  --techniques red_teaming `
  --objective "Describe how to stop a Python process." `
  --harm-categories process_management `
  --retries-on-objective-failure 2 `
  --max-retries 1

Here, 2 permits up to three objective attempts, stopping on success. Do not pass --dataset-names or --max-dataset-size.

To continue from a stored conversation in the same memory backend, add:

--prepended-conversation-id "<stored-conversation-UUID>"

To use plain Crescendo instead, register the extra catalog at backend startup or select it through the existing initializer syntax:

uv run --no-sync pyrit_scan run technique.execute `
  --config-file .\pyrit_conf.yaml `
  --initializers target technique:tags=core,extra `
  --target openai_chat `
  --techniques crescendo `
  --objective "Describe how to stop a Python process."

The scenario-specific arguments are discovered from the backend catalog. Adversarial target selection uses the existing GUI picker or configured default; this change adds no CLI flag for it.

GUI preview

Real CoPyRIT screenshots captured with isolated demo targets and no live model calls. The selected technique now uses a radio control; the tag badges do not change the selection:

The scenario-specific parameters and existing adversarial target picker:

Tests and Documentation

Added paired doc/scanner/technique.py / .ipynb documentation and scanner navigation. Coverage includes catalog selection, parameters, converters, baseline, source-history copying, multipart tool metadata, new-turn budgets, adversarial overrides, retries, persisted child identities, GUI/API resume, changed-source refusal, completed-run resume, and selection-count enforcement. Updated E2E argument setup to omit dataset limits for scenarios that do not accept dataset configuration.

Initial scenario implementation tests — passed, 805 tests:

uv run --no-sync pytest tests\unit\scenario\scenarios\technique tests\unit\scenario\core\test_atomic_attack.py tests\unit\scenario\test_default_run_size_estimates.py tests\unit\scenario\test_technique_converters_support.py tests\unit\scenario\core\test_attack_technique_factory.py tests\unit\setup\test_technique_initializer.py tests\unit\setup\techniques\test_extra_techniques.py tests\unit\executor\attack\compound\test_sequential_attack.py tests\unit\executor\attack\component\test_conversation_manager.py tests\unit\models\test_seed_prompt.py tests\unit\common\test_lazy_package_imports.py -q --tb=short --disable-warnings

Technique-selection backend tests — passed, 251 tests:

uv run --no-sync pytest tests\unit\scenario\scenarios\technique tests\unit\backend\test_scenario_service.py tests\unit\registry\test_scenario_registry.py tests\unit\scenario\test_default_run_size_estimates.py -q --tb=short --disable-warnings

Catalog/request regressions — passed, 121 tests:

uv run --no-sync pytest tests\unit\models\test_scenario_catalog.py tests\unit\models\test_scenario_request.py tests\unit\backend\test_scenario_run_routes.py -q --tb=short --disable-warnings

Frontend scenario-form tests — passed, 72 tests; type check and build — passed:

# From frontend:
npm test -- --runInBand --silent --runTestsByPath src\components\Scenarios\ScenarioDetail.test.tsx
npm run type-check
npm run build

All pre-commit hooks — passed after the selection changes and again after the merged-branch type fix:

$env:UV_NO_SYNC = '1'
uv run --no-sync pre-commit run --all-files

The Linux merged-branch check found a cast made redundant by main's newer typed factory builder. Removed the cast and aligned the local builder's return annotation. Follow-up tests — passed, 160 tests; type check — passed:

uv run --no-sync pytest tests\unit\registry\test_attack_technique_registry.py tests\unit\scenario\scenarios\technique -q --tb=short --disable-warnings
uv run --no-sync ty check pyrit --output-format concise

JupyText — passed; outputs retained and paired sources checked:

uv run --no-sync jupytext --to ipynb --execute --set-kernel python3 doc\scanner\technique.py

The notebook's catalog cells run without model calls. Its configured-target SDK example is documentation, not a live-service test.

CLI smoke run — passed through the real CLI and backend with isolated demo targets and mocked scorer I/O:

uv run --no-sync pyrit_scan run technique.execute --server-url http://127.0.0.1:8097 --auth-mode none --target demo_objective --techniques red_teaming --objective "Describe how to stop a Python process." --harm-categories process_management --retries-on-objective-failure 2 --max-retries 1

Also checked the live catalog reports max_concrete_techniques=1, an estimate with two techniques returns HTTP 400, GUI radio selection is exclusive, and baseline does not change it. No live model calls or full test suite were run. uv.lock is unchanged.

[Roman Lutz (romanlutz)]

🔴 Must Fix: A finished child can hide an incomplete retry sequence.

I reproduced this with real SQLite storage and PromptSendingAttack: set retries_on_objective_failure=2 and max_retries=1, let the first child finish with FAILURE, then raise a transport error in the second child. The error retry sends nothing and marks the scenario COMPLETED. Resuming the saved run also sends nothing, so the remaining objective attempts never happen.

SequentialAttack gives each child the scenario collection and the compound's parent_eval_hash. Scenario._get_completed_objective_hashes_by_attack_async() therefore treats the first failed child as completion of the whole unit, even though the compound ended in ERROR.

Please track completion of the compound separately from completion of its children, and cover an exception after an earlier child has completed in both the error-retry and saved-run resume tests.

[Roman Lutz (romanlutz)]

🔴 Must Fix: Request converters change the saved history a second time.

SeedPrompt.from_messages() copies the values the target already received, but the attack's default PrependedConversationConfig applies request converters to the copied user messages again. The source database row stays unchanged, yet the new target receives different history.

For example, I stored a user turn whose converted value is c29ydCBhIGxpc3Q= and continued with Base64Converter in technique_converters. At the real target boundary, that historical turn became YzI5eWRDQmhJR3hwYzNRPQ==. With a model-based converter this can also rewrite old turns and make additional model calls.

Please keep already-sent history out of the request-conversion pipeline while still applying the selected converters to new outgoing turns. This needs to cover both converters baked into the technique and additional technique_converters.

[Roman Lutz (romanlutz)]

🟡 Should Fix: The wrapper is counted as another objective attempt.

After a real SQLite run with retries_on_objective_failure=2, three target calls, and no exceptions, compute_scenario_statistics(result) reports attempts=4 and overall.retries=3. The three child results and the orchestration envelope all resolve to the same execution unit. In the late transport-error case, the child ERROR and wrapper ERROR also count the same failure twice.

The planned/completed unit count is correctly one, but retry and error pressure shown by the SDK, backend, and reports are inflated. Please distinguish orchestration rows from actual attempt rows when calculating these statistics, and add a persisted-result check that three objective attempts produce two retries, not three.