Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions .github/instructions/targets.instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,41 @@ class MyTarget(PromptTarget):
``send_prompt_async`` (the public entry point) is ``@final`` and MUST NOT
be overridden. Override ``_send_prompt_to_target_async`` instead.

## Releasing per-conversation state

Attacks call ``reset_conversation_async(*, conversation_id)`` from
``_teardown_async`` when they are done with a conversation id. The base
implementation is a no-op, so a target that keeps no state between calls
does not need to do anything.

The attack decides when a conversation is over and says so; the target only
releases what it holds. Only the **objective** target is reset. Adversarial,
scorer and converter targets have their own lifetimes and are out of scope.

An attack whose context keeps the live conversation somewhere other than
``conversation_id`` or ``session.conversation_id`` should expose it as a
``conversation_id`` property, the way ``TAPAttackContext`` reports the best
branch. That is the same property the error-result builder reads.

Targets that hold external state keyed by conversation (a websocket
connection, a browser page, an upstream session) SHOULD override it to
release that state:

```python
async def reset_conversation_async(self, *, conversation_id: str) -> None:
connection = self._connections.pop(conversation_id, None)
if connection:
await connection.close()
```

It is best-effort cleanup, so an implementation SHOULD NOT raise for an
unknown conversation id and SHOULD be safe to call more than once for the
same id. The attack logs and swallows anything that does raise, so a
failure here never replaces the error the attack was reporting.

Closing the whole target rather than one conversation is a different
concern and stays in ``cleanup_target_async``.

## Keyword-only ``__init__`` is enforced

Every ``PromptTarget`` subclass MUST make all ``__init__`` parameters
Expand Down
16 changes: 16 additions & 0 deletions doc/code/targets/0_prompt_targets.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,22 @@ Target-side rate limiting remains independent and continues to use `limit_reques
`_send_prompt_to_target_async(*, normalized_conversation: list[Message]) -> list[Message]` instead of
overriding `send_prompt_async`.

## Releasing per-conversation state

Some targets hold state for a conversation outside of PyRIT's memory: an open websocket, a browser page, a session on the far side of an HTTP API. When an attack is finished with a conversation, it calls

```
async def reset_conversation_async(self, *, conversation_id: str) -> None:
```

on the objective target for every conversation the run used. That includes conversations the attack abandoned partway through, such as a `PromptSendingAttack` retry or a `CrescendoAttack` backtrack.

The base implementation does nothing, so a target that keeps no state between calls does not need to override it. `RealtimeTarget` and `WebsocketTarget` override it to close the websocket each caches per conversation. If you write a target that holds something similar, override it and release that state there. Do not raise for a conversation id you do not recognize, since the attack calls this while it is tearing down and treats it as best effort.

The reset runs from the attack's teardown, which is in the `finally` of the execution lifecycle, so it covers runs that succeed, runs that raise and runs that are cancelled. An error from your implementation is logged and swallowed, but a `CancelledError` is not: cancelling a run while it is releasing stops the release, and whatever is left is `cleanup_target_async`'s job.

Two things are deliberately out of scope. **Only the objective target is reset.** An attack can also drive an adversarial chat target, a scorer target and converter targets; those have their own lifetimes and are not released here, which is why the adversarial conversations an attack records are skipped. And **closing the target as a whole** is a different lifetime from releasing one conversation, so it stays where it is rather than moving into this hook.

## Chat-style targets vs general targets

A `PromptTarget` is a generic place to send a prompt. With PyRIT, the idea is that it will eventually be consumed by an AI application, but that doesn't have to be immediate. For example, you could have a SharePoint target. Everything you send a prompt to is a `PromptTarget`. Many attacks work generically with any `PromptTarget` including `RedTeamingAttack` and `PromptSendingAttack`.
Expand Down
3 changes: 0 additions & 3 deletions pyrit/executor/attack/compound/sequential_attack.py
Original file line number Diff line number Diff line change
Expand Up @@ -242,9 +242,6 @@ def _validate_context(self, *, context: AttackContext[AttackParameters]) -> None
async def _setup_async(self, *, context: AttackContext[AttackParameters]) -> None:
"""No-op: per-child-attack setup is owned by each inner strategy's executor."""

async def _teardown_async(self, *, context: AttackContext[AttackParameters]) -> None:
"""No-op: per-child-attack teardown is owned by each inner strategy's executor."""

async def _perform_async(self, *, context: AttackContext[AttackParameters]) -> SequentialAttackResult:
results: list[AttackResult] = []

Expand Down
109 changes: 104 additions & 5 deletions pyrit/executor/attack/core/attack_strategy.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
AttackResult,
ComponentIdentifier,
ConversationReference,
ConversationType,
ConverterIdentifier,
Identifiable,
Message,
Expand Down Expand Up @@ -168,6 +169,31 @@ def next_message(self, value: Message | None) -> None:
self._next_message_override = value


def _resolve_live_conversation_id(*, context: AttackContext[Any]) -> str | None:
"""
Return the objective-target conversation the run is currently using.

Single-turn contexts expose it directly and multi-turn contexts keep it on
their conversation session. ``TAPAttackContext`` overrides
``conversation_id`` to report the best branch, so the first lookup covers it.

This is the lookup #2322 gave the error-result builder, moved here so that
builder and the teardown reset resolve a run's conversation the same way
rather than walking the context twice.

Args:
context (AttackContext[Any]): The context for the attack.

Returns:
str | None: The conversation id, or ``None`` when the context exposes
neither layout.
"""
candidate = getattr(context, "conversation_id", None) or getattr(
getattr(context, "session", None), "conversation_id", None
)
return candidate if isinstance(candidate, str) and candidate else None


class _DefaultAttackStrategyEventHandler(StrategyEventHandler[AttackStrategyContextT, AttackStrategyResultT]):
"""
Default event handler for attack strategies.
Expand Down Expand Up @@ -384,11 +410,7 @@ async def _on_error_async(
collector = get_retry_collector()
retry_events = collector.events if collector else []

# Multi-turn contexts keep the active ID on their conversation session.
conversation_id = getattr(context, "conversation_id", None)
if not conversation_id:
conversation_id = getattr(getattr(context, "session", None), "conversation_id", None)
conversation_id = conversation_id or str(uuid.uuid4())
conversation_id = _resolve_live_conversation_id(context=context) or str(uuid.uuid4())

error_result = AttackResult(
conversation_id=conversation_id,
Expand Down Expand Up @@ -685,6 +707,83 @@ def get_request_converters(self) -> list[Any]:
"""
return self._request_converters

def _get_objective_conversation_ids(self, *, context: AttackStrategyContextT) -> list[str]:
"""
Collect every objective-target conversation id this run used.

This is ``AttackResult.get_active_conversation_ids()`` read off the
context instead of the result: the live conversation plus the ones
recorded as ``PRUNED``. A run leaves conversations behind whenever it
mints a fresh id mid-run, which a ``PromptSendingAttack`` retry, a
Crescendo backtrack, the single-turn rotation in multi-turn attacks and
TAP branching all do. Those still hold target-side state.

The two are pinned equal by test. Reading the context rather than the
result is what lets teardown run on the paths where no result exists,
which is every failed and every cancelled run.

Adversarial, scorer and converter conversations belong to other targets
and are deliberately not included; ``get_active_conversation_ids()``
excludes them for the same reason.

An attack that keeps its live conversation somewhere else should expose
it as ``conversation_id`` on its context, the way ``TAPAttackContext``
reports the best branch, rather than overriding this. That keeps one
lookup, and it is the same property the error-result builder reads.

Args:
context (AttackStrategyContextT): The context for the attack.

Returns:
list[str]: Conversation ids to release, in no particular order and
without duplicates.
"""
ids: list[str] = []

live = _resolve_live_conversation_id(context=context)
if live:
ids.append(live)

ids.extend(
ref.conversation_id
for ref in context.related_conversations
if ref.conversation_type == ConversationType.PRUNED
)
return list(dict.fromkeys(ids))

async def _teardown_async(self, *, context: AttackStrategyContextT) -> None:
"""
Release the objective target's state for the run's conversations.

Hands each conversation id to ``PromptTarget.reset_conversation_async``
so targets holding external state keyed by conversation (a websocket
connection, a browser page) can close it. The base target
implementation is a no-op, so this is inert for stateless targets.

This pass covers the objective target only. Adversarial, scorer and
converter targets have their own lifetimes and are not released here.

This runs in the ``finally`` of the execution lifecycle, so it covers
runs that succeed, runs that raise and runs that are cancelled. An
``Exception`` from a target is logged rather than allowed to replace
whatever error the attack was already reporting. Cancellation is not
caught: if the run is cancelled while this is releasing, it propagates
and the conversations after it are left to ``cleanup_target_async``,
because swallowing a ``CancelledError`` to finish a cleanup loop is
worse than not finishing it.

Subclasses that need their own teardown should override this and call
``await super()._teardown_async(context=context)``.

Args:
context (AttackStrategyContextT): The context for the attack.
"""
for conversation_id in self._get_objective_conversation_ids(context=context):
try:
await self._objective_target.reset_conversation_async(conversation_id=conversation_id)
except Exception as e: # noqa: BLE001 - teardown runs in a finally; never mask the attack's own error
self._logger.warning(f"Error resetting conversation {conversation_id} on the objective target: {e}")

async def execute_with_context_async(self, *, context: AttackStrategyContextT) -> AttackStrategyResultT:
"""
Execute an attack and persist its completed result after teardown.
Expand Down
8 changes: 0 additions & 8 deletions pyrit/executor/attack/multi_turn/chunked_request.py
Original file line number Diff line number Diff line change
Expand Up @@ -387,11 +387,3 @@ async def _score_combined_value_async(
):
scores = await self._objective_scorer.score_text_async(text=combined_value, objective=objective)
return scores[0] if scores else None

async def _teardown_async(self, *, context: ChunkedRequestAttackContext) -> None:
"""
Teardown the attack by cleaning up conversation context.

Args:
context (ChunkedRequestAttackContext): The attack context containing conversation session.
"""
9 changes: 0 additions & 9 deletions pyrit/executor/attack/multi_turn/crescendo.py
Original file line number Diff line number Diff line change
Expand Up @@ -462,15 +462,6 @@ async def _perform_async(self, *, context: CrescendoAttackContext) -> CrescendoA
result.backtrack_count = context.backtrack_count
return result

async def _teardown_async(self, *, context: CrescendoAttackContext) -> None:
"""
Clean up after attack execution.

Args:
context (CrescendoAttackContext): The attack context.
"""
# Nothing to be done here, no-op

def _build_adversarial_manager(self, *, context: CrescendoAttackContext) -> _AdversarialConversationManager:
"""
Build the adversarial-conversation manager that owns Crescendo's adversarial-chat turn.
Expand Down
4 changes: 0 additions & 4 deletions pyrit/executor/attack/multi_turn/multi_prompt_sending.py
Original file line number Diff line number Diff line change
Expand Up @@ -340,10 +340,6 @@ def _determine_attack_outcome(
# At least one prompt was filtered or failed to get a response
return AttackOutcome.FAILURE, "At least one prompt was filtered or failed to get a response"

async def _teardown_async(self, *, context: MultiTurnAttackContext[Any]) -> None:
"""Clean up after attack execution."""
# Nothing to be done here, no-op

async def _send_prompt_to_objective_target_async(
self, *, current_message: Message, context: MultiTurnAttackContext[Any]
) -> Message | None:
Expand Down
4 changes: 0 additions & 4 deletions pyrit/executor/attack/multi_turn/red_teaming.py
Original file line number Diff line number Diff line change
Expand Up @@ -377,10 +377,6 @@ async def _perform_async(self, *, context: MultiTurnAttackContext[Any]) -> Attac
labels=context.memory_labels,
)

async def _teardown_async(self, *, context: MultiTurnAttackContext[Any]) -> None:
"""Clean up after attack execution."""
# Nothing to be done here, no-op

def _build_adversarial_manager(self, *, context: MultiTurnAttackContext[Any]) -> _AdversarialConversationManager:
"""
Build the adversarial conversation manager for this execution.
Expand Down
Loading