Skip to content

TEST: Separate score cancellation watchdog from attack startup - #3139

Open
Richard Lundeen (richlundeen) wants to merge 1 commit into
microsoft:mainfrom
richlundeen:richlundeen-score-cancellation-timeout
Open

Richard Lundeen (richlundeen) wants to merge 1 commit into
microsoft:mainfrom
richlundeen:richlundeen-score-cancellation-timeout

Conversation

@richlundeen

Copy link
Copy Markdown
Contributor

Description

The merge queue for #3121 failed in an unrelated score cancellation test on macOS / Python 3.13. Failed job log.

The five-second watchdog covered the whole attack, including target setup and conversation persistence. The failed stack was still in PromptNormalizer.add_conversation_to_memory_async, reading ConversationEntry through the aiosqlite worker queue. It had not reached score validation. The log does not show a score rollback or error-result write failure.

Wait up to 30 seconds for the existing validation-ready event, then keep the five-second cleanup deadline. Use the existing non-cancelling completion helper so the watchdog does not add another cancellation to the operation under test. Assert that the held validation read received cancellation and that error propagation did not require releasing the read. Query scores with an explicit type filter; an unfiltered query returns no rows by design and cannot check rollback.

This is a test-only change. The auxiliary-scorer failure, transaction-start event, blocked validation read, exception cause, and persisted attack-error assertions remain in place. No production code or changes from #3121 are included.

Tests and Documentation

Validation on Windows / uv-managed Python 3.12.12:

  • uv run --no-sync pytest tests\unit\executor\attack\test_scoring_expectation_transport.py -q --durations=5 — passed, 41 tests.
  • uv run --no-sync pytest tests\unit\executor\attack\test_scoring_expectation_transport.py tests\unit\memory\test_sqlite_cancellation.py -n 4 --dist=loadfile -q — passed, 78 tests on the final diff.
  • $env:UV_NO_SYNC='1'; uv run --no-sync pre-commit run --all-files — all hooks passed on the final diff.

A temporary pytest plugin outside the repository added a 5.1-second async delay before target sending. The original test failed with TimeoutError before scoring; the revised test passed with the same delay. The revised test does not add this delay to normal CI runs. The original macOS / Python 3.13 environment was not available locally; CI must confirm that platform.

No product documentation changes apply to this test-only fix. JupyText was not run.

Wait for validation readiness before timing cleanup. Assert the validation read was cancelled and query stored scores with an explicit filter.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 226dfd07-b984-4cc9-a0f3-502d3c02d318

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant