Skip to content

Python: Background completion waiter can finalize a newer continuation under the same task ID #9441

Closed

Description

@CorgiBoyG

Describe the bug

background_agents_wait_for_first_completion can finalize a different execution from the one it waited for when another caller continues the same background task before the waiter resumes.

The tool records (task_id, asyncio.Task) pairs before calling asyncio.wait(). After the wait completes, it finds the completed ID using those original task objects, but fetches the task to finalize again from runtime.in_flight_tasks[task_id]. The continuation tool reuses that ID and replaces its runtime task.

Consequently:

  1. A waiter starts waiting for execution A of task 1.
  2. Execution A finishes.
  3. Another caller retrieves A's result and continues task 1, starting execution B.
  4. The waiter resumes, recognizes A as completed, then tries to finalize B.

If B is still running, Task.exception() raises asyncio.InvalidStateError: Exception is not set. The wait tool fails even though the execution it waited for completed successfully.

Expected behavior

A wait operation should report the outcome of the execution it actually observed. A stale waiter must not consume or remove a newer execution registered under the same task ID.

Reproduction and evidence

I reproduced this on current main, commit 9a1d66b3e973bdbfdcd4359d72c5f6fd4ff4d931, with Python 3.13.15 on macOS.

The deterministic regression uses the provider's public start, wait, get-results, and continue tools with an offline child agent. Events pause the waiter after the real asyncio.wait() returns, allowing the result retrieval and continuation to run before the waiter processes its completed set. No runtime task or persisted state is replaced directly by the test.

The regression currently fails through:

background_agents_wait_for_first_completion
  -> _finalize_task(task_info, completed_task, runtime)
  -> completed_task.exception()
asyncio.exceptions.InvalidStateError: Exception is not set.

Proposed scoped fix

Finalize the exact task object returned in the original wait set, and only update/remove runtime state if it still refers to that same execution. Add coverage for a pending and a completed continuation, ordinary waits, and concurrent waiters.

This concerns the local wait/continuation race. It is separate from completed-task retention in #9387, distributed runtime ownership in #8760/#9040, and the release-timeout issue in #9231.

Activity

  1. CorgiBoyG commented on Oct 16, 2026

    @CorgiBoyG
    ContributorAuthor

    I can work on this. I have a deterministic failing regression and am preparing a scoped fix that binds completion handling to the task object actually waited on, with coverage for continuation and concurrent waiters.

  2. added
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Oct 16, 2026
  3. CorgiBoyG commented on Oct 16, 2026

    @CorgiBoyG
    ContributorAuthor

    The local fix is now validated:

    • Completion handling uses the exact task object from the original wait set.
    • Runtime removal and persisted-state updates are conditional on that task still being the current execution for the ID.
    • Twelve deterministic cases cover the original execution succeeding, failing, or being cancelled, with a running, completed, already-finalized, or failed continuation. A separate case covers concurrent waiters.
    • With the unchanged main implementation, 11 of the 12 continuation cases fail; all pass with the fix.
    • Core non-integration tests: 8768 passed, 20 skipped, 2 xfailed. Ruff and source/test Pyright checks pass.

    Eduard van Valkenburg (@eavanvalkenburg), could you confirm this scoped direction before I open the PR? It preserves the current retention and release policies; #9387 and #9231 remain separate.

  4. CorgiBoyG commented on Oct 16, 2026

    @CorgiBoyG
    ContributorAuthor

    Opened draft PR #9442 with the scoped fix and regression coverage. The draft remains pending confirmation of the direction requested above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    pythonUsage: [Issues, PRs], Target: PythontriageUsage: [Issues], Target: All issues that still need to be triaged

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions