Skip to content

bug: reused thread pools propagate creation context instead of each request #1473

Open

Description

@LJK-ikun

Description

ConcurrentFuturesInstrumentor copies context in the executor initializer, but its submit wrapper forwards tasks without capturing the caller's current context. A pool created before a request, or reused across requests, therefore assigns worker spans to the pool's old parent rather than the parent active at submission. ContextVar writes in one task also remain visible to subsequent tasks on the same worker.

This breaks trace attribution for concurrent agent operations when an instrumented executor is reused.

Reproduction

Verified on current main (f8e907b92dabe47232978023fdcb01e2a7d4b752) with Python 3.12.14 / OpenTelemetry SDK 1.45.1, using the real ConcurrentFuturesInstrumentor, ThreadPoolExecutor(max_workers=1) and an isolated TracerProvider. No backend, LLM, API key or network request is involved.

  1. Enable ConcurrentFuturesInstrumentor.
  2. Create a pool under a pool-creation span, then exit that span.
  3. Under a new request-a span, submit a task that creates a worker span and reads a caller ContextVar.
  4. Repeat under a separate request-b span using the same pool.

Both worker spans retain the pool-creation trace/parent and read the ContextVar default, instead of their respective request context. A second reproduction changes a ContextVar inside a worker task; the next task sees that previous task's value even though the caller never set it.

Five executable local regression/control cases on unchanged main: four fail (request parent, queued submission snapshot, isolation after success, isolation after failure), one passes (initializer arguments and task arguments).

Expected behavior / proposed fix

Capture a fresh contextvars.copy_context() for each submission and execute that task through the copied context's run. This preserves the parent active at submission and isolates task-local changes, including when a task raises, while keeping normal Future exception/return behavior and initializer support.

I can prepare this small fix with real-thread regression tests. Current open PRs and all-state ThreadPoolExecutor / concurrent_futures searches did not reveal an equivalent change. Historical #1018 introduced the feature; #1113 changes activation rules rather than per-submission propagation.

Activity

  1. NaaeemMalik commented on Nov 16, 2026

    @NaaeemMalik

    The issue is missing per-submission context capture combined with unmanaged thread-local state recycling.

    When wrapping ThreadPoolExecutor, capturing context once at pool creation binds all future jobs to the initialization scope. Python worker threads do not inherit caller contextvars across thread boundaries by default.

    The standard pattern requires intercepting submit and map directly:

    1. Capture caller context on submission: ctx = contextvars.copy_context() and otel_ctx = context.get_current().
    2. Wrap the submitted callable in a closure executing under ctx.run.
    3. Inside the worker thread, attach the captured OpenTelemetry context via token = context.attach(otel_ctx) wrapped in a try/finally block that calls context.detach(token).

    The try/finally detachment is critical. Without explicit detachment, any context mutations performed inside task A remain active on the worker thread and pollute task B when that worker is recycled from the pool.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions