You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
bug: reused thread pools propagate creation context instead of each request #1473
ConcurrentFuturesInstrumentor copies context in the executor initializer, but its submit wrapper forwards tasks without capturing the caller's current context. A pool created before a request, or reused across requests, therefore assigns worker spans to the pool's old parent rather than the parent active at submission. ContextVar writes in one task also remain visible to subsequent tasks on the same worker.
This breaks trace attribution for concurrent agent operations when an instrumented executor is reused.
Reproduction
Verified on current main (f8e907b92dabe47232978023fdcb01e2a7d4b752) with Python 3.12.14 / OpenTelemetry SDK 1.45.1, using the real ConcurrentFuturesInstrumentor, ThreadPoolExecutor(max_workers=1) and an isolated TracerProvider. No backend, LLM, API key or network request is involved.
Enable ConcurrentFuturesInstrumentor.
Create a pool under a pool-creation span, then exit that span.
Under a new request-a span, submit a task that creates a worker span and reads a caller ContextVar.
Repeat under a separate request-b span using the same pool.
Both worker spans retain the pool-creation trace/parent and read the ContextVar default, instead of their respective request context. A second reproduction changes a ContextVar inside a worker task; the next task sees that previous task's value even though the caller never set it.
Five executable local regression/control cases on unchanged main: four fail (request parent, queued submission snapshot, isolation after success, isolation after failure), one passes (initializer arguments and task arguments).
Expected behavior / proposed fix
Capture a fresh contextvars.copy_context() for each submission and execute that task through the copied context's run. This preserves the parent active at submission and isolates task-local changes, including when a task raises, while keeping normal Future exception/return behavior and initializer support.
I can prepare this small fix with real-thread regression tests. Current open PRs and all-state ThreadPoolExecutor / concurrent_futures searches did not reveal an equivalent change. Historical #1018 introduced the feature; #1113 changes activation rules rather than per-submission propagation.
The issue is missing per-submission context capture combined with unmanaged thread-local state recycling.
When wrapping ThreadPoolExecutor, capturing context once at pool creation binds all future jobs to the initialization scope. Python worker threads do not inherit caller contextvars across thread boundaries by default.
The standard pattern requires intercepting submit and map directly:
Capture caller context on submission: ctx = contextvars.copy_context() and otel_ctx = context.get_current().
Wrap the submitted callable in a closure executing under ctx.run.
Inside the worker thread, attach the captured OpenTelemetry context via token = context.attach(otel_ctx) wrapped in a try/finally block that calls context.detach(token).
The try/finally detachment is critical. Without explicit detachment, any context mutations performed inside task A remain active on the worker thread and pollute task B when that worker is recycled from the pool.
Activity
NaaeemMalik commentedon Nov 16, 2026
The issue is missing per-submission context capture combined with unmanaged thread-local state recycling.
When wrapping ThreadPoolExecutor, capturing context once at pool creation binds all future jobs to the initialization scope. Python worker threads do not inherit caller contextvars across thread boundaries by default.
The standard pattern requires intercepting submit and map directly:
The try/finally detachment is critical. Without explicit detachment, any context mutations performed inside task A remain active on the worker thread and pollute task B when that worker is recycled from the pool.