`BackendSpanExporter._shutdown_event` is never cleared, permanently disabling retries after one shutdown

[rajarshidattapy]

Area: src/agents/tracing/processors.py:87, 236-241, 537-538
Type: bug
Severity: low

Description

BackendSpanExporter has a one-way threading.Event used to abandon retry backoff during
shutdown:

def _request_shutdown(self) -> None:
    self._shutdown_event.set()

def _sleep_before_retry(self, sleep_time, deadline) -> bool:
    if deadline is None:
        if self._shutdown_event.wait(sleep_time):
            logger.warning("[non-fatal] Tracing: shutdown requested during retry backoff, giving up.")
            return False
        return not self._shutdown_event.is_set()
    ...

The event is set by BatchTraceProcessor.shutdown(timeout=...) and is never cleared.
Because default_exporter() caches a module-level singleton (_global_exporter), the same
exporter instance survives the shutdown.

Repro

from unittest.mock import MagicMock, patch

from agents.tracing.processors import BackendSpanExporter, BatchTraceProcessor
from agents.tracing.span_data import AgentSpanData
from agents.tracing.spans import SpanImpl

with patch("httpx2.Client") as mock_client:
    response = MagicMock()
    response.status_code = 504
    mock_client.return_value.post.return_value = response

    exporter = BackendSpanExporter(api_key="test_key", max_retries=3, base_delay=0.01,
                                   max_delay=0.02)

    first = BatchTraceProcessor(exporter=exporter)
    first.shutdown(timeout=1.0)

    # A fresh processor over the same (singleton) exporter.
    second = BatchTraceProcessor(exporter=exporter)
    span = SpanImpl(trace_id="t", span_id="s", parent_id=None, processor=second,
                    span_data=AgentSpanData(name="a"), tracing_api_key=None)
    second._queue.put_nowait(span)
    second.force_flush()

    print("post attempts for a 504 batch (max_retries=3):",
          mock_client.return_value.post.call_count)

[non-fatal] Tracing: server error 504, retrying.
[non-fatal] Tracing: shutdown requested during retry backoff, giving up.
post attempts for a 504 batch (max_retries=3): 1

Impact

Any code path that shuts a processor down and then continues exporting through the same
exporter instance silently loses every batch that hits a transient failure — the first 5xx
or network error aborts the retry loop immediately instead of backing off. Concretely:

• A processor is registered, shutdown(timeout=...) is called (e.g. an explicit
  force_flush/teardown, or a test that tears down the provider), then a new
  BatchTraceProcessor is registered over the same default_exporter().
• Test suites that shut down tracing between cases and then re-enable it.

The failure is silent: the retry is skipped and the batch is dropped with a warning that
blames "shutdown requested" long after shutdown finished.

Suggested fix

Either clear the event when the exporter is reused (add a _reset_shutdown() /
self._shutdown_event.clear() when a new processor attaches), or make the shutdown state
terminal and explicit — have _request_shutdown() also mark the exporter closed so further
export() calls fail loudly rather than degrading into "no retries, ever".

[mikemikimike]

Implemented in PR #4711.

• Changed: Clear the exporter shutdown signal when a new BatchTraceProcessor attaches, so a reused exporter resumes transient-failure retries.
• Tests: uv run pytest -q tests/test_trace_processor.py (57 passed).
• Validation: Ruff formatting/checks and git diff --check passed.

[rajarshidattapy]

@seratch should I retry solving this issue in a new PR?