feat(claude-agent-sdk): one inference span per model response (incl. tool-call-only), capture thinking, per-call usage #293

What problem do you want to solve?

In a ReAct loop where the model often answers with only thinking + tool_use (no text block), the instrumentation produces one chat span per agent invocation instead of one per model call. There are three related problems:

1. No inference span for tool-call-only responses (by design today, but out of line with semconv).
_process_assistant_message only starts an LLM turn when the message has a TextBlock. A response that only requests tool calls gets execute_tool spans under the agent and no chat span. This is the behavior #237 intentionally locked in with test_tool_use_without_text_does_not_create_synthetic_llm_span. The GenAI semantic conventions (v1.37.0, Inference span), however, define the span as:

This span represents a client call to Generative AI model or service that generates a response or requests a tool call based on the input prompt.

So each tool-call-only model call should have its own inference span. We'd like to ask you to revisit this decision; see Additional context for why this needn't be a synthetic span.

2. ThinkingBlock is dropped.
_extract_message_parts handles only TextBlock and ToolUseBlock. Thinking never reaches gen_ai.output.messages, even on the spans that are created.

3. Cumulative usage is put on one LLM span (independent of 1).
LLM-span usage comes only from ResultMessage, which is the total for the whole invocation, and AssistantTurnTracker.update_usage writes it to the last turn. This is wrong even with the current text-anchored segmentation: if all three calls in the repro below start with a text block, you get three chat spans, but the first two have no gen_ai.usage.* at all and the last has input_tokens=300 output_tokens=126, the sum of all three calls.

For context: the Claude Code CLI emits one AssistantMessage per content block. Blocks from the same API response share message_id and usage; claude-agent-sdk fills both from message.id and message.usage in message_parser.py. In real expert-agent transcripts from our deployment, 357 of 502 model calls (71%) had no text block: 275 were thinking + tool_use and 82 were tool_use only. Those runs used DeepSeek and Kimi models through an Anthropic-compatible gateway. As a result, most model calls are missing from our traces, and the token and latency data on the remaining span are misleading.

Reproduction

This uses the real claude-agent-sdk message dataclasses and the instrumentation's stream processor, the same way tests/test_session_capture.py does. The message sequence matches what the CLI emits.

import asyncio
import json
import os

os.environ["OTEL_SEMCONV_STABILITY_OPT_IN"] = "gen_ai_latest_experimental"
os.environ["OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT"] = "SPAN_ONLY"

from claude_agent_sdk.types import (
    AssistantMessage, ResultMessage, SystemMessage, TextBlock,
    ThinkingBlock, ToolResultBlock, ToolUseBlock, UserMessage,
)
from opentelemetry.instrumentation.claude_agent_sdk.patch import _process_agent_invocation_stream
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.util.genai.extended_handler import ExtendedTelemetryHandler

M = "claude-sonnet-4-5"
U = {"input_tokens": 100, "output_tokens": 42}

# Same shape the Claude Code CLI emits: one AssistantMessage per content block,
# blocks of the same API response share message_id and usage.
MESSAGES = [
    SystemMessage("init", {"session_id": "s1", "model": M}),
    AssistantMessage([ThinkingBlock("Need txt files first.", "sig")], M, usage=U, message_id="msg_01"),
    AssistantMessage([ToolUseBlock("toolu_01", "Glob", {"pattern": "*.txt"})], M, usage=U, message_id="msg_01"),
    UserMessage([ToolResultBlock("toolu_01", "a.txt")]),
    AssistantMessage([ThinkingBlock("Now md files.", "sig")], M, usage=U, message_id="msg_02"),
    AssistantMessage([ToolUseBlock("toolu_02", "Glob", {"pattern": "*.md"})], M, usage=U, message_id="msg_02"),
    UserMessage([ToolResultBlock("toolu_02", "b.md")]),
    AssistantMessage([ThinkingBlock("I have both lists.", "sig")], M, usage=U, message_id="msg_03"),
    AssistantMessage([TextBlock("Found a.txt and b.md.")], M, usage=U, message_id="msg_03"),
    ResultMessage("success", 10, 10, False, 3, "s1",
                  usage={"input_tokens": 300, "output_tokens": 126}),
]


async def stream():
    for message in MESSAGES:
        yield message


async def main():
    exporter = InMemorySpanExporter()
    provider = TracerProvider()
    provider.add_span_processor(SimpleSpanProcessor(exporter))
    handler = ExtendedTelemetryHandler(tracer_provider=provider)
    async for _ in _process_agent_invocation_stream(stream(), handler=handler, model=M, prompt="List files"):
        pass
    for span in sorted(exporter.get_finished_spans(), key=lambda s: s.start_time):
        attrs = span.attributes
        line = span.name
        if attrs.get("gen_ai.operation.name") == "chat":
            parts = [p["type"] for m in json.loads(attrs["gen_ai.output.messages"]) for p in m["parts"]]
            line += (f"  output_parts={parts}  input_tokens={attrs.get('gen_ai.usage.input_tokens')}"
                     f"  output_tokens={attrs.get('gen_ai.usage.output_tokens')}")
        print(line)


asyncio.run(main())

For problem 3 on its own, replace the ThinkingBlock of msg_01 and msg_02 with a TextBlock (e.g. TextBlock("Let me list txt files.")) and run again.

We also checked this end to end with the public API: ClaudeAgentSDKInstrumentor().instrument() + claude_agent_sdk.query() running the bundled CLI 2.1.138 against a local fake Anthropic endpoint that returns three streamed responses (thinking + tool_use, thinking + tool_use, thinking + text). The CLI emitted the split messages shown above, and the exported spans matched the actual result below exactly.

Current result

As written:

invoke_agent claude-agent
execute_tool Glob
execute_tool Glob
chat claude-sonnet-4-5  output_parts=['text']  input_tokens=300  output_tokens=126

With msg_01/msg_02 starting with a TextBlock (problem 3 only):

chat claude-sonnet-4-5  output_parts=['text', 'tool_call']  input_tokens=None  output_tokens=None
invoke_agent claude-agent
execute_tool Glob
chat claude-sonnet-4-5  output_parts=['text', 'tool_call']  input_tokens=None  output_tokens=None
execute_tool Glob
chat claude-sonnet-4-5  output_parts=['text']  input_tokens=300  output_tokens=126

Describe the solution you'd like

One chat span per API response (message_id), with thinking captured as Reasoning and its own usage. For the repro above:

invoke_agent claude-agent
chat claude-sonnet-4-5  output_parts=['reasoning', 'tool_call']  input_tokens=100  output_tokens=42
execute_tool Glob
chat claude-sonnet-4-5  output_parts=['reasoning', 'tool_call']  input_tokens=100  output_tokens=42
execute_tool Glob
chat claude-sonnet-4-5  output_parts=['reasoning', 'text']       input_tokens=100  output_tokens=42

(The agent span keeps the cumulative usage from ResultMessage.)

A possible direction:

• Start a new LLM turn when message_id changes, not when a text block appears, and append later blocks with the same message_id to that turn.
• Set gen_ai.response.id from message_id.
• Map ThinkingBlock to Reasoning, without the signature.
• Take LLM-span usage from AssistantMessage.usage and keep ResultMessage usage for the agent span.
• When message_id is missing (older SDKs), keep the current behavior.

Describe alternatives you've considered

• Keep text-anchored turns and only fix usage: fixes problem 3 but still leaves most model calls without an inference span when models skip text before tool calls.
• Make message_id-based segmentation opt-in (e.g. an env var), if changing the default from fix: capture Claude SDK stream model metadata #237 is a concern. The current behavior stays as the default, and users who need one span per model call can turn it on.
• Instrument below the SDK: not practical, because model requests are made by the Claude Code CLI subprocess and aren't visible to Python instrumentation.

Additional Context

• On "synthetic" spans (fix: capture Claude SDK stream model metadata #237): avoiding fabricated LLM spans makes sense when there's no reliable boundary. Here there is one. AssistantMessage.message_id is the provider's response id, which maps naturally to gen_ai.response.id ("The unique identifier for the completion"), and AssistantMessage.usage is that response's usage. A span keyed by message_id stands for a real model call rather than a synthetic one.
• On reasoning: bug(strands): normalize public reasoning and keep Agent output final-only #282/fix(strands): normalize reasoning and Agent output boundaries #283 normalize provider-public reasoning in the strands instrumentation into the shared Reasoning part and exclude signatures and redacted/encrypted reasoning. Doing the same for ThinkingBlock here would make the two instrumentations consistent.

Environment used for the reproduction:

OS: Windows 10 for the reproduction above; we see the same behavior on Linux containers in production
Python version: 3.10.11
Package version:

• loongsuite-instrumentation-claude-agent-sdk 0.9.0 (patch.py on main @ 243bd6e is identical)
• loongsuite-otel-util-genai 0.9.0
• claude-agent-sdk 0.1.80 (bundled Claude Code CLI 2.1.138)
• opentelemetry-sdk 1.45.1

Would you like to implement a fix?

No

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}