Skip to content

feat(telemetry): trace RPC calls made and handled - #7134

Draft
davidzhao wants to merge 3 commits into
dz/telemetry-startup-shutdownfrom
dz/telemetry-rpc
Draft

feat(telemetry): trace RPC calls made and handled#7134
davidzhao wants to merge 3 commits into
dz/telemetry-startup-shutdownfrom
dz/telemetry-rpc

Conversation

@davidzhao

@davidzhao davidzhao commented Sep 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

What

Neither user-registered RPC methods nor the avatar datastream RPCs appeared in a trace. This installs a tracing interceptor on the job's local participant and turns each RPC into a span.

Depends on the RpcInterceptor hook in livekit-rtc (livekit/python-sdks#806). On the current livekit==1.1.17 pin install is a no-op with one debug log, so this can land first; tracing activates once the pin moves to the SDK release carrying the hook.

Spans

Span Kind Parent
rpc_call CLIENT the current span, so an RPC issued from a tool nests under function_tool
rpc_handler SERVER the primary session's root span, so it lands on the session timeline

Attributes: rpc.method, lk.rpc.request_id, lk.rpc.caller_identity, lk.rpc.destination_identity, lk.rpc.payload_size, lk.rpc.response_size, lk.rpc.response_timeout, lk.rpc.error_code, lk.rpc.handler_registered (false when a client called a method the agent never registered). Request and response bodies are under PII keys, lk.pii.rpc.payload and lk.pii.rpc.response, truncated to 1 KiB. RpcError and handler exceptions set error status.

Where

  • telemetry/rpc.py: TracingRpcInterceptor (subclasses rtc.RpcInterceptor when present) and install(local_participant).
  • Installed from JobContext.connect() after the room connects, and from RoomIO.start() for a room connected elsewhere. One singleton interceptor; the SDK dedups registrations by identity, so repeated installs are no-ops.

Tests

tests/test_rpc_tracing.py drives the interceptor with fake continuations: attributes and parenting for both directions, request and response truncation, error codes and status, the unregistered-method flag, handler exceptions, and install registering once or degrading. Passes against both the pinned SDK and the interceptor-capable SDK source tree.

Stacked on #7131.

🤖 Generated with Claude Code

@davidzhao davidzhao mentioned this pull request Sep 5, 2026
Draft
@davidzhao
davidzhao force-pushed the dz/telemetry-rpc branch 2 times, most recently from 1306080 to aa343e0 Compare September 5, 2026 21:10
@davidzhao
davidzhao force-pushed the dz/telemetry-rpc branch 3 times, most recently from 3371ae3 to c690ca5 Compare September 6, 2026 05:23
@davidzhao
davidzhao force-pushed the dz/telemetry-rpc branch 3 times, most recently from 3b4d43e to 78b2948 Compare September 6, 2026 05:56
@davidzhao
davidzhao force-pushed the dz/telemetry-rpc branch 2 times, most recently from 371086d to dc1ab37 Compare September 6, 2026 08:09
davidzhao and others added 3 commits September 6, 2026 10:10
@davidzhao @claude
feat(telemetry): trace RPC calls made and handled
60a0e0c 
Neither user-registered RPC methods nor the avatar datastream RPCs appeared
in a trace. livekit-rtc is gaining an RpcInterceptor hook (add_rpc_interceptor
on LocalParticipant) that wraps every perform_rpc call and every dispatched
handler invocation; this installs a tracing interceptor on the job's local
participant and turns each call into a span following the OpenTelemetry RPC
semantic conventions:

- rpc_call (SpanKind.CLIENT) for outgoing calls, parented to the current
  span so an RPC issued from a tool nests under function_tool;
- rpc_handler (SpanKind.SERVER) for incoming invocations, parented to the
  primary session's root span so it lands on the session timeline.

Attributes: rpc.system=livekit, rpc.method, lk.rpc.request_id,
lk.pii.rpc.caller_identity / lk.pii.rpc.destination_identity,
lk.rpc.payload_size, lk.pii.rpc.payload (truncated to 1 KiB),
lk.rpc.response_size, lk.rpc.response_timeout, lk.rpc.error_code, and
lk.rpc.handler_registered (false when a client called a method the agent
never registered). RpcError and handler exceptions set error status.

Installed from JobContext.connect() after the room connects, and from
RoomIO.start() for a room connected elsewhere. On an SDK without the hook
(the current livekit==1.1.17 pin) install is a no-op with one debug log, so
this can land ahead of the SDK release; tracing activates once the pin moves
to the release carrying RpcInterceptor.

Tests exercise the interceptor with fake continuations (attributes, parenting,
truncation, error codes, unregistered method, handler exceptions) and the
install/degrade behavior; they pass against both the pinned SDK and the
interceptor-capable SDK source tree.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@davidzhao @claude
telemetry(rpc): drop rpc.system, record responses under a PII key
efa6cff 
rpc.system is always livekit here, so it carried no information. Responses
are now recorded like requests: lk.pii.rpc.response, truncated to 1 KiB,
alongside the existing size, so a client/agent disagreement about a message
format is readable from both sides of the span.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@davidzhao @claude
telemetry(rpc): record caller and destination identities untagged
22ff30c 
Participant identities are application identifiers, not end-user data.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@davidzhao