Repository navigation
test(live-provider): request stream usage from OpenAI in streaming tests - #2124
Conversation
The Langfuse server stopped inferring token usage from content (langfuse/langfuse#19802). OpenAI streams only report usage when stream_options.include_usage is set, so these tests got empty usage_details and failed with KeyError: 'input'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@claude review |
There was a problem hiding this comment.
Claude Code Review
No review was started: this request came from a bot account. Manual reviews can only be requested by someone with write access to this repository. Ask a maintainer to comment @claude review, or have your automation post the comment from a user account with write access.
Tip: disable this comment in your organization's Code Review settings.
There was a problem hiding this comment.
LGTM. This is a test-only change that asks OpenAI for usage data in the six streaming tests that assert on usage_details.
What was reviewed:
- The final usage chunk has an empty
choiceslist. The loops that indexchoices[0]already skip that case, and the async loops only print chunks. - The
model_parametersassertions are unaffected becauselangfuse/openai.pydoesn't recordstream_optionsthere. - The streaming tests left unchanged either don't assert on usage, or are skipped (
test_openai_completion_stream, viarequires_legacy_completion_model), or use the Responses API. - Not run here: the live-provider tests themselves. CI's live-provider job is the check.
Extended reasoning...
The change adds stream_options={"include_usage": True} to six streaming chat completion calls in tests/live_provider/test_openai.py and tests/live_provider/test_decorators.py, and changes no library code. It touches no security-sensitive surface. CODEOWNERS only covers .github/, so none of the changed paths has an owner. The change is small and mechanical, the tests still hold with the extra final chunk, and the timeline has no other reviewer activity.
Fixes the
E2E live-provider testsjob, which fails onmain(8e6c60db) and on every PR right now, including #2123.Why it fails
6 OpenAI streaming tests fail with
KeyError: 'input'onusage_details["input"]:test_openai.py:test_openai_chat_completion_stream,test_openai_chat_completion_stream_with_next_iteration,test_async_chat_stream,test_async_chat_stream_with_anext,test_close_async_streamtest_decorators.py:test_async_nested_openai_chat_streamThese tests stream without
stream_options={"include_usage": True}, so OpenAI sends no usage. The server used to fill in token counts by tokenizing the content, and langfuse/langfuse#19802 (Nov 6) removed that. CI runs against the latest server, sousage_detailsnow comes back empty for these streams. The non-streaming tests with the same assertion still pass.Fix
Pass
stream_options={"include_usage": True}in those 6 calls, so the usage assertions keep checking real data instead of being dropped.langfuse/openai.py).stream_optionsisn't captured inmodel_parameters, so the exactmodel_parametersassertions don't change.choices[0]already guard against the final chunk having no choices.Not in this PR: whether
langfuse.openaishould addinclude_usageautomatically for streams. That would change SDK behavior.Verification
uv run --frozen ruff check tests/live_providerandruff format --check tests/live_provider: pass🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge.Summary
Adds
stream_options={"include_usage": True}to six live-provider streaming tests so their existing usage checks use counts returned by OpenAI.Reviews (1) · Last reviewed commit: "test(live-provider): request stream usag..." · Reviewed by Greptile