Skip to content

test(live-provider): request stream usage from OpenAI in streaming tests - #2124

Merged
niklassemmler merged 1 commit into
mainfrom
test/live-provider-stream-include-usage
Nov 10, 2026
Merged

niklassemmler merged 1 commit into
mainfrom
test/live-provider-stream-include-usage

Conversation

@niklassemmler

@niklassemmler niklassemmler commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Fixes the E2E live-provider tests job, which fails on main (8e6c60db) and on every PR right now, including #2123.

Why it fails

6 OpenAI streaming tests fail with KeyError: 'input' on usage_details["input"]:

  • test_openai.py: test_openai_chat_completion_stream, test_openai_chat_completion_stream_with_next_iteration, test_async_chat_stream, test_async_chat_stream_with_anext, test_close_async_stream
  • test_decorators.py: test_async_nested_openai_chat_stream

These tests stream without stream_options={"include_usage": True}, so OpenAI sends no usage. The server used to fill in token counts by tokenizing the content, and langfuse/langfuse#19802 (Nov 6) removed that. CI runs against the latest server, so usage_details now comes back empty for these streams. The non-streaming tests with the same assertion still pass.

Fix

Pass stream_options={"include_usage": True} in those 6 calls, so the usage assertions keep checking real data instead of being dropped.

  • The SDK already reads usage from the final chunk (langfuse/openai.py).
  • stream_options isn't captured in model_parameters, so the exact model_parameters assertions don't change.
  • The loops that index choices[0] already guard against the final chunk having no choices.

Not in this PR: whether langfuse.openai should add include_usage automatically for streams. That would change SDK behavior.

Verification

  • uv run --frozen ruff check tests/live_provider and ruff format --check tests/live_provider: pass
  • Live-provider tests not run locally (no OpenAI key or local server). CI's live-provider job on this PR is the check.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge.

Summary

Adds stream_options={"include_usage": True} to six live-provider streaming tests so their existing usage checks use counts returned by OpenAI.

  • OpenAI streaming tests now request usage data.

Reviews (1) · Last reviewed commit: "test(live-provider): request stream usag..." · Reviewed by Greptile

The Langfuse server stopped inferring token usage from content
(langfuse/langfuse#19802). OpenAI streams only report usage when
stream_options.include_usage is set, so these tests got empty
usage_details and failed with KeyError: 'input'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

@claude review

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

No review was started: this request came from a bot account. Manual reviews can only be requested by someone with write access to this repository. Ask a maintainer to comment @claude review, or have your automation post the comment from a user account with write access.

Tip: disable this comment in your organization's Code Review settings.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. This is a test-only change that asks OpenAI for usage data in the six streaming tests that assert on usage_details.

What was reviewed:

  • The final usage chunk has an empty choices list. The loops that index choices[0] already skip that case, and the async loops only print chunks.
  • The model_parameters assertions are unaffected because langfuse/openai.py doesn't record stream_options there.
  • The streaming tests left unchanged either don't assert on usage, or are skipped (test_openai_completion_stream, via requires_legacy_completion_model), or use the Responses API.
  • Not run here: the live-provider tests themselves. CI's live-provider job is the check.
Extended reasoning...

The change adds stream_options={"include_usage": True} to six streaming chat completion calls in tests/live_provider/test_openai.py and tests/live_provider/test_decorators.py, and changes no library code. It touches no security-sensitive surface. CODEOWNERS only covers .github/, so none of the changed paths has an owner. The change is small and mechanical, the tests still hold with the extra final chunk, and the timeline has no other reviewer activity.

@niklassemmler
niklassemmler merged commit e3ffb85 into main Nov 10, 2026
25 checks passed
@niklassemmler
niklassemmler deleted the test/live-provider-stream-include-usage branch November 10, 2026 19:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant