Skip to content

Surface chat system_fingerprint and logprobs as response metadata - #1104

Open
PratikDhanave (PratikDhanave) wants to merge 4 commits into
microsoft:mainfrom
PratikDhanaveFork:fix/openai-chat-response-metadata
Open

PratikDhanave (PratikDhanave) wants to merge 4 commits into
microsoft:mainfrom
PratikDhanaveFork:fix/openai-chat-response-metadata

Conversation

@PratikDhanave

Copy link
Copy Markdown
Contributor

The Chat path built its ResponseUpdate with ResponseID/FinishReason/usage but never set AdditionalProperties, so two pieces of response metadata were dropped:

  • system_fingerprint (on the completion / chunk), and
  • per-choice logprobs (returned when a caller requests them via Logprobs/TopLogprobs).

The Python client carries both as chat response metadata (_get_metadata_from_chat_response → system_fingerprint, _get_metadata_from_chat_choice → logprobs). The Responses path in this provider already populates AdditionalProperties, so this was also a chat-vs-responses asymmetry — the existing chat fixtures even contain system_fingerprint/logprobs values that nothing surfaced.

Change

  • Populate AdditionalProperties with SystemFingerprint (when non-empty) and Logprobs (when the choice carries content/refusal logprobs) on both the non-streaming and streaming paths. Keys follow the PascalCase convention already used by the Responses path helper (responsesPopulateAdditionalProperties).

Tests

  • TestChatResponseMetadataSurfaced_NonStreaming and _Streaming: assert SystemFingerprint (and, non-streaming, Logprobs) reach the message AdditionalProperties. Both fail before the change (no message carries any), pass after.

The Chat path set ResponseID, FinishReason, and usage but never populated
AdditionalProperties, dropping the response system_fingerprint and the
per-choice logprobs that a caller requesting logprobs would expect back. The
Responses path already populates AdditionalProperties, so this was also a
chat-vs-responses asymmetry.

Populate AdditionalProperties with SystemFingerprint (when present) and
Logprobs (when the choice carries any) on both the non-streaming and
streaming paths, matching the Python client which surfaces both as chat
response metadata.
Copilot AI lite review requested due to automatic review settings September 18, 2026 12:28
@github-actions github-actions Bot added area:provider Changes files in the provider area area:provider/openai Changes files in the provider / openai area size:large At most 300 changed lines across at most 10 files labels Sep 18, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Streaming logprobs may be overwritten across chunks, and this path lacks coverage.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Surfaces Chat response system_fingerprint and logprobs metadata through AdditionalProperties.

Changes:

  • Propagates metadata for streaming and non-streaming responses.
  • Adds tests for metadata exposure.
File summaries
File Summary
provider/openaiprovider/chat.go Populates Chat response metadata.
provider/openaiprovider/chat_test.go Tests metadata propagation.
Review details

Suppressed comments (2)

provider/openaiprovider/chat.go:227

  • The new streaming Logprobs branch is not exercised: the added streaming test only sends/asserts SystemFingerprint, while all existing streaming fixtures use logprobs: null. Add a streamed chunk containing content or refusal logprobs and assert that the yielded update (and collected message) preserves the Logprobs metadata, so this path cannot regress unnoticed.
				if logprobs := chunk.Choices[0].Logprobs; len(logprobs.Content) > 0 || len(logprobs.Refusal) > 0 {
					additionalProperties = map[string]any{"Logprobs": logprobs}
				}

provider/openaiprovider/chat_test.go:457

  • This streaming test only sends system_fingerprint; it never supplies a non-empty chunk logprobs value, so it would pass if the new streaming Logprobs handling were removed or broken. Add a chunk with content/refusal logprobs and assert that the streamed update (and, if collection is supported, the collected response) preserves them.
func TestChatResponseMetadataSurfaced_Streaming(t *testing.T) {
	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
		w.Header().Set("Content-Type", "text/event-stream")
		_, _ = io.WriteString(w, "data: {\"id\":\"chatcmpl-md\",\"object\":\"chat.completion.chunk\",\"created\":1727888631,\"model\":\"gpt-4o-mini\",\"system_fingerprint\":\"fp_test\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"ok\"},\"finish_reason\":\"stop\"}]}\n\n")
  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread provider/openaiprovider/chat.go Outdated
Comment on lines +225 to +226
if logprobs := chunk.Choices[0].Logprobs; len(logprobs.Content) > 0 || len(logprobs.Refusal) > 0 {
additionalProperties = map[string]any{"Logprobs": logprobs}
@github-actions github-actions Bot added kind:code Changes production behavior or code kind:tests Changes tests, fixtures, or test infrastructure labels Sep 18, 2026
@github-actions

This comment has been minimized.

Streaming logprobs arrive incrementally (each chunk carries only its own
tokens), and Response.Update merges AdditionalProperties with maps.Copy, so
attaching per-chunk logprobs kept only the final chunk's tokens after
Collect(). Accumulate content/refusal token logprobs across chunks and attach
the running total, so the collected message holds the complete per-choice
list.
@github-actions

This comment has been minimized.

@github-actions github-actions Bot added the kind:dependencies Changes dependencies or manifests label Sep 21, 2026
…ponse-metadata

# Conflicts:
#	go.mod
#	go.sum
#	provider/openaiprovider/chat.go
#	provider/openaiprovider/chat_test.go
@github-actions github-actions Bot removed the kind:dependencies Changes dependencies or manifests label Oct 3, 2026
@github-actions

This comment has been minimized.

@github-actions github-actions Bot added the public-api-change Pull Request changes public APIs label Oct 3, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generated by Go API Consistency Review Agent for #1104 · copilot · auto · 188.9 AIC · ⌖ 8.09 AIC · ⊞ 13.9K

MessageID: resp.ID,
FinishReason: finishReason,
CreatedAt: time.Unix(resp.Created, 0),
AdditionalProperties: additionalProperties,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parity issue: SystemFingerprint/Logprobs are scoped to the message, not the response

Because this update sets MessageID: resp.ID (non-empty) alongside AdditionalProperties, Response.Update (see agent/response.go's own doc comment: "A non-empty MessageID scopes these properties to that message. Otherwise, they are response-level properties.") copies SystemFingerprint/Logprobs into msg.AdditionalProperties, not resp.AdditionalProperties. The PR's own test (lastMessageAdditionalProperties) confirms this — it has to scan resp.Messages rather than reading resp.AdditionalProperties.

Upstream Python treats both as response-level metadata: _get_metadata_from_chat_response/_get_metadata_from_chat_choice populate a single response_metadata dict that is passed as ChatResponse(..., additional_properties=response_metadata, ...) (_chat_completion_client.py lines ~879-904, 1007-1022), never on an individual Message.

This same provider package already has the correct pattern for this exact situation: responsesPopulateAdditionalProperties in provider/openaiprovider/responses.go (around line 1265) explicitly yields the message-bearing update first, then emits a second, metadata-only update with an empty MessageID so the properties land on Response.AdditionalProperties instead of the message.

Suggested fix: follow the same split used in responses.go — yield the content-bearing update with MessageID set, then emit a follow-up update (or restructure this single update) with MessageID unset so SystemFingerprint/Logprobs land in Response.AdditionalProperties, matching Python's ChatResponse.additional_properties semantics. The same issue applies to the streaming path at line 250.

…ponse-metadata

# Conflicts:
#	provider/openaiprovider/chat.go
#	provider/openaiprovider/chat_test.go
@github-actions github-actions Bot removed the public-api-change Pull Request changes public APIs label Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

API consistency review

Result Scope
unable to determine user-visible behavior

Changed Go contract

provider/openaiprovider/chat.go: Chat responses (streaming and non-streaming) now set AdditionalProperties with SystemFingerprint and Logprobs. Streaming accumulates logprobs across chunks. No exported identifiers changed.

Upstream evidence reviewed

A code search found system_fingerprint in python/packages/openai/agent_framework_openai/_chat_completion_client.py in microsoft/agent-framework. I could not read the exact Python metadata helpers (_get_metadata_from_chat_response / _get_metadata_from_chat_choice), including their keys, value shapes, and streaming behavior. The file is too large for the available tools, and code search was rate-limited. I did not review the .NET MEAI OpenAIChatClient equivalent.

Assessment

The change looks consistent with the Python behavior the PR describes, but I could not verify key names, value shapes, or per-chunk versus accumulated streaming semantics against upstream. I am not reporting a finding or approving parity. No exported Go APIs changed, so public-api-change is removed.

Generated by Go API Consistency Review Agent · copilot · auto · 28.5 AIC · ⌖ 0.619 AIC · ⊞ 5.1K · ◷

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:provider/openai Changes files in the provider / openai area area:provider Changes files in the provider area kind:code Changes production behavior or code kind:tests Changes tests, fixtures, or test infrastructure size:large At most 300 changed lines across at most 10 files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants