Skip to content

fix: retry OTLP gRPC responses with trailer status - #8854

Merged
jack-berg merged 12 commits into
open-telemetry:mainfrom
efegokdemir:codex/issue-8843-trailer-retry
Oct 8, 2026
Merged

jack-berg merged 12 commits into
open-telemetry:mainfrom
efegokdemir:codex/issue-8843-trailer-retry

Conversation

@efegokdemir

Copy link
Copy Markdown
Contributor

Problem

OkHttpGrpcSender.isRetryable only inspected the grpc-status response header. When a retryable status such as UNAVAILABLE was delivered in HTTP/2 trailers, the exporter reported the status but did not retry.

Change

Read grpc-status from trailers when it is absent from headers, matching the existing status-resolution path. An IOException while reading trailers remains non-retryable. Added a regression test and an Unreleased changelog entry.

Fixes #8843

Validation

  • Baseline regression test on upstream/main: failed as expected (false instead of true).
  • JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :exporters:sender:okhttp:test --tests io.opentelemetry.exporter.sender.okhttp.internal.OkHttpGrpcSenderTest: passed (49 tests).
  • JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :exporters:sender:okhttp:spotlessCheck: passed.
  • git diff --check: passed.
  • :exporters:sender:okhttp:check: one unrelated existing timing-sensitive failure in shutdown_CompletableResultCodeShouldWaitForThreads; the changed regression and remaining tests passed.

Provenance

Codex assisted with repository analysis, implementation, regression testing, and validation. The submitter reviewed the complete diff and is responsible for the contribution.

Signed-off-by: Efe Gökdemir <efe@rexcode.co.uk>
@efegokdemir
efegokdemir requested a review from a team as a code owner September 23, 2026 11:14
@linux-foundation-easycla

linux-foundation-easycla Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

Signed-off-by: Efe Gökdemir <efe@rexcode.co.uk>
@opentelemetry-pr-dashboard

opentelemetry-pr-dashboard Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Pull request dashboard status

Merged · refreshed 2026-10-08 22:29 UTC

Status above doesn't look right?
  • Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

@codecov

codecov Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.45299% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.64%. Comparing base (a13bc21) to head (023bcc0).
⚠️ Report is 55 commits behind head on main.

Files with missing lines Patch % Lines
...orter/sender/okhttp/internal/OkHttpGrpcSender.java 83.33% 5 Missing and 5 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #8854      +/-   ##
============================================
- Coverage     91.44%   90.64%   -0.80%     
- Complexity    10667    10876     +209     
============================================
  Files          1007     1015       +8     
  Lines         28686    29712    +1026     
  Branches       3676     3874     +198     
============================================
+ Hits          26231    26932     +701     
- Misses         1657     1863     +206     
- Partials        798      917     +119     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@efegokdemir
efegokdemir force-pushed the codex/issue-8843-trailer-retry branch from a90f00d to a7ee760 Compare September 30, 2026 13:34
Signed-off-by: Efe Gökdemir <efe@rexcode.co.uk>
@efegokdemir

Copy link
Copy Markdown
Contributor Author

Fixed the patch-caused response-size regression in be1cea4. The new resolved-response retry path was retrying the sender-generated RESOURCE_EXHAUSTED response used when the local response-body limit is exceeded; that allowed the existing response-body-bound tests to observe a later successful attempt.

handleResponse now marks only responses decoded from the server as retryable candidates. Local size-limit, unsupported-encoding, and invalid-frame responses remain terminal failures.

Validation: git diff --check passed. The focused Gradle test could not run locally because this environment has no Java runtime; CI should validate the focused and matrix suites.

@efegokdemir

Copy link
Copy Markdown
Contributor Author

I inspected the current required-status-check and failed matrix logs. The failures are broad across macOS, Ubuntu, and Windows and across JDK 8, 11, 17, 21, 25, and 27. The logs show unrelated SDK metrics stress-test timeouts (for example SynchronousInstrumentStressTest partialWriteStressTest) and the required-status-check itself is an intentional exit 1; security/API checks pass. I found no evidence that the PR changes caused these failures, so I made no unrelated code change.

Signed-off-by: Efe Gökdemir <efe@rexcode.co.uk>
@efegokdemir

Copy link
Copy Markdown
Contributor Author

The retry regression was caused by OkHttp treating bodyless gRPC responses with an explicit trailer status as non-retryable. Updated the sender to preserve the decoded trailer status and mark only explicit server statuses retryable, while local size-limit/encoding failures remain non-retryable. Added focused coverage for an empty body with grpc-status: 14. New HEAD: 3f95082a. Local validation: :exporters:sender:okhttp:test PASS, :exporters:otlp:profiles:test --tests ...retryableError_tooManyAttempts PASS, spotlessCheck PASS, git diff --check PASS.

efegokdemir and others added 3 commits October 8, 2026 01:33
Signed-off-by: Efe Gökdemir <gokdemirefe1903@gmail.com>
Signed-off-by: Efe Gökdemir <gokdemirefe1903@gmail.com>
@jack-berg
jack-berg merged commit 49dd9f3 into open-telemetry:main Oct 8, 2026
34 checks passed
@otelbot

otelbot Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution @efegokdemir! 🎉 We would like to hear from you about your experience contributing to OpenTelemetry by taking a few minutes to fill out this survey.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OTLP gRPC: isRetryable ignores grpc-status in trailers, so UNAVAILABLE is never retried

2 participants