Skip to content

test(storage): join metrics reader threads before tests exit - #16604

Open
kalragauri wants to merge 3 commits into
googleapis:mainfrom
kalragauri:fix-16587
Open

kalragauri wants to merge 3 commits into
googleapis:mainfrom
kalragauri:fix-16587

Conversation

@kalragauri

@kalragauri kalragauri commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

EnableGrpcMetricsImpl() stores its MeterProvider in ExporterRegistry::Singleton() and registers it with gRPC via OpenTelemetryPluginBuilder::BuildAndRegisterGlobal(). Neither singleton is ever destroyed, so the provider's PeriodicExportingMetricReader thread is still running when a test binary exits. That thread runs its first collection immediately upon startup; if the collection overlaps exit(), SyncMetricStorage::Collect() reads cardinality_limit_ from the function-local static in AggregationConfig::GetOrDefault() after ~AggregationConfig() has poisoned it under MemorySanitizer. Three unit tests that call EnableGrpcMetricsImpl() hit this race intermittently and were skipped in #16588.

This PR adds a ScopedProviderShutdown RAII guard in channel_metrics_test.cc and metrics_exporter_impl_test.cc that downcasts the MeterProvider to opentelemetry::sdk::metrics::MeterProvider and calls Shutdown() when the test ends (including after a failed ASSERT_*), joining the reader thread before exit() runs.

Fixes #16587

@product-auto-label product-auto-label Bot added the api: storage Issues related to the Cloud Storage API. label Oct 29, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request re-enables several previously flaky gRPC metrics tests by introducing a ScopedProviderShutdown helper class. This helper ensures that process-wide MeterProvider instances are properly shut down when a test ends, preventing background collection threads from racing with test exit. The review feedback suggests optimizing the RegisteredAuthorityHonorsOption test by avoiding redundant calls to FindMeterProvider and instead reusing the retrieved provider.

Comment thread google/cloud/storage/internal/grpc/channel_metrics_test.cc Outdated
@kalragauri
kalragauri marked this pull request as ready for review October 29, 2026 11:21
@kalragauri
kalragauri requested review from a team as code owners October 29, 2026 11:21
@codecov

codecov Bot commented Oct 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.40%. Comparing base (608aeef) to head (5e05556).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff            @@
##             main   #16604    +/-   ##
========================================
  Coverage   92.40%   92.40%            
========================================
  Files        2266     2266            
  Lines      220810   221043   +233     
========================================
+ Hits       204036   204255   +219     
- Misses      16774    16788    +14     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch was successfully deployed

1 active deployment
false — 5e05556b Deployed Oct 29, 2026 by kalragauri via Save PR ref #12709
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the Cloud Storage API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

storage: MSan use-after-dtor flake at exit in tests calling EnableGrpcMetricsImpl

2 participants