Skip to content

fix(warehouse): close MetricsData after processing to release Arrow buffers - #4335

Open
paultanay wants to merge 6 commits into
apache:masterfrom
paultanay:fix/arrow-resource-leak
Open

fix(warehouse): close MetricsData after processing to release Arrow buffers#4335
paultanay wants to merge 6 commits into
apache:masterfrom
paultanay:fix/arrow-resource-leak

Conversation

@paultanay

@paultanay paultanay commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

What's changed?

DataStorageDispatch.startPersistentDataStorage() polls MetricsData objects from the queue and passes them through the history, plugin, and realtime writers, but never called close() afterward. MetricsData implements AutoCloseable and wraps an ArrowTable backed by Arrow column vectors - without close(), those vectors and their buffers are retained until GC decides to collect them. Under production load with many monitors this compounds into the steady heap growth reported in #4136.

Wrapped the polled object in try-with-resources so MetricsData.close() is guaranteed on all exit paths: normal completion, writer exception, and thread interrupt. DataStorageDispatch is the terminal consumer, so closing here is safe.

Fixes: #4136


Checklist

…uffers

MetricsData implements AutoCloseable and wraps an ArrowTable backed by
Arrow column vectors. The persistent-data-storage consumer loop was
polling MetricsData objects and passing them through writers without
ever calling close(), preventing the backing Arrow buffers from being
released and causing steady heap growth under load (issue apache#4136).

Wrap the polled object in try-with-resources so close() is guaranteed
on all exit paths—normal completion, writer exception, and interrupt.
DataStorageDispatch is the terminal consumer of each object, so closing
here is correct and safe.
@paultanay
paultanay marked this pull request as ready for review August 21, 2026 05:42

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: close MetricsData after processing to release Arrow buffers

Clean, minimal, and correct resource-leak fix.

What's good

  • commonDataQueue.pollMetricsDataToStorage() is now wrapped in try-with-resources, so MetricsData.close() (releasing the underlying Arrow column-vector buffers) is guaranteed on every exit path: normal completion, writer exception, and thread interrupt. This directly addresses the steady heap growth reported in #4136.
  • DataStorageDispatch.startPersistentDataStorage() is the terminal consumer of these MetricsData objects, so closing them here is safe — no downstream reader needs the Arrow buffers afterward.
  • The original control flow is preserved exactly: the inner try/finally still ensures realTimeDataWriter.saveData(...) runs even if calculateMonitorStatus/history/plugin steps throw. The null-guarded continue is still valid under try-with-resources (a null resource is simply not closed).
  • MetricsData implements AutoCloseable, so the compiler-enforced contract holds.

Non-blocking note

  • Standard try-with-resources behavior means if close() itself throws, it would suppress an earlier exception from the body. That is an acceptable, rare edge for this terminal-sink path.

No blocking issues. Approve.

@paultanay

Copy link
Copy Markdown
Contributor Author

CI failures are in the setup-deps infrastructure step (fails before any compilation). Build and all 88 tests pass locally under the release profile (mvn clean package -Prelease). Looks like a runner issue.

@paultanay
paultanay requested a review from Aias00 August 28, 2026 12:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants