Fix native OOM from unbounded CallTraceHashTable expansion#670
Open
jbachorik wants to merge 1 commit into
Open
Fix native OOM from unbounded CallTraceHashTable expansion#670jbachorik wants to merge 1 commit into
jbachorik wants to merge 1 commit into
Conversation
CallTraceHashTable grows its backing LongHashTable by doubling capacity from a LinearAllocator whose chunks are a fixed 8MB. Once a table exceeds one chunk (capacity >= 512K entries), LinearAllocator::alloc() can never place it: the bump-allocator loops allocating fresh chunks that still cannot fit the oversized request, until mmap fails - a native OOM / vm.max_map_count exhaustion, not a heap-space error. This is reachable under live-heap profiling (memory=...:l) with a long, unrotated recording, where the active table grows past that threshold between processTraces() rotations. Cap expansion at the largest table that fits in one allocator chunk (LinearAllocator::maxAllocatableSize()); further growth degrades put() to a higher load factor and recovers on the next rotation. Deriving the bound from the chunk size keeps it correct if the chunk size ever changes. Also fix a per-thread native leak in LivenessTracker's subsampling RNG state: the gen/dis/skipped ThreadLocals were reclaimed only via pthread-key destructors, which never fire when a JNI-attached thread detaches on a reused OS thread. Release them explicitly from Profiler::onThreadEnd(), matching how the CPU/wall engines and ProfiledThread are already torn down. Environment: Datadog workspace Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Benchmark Results (commit 75285e9)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125374352 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Under live-heap profiling (
memory=...:l) with a long-running, unrotated recording, the profiler can exhaust native address space and crash withos::commit_memory ... failed; error='Not enough space' (errno=12)/pthread_create failed (EAGAIN). The failure isvm.max_map_count(VMA) exhaustion, not RAM: a captured memory map showed tens of thousands of 8 MB (and 8 MB-multiple) anonymous regions.Root cause
CallTraceHashTablegrows its backingLongHashTableby doubling capacity, allocating each generation from aLinearAllocatorwhose chunks are a fixed 8 MB (CALL_TRACE_CHUNK). Once a table's byte size exceeds one chunk (capacity ≥ ~512K entries),LinearAllocator::alloc()can never place it: the bump-allocator's inner loop only fitsoffs + size <= _chunk_size, so an oversized request falls through togetNextChunk(), which allocates a fresh 8 MB chunk that also can't fit it — looping and allocating chunks untilmmapfails.The active table only grows this large between
processTraces()rotations (which reset it to the initial capacity). In a single start-to-end JFR recording no rotation happens, so under the high distinct-trace volume that live-heap sampling sustains, the table crosses the one-chunk threshold and the allocator runs away.Fix
expandTableIfNeeded()now refuses to expand when the next generation'sgetSize()would exceedLinearAllocator::maxAllocatableSize()(one chunk minus its header). Instead of an unbounded allocation loop,put()keeps working on the current table at a higher load factor; the next rotation resets it. The bound is derived from the chunk size, so it stays correct ifCALL_TRACE_CHUNKchanges. A newcalltrace_storage_expansion_skippedcounter records when the cap engages.This also includes a related per-thread native leak fix in
LivenessTracker: its subsampling RNGThreadLocals (gen/dis/skipped) were reclaimed only via pthread-key destructors, which never fire when a JNI-attached thread detaches on a reused OS thread. They are now released explicitly fromProfiler::onThreadEnd(), matching how the CPU/wall engines andProfiledThreadare already torn down.Validation
Reproducer (
renaissance akka-uct -r 20undermemory=64:l) OOM'd at ~90 s / 13 iterations before the fix. With the fix it completes all 20 iterations with peak VMA count in the hundreds (was pinned at the 65530 limit). Compiles clean; verified the change is compatible with the existingexpandTableIfNeeded/overflow-guard unit tests by inspection (the new check sits alongside the existing guards and only engages for tables that exceed one chunk).🤖 Generated with Claude Code