Skip to content

Reference chains for surviving live-heap samples (PROF-15341)#644

Draft
jbachorik wants to merge 5 commits into
mainfrom
jb/reference-chains
Draft

Reference chains for surviving live-heap samples (PROF-15341)#644
jbachorik wants to merge 5 commits into
mainfrom
jb/reference-chains

Conversation

@jbachorik

@jbachorik jbachorik commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?:
Implements reference-chain reconstruction for live-heap samples that survive
past their allocation window (PROF-15341): a JVMTI FollowReferences-based
BFS walk (ReferenceChainTracker) tags reachable objects and records
referrer-type chains back toward a GC root, bridged to
LivenessTracker::selectLeakCandidates()'s population-slope leak detection.

This revision also closes a structural gap found while adding end-to-end
coverage: a one-shot search could finish walking the whole reachable graph
before population-trend detection had accumulated enough GC epochs to flag a
candidate, making any object allocated after that point permanently
undiscoverable. Search restarts are now gated on an actual leak indication
existing (so a restarted walk is very likely to still find the object) plus
a PainBudget cooldown - a leaky bucket over cumulative safepoint time
spent, not a fixed interval, so a cheap search can restart sooner than an
expensive one.

Motivation:
PROF-15341 - give live-heap samples that survive long enough to look
leak-shaped an actual referrer chain, not just "this object is still alive".

Additional Notes:
See doc/architecture/LiveHeapReferenceChains.md and
doc/architecture/LiveHeapReferenceChains-RemainingWorkPlan.md for the full
design history, open questions, and what's still provisional (per-pass
budget/hop-cap/TTL/frontier-cap/pain-budget-refill-rate defaults are all
round, unbenchmarked placeholders pending a future measurement pass).

How to test the change?:

  • ./gradlew :ddprof-lib:gtestDebug - full native unit suite, including new
    PainBudgetTest/SearchRestartTest coverage for the restart gate.
  • ./gradlew :ddprof-test:testDebug --tests "*ExternalProcessReferenceChainTest*" -
    genuinely separate-process end-to-end test: runs a real leaking-cache Java
    app in a child JVM and asserts the reconstructed chain's leaf class.
  • ./gradlew :ddprof-test:testDebug --tests "*ReferenceChainTrackingTest*" -
    in-process coverage for the BFS engine, target-selection bridging, and
    abandonment reporting.

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a security review (run the dd:platform-security-review
    skill, or file a request via the PSEC review form).
    bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: PROF-15341

Unsure? Have a question? Request a review!

🤖 Generated with Claude Code

@jbachorik jbachorik added the AI label Jul 8, 2026
@datadog-datadog-prod-us1

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #29515196421 | Commit: b8068c7 | Duration: 18m 12s (longest job)

All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - - -
8-ibm - - -
8-j9 - -
8-librca - -
8-orcl - - -
11 - - -
11-j9 - -
11-librca - -
17 - -
17-graal - -
17-j9 - -
17-librca - -
21 - -
21-graal - -
21-librca - -
25 - -
25-graal - -
25-librca - -

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0


Updated: 2026-07-16 16:42:51 UTC

@dd-octo-sts

dd-octo-sts Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Reliability & Chaos Results

All reliability & chaos checks passed Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/125072101

@dd-octo-sts

dd-octo-sts Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit cfec595)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/123822759 Commit: cfec5956f438cc43b62cd2c81cbe46077ef66168

⚠️ Significant outliers

  • 💥 akka-uct (JDK 25): latest crashed
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 25 💥 0 ms (0 iters) 💥 0 ms (0 iters) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 3 / 1 1949 / 2055 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / · ✅ / · ✅ / · ✅ / · ✅ / · ✅ / ·
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 4 / 1 8727 / 8389 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 1 8166 / 8396 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 3 / 2 1292 / 1256 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 3 / 1 1288 / 1290 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 2 / 1 2830 / 3023 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 2976 / 2821 ✅ / ✅ ✅ / ✅
naive-bayes 21 · / ✅ · / ✅ · / 3 · / 3555 · / ✅ · / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 2 / 3 3499 / 3485 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 1 / 1 1592 / 1697 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / 1 1922 / 1854 ✅ / ✅ ✅ / ✅

jbachorik added a commit that referenced this pull request Jul 10, 2026
…racking

Addresses review findings from PR #644 (reference chains for surviving
live-heap samples): FrontierTable's shared-lock mutation race, a
single-byte JFR event size prefix that silently truncates chains longer
than 255 bytes, blocking sample-lock retries in writeReferenceChain()
now bounded by a shared per-batch deadline with a drop counter,
unvalidated referencechains sub-option values, a startThread()/
pthread_kill() race publishing _running before the thread handle is
initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags()
now surfacing GetObjectsWithTags() failures so restartSearch() never
resets tag state prematurely, resolveLoadedClasses() skipping its
per-class scan only when the loaded-class count is unchanged (not just
non-decreasing), a spurious COMPLETED state after a failed first-pass
FollowReferences call, and removal of a leftover debug helper in
ExternalProcessReferenceChainTest. Adds regression test coverage for
release-failure and negative-value option paths.

Verified via the full ddprof-lib gtestDebug suite (149/149 tasks,
57/57 referenceChains_ut tests).

Environment: Datadog workspace

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dd-octo-sts

dd-octo-sts Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 0714bd3)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124004132 Commit: 0714bd3166261f2710c22414dbfa68c11c699bdc

⚠️ Significant outliers

  • 🟢 fj-kmeans (JDK 21): runtime -2.6% (2834→2761 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10184 ms (21 iters) ✅ 10371 ms (21 iters) ≈ +1.8% (±11.5%) — / —
akka-uct 25 ✅ 8891 ms (24 iters) ✅ 8865 ms (24 iters) ≈ -0.3% (±10.2%) — / —
finagle-chirper 21 ✅ 6003 ms (33 iters) ✅ 6034 ms (33 iters) ≈ +0.5% (±25.2%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5507 ms (36 iters) ✅ 5426 ms (36 iters) ≈ -1.5% (±23.8%) ⚠️ W:4 / ⚠️ W:3
fj-kmeans 21 ✅ 2834 ms (66 iters) ✅ 2761 ms (67 iters) 🟢 -2.6% — / —
fj-kmeans 25 ✅ 2824 ms (66 iters) ✅ 2838 ms (66 iters) ≈ +0.5% (±2.7%) — / —
future-genetic 21 ✅ 2113 ms (88 iters) ✅ 2065 ms (89 iters) ≈ -2.3% (±2.6%) — / —
future-genetic 25 ✅ 2059 ms (90 iters) ✅ 2102 ms (89 iters) ≈ +2.1% (±2.6%) — / —
naive-bayes 21 ✅ 1283 ms (134 iters) ✅ 1260 ms (135 iters) ≈ -1.8% (±32%) — / —
naive-bayes 25 ✅ 1016 ms (169 iters) ✅ 1017 ms (168 iters) ≈ +0.1% (±31.5%) — / —
reactors 21 ✅ 16658 ms (15 iters) ✅ 16371 ms (16 iters) ≈ -1.7% (±9%) — / —
reactors 25 ✅ 18507 ms (15 iters) ✅ 18056 ms (15 iters) ≈ -2.4% (±5.6%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 2 / 1 1929 / 2034 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 3 / 1 2310 / 2113 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 4 / 5 8529 / 8852 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 2 / 2 8635 / 8390 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 1 / 5 1284 / 1281 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 5 / 2 1283 / 1285 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / 1 2930 / 2864 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 2963 / 2953 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 7 / 6 3558 / 3510 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 2 / 4 3463 / 3469 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 1 / 1 1564 / 1782 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1905 / 1681 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 700d838)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124275960 Commit: 700d838fc33298d5789fd9e1b29c8a142c11f41e

⚠️ Significant outliers

  • 🔴 fj-kmeans (JDK 21): runtime +4.5% (2598→2715 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10425 ms (21 iters) ✅ 10259 ms (21 iters) ≈ -1.6% (±11.1%) — / —
akka-uct 25 ✅ 8748 ms (24 iters) ✅ 8797 ms (24 iters) ≈ +0.6% (±10.5%) — / —
finagle-chirper 21 ✅ 5970 ms (33 iters) ✅ 5982 ms (33 iters) ≈ +0.2% (±25.3%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5469 ms (36 iters) ✅ 5468 ms (36 iters) ≈ -0% (±24.3%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2598 ms (73 iters) ✅ 2715 ms (69 iters) 🔴 +4.5% — / —
fj-kmeans 25 ✅ 2784 ms (67 iters) ✅ 2847 ms (66 iters) ≈ +2.3% (±2.7%) — / —
future-genetic 21 ✅ 2122 ms (87 iters) ✅ 2126 ms (88 iters) ≈ +0.2% (±2.7%) — / —
future-genetic 25 ✅ 2035 ms (91 iters) ✅ 2057 ms (91 iters) ≈ +1.1% (±2.7%) — / —
naive-bayes 21 ✅ 1304 ms (131 iters) ✅ 1253 ms (136 iters) ≈ -3.9% (±32.2%) — / —
naive-bayes 25 ✅ 1015 ms (169 iters) ✅ 1018 ms (168 iters) ≈ +0.3% (±31.8%) — / —
reactors 21 ✅ 16107 ms (15 iters) ✅ 16847 ms (15 iters) ≈ +4.6% (±7.9%) — / —
reactors 25 ✅ 18469 ms (15 iters) ✅ 19075 ms (14 iters) ≈ +3.3% (±4.9%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 3 / ✅ 1950 / 1991 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ ✅ / 2 2228 / 2453 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 6 / 2 8686 / 8417 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 1 8561 / 8290 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 2 / 2 1272 / 1289 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 1287 / 1298 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / ✅ 2912 / 3017 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ ✅ / 1 2907 / 2872 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 2 / 3 3515 / 3489 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 2 / 7 3498 / 3463 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 2 / 2 1732 / 1701 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1780 / 1924 ✅ / ✅ ✅ / ✅

jbachorik added a commit that referenced this pull request Jul 14, 2026
…racking

Addresses review findings from PR #644 (reference chains for surviving
live-heap samples): FrontierTable's shared-lock mutation race, a
single-byte JFR event size prefix that silently truncates chains longer
than 255 bytes, blocking sample-lock retries in writeReferenceChain()
now bounded by a shared per-batch deadline with a drop counter,
unvalidated referencechains sub-option values, a startThread()/
pthread_kill() race publishing _running before the thread handle is
initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags()
now surfacing GetObjectsWithTags() failures so restartSearch() never
resets tag state prematurely, resolveLoadedClasses() skipping its
per-class scan only when the loaded-class count is unchanged (not just
non-decreasing), a spurious COMPLETED state after a failed first-pass
FollowReferences call, and removal of a leftover debug helper in
ExternalProcessReferenceChainTest. Adds regression test coverage for
release-failure and negative-value option paths.

Verified via the full ddprof-lib gtestDebug suite (149/149 tasks,
57/57 referenceChains_ut tests).

Environment: Datadog workspace

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@jbachorik
jbachorik force-pushed the jb/reference-chains branch from 700d838 to 9972bff Compare July 14, 2026 12:03
@dd-octo-sts

dd-octo-sts Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 9972bff)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124511782 Commit: 9972bff9464d4d2a8ee4a279dc8f8d35c88dd2e9

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10365 ms (21 iters) ✅ 10456 ms (21 iters) ≈ +0.9% (±12%) — / —
akka-uct 25 ✅ 8820 ms (24 iters) ✅ 8890 ms (24 iters) ≈ +0.8% (±9.9%) — / —
finagle-chirper 21 ✅ 5990 ms (33 iters) ✅ 6050 ms (33 iters) ≈ +1% (±24.8%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5495 ms (36 iters) ✅ 5462 ms (36 iters) ≈ -0.6% (±24.2%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2745 ms (68 iters) ✅ 2714 ms (69 iters) ≈ -1.1% (±2.7%) — / —
fj-kmeans 25 ✅ 2831 ms (66 iters) ✅ 2828 ms (66 iters) ≈ -0.1% (±2.6%) — / —
future-genetic 21 ✅ 2150 ms (87 iters) ✅ 2112 ms (87 iters) ≈ -1.8% (±2.7%) — / —
future-genetic 25 ✅ 2073 ms (90 iters) ✅ 2121 ms (87 iters) ≈ +2.3% (±2.5%) — / —
naive-bayes 21 ✅ 1287 ms (133 iters) ✅ 1312 ms (131 iters) ≈ +1.9% (±33%) — / —
naive-bayes 25 ✅ 1017 ms (168 iters) ✅ 1030 ms (166 iters) ≈ +1.3% (±32.1%) — / —
reactors 21 ✅ 15980 ms (15 iters) ✅ 16718 ms (15 iters) ≈ +4.6% (±7%) — / —
reactors 25 ✅ 18712 ms (15 iters) ✅ 18346 ms (15 iters) ≈ -2% (±3.1%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 1 / 1 1902 / 2028 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 2 / 1 2103 / 2152 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 3 / 5 8628 / 8249 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 2 8833 / 8700 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 1 / 3 1256 / 1240 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 4 / 5 1264 / 1305 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ ✅ / 2 3046 / 3002 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ ✅ / 3 2879 / 2830 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ ✅ / 2 3479 / 3463 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 1 / 5 3476 / 3482 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / 1 1862 / 1833 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit a636398)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124700626 Commit: a636398f93c3fd15fb9fd3b6253da66f36e56742

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10127 ms (21 iters) ✅ 10208 ms (21 iters) ≈ +0.8% (±11.6%) — / —
akka-uct 25 ✅ 9010 ms (24 iters) ✅ 8842 ms (24 iters) ≈ -1.9% (±10.3%) — / —
finagle-chirper 21 ✅ 6004 ms (33 iters) ✅ 5987 ms (33 iters) ≈ -0.3% (±25%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5500 ms (36 iters) ✅ 5500 ms (36 iters) ≈ 0% (±24.6%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2714 ms (69 iters) ✅ 2715 ms (68 iters) ≈ +0% (±2.7%) — / —
fj-kmeans 25 ✅ 2812 ms (66 iters) ✅ 2837 ms (66 iters) ≈ +0.9% (±2.6%) — / —
future-genetic 21 ✅ 2107 ms (88 iters) ✅ 2065 ms (90 iters) ≈ -2% (±2.6%) — / —
future-genetic 25 ✅ 2018 ms (92 iters) ✅ 2025 ms (92 iters) ≈ +0.3% (±2.8%) — / —
naive-bayes 21 ✅ 1246 ms (137 iters) ✅ 1227 ms (139 iters) ≈ -1.5% (±32.3%) — / —
naive-bayes 25 ✅ 1003 ms (170 iters) ✅ 1028 ms (166 iters) ≈ +2.5% (±32%) — / —
reactors 21 ✅ 15862 ms (15 iters) ✅ 16456 ms (15 iters) ≈ +3.7% (±9.3%) — / —
reactors 25 ✅ 18376 ms (15 iters) 💥 0 ms (0 iters) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 3 / 4 1972 / 1963 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 2 / 6 2396 / 2157 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 2 / 2 8557 / 8411 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 4 / ✅ 8575 / 8705 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 1 / ✅ 1288 / 1228 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 1 / 3 1268 / 1277 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 2 / 2 2972 / 2870 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 1 / 2 2818 / 2994 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 3 / 2 3519 / 3503 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 2 / 3 3468 / 3470 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / 1 1567 / 1727 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1929 / 1873 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit f2d8978)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124747648 Commit: f2d8978c23b568eb4ce8006e8e9b901cdaed11c3

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10331 ms (21 iters) ✅ 10246 ms (21 iters) ≈ -0.8% (±11.2%) — / —
akka-uct 25 ✅ 8728 ms (24 iters) ✅ 8956 ms (24 iters) ≈ +2.6% (±10.6%) — / —
finagle-chirper 21 ✅ 6011 ms (33 iters) ✅ 5985 ms (33 iters) ≈ -0.4% (±25.6%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5522 ms (36 iters) ✅ 5441 ms (36 iters) ≈ -1.5% (±24.4%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2699 ms (70 iters) ✅ 2742 ms (68 iters) ≈ +1.6% (±2.7%) — / —
fj-kmeans 25 ✅ 2806 ms (66 iters) ✅ 2796 ms (66 iters) ≈ -0.4% (±2.6%) — / —
future-genetic 25 ✅ 2082 ms (89 iters) ✅ 2051 ms (90 iters) ≈ -1.5% (±2.6%) — / —
naive-bayes 21 ✅ 1252 ms (136 iters) ✅ 1252 ms (136 iters) ≈ 0% (±32.7%) — / —
reactors 21 ✅ 16483 ms (15 iters) ✅ 17081 ms (15 iters) ≈ +3.6% (±7.9%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 1 / 4 1998 / 1935 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 3 / 2 2240 / 2356 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 3 / 5 8506 / 8471 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 1 8384 / 7986 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 5 / 3 1293 / 1281 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 2 / 2 1255 / 1280 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ ✅ / 2 2987 / 2963 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ ✅ / 2 2916 / 2851 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 5 / 4 3501 / 3519 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 6 / 9 3486 / 3432 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 1 / 1 1617 / 1818 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / 1 1931 / 1868 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit a48899b)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124773626 Commit: a48899b982ca5139081f665d6ffe1b8b5c3790fa

⚠️ Significant outliers

  • 🔴 fj-kmeans (JDK 21): runtime +2.8% (2687→2761 ms)
  • 🔴 future-genetic (JDK 21): runtime +3.4% (2047→2116 ms)
  • 🔴 future-genetic (JDK 25): runtime +4.2% (2000→2083 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10376 ms (21 iters) ✅ 10228 ms (21 iters) ≈ -1.4% (±11%) — / —
akka-uct 25 ✅ 8829 ms (24 iters) ✅ 8859 ms (24 iters) ≈ +0.3% (±10%) — / —
finagle-chirper 21 ✅ 6001 ms (33 iters) ✅ 5928 ms (33 iters) ≈ -1.2% (±25.2%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5469 ms (36 iters) ✅ 5475 ms (36 iters) ≈ +0.1% (±24.5%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2687 ms (70 iters) ✅ 2761 ms (68 iters) 🔴 +2.8% — / —
fj-kmeans 25 ✅ 2810 ms (66 iters) ✅ 2805 ms (66 iters) ≈ -0.2% (±2.6%) — / —
future-genetic 21 ✅ 2047 ms (91 iters) ✅ 2116 ms (88 iters) 🔴 +3.4% — / —
future-genetic 25 ✅ 2000 ms (93 iters) ✅ 2083 ms (89 iters) 🔴 +4.2% — / —
naive-bayes 21 ✅ 1256 ms (136 iters) ✅ 1253 ms (136 iters) ≈ -0.2% (±32.8%) — / —
naive-bayes 25 ✅ 1016 ms (169 iters) ✅ 1016 ms (169 iters) ≈ 0% (±31.3%) — / —
reactors 21 ✅ 16870 ms (15 iters) ✅ 16463 ms (15 iters) ≈ -2.4% (±7.3%) — / —
reactors 25 ✅ 18270 ms (15 iters) ✅ 18333 ms (15 iters) ≈ +0.3% (±4.5%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 4 / 2 1988 / 2024 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 2 / 3 2263 / 2395 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 1 / 6 8440 / 8679 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ ✅ / 2 8283 / 8410 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 2 / 1 1288 / 1260 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 4 / 4 1250 / 1295 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 3 / 2 2968 / 3003 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ ✅ / 1 2843 / 2902 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 4 / 5 3474 / 3510 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 3 / 2 3495 / 3488 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1726 / 1721 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / 3 1893 / 1750 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 787f29e)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124811024 Commit: 787f29e84914d8da397ed2a644c91d59a2f1e08e

⚠️ Significant outliers

  • 🔴 fj-kmeans (JDK 21): runtime +5.3% (2693→2836 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10215 ms (21 iters) ✅ 10475 ms (21 iters) ≈ +2.5% (±11.9%) — / —
akka-uct 25 ✅ 8843 ms (24 iters) ✅ 8861 ms (24 iters) ≈ +0.2% (±10.5%) — / —
finagle-chirper 21 ✅ 5986 ms (33 iters) ✅ 5971 ms (33 iters) ≈ -0.3% (±25.5%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5515 ms (36 iters) ✅ 5479 ms (36 iters) ≈ -0.7% (±24.2%) ⚠️ W:3 / ⚠️ W:4
fj-kmeans 21 ✅ 2693 ms (70 iters) ✅ 2836 ms (66 iters) 🔴 +5.3% — / —
fj-kmeans 25 ✅ 2820 ms (66 iters) ✅ 2837 ms (66 iters) ≈ +0.6% (±2.6%) — / —
future-genetic 21 ✅ 2059 ms (90 iters) ✅ 2070 ms (89 iters) ≈ +0.5% (±2.6%) — / —
future-genetic 25 ✅ 2089 ms (89 iters) ✅ 2130 ms (87 iters) ≈ +2% (±2.6%) — / —
naive-bayes 21 ✅ 1284 ms (133 iters) ✅ 1313 ms (131 iters) ≈ +2.3% (±33%) — / —
naive-bayes 25 ✅ 1009 ms (170 iters) ✅ 988 ms (173 iters) ≈ -2.1% (±31.4%) — / —
reactors 21 ✅ 16540 ms (15 iters) ✅ 16155 ms (15 iters) ≈ -2.3% (±7.1%) — / —
reactors 25 ✅ 18545 ms (15 iters) ✅ 18816 ms (15 iters) ≈ +1.5% (±4.2%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 3 / 2 1987 / 1990 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 1 / 4 2238 / 2323 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 2 / 2 8290 / 8529 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ ✅ / 1 8323 / 8386 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 3 / 1 1299 / 1277 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1274 / 1299 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 2 / ✅ 2920 / 2982 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / 4 2886 / 2864 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 4 / 1 3505 / 3499 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1593 / 1669 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 1 / 1 1849 / 1940 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit fede524)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124833099 Commit: fede5244169eddd7f6280cd0a9fe39b618080bd7

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10349 ms (21 iters) ✅ 10262 ms (21 iters) ≈ -0.8% (±12.1%) — / —
akka-uct 25 ✅ 8820 ms (24 iters) ✅ 8733 ms (24 iters) ≈ -1% (±10.2%) — / —
finagle-chirper 21 ✅ 6023 ms (33 iters) ✅ 6016 ms (33 iters) ≈ -0.1% (±25.6%) ⚠️ W:4 / ⚠️ W:3
finagle-chirper 25 ✅ 5517 ms (36 iters) ✅ 5419 ms (36 iters) ≈ -1.8% (±24.1%) ⚠️ W:4 / ⚠️ W:3
fj-kmeans 21 ✅ 2713 ms (69 iters) ✅ 2649 ms (71 iters) ≈ -2.4% (±2.5%) — / —
fj-kmeans 25 ✅ 2826 ms (66 iters) ✅ 2808 ms (66 iters) ≈ -0.6% (±2.5%) — / —
future-genetic 21 ✅ 2076 ms (89 iters) ✅ 2092 ms (89 iters) ≈ +0.8% (±2.7%) — / —
future-genetic 25 ✅ 2029 ms (92 iters) ✅ 2024 ms (91 iters) ≈ -0.2% (±2.7%) — / —
naive-bayes 21 ✅ 1253 ms (136 iters) ✅ 1279 ms (134 iters) ≈ +2.1% (±33%) — / —
naive-bayes 25 ✅ 1008 ms (170 iters) ✅ 1012 ms (169 iters) ≈ +0.4% (±31.6%) — / —
reactors 21 ✅ 16287 ms (15 iters) ✅ 16063 ms (16 iters) ≈ -1.4% (±6.5%) — / —
reactors 25 ✅ 18328 ms (15 iters) ✅ 18575 ms (15 iters) ≈ +1.3% (±4.4%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 2 / 2 1913 / 1895 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 2 / 3 2116 / 2231 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 4 / 4 8775 / 8877 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 2 / 1 8359 / 8415 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 2 / 2 1260 / 1279 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 1275 / 1273 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 1 / 1 2874 / 2807 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 5 / ✅ 3520 / 3527 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 5 / 5 3466 / 3469 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / 1 1547 / 1755 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / 1 1873 / 1890 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit ef0f5c6)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124961539 Commit: ef0f5c6cd6793c440715d8f51a20218afa8cd188

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10321 ms (21 iters) ✅ 10247 ms (21 iters) ≈ -0.7% (±11.2%) — / —
akka-uct 25 ✅ 8827 ms (24 iters) ✅ 8856 ms (24 iters) ≈ +0.3% (±9.9%) — / —
finagle-chirper 21 ✅ 5961 ms (33 iters) ✅ 5961 ms (33 iters) ≈ 0% (±25.2%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5473 ms (36 iters) ✅ 5482 ms (36 iters) ≈ +0.2% (±24.2%) ⚠️ W:3 / ⚠️ W:4
fj-kmeans 21 ✅ 2721 ms (69 iters) ✅ 2776 ms (67 iters) ≈ +2% (±2.8%) — / —
fj-kmeans 25 ✅ 2805 ms (66 iters) ✅ 2801 ms (66 iters) ≈ -0.1% (±2.6%) — / —
future-genetic 21 ✅ 2074 ms (89 iters) ✅ 2054 ms (90 iters) ≈ -1% (±2.5%) — / —
future-genetic 25 ✅ 2082 ms (89 iters) ✅ 2065 ms (90 iters) ≈ -0.8% (±2.6%) — / —
naive-bayes 21 ✅ 1255 ms (137 iters) ✅ 1266 ms (135 iters) ≈ +0.9% (±32.6%) — / —
naive-bayes 25 ✅ 1006 ms (170 iters) ✅ 979 ms (174 iters) ≈ -2.7% (±31.6%) — / —
reactors 21 ✅ 16850 ms (15 iters) ✅ 16138 ms (15 iters) ≈ -4.2% (±7%) — / —
reactors 25 ✅ 18309 ms (15 iters) ✅ 18522 ms (15 iters) ≈ +1.2% (±5.7%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 1 / 1 1878 / 1946 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ ✅ / 3 2374 / 2471 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 3 / 3 8174 / 8713 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 6 / 2 8354 / 8759 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ ✅ / 2 1279 / 1258 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 2 / 2 1264 / 1263 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / 3 2985 / 2919 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / 5 2902 / 2849 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 4 / 5 3582 / 3490 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 9 / 3 3490 / 3517 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 2 / 1 1749 / 1686 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 6 / 2 1866 / 1903 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 1299316)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124967829 Commit: 1299316723d6feb8baf7e573b7925f816b772b79

⚠️ Significant outliers

  • 🔴 fj-kmeans (JDK 21): runtime +4.7% (2696→2822 ms)
  • 🟢 fj-kmeans (JDK 25): runtime -3.4% (2842→2746 ms)
  • 🟢 future-genetic (JDK 25): runtime -3.5% (2134→2059 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10336 ms (21 iters) ✅ 10245 ms (21 iters) ≈ -0.9% (±11.5%) — / —
akka-uct 25 ✅ 8868 ms (24 iters) ✅ 8789 ms (24 iters) ≈ -0.9% (±10.7%) — / —
finagle-chirper 21 ✅ 5968 ms (33 iters) ✅ 5971 ms (33 iters) ≈ +0.1% (±25.8%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5510 ms (36 iters) ✅ 5537 ms (36 iters) ≈ +0.5% (±24.5%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2696 ms (69 iters) ✅ 2822 ms (66 iters) 🔴 +4.7% — / —
fj-kmeans 25 ✅ 2842 ms (66 iters) ✅ 2746 ms (68 iters) 🟢 -3.4% — / —
future-genetic 21 ✅ 2068 ms (90 iters) ✅ 2051 ms (90 iters) ≈ -0.8% (±2.6%) — / —
future-genetic 25 ✅ 2134 ms (87 iters) ✅ 2059 ms (90 iters) 🟢 -3.5% — / —
naive-bayes 21 ✅ 1318 ms (130 iters) ✅ 1253 ms (136 iters) ≈ -4.9% (±31.8%) — / —
naive-bayes 25 ✅ 998 ms (171 iters) ✅ 1025 ms (167 iters) ≈ +2.7% (±32.1%) — / —
reactors 21 ✅ 16264 ms (15 iters) ✅ 16625 ms (15 iters) ≈ +2.2% (±7.4%) — / —
reactors 25 ✅ 17926 ms (15 iters) ✅ 18917 ms (15 iters) ≈ +5.5% (±5.6%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ ✅ / 1 1917 / 1934 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 3 / 3 2285 / 2434 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 6 / 2 8503 / 8628 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 2 / 2 8331 / 8664 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 2 / 1 1287 / 1266 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 2 / 2 1277 / 1290 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / 3 3037 / 2981 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 1 / ✅ 2897 / 2943 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 3 / 12 3507 / 3493 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 5 / 5 3446 / 3481 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 4 / ✅ 1765 / 1665 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 1 / ✅ 1855 / 1920 ✅ / ✅ ✅ / ✅

jbachorik added a commit that referenced this pull request Jul 16, 2026
…racking

Addresses review findings from PR #644 (reference chains for surviving
live-heap samples): FrontierTable's shared-lock mutation race, a
single-byte JFR event size prefix that silently truncates chains longer
than 255 bytes, blocking sample-lock retries in writeReferenceChain()
now bounded by a shared per-batch deadline with a drop counter,
unvalidated referencechains sub-option values, a startThread()/
pthread_kill() race publishing _running before the thread handle is
initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags()
now surfacing GetObjectsWithTags() failures so restartSearch() never
resets tag state prematurely, resolveLoadedClasses() skipping its
per-class scan only when the loaded-class count is unchanged (not just
non-decreasing), a spurious COMPLETED state after a failed first-pass
FollowReferences call, and removal of a leftover debug helper in
ExternalProcessReferenceChainTest. Adds regression test coverage for
release-failure and negative-value option paths.

Verified via the full ddprof-lib gtestDebug suite (149/149 tasks,
57/57 referenceChains_ut tests).

Environment: Datadog workspace

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@jbachorik
jbachorik force-pushed the jb/reference-chains branch from 1299316 to baba070 Compare July 16, 2026 11:31
@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit baba070)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124983919 Commit: baba0702198bb700cacfe9ab28c900eaf7d3f729

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10403 ms (21 iters) ✅ 10169 ms (21 iters) ≈ -2.2% (±10.2%) — / —
akka-uct 25 ✅ 8907 ms (24 iters) ✅ 8843 ms (24 iters) ≈ -0.7% (±10.4%) — / —
finagle-chirper 21 ✅ 5989 ms (33 iters) ✅ 5996 ms (33 iters) ≈ +0.1% (±25.4%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5542 ms (36 iters) ✅ 5550 ms (36 iters) ≈ +0.1% (±24.6%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2791 ms (67 iters) ✅ 2838 ms (66 iters) ≈ +1.7% (±2.7%) — / —
fj-kmeans 25 ✅ 2852 ms (66 iters) ✅ 2834 ms (66 iters) ≈ -0.6% (±2.6%) — / —
future-genetic 21 ✅ 2086 ms (89 iters) ✅ 2064 ms (90 iters) ≈ -1.1% (±2.6%) — / —
future-genetic 25 ✅ 2100 ms (88 iters) ✅ 2103 ms (88 iters) ≈ +0.1% (±2.5%) — / —
naive-bayes 21 ✅ 1266 ms (135 iters) ✅ 1263 ms (135 iters) ≈ -0.2% (±32.4%) — / —
naive-bayes 25 ✅ 1016 ms (168 iters) ✅ 1017 ms (168 iters) ≈ +0.1% (±31.8%) — / —
reactors 21 ✅ 16362 ms (15 iters) ✅ 16222 ms (15 iters) ≈ -0.9% (±8.7%) — / —
reactors 25 ✅ 18867 ms (15 iters) ✅ 19004 ms (15 iters) ≈ +0.7% (±4.9%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 2 / 1 1936 / 1969 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ ✅ / 3 2258 / 2003 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 4 / 5 8622 / 8930 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 2 8497 / 8401 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 3 / 2 1283 / 1288 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 1 / ✅ 1282 / 1292 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 4 / 1 2933 / 3028 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / 3 2856 / 2850 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 2 / 4 3539 / 3501 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 7 / 5 3446 / 3450 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 2 / 1 1682 / 1519 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1907 / 1977 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 35c49c2)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124997895 Commit: 35c49c2663a2e6e270a0287e4e059b8dab047faa

⚠️ Significant outliers

  • 🟢 fj-kmeans (JDK 25): runtime -3% (2842→2756 ms)
  • 🟢 future-genetic (JDK 21): runtime -2.9% (2142→2080 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10367 ms (21 iters) ✅ 10371 ms (21 iters) ≈ +0% (±11.3%) — / —
akka-uct 25 ✅ 8917 ms (24 iters) ✅ 8766 ms (24 iters) ≈ -1.7% (±10.2%) — / —
finagle-chirper 21 ✅ 5986 ms (33 iters) ✅ 5972 ms (33 iters) ≈ -0.2% (±25.3%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2801 ms (67 iters) ✅ 2750 ms (68 iters) ≈ -1.8% (±2.6%) — / —
fj-kmeans 25 ✅ 2842 ms (66 iters) ✅ 2756 ms (68 iters) 🟢 -3% — / —
future-genetic 21 ✅ 2142 ms (87 iters) ✅ 2080 ms (89 iters) 🟢 -2.9% — / —
future-genetic 25 ✅ 1991 ms (94 iters) ✅ 2035 ms (91 iters) ≈ +2.2% (±2.7%) — / —
naive-bayes 25 ✅ 1025 ms (167 iters) ✅ 1010 ms (169 iters) ≈ -1.5% (±31.4%) — / —
reactors 21 ✅ 16244 ms (15 iters) ✅ 16474 ms (15 iters) ≈ +1.4% (±7.9%) — / —
reactors 25 ✅ 18288 ms (15 iters) ✅ 18332 ms (15 iters) ≈ +0.2% (±3.9%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 2 / 4 1950 / 1991 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 1 / 1 2303 / 2180 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 2 / 6 8767 / 8881 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 2 / 1 8034 / 8300 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 1 / 1 1316 / 1287 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 1 / 2 1289 / 1282 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 2 / ✅ 2889 / 2882 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 3 / 1 2935 / 2838 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 8 / 2 3496 / 3495 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 4 / 4 3477 / 3510 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 2 / 1 1642 / 1586 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 1 / 3 1802 / 1759 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit a9fb05f)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125008807 Commit: a9fb05fe90d68c1097e48893b1256661ecdfce30

⚠️ Significant outliers

  • 🟢 fj-kmeans (JDK 21): runtime -3.1% (2772→2687 ms)
  • 💥 naive-bayes (JDK 25): latest crashed
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10309 ms (21 iters) ✅ 10453 ms (21 iters) ≈ +1.4% (±11%) — / —
akka-uct 25 ✅ 8814 ms (24 iters) ✅ 8779 ms (24 iters) ≈ -0.4% (±10.4%) — / —
finagle-chirper 21 ✅ 6006 ms (33 iters) ✅ 6047 ms (33 iters) ≈ +0.7% (±25.8%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5467 ms (36 iters) ✅ 5455 ms (36 iters) ≈ -0.2% (±24.8%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2772 ms (68 iters) ✅ 2687 ms (70 iters) 🟢 -3.1% — / —
fj-kmeans 25 ✅ 2810 ms (66 iters) ✅ 2801 ms (66 iters) ≈ -0.3% (±2.6%) — / —
future-genetic 21 ✅ 2087 ms (88 iters) ✅ 2055 ms (90 iters) ≈ -1.5% (±2.7%) — / —
future-genetic 25 ✅ 2088 ms (89 iters) ✅ 2046 ms (91 iters) ≈ -2% (±2.5%) — / —
naive-bayes 21 ✅ 1233 ms (139 iters) ✅ 1257 ms (136 iters) ≈ +1.9% (±32.9%) — / —
naive-bayes 25 💥 0 ms (0 iters) 💥 0 ms (0 iters) — / —
reactors 21 ✅ 16947 ms (15 iters) ✅ 15936 ms (15 iters) ≈ -6% (±8.2%) — / —
reactors 25 ✅ 18681 ms (15 iters) ✅ 18731 ms (15 iters) ≈ +0.3% (±5.3%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 1 / 1 2008 / 2140 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 1 / 1 2058 / 2262 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 3 / 3 8437 / 8783 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 3 8064 / 8228 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 6 / ✅ 1275 / 1291 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ ✅ / 2 1250 / 1261 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ ✅ / 2 2876 / 3036 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / ✅ 2912 / 2869 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 2 / 4 3492 / 3491 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1892 / 1633 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 2 / 2 1914 / 1937 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 2033006)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125072159 Commit: 20330063bf66bf581cf9ae47f94cff709e5aa6be

⚠️ Significant outliers

  • 🟢 fj-kmeans (JDK 25): runtime -3.7% (2848→2743 ms)
  • 🔴 future-genetic (JDK 25): runtime +3.2% (1994→2057 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10363 ms (21 iters) ✅ 10147 ms (21 iters) ≈ -2.1% (±11.6%) — / —
akka-uct 25 ✅ 8966 ms (24 iters) ✅ 8873 ms (24 iters) ≈ -1% (±10.2%) — / —
finagle-chirper 21 ✅ 6007 ms (33 iters) ✅ 5986 ms (33 iters) ≈ -0.3% (±26.1%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5488 ms (36 iters) ✅ 5433 ms (36 iters) ≈ -1% (±24.3%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2726 ms (68 iters) ✅ 2760 ms (68 iters) ≈ +1.2% (±2.6%) — / —
fj-kmeans 25 ✅ 2848 ms (66 iters) ✅ 2743 ms (69 iters) 🟢 -3.7% — / —
future-genetic 21 ✅ 2062 ms (90 iters) ✅ 2047 ms (90 iters) ≈ -0.7% (±2.6%) — / —
future-genetic 25 ✅ 1994 ms (93 iters) ✅ 2057 ms (90 iters) 🔴 +3.2% — / —
naive-bayes 21 ✅ 1269 ms (135 iters) ✅ 1258 ms (136 iters) ≈ -0.9% (±32.5%) — / —
naive-bayes 25 ✅ 1012 ms (169 iters) ✅ 1016 ms (168 iters) ≈ +0.4% (±31.6%) — / —
reactors 21 ✅ 16554 ms (15 iters) ✅ 16630 ms (15 iters) ≈ +0.5% (±8.5%) — / —
reactors 25 ✅ 18407 ms (15 iters) ✅ 18308 ms (15 iters) ≈ -0.5% (±4.1%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 1 / 2 2101 / 2026 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 4 / 3 2336 / 2365 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 4 / 3 8772 / 8497 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 1 8397 / 8470 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 3 / 1 1260 / 1274 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 1 / 3 1303 / 1307 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / 2 2893 / 2882 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 1 / 2 2958 / 2912 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ ✅ / 1 3540 / 3527 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 5 / 6 3458 / 3485 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 2 / 1 1736 / 1754 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1990 / 1872 ✅ / ✅ ✅ / ✅

GC-root reconstruction for live-heap allocation samples: track surviving
objects, walk the reference graph back to a GC root, and emit the resulting
chain via JFR. Core support spans the liveness tracker, object sampler,
flight recorder, and JVMTI-facing safe-memory-access helpers used while
walking potentially-stale references.
Gtest coverage for the liveness tracker, reference-chain BFS
reconstruction, JFR round-trip encoding, and the line-number-table
copy bounds guard (catches an off-by-one mutation at the
MAX_LINE_NUMBER_TABLE_ENTRIES boundary).
In-process and external-process end-to-end coverage: JFR event parsing,
a leaking-cache scenario that grows a live population over time, and
test seams that isolate frontier/BFS state across shared-JVM test runs.
Also fixes a stale-jmethodID stress test's javadoc to correctly attribute
which SafeAccess guard rejects which access.
Wire reference-chain tests into the test workflow and fix quoting/allocator
existence-check gaps in run-chaos-harness.sh surfaced by review.
@jbachorik
jbachorik force-pushed the jb/reference-chains branch from 2033006 to 02a0a93 Compare July 17, 2026 11:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant