From 37262c516d3182d3afb2d269a795c4311d198d1a Mon Sep 17 00:00:00 2001
From: Douglas Q Hawkins <douglas.hawkins@datadoghq.com>
Date: Wed, 15 Jul 2026 06:30:04 -0400
Subject: [PATCH 1/2] Add self-serve PetClinic macro-benchmark harness

One-command PetClinic macro benchmark extending benchmark-integration: fetch + build
PetClinic (pinned commit), run it under the agent across version and heap sweeps with a
wrk load, capture JFR, and analyze into throughput plus the tracer-overhead split
(foreground/background x instrumentation/core, CPU + sampled allocation).

Detects a local Datadog Agent on the (configurable) APM port and picks the writer
accordingly -- agent-reachable = full pipeline incl. the background serializer,
unreachable = the agent-down regime -- with an EXPECT_AGENT guard so an unattended
sweep fails fast rather than silently measuring the wrong regime.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 dd-java-agent/benchmark-integration/README.md |  73 +++++++++
 .../benchmark-integration/analyze-jfr.sh      | 112 ++++++++++++++
 .../benchmark-integration/fetch-petclinic.sh  |  53 +++++++
 .../perf-test-petclinic-settings.rc           |  27 ++++
 .../benchmark-integration/run-perf-test.sh    | 128 +++++++++++++---
 .../benchmark-integration/run-petclinic.sh    | 145 ++++++++++++++++++
 6 files changed, 513 insertions(+), 25 deletions(-)
 create mode 100755 dd-java-agent/benchmark-integration/analyze-jfr.sh
 create mode 100755 dd-java-agent/benchmark-integration/fetch-petclinic.sh
 create mode 100644 dd-java-agent/benchmark-integration/perf-test-petclinic-settings.rc
 create mode 100755 dd-java-agent/benchmark-integration/run-petclinic.sh

diff --git a/dd-java-agent/benchmark-integration/README.md b/dd-java-agent/benchmark-integration/README.md
index 00b1c6b7ec9..6a474276f4d 100644
--- a/dd-java-agent/benchmark-integration/README.md
+++ b/dd-java-agent/benchmark-integration/README.md
@@ -29,3 +29,76 @@ cp /tmp/perf_results.csv ~/somewhere_else/
 /usr/local/bin/bash ./run-perf-test.sh play-zip play-perftest/build/distributions/main-*.zip NoAgent ~/Downloads/dd-java-agent-0.18.0.jar ~/Downloads/dd-java-agent-0.19.0.jar
 cp /tmp/perf_results.csv ~/somewhere_else/
 ```
+
+## PetClinic macro benchmark (self-serve)
+
+`run-petclinic.sh` is the one-command version of the above against **Spring PetClinic** — a real
+Spring MVC app that exercises a broad slice of the agent's instrumentation. It downloads and builds
+PetClinic, runs it under load with the agent, captures a JFR per variant, and prints **throughput**
+plus the **tracer-overhead split** (CPU + sampled allocation, bucketed foreground/background and
+instrumentation/core). The goal is that any developer — not just perf specialists — can reproduce the
+macro measurement that keeps micro benchmarks honest.
+
+This is the fast, **local, dev-iteration** tool. It complements, and does not replace, the org's
+automated benchmarking platform (the GitLab `apm-sdks-benchmarks-java-slo` track).
+
+### Extra dependencies
+On top of `wrk` and `nc`: `git`, a JDK (provides the `jfr` CLI), and `curl`. Maven is supplied by
+PetClinic's own wrapper. First run needs network access (clone + build + optional release download).
+
+### Trace destination — what actually gets measured
+The harness auto-detects a Datadog Agent on `localhost:8126` and picks the writer accordingly. This
+is a deliberate knob — it lets one tool measure two regimes:
+- **Agent reachable** (`DDAgentWriter`) — the *full pipeline*: span work on the request threads **and**
+  the background serializer (`TraceMapperV0_4`, msgpack, cache recalibration). Run your **local
+  Datadog Agent** on `:8126` for this; the analyzer's `background/*` buckets then reflect real
+  serialization cost.
+- **Agent unreachable** (`LoggingWriter`) — traces are discarded, so you measure the **agent-down
+  regime**: foreground overhead only, no serializer. Useful on its own for understanding behavior
+  when the Agent is missing.
+
+Pick intentionally based on what you want to see, and keep it consistent across a sweep. The harness
+prints a `>> REGIME:` banner so the choice is never silent, and re-checks before every variant. Set
+`EXPECT_AGENT=up` (or `down`) to make it **fail fast** if reality differs — e.g. a full-pipeline sweep
+aborts immediately if the Agent isn't up, and aborts mid-sweep if it dies, instead of quietly
+degrading to `LoggingWriter`. Default is `any` (auto-detect, no assertion).
+
+### Usage
+```
+# NoAgent vs the current checkout:
+./run-petclinic.sh
+
+# A version sweep for the historic curve — each token is a jar path, `current`
+# (builds this checkout's shadow jar), or a release version (downloaded from Maven Central):
+./run-petclinic.sh 1.54.0 1.58.0 1.62.0 current
+```
+Artifacts (throughput CSV + per-variant JFRs) are copied to `build/petclinic-results/results-<ts>/`.
+Analyze any recording on its own with `./analyze-jfr.sh <recording.jfr>`.
+
+#### Heap regime (ample vs tight)
+Server heap is a fixed `Xms=Xmx` knob (default 256m) — the regime where allocation overhead turns
+into throughput cost. Set one heap with `server_heap=` in the `.rc` (or `PERF_HEAP=`), or **sweep**
+several by passing a list — the full variant sweep runs once per heap:
+```
+PETCLINIC_HEAPS="64m 256m 512m" ./run-petclinic.sh 1.54.0 1.58.0 current
+```
+Results land in per-heap subdirs (`heap-64m/`, …) plus a combined `throughput-grid.csv` (leading
+`Heap` column). **Runtime is heaps × variants** — a heap list multiplies the wall clock, so size the
+version list accordingly.
+
+PetClinic is pinned to a specific commit in `fetch-petclinic.sh` (bump deliberately); JFR is opt-in in
+`run-perf-test.sh` via `PERF_JFR=1`, and endpoints/load live in `perf-test-petclinic-settings.rc`.
+
+### Reading the results honestly
+- **Allocation is the anchor; throughput is directional.** Allocation trends are stable across runs;
+  throughput on a shared dev box is noisy and swings with the CPU-contention regime — read it as a
+  direction, not a precise number.
+- Allocation here is **TLAB-sampled** (via JFR) — authoritative for *ratios* (which thread, which
+  package), with only an *estimated* total. That's sufficient for a macro trend.
+- Run on an otherwise-**quiet machine**, and respect the warmup (the harness warms each endpoint
+  before measuring). Results depend on the machine's core count / heap regime.
+
+### Known limitations (follow-ups)
+- No containerization yet — no pinned cores/heap, so cross-machine comparisons are approximate.
+- `run-perf-test.sh` is still lightly macOS-flavored (`/usr/bin/time`, `lsof`, `nc`); there is a basic
+  Linux guard but not full cross-platform hardening.
diff --git a/dd-java-agent/benchmark-integration/analyze-jfr.sh b/dd-java-agent/benchmark-integration/analyze-jfr.sh
new file mode 100755
index 00000000000..602f49f319e
--- /dev/null
+++ b/dd-java-agent/benchmark-integration/analyze-jfr.sh
@@ -0,0 +1,112 @@
+#!/usr/bin/env bash
+#
+# Analyze a perf-test JFR recording and print the tracer-overhead split that our methodology cares
+# about: self-time (and sampled allocation) bucketed by thread role (foreground request threads vs
+# background agent threads) and by code owner (instrumentation vs tracer-core vs app/JDK).
+#
+# This encodes the analysis so the numbers are trustworthy to someone running it blindly -- the tool
+# emits the meaningful split, not a raw JFR to hand-parse.
+#
+# Usage:   ./analyze-jfr.sh /tmp/perf_<label>.jfr
+# Requires: the JDK 'jfr' CLI (ships with the JDK) and awk.
+
+set -euo pipefail
+
+jfr_file="${1:-}"
+if [ -z "$jfr_file" ] || [ ! -f "$jfr_file" ]; then
+  echo "usage: ./analyze-jfr.sh <recording.jfr>" >&2
+  exit 1
+fi
+# The jfr CLI ships with JDK 11+. If it is not already on PATH, try the JDK env vars.
+if ! command -v jfr >/dev/null 2>&1; then
+  for jh in "${BENCH_JAVA_HOME:-}" "${JAVA_17_HOME:-}" "${JAVA_21_HOME:-}" "${JAVA_HOME:-}"; do
+    if [ -n "$jh" ] && [ -x "$jh/bin/jfr" ]; then PATH="$jh/bin:$PATH"; break; fi
+  done
+fi
+if ! command -v jfr >/dev/null 2>&1; then
+  echo "ERROR: the 'jfr' CLI was not found (ships with JDK 11+); set BENCH_JAVA_HOME to a JDK 17+." >&2
+  exit 1
+fi
+
+# Shared awk classifier. Reads a stream of "THREAD<TAB>TOPFRAME" lines and prints the bucketed
+# breakdown. Foreground = servlet-container request threads; background = agent + JFR/GC threads.
+# Owner: instrumentation (datadog.trace.instrumentation.*) vs core (other datadog.*) vs app/other.
+classify_awk='
+function role(t) {
+  if (t ~ /^http-nio/ || t ~ /^http-bio/ || t ~ /exec-/ || t ~ /^tomcat/ || t ~ /^XNIO/ || t ~ /^reactor-http/ || t ~ /^qtp/)
+    return "foreground";
+  if (t ~ /^dd-/ || t ~ /^dd\./ || t ~ /datadog/ || t ~ /^JFR/ || t ~ /Flight Recorder/ || t ~ /^GC / || t ~ /G1 / || t ~ /Reference Handl/)
+    return "background";
+  return "other";
+}
+function owner(f) {
+  if (f ~ /^datadog\.trace\.instrumentation\./) return "instrumentation";
+  if (f ~ /^datadog\./) return "core";
+  return "app";
+}
+{
+  total++;
+  r = role($1); o = owner($2);
+  role_tot[r]++;
+  if (o == "instrumentation" || o == "core") {
+    tracer_tot++;
+    split_ct[r SUBSEP o]++;
+  }
+}
+END {
+  printf "  samples total: %d   tracer (instrumentation+core): %d (%.1f%% of all)\n",
+    total, tracer_tot, total ? 100.0*tracer_tot/total : 0;
+  printf "  by thread role:  foreground=%d  background=%d  other=%d\n",
+    role_tot["foreground"]+0, role_tot["background"]+0, role_tot["other"]+0;
+  printf "\n  tracer self-%s split (share of tracer, and of all samples):\n", METRIC;
+  printf "    %-28s %10s %10s\n", "bucket", "of-tracer", "of-all";
+  emit(tracer_tot, total, "foreground", "instrumentation");
+  emit(tracer_tot, total, "foreground", "core");
+  emit(tracer_tot, total, "background", "instrumentation");
+  emit(tracer_tot, total, "background", "core");
+  emit(tracer_tot, total, "other",      "instrumentation");
+  emit(tracer_tot, total, "other",      "core");
+}
+function emit(tt, all, r, o,   c) {
+  c = split_ct[r SUBSEP o] + 0;
+  if (c == 0) return;
+  printf "    %-28s %9.1f%% %9.1f%%\n", r"/"o, tt ? 100.0*c/tt : 0, all ? 100.0*c/all : 0;
+}
+'
+
+# jfr print emits an event block per sample; the first stackTrace frame line after the event is the
+# leaf. We reduce each ExecutionSample to "sampledThread<TAB>leafFrame". The sampledThread field and
+# the stackTrace appear as fields; parse defensively across JDK jfr output formatting.
+extract_stream() {
+  local event="$1"
+  jfr print --events "$event" "$jfr_file" 2>/dev/null | awk '
+    /sampledThread = / { thr=$0; sub(/.*sampledThread = /,"",thr); sub(/ .*/,"",thr); gsub(/"/,"",thr); next }
+    /eventThread = /   { if (thr=="") { thr=$0; sub(/.*eventThread = /,"",thr); sub(/ .*/,"",thr); gsub(/"/,"",thr) } next }
+    /stackTrace = \[/  { want=1; next }
+    want==1 {
+      leaf=$1; sub(/\(.*/,"",leaf);           # strip "(params)"
+      gsub(/^[ \t]+/,"",leaf);
+      if (leaf != "" && leaf != "...") { print thr "\t" leaf }
+      want=0; thr="";
+    }
+  '
+}
+
+echo "== JFR analysis: $jfr_file =="
+echo
+echo "[CPU] jdk.ExecutionSample"
+cpu_stream=$(extract_stream jdk.ExecutionSample)
+if [ -n "$cpu_stream" ]; then
+  echo "$cpu_stream" | awk -F'\t' -v METRIC="cpu" "$classify_awk"
+else
+  echo "  (no ExecutionSample events found)"
+fi
+
+echo
+echo "[ALLOC] jdk.ObjectAllocationSample (TLAB-sampled -- ratios authoritative, total estimated)"
+alloc_stream=$(extract_stream jdk.ObjectAllocationSample)
+if [ -n "$alloc_stream" ]; then
+  echo "$alloc_stream" | awk -F'\t' -v METRIC="alloc" "$classify_awk"
+else
+  echo "  (no ObjectAllocationSample events found)"
+fi
diff --git a/dd-java-agent/benchmark-integration/fetch-petclinic.sh b/dd-java-agent/benchmark-integration/fetch-petclinic.sh
new file mode 100755
index 00000000000..0584402edde
--- /dev/null
+++ b/dd-java-agent/benchmark-integration/fetch-petclinic.sh
@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+#
+# Clone + build Spring PetClinic at a pinned commit, and print the path to the runnable fat jar.
+#
+# Idempotent: if the jar already exists it is reused (no clone, no rebuild). All progress output
+# goes to stderr so that stdout is ONLY the jar path -- callers capture it with:
+#     petclinic_jar=$(./fetch-petclinic.sh)
+#
+# Requires: git, a JDK (for the Maven wrapper), and network access on the first run.
+
+set -euo pipefail
+
+# Pinned so results are reproducible across machines and over time. Bump deliberately.
+PETCLINIC_REPO="https://github.com/spring-projects/spring-petclinic.git"
+PETCLINIC_SHA="51045d1648dad955df586150c1a1a6e22ef400c2"
+
+script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# build/ is already git-ignored by the module, so the checkout is never committed.
+src_dir="$script_dir/build/spring-petclinic"
+
+log() { echo "[fetch-petclinic] $*" >&2; }
+
+# Reuse an existing build if present.
+existing_jar=$(ls "$src_dir"/target/spring-petclinic-*.jar 2>/dev/null \
+  | grep -v -- '-sources.jar$' | grep -v -- '.original$' | head -1 || true)
+if [ -n "$existing_jar" ]; then
+  log "reusing existing jar: $existing_jar"
+  echo "$existing_jar"
+  exit 0
+fi
+
+# Fetch exactly the pinned commit (shallow) -- avoids pulling the whole history.
+if [ ! -d "$src_dir/.git" ]; then
+  log "cloning $PETCLINIC_REPO @ $PETCLINIC_SHA"
+  mkdir -p "$src_dir"
+  git -C "$src_dir" init -q
+  git -C "$src_dir" remote add origin "$PETCLINIC_REPO" 2>/dev/null || true
+fi
+log "fetching pinned commit"
+git -C "$src_dir" fetch -q --depth 1 origin "$PETCLINIC_SHA"
+git -C "$src_dir" checkout -q FETCH_HEAD
+
+log "building PetClinic (this can take a few minutes on first run)"
+( cd "$src_dir" && ./mvnw -q -B -DskipTests package )
+
+jar=$(ls "$src_dir"/target/spring-petclinic-*.jar 2>/dev/null \
+  | grep -v -- '-sources.jar$' | grep -v -- '.original$' | head -1 || true)
+if [ -z "$jar" ]; then
+  log "ERROR: build finished but no runnable jar found under $src_dir/target"
+  exit 1
+fi
+log "built jar: $jar"
+echo "$jar"
diff --git a/dd-java-agent/benchmark-integration/perf-test-petclinic-settings.rc b/dd-java-agent/benchmark-integration/perf-test-petclinic-settings.rc
new file mode 100644
index 00000000000..81d59cdaeb6
--- /dev/null
+++ b/dd-java-agent/benchmark-integration/perf-test-petclinic-settings.rc
@@ -0,0 +1,27 @@
+# Settings for running the perf test against Spring PetClinic.
+# Sourced by run-perf-test.sh (copy to perf-test-settings.rc, or point PERF_SETTINGS at it).
+
+# wrk settings
+test_warmup_seconds=60     # reach C2 steady state before measuring
+test_time_seconds=120      # stable throughput + ample JFR samples per endpoint
+test_num_connections=8
+test_num_threads=4
+
+# Quiet gap between version runs, so thermals/GC/OS settle and one run's tail does not
+# bleed into the next JVM. Only honored by run-perf-test.sh when set (>0).
+test_cooldown_seconds=60
+
+# Server heap, fixed Xms=Xmx (default 256m). A single run's regime. For a heap SWEEP use
+# run-petclinic.sh with PETCLINIC_HEAPS="64m 256m 512m"; PERF_HEAP overrides this per run.
+server_heap=256m
+
+# PetClinic endpoints -- a read-heavy mix spanning view rendering, a form, a DB-backed query,
+# a detail page, and the error path. PetClinic's default profile is in-memory H2 (no external DB).
+declare -A endpoints
+endpoints['home']='http://localhost:8080/'
+endpoints['vets']='http://localhost:8080/vets.html'
+endpoints['find-owners']='http://localhost:8080/owners/find'
+endpoints['owners-query']='http://localhost:8080/owners?lastName='
+endpoints['owner-detail']='http://localhost:8080/owners/1'
+endpoints['error']='http://localhost:8080/oups'
+test_order=( 'home' 'vets' 'find-owners' 'owners-query' 'owner-detail' 'error' )
diff --git a/dd-java-agent/benchmark-integration/run-perf-test.sh b/dd-java-agent/benchmark-integration/run-perf-test.sh
index 46dc8813ca8..0fe9dbe087f 100755
--- a/dd-java-agent/benchmark-integration/run-perf-test.sh
+++ b/dd-java-agent/benchmark-integration/run-perf-test.sh
@@ -7,7 +7,8 @@ server_type=$1
 server_package=$2
 agent_jars="${@:3}"
 server_pid=""
-agent_pid=$(lsof -i tcp:8126 | awk '$8 == "TCP" { print $2 }')
+agent_pid=""
+agent_state=""
 if [[ "$server_package" = "" ]] || [[ "$server_type" != "play-zip" && "$server_type" != "jar" ]]; then
     echo "usage: ./run-perf-test.sh [play-zip|jar] path-to-server-package path-to-agent1 path-to-agent2..."
     echo ""
@@ -23,40 +24,102 @@ if [[ "$server_package" = "" ]] || [[ "$server_type" != "play-zip" && "$server_t
     exit 1
 fi
 
-if [ "$agent_pid" = "" ]; then
-    echo "discarding traces"
-    writer_type="LoggingWriter"
-else
-    echo "sending traces to local trace agent: $agent_pid"
-    writer_type="DDAgentWriter"
-fi
+# Datadog Agent detection drives the writer, and therefore WHICH REGIME we measure:
+#   agent up   -> DDAgentWriter  -> full pipeline (request-thread work + background serializer)
+#   agent down -> LoggingWriter  -> agent-unreachable regime (foreground only, traces discarded)
+# check_agent refreshes this; it is called before every variant so a mid-run agent death is not silent.
+function check_agent {
+    # Probe the APM receiver rather than lsof: the Datadog Agent runs as user _dd-agent, so lsof (run
+    # as us) can't see its socket. Port is TRACE_AGENT_PORT (default 8126). agent_pid stays empty --
+    # we don't attribute the system agent's CPU here; the JFR is the CPU signal.
+    if nc -z localhost "${TRACE_AGENT_PORT:-8126}" >/dev/null 2>&1; then
+        agent_state="up"; writer_type="DDAgentWriter"
+    else
+        agent_state="down"; writer_type="LoggingWriter"
+    fi
+    agent_pid=""
+}
+function regime_banner {
+    local port="${TRACE_AGENT_PORT:-8126}"
+    if [ "$agent_state" = "up" ]; then
+        echo ">> REGIME: full pipeline -- Datadog Agent reachable on :$port, DDAgentWriter"
+    else
+        echo ">> REGIME: agent-unreachable -- nothing on :$port, LoggingWriter (traces discarded)"
+    fi
+}
+# EXPECT_AGENT=up|down|any (default any). When up/down, abort if reality differs -- guards an
+# unattended sweep from silently measuring the wrong regime.
+function assert_expected_agent {
+    local expect="${EXPECT_AGENT:-any}"
+    if [ "$expect" != "any" ] && [ "$expect" != "$agent_state" ]; then
+        echo "ERROR: EXPECT_AGENT=$expect but the Datadog Agent on :8126 is '$agent_state'." >&2
+        regime_banner >&2
+        echo "Aborting to avoid measuring the wrong regime (set EXPECT_AGENT=any to override)." >&2
+        [ -n "$server_pid" ] && kill "$server_pid" 2>/dev/null
+        exit 3
+    fi
+}
 
-if [ -f perf-test-settings.rc ]; then
-    echo "loading custom settings"
-    cat ./perf-test-settings.rc
-    . ./perf-test-settings.rc
-else
-    echo "loading default settings"
-    cat ./perf-test-default-settings.rc
-    . ./perf-test-default-settings.rc
+check_agent
+regime_banner
+assert_expected_agent
+
+# Settings file precedence: $PERF_SETTINGS env override, then a local perf-test-settings.rc,
+# then the checked-in defaults. (run-petclinic.sh points PERF_SETTINGS at the PetClinic rc.)
+settings_file="${PERF_SETTINGS:-}"
+if [ -z "$settings_file" ]; then
+    if [ -f perf-test-settings.rc ]; then
+        settings_file="perf-test-settings.rc"
+    else
+        settings_file="perf-test-default-settings.rc"
+    fi
 fi
+echo "loading settings from $settings_file"
+cat "$settings_file"
+. "$settings_file"
+
+# Server heap. Fixed (Xms=Xmx) so GC behavior is stable and comparable across a sweep. Override via
+# the settings file (server_heap=...) or the PERF_HEAP env var; defaults to 256m. Sweep the same
+# versions at different heaps to expose the ample-vs-tight-heap regime.
+server_heap="${PERF_HEAP:-${server_heap:-256m}}"
+echo "server heap: -Xms$server_heap -Xmx$server_heap"
 echo ""
 echo ""
 
+# /usr/bin/time differs by platform: macOS (BSD) uses -l and prints RSS in bytes as a leading
+# number; GNU/Linux uses -v and prints "Maximum resident set size (kbytes): N". Detect which.
+if /usr/bin/time -l true >/dev/null 2>&1; then
+    time_flag="-l"
+else
+    time_flag="-v"
+fi
+
 unzipped_server_path=""
 
 # Start up server passed into the script
 # Blocks until server is bound to local port 8080
 function start_server {
     agent_jar="$1"
+    variant_label="${2:-run}"
     javaagent_arg=""
     if [ "$agent_jar" != "" -a -f "$agent_jar" ]; then
-        javaagent_arg="-javaagent:$agent_jar -Ddatadog.slf4j.simpleLogger.defaultLogLevel=off -Ddd.writer.type=$writer_type -Ddd.service.name=perf-test-app"
+        javaagent_arg="-javaagent:$agent_jar -Ddatadog.slf4j.simpleLogger.defaultLogLevel=off -Ddd.writer.type=$writer_type -Ddd.trace.agent.port=${TRACE_AGENT_PORT:-8126} -Ddd.service.name=perf-test-app"
     fi
 
+    # Opt-in JFR (PERF_JFR=1): one recording per variant, named by label so runs don't collide.
+    # settings=profile captures jdk.ExecutionSample + jdk.ObjectAllocationSample for analyze-jfr.sh.
+    jfr_arg=""
+    if [ "${PERF_JFR:-}" = "1" ]; then
+        jfr_file="/tmp/perf_${variant_label}.jfr"
+        rm -f "$jfr_file"
+        jfr_arg="-XX:StartFlightRecording=settings=profile,filename=$jfr_file,dumponexit=true"
+        echo "JFR enabled -> $jfr_file"
+    fi
+    run_java_args="$javaagent_arg $jfr_arg -Xms$server_heap -Xmx$server_heap"
+
     if [ "$server_type" = "jar" ]; then
-      echo "starting server: java $javaagent_arg -jar $server_package"
-      { /usr/bin/time -l java $javaagent_arg -Xms256m -Xmx256m -jar $server_package ; } 2> $server_output  &
+      echo "starting server: java $run_java_args -jar $server_package"
+      { /usr/bin/time $time_flag java $run_java_args -jar $server_package ; } 2> $server_output  &
     else
       # make a temp directory to hold the unzipped server
       unzip_temp=`mktemp -d`
@@ -75,21 +138,21 @@ function start_server {
       # unzipped server location, will be removed when the server is stopped
       unzipped_server_path=${unzip_temp}
 
-      java_opts_env='JAVA_OPTS="'${javaagent_arg}'"'
+      java_opts_env='JAVA_OPTS="'${run_java_args}'"'
       # it appears the binary script will always be named main at the time of writing
       # no matter what the zip file is named.
       play_script=${unzipped_server_path}/${unzipped_dirname}/bin/main
 
       # have to use env to set JAVA_OPTS because of a gradle play plugin bug:
       # https://github.com/gradle/gradle/issues/4471
-      if [ "$agent_jar" != "" -a -f "$agent_jar" ]; then
-        utility_cmd="env JAVA_OPTS=${javaagent_arg} ${play_script}"
+      if [ -n "$(echo $run_java_args | tr -d ' ')" ]; then
+        utility_cmd="env JAVA_OPTS=${run_java_args} ${play_script}"
       else
         utility_cmd="${play_script}"
       fi
 
       echo "starting server: ${utility_cmd}"
-      { /usr/bin/time -l ${utility_cmd} ; } 2> $server_output &
+      { /usr/bin/time $time_flag ${utility_cmd} ; } 2> $server_output &
     fi
 
     # Block until server is up
@@ -139,14 +202,29 @@ header="$header,Agent CPU Burn,Server CPU Burn,Agent RSS Delta,Server Max RSS,Se
 echo $header > $test_csv_file
 
 for agent_jar in $agent_jars; do
+    # Quiet cooldown between variants (skip before the first). Off unless the settings set it.
+    if [ "${_did_first_variant:-}" = "1" ] && [ "${test_cooldown_seconds:-0}" -gt 0 ] 2>/dev/null; then
+        echo "cooldown ${test_cooldown_seconds}s before next variant..."
+        sleep "$test_cooldown_seconds"
+    fi
+    _did_first_variant=1
+    # Re-detect the agent before each variant so a mid-sweep agent death aborts (with EXPECT_AGENT)
+    # rather than silently degrading later runs to LoggingWriter.
+    check_agent
+    assert_expected_agent
     echo "----Testing agent $agent_jar----"
+    regime_banner
     if [ "$agent_jar" == "NoAgent" ]; then
         result_row="NoAgent"
-        start_server ""
+        variant_label="noagent"
+        start_server "" "$variant_label"
     else
         agent_version=$(java -jar $agent_jar 2>/dev/null)
         result_row="$agent_version"
-        start_server $agent_jar
+        # sanitize the version into a filesystem-safe JFR label; fall back to the jar basename.
+        variant_label=$(echo "$agent_version" | tr -c 'A-Za-z0-9._-' '_' | sed 's/_*$//')
+        [ -z "$variant_label" ] && variant_label=$(basename "$agent_jar" .jar)
+        start_server "$agent_jar" "$variant_label"
     fi
 
 
diff --git a/dd-java-agent/benchmark-integration/run-petclinic.sh b/dd-java-agent/benchmark-integration/run-petclinic.sh
new file mode 100755
index 00000000000..f50ae46d94c
--- /dev/null
+++ b/dd-java-agent/benchmark-integration/run-petclinic.sh
@@ -0,0 +1,145 @@
+#!/usr/bin/env bash
+#
+# One command to download, run, and analyze Spring PetClinic under the Datadog Java agent -- the
+# self-serve macro benchmark. Sweeps one or more agent variants against a NoAgent baseline, captures
+# a JFR per variant, and prints throughput + the tracer-overhead split (CPU + sampled allocation).
+#
+# Usage:
+#   ./run-petclinic.sh [variant ...]
+#
+# Each variant resolves to an agent jar:
+#   <path/to/agent.jar>   use that jar as-is
+#   current               build the current checkout's shadow jar
+#   <version> (e.g. 1.62.0)   download that release from Maven Central
+#
+# With no variants it defaults to `current`. A NoAgent baseline is always run first.
+# Examples:
+#   ./run-petclinic.sh                       # NoAgent vs the current branch
+#   ./run-petclinic.sh 1.54.0 1.58.0 current # a version sweep for the historic curve
+#
+# Requires: git, a JDK (jfr CLI + Maven wrapper), wrk, nc, curl.
+
+set -euo pipefail
+
+script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+repo_root="$(cd "$script_dir/../.." && pwd)"
+agents_dir="$script_dir/build/agents"
+mkdir -p "$agents_dir"
+
+log() { echo "[run-petclinic] $*" >&2; }
+
+# PetClinic (Spring Boot 3) requires JDK 17+, and the jfr CLI requires 11+. Default to JAVA_17_HOME
+# (override with BENCH_JAVA_HOME) and put it first on PATH so java / jfr / mvnw are all consistent.
+# All variants in a sweep run under this one JDK -- keep it fixed for comparability.
+bench_java_home="${BENCH_JAVA_HOME:-${JAVA_17_HOME:-}}"
+if [ -z "$bench_java_home" ] || [ ! -x "$bench_java_home/bin/java" ] || [ ! -x "$bench_java_home/bin/jfr" ]; then
+  echo "[run-petclinic] ERROR: need a JDK 17+ with the jfr CLI." >&2
+  echo "[run-petclinic]   set BENCH_JAVA_HOME (JAVA_17_HOME is currently '${JAVA_17_HOME:-unset}')" >&2
+  exit 1
+fi
+export JAVA_HOME="$bench_java_home"
+export PATH="$JAVA_HOME/bin:$PATH"
+log "using JDK: $(java -version 2>&1 | head -1)  ($JAVA_HOME)"
+
+# Resolve a variant token to a jar path (echoes the path). "NoAgent" passes through as a baseline
+# marker that run-perf-test.sh understands.
+resolve_variant() {
+  local v="$1"
+  if [ "$v" = "NoAgent" ]; then
+    echo "NoAgent"; return
+  fi
+  if [ -f "$v" ]; then
+    echo "$v"; return
+  fi
+  if [ "$v" = "current" ]; then
+    log "building current shadow jar (:dd-java-agent:shadowJar)"
+    # Redirect the build's stdout to stderr: resolve_variant's stdout is captured as the jar path by
+    # the caller, so any gradle output on stdout would corrupt it (that broke the first sweep).
+    ( cd "$repo_root" && ./gradlew -q :dd-java-agent:shadowJar >&2 )
+    local jar
+    jar=$(ls -t "$repo_root"/dd-java-agent/build/libs/dd-java-agent-*.jar 2>/dev/null \
+      | grep -v -- '-sources' | grep -v -- '-javadoc' | head -1 || true)
+    [ -n "$jar" ] || { log "ERROR: shadow jar not found after build"; exit 1; }
+    echo "$jar"; return
+  fi
+  # Otherwise treat as a released version -> download from Maven Central.
+  local dest="$agents_dir/dd-java-agent-$v.jar"
+  if [ ! -f "$dest" ]; then
+    local url="https://repo1.maven.org/maven2/com/datadoghq/dd-java-agent/$v/dd-java-agent-$v.jar"
+    log "downloading dd-java-agent $v from Maven Central"
+    curl -fsSL "$url" -o "$dest" \
+      || { log "ERROR: could not download version '$v' ($url)"; rm -f "$dest"; exit 1; }
+  fi
+  echo "$dest"
+}
+
+variants=("$@")
+[ ${#variants[@]} -eq 0 ] && variants=("current")
+# Always lead with a NoAgent baseline unless the caller already did.
+if [ "${variants[0]}" != "NoAgent" ]; then
+  variants=("NoAgent" "${variants[@]}")
+fi
+
+log "resolving ${#variants[@]} variant(s): ${variants[*]}"
+resolved=()
+for v in "${variants[@]}"; do
+  resolved+=("$(resolve_variant "$v")")
+done
+
+log "fetching + building PetClinic"
+petclinic_jar="$(bash "$script_dir/fetch-petclinic.sh")"
+log "PetClinic jar: $petclinic_jar"
+
+# Heap-sweep dimension: run the full variant sweep once per heap size. PETCLINIC_HEAPS is a
+# space-separated list (e.g. "64m 256m 512m"); default is a single 256m (no heap sweep).
+# NOTE: total runtime is heaps x variants -- a heap list multiplies the wall clock.
+heaps=(${PETCLINIC_HEAPS:-256m})
+
+# Honor a pre-set PERF_SETTINGS (e.g. a smoke-test rc); otherwise use the PetClinic settings.
+settings="${PERF_SETTINGS:-$script_dir/perf-test-petclinic-settings.rc}"
+
+results_dir="$script_dir/build/petclinic-results/results-$(date +%Y%m%d-%H%M%S)"
+mkdir -p "$results_dir"
+combined_csv="$results_dir/throughput-grid.csv"
+
+log "sweeping ${#resolved[@]} variant(s) x ${#heaps[@]} heap(s) [${heaps[*]}], JFR enabled, settings=$settings"
+for heap in "${heaps[@]}"; do
+  heap_dir="$results_dir/heap-$heap"
+  mkdir -p "$heap_dir"
+  rm -f /tmp/perf_*.jfr /tmp/perf_results.csv
+  log "=== HEAP $heap ==="
+  (
+    cd "$script_dir"
+    PERF_JFR=1 PERF_HEAP="$heap" PERF_SETTINGS="$settings" \
+      bash ./run-perf-test.sh jar "$petclinic_jar" "${resolved[@]}"
+  )
+  cp -f /tmp/perf_results.csv "$heap_dir/" 2>/dev/null || true
+  cp -f /tmp/perf_*.jfr "$heap_dir/" 2>/dev/null || true
+
+  # Fold this heap's throughput into the combined grid CSV with a leading Heap column.
+  if [ -f /tmp/perf_results.csv ]; then
+    [ -f "$combined_csv" ] || echo "Heap,$(head -1 /tmp/perf_results.csv)" > "$combined_csv"
+    tail -n +2 /tmp/perf_results.csv | sed "s/^/$heap,/" >> "$combined_csv"
+  fi
+
+  echo
+  echo "================ HEAP $heap: THROUGHPUT / LATENCY ================"
+  cat "$heap_dir/perf_results.csv" 2>/dev/null || echo "(no throughput CSV produced)"
+  echo
+  echo "================ HEAP $heap: TRACER-OVERHEAD SPLIT ================"
+  for jfr in "$heap_dir"/perf_*.jfr; do
+    [ -e "$jfr" ] || continue
+    echo
+    echo "---- $(basename "$jfr") ----"
+    bash "$script_dir/analyze-jfr.sh" "$jfr" || true
+  done
+done
+
+if [ ${#heaps[@]} -gt 1 ]; then
+  echo
+  echo "================ COMBINED THROUGHPUT GRID (heap x version) ================"
+  cat "$combined_csv" 2>/dev/null || true
+fi
+
+echo
+log "artifacts saved to: $results_dir"

From d3bbc4ea4523a33e3fb1efb9735878428e9ecf32 Mon Sep 17 00:00:00 2001
From: Douglas Q Hawkins <douglas.hawkins@datadoghq.com>
Date: Thu, 16 Jul 2026 06:42:52 -0400
Subject: [PATCH 2/2] Gate wrk on a real HTTP 200 + abort on socket connect
 errors
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The readiness check was TCP-only (`nc -z localhost 8080`), so wrk could start
before the app was actually serving under load. Under the java agent (slower
startup) this produced `connect` socket errors and a bogus ~26x throughput
collapse on the agent runs — a measurement artifact, not tracer overhead (a
direct sustained-load repro shows agent ~= NoAgent on this workload).

- Readiness gate now polls the home endpoint for a real HTTP 200 (curl if
  present, else python3), with a 180s timeout and a brief settle before wrk.
- Add a sanity gate: if wrk reports socket connect errors on any endpoint, the
  server was unhealthy under load — abort loudly rather than record a garbage
  row.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 .../benchmark-integration/run-perf-test.sh    | 23 +++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

diff --git a/dd-java-agent/benchmark-integration/run-perf-test.sh b/dd-java-agent/benchmark-integration/run-perf-test.sh
index 0fe9dbe087f..a5d444c0f06 100755
--- a/dd-java-agent/benchmark-integration/run-perf-test.sh
+++ b/dd-java-agent/benchmark-integration/run-perf-test.sh
@@ -155,10 +155,21 @@ function start_server {
       { /usr/bin/time $time_flag ${utility_cmd} ; } 2> $server_output &
     fi
 
-    # Block until server is up
-    until nc -z localhost 8080; do
+    # Block until the server actually SERVES HTTP 200 -- not just binds the TCP port. The java agent
+    # slows startup, so a TCP-only check (nc -z) let wrk start before the app was serving under
+    # load, producing `connect` socket errors and a bogus ~26x throughput collapse on the agent
+    # runs. Poll the home endpoint for a real 200 (curl if present, else python3), with a timeout.
+    ready_deadline=$((SECONDS + 180))
+    until { command -v curl >/dev/null 2>&1 && curl -sf -o /dev/null http://localhost:8080/ ; } \
+        || python3 -c "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:8080/',timeout=2).getcode()==200 else 1)" >/dev/null 2>&1 ; do
+        if [ "$SECONDS" -ge "$ready_deadline" ]; then
+            echo "ERROR: server did not serve HTTP 200 on / within 180s -- aborting" >&2
+            exit 1
+        fi
         sleep 0.5
     done
+    # Brief settle so first-hit JIT/classloading doesn't skew the wrk warmup.
+    sleep 2
     server_pid=$(lsof -i tcp:8080 | awk '$8 == "TCP" { print $2 }' | uniq)
     echo "server $server_pid started"
 }
@@ -188,6 +199,14 @@ function test_endpoint {
     # run test
     wrk_results=/tmp/wrk_results.`date +%s`
     wrk -c $test_num_connections -t$test_num_threads -d ${test_time_seconds}s $url > $wrk_results
+    # Sanity gate: with the HTTP-200 readiness check above, wrk should never see socket connect
+    # errors. If it does, the app was not healthy under load and the row would be garbage (the
+    # failure mode behind the bogus 26x sweep) -- abort loudly instead of recording it.
+    if grep -qE 'Socket errors:.*connect [1-9]' "$wrk_results"; then
+        echo "ERROR: wrk reported socket connect errors on $url -- server unhealthy under load:" >&2
+        grep -E 'Socket errors|Requests/sec' "$wrk_results" >&2
+        exit 1
+    fi
     echo $wrk_results
 }