Add self-serve PetClinic macro-benchmark harness

dd-java-agent/benchmark-integration/README.md

@@ -29,3 +29,76 @@ cp /tmp/perf_results.csv ~/somewhere_else/
 /usr/local/bin/bash ./run-perf-test.sh play-zip play-perftest/build/distributions/main-*.zip NoAgent ~/Downloads/dd-java-agent-0.18.0.jar ~/Downloads/dd-java-agent-0.19.0.jar
 cp /tmp/perf_results.csv ~/somewhere_else/
 ```
+
+## PetClinic macro benchmark (self-serve)
+
+`run-petclinic.sh` is the one-command version of the above against **Spring PetClinic** — a real
+Spring MVC app that exercises a broad slice of the agent's instrumentation. It downloads and builds
+PetClinic, runs it under load with the agent, captures a JFR per variant, and prints **throughput**
+plus the **tracer-overhead split** (CPU + sampled allocation, bucketed foreground/background and
+instrumentation/core). The goal is that any developer — not just perf specialists — can reproduce the
+macro measurement that keeps micro benchmarks honest.
+
+This is the fast, **local, dev-iteration** tool. It complements, and does not replace, the org's
+automated benchmarking platform (the GitLab `apm-sdks-benchmarks-java-slo` track).
+
+### Extra dependencies
+On top of `wrk` and `nc`: `git`, a JDK (provides the `jfr` CLI), and `curl`. Maven is supplied by
+PetClinic's own wrapper. First run needs network access (clone + build + optional release download).
+
+### Trace destination — what actually gets measured
+The harness auto-detects a Datadog Agent on `localhost:8126` and picks the writer accordingly. This
+is a deliberate knob — it lets one tool measure two regimes:
+- **Agent reachable** (`DDAgentWriter`) — the *full pipeline*: span work on the request threads **and**
+  the background serializer (`TraceMapperV0_4`, msgpack, cache recalibration). Run your **local
+  Datadog Agent** on `:8126` for this; the analyzer's `background/*` buckets then reflect real
+  serialization cost.
+- **Agent unreachable** (`LoggingWriter`) — traces are discarded, so you measure the **agent-down
+  regime**: foreground overhead only, no serializer. Useful on its own for understanding behavior
+  when the Agent is missing.
+
+Pick intentionally based on what you want to see, and keep it consistent across a sweep. The harness
+prints a `>> REGIME:` banner so the choice is never silent, and re-checks before every variant. Set
+`EXPECT_AGENT=up` (or `down`) to make it **fail fast** if reality differs — e.g. a full-pipeline sweep
+aborts immediately if the Agent isn't up, and aborts mid-sweep if it dies, instead of quietly
+degrading to `LoggingWriter`. Default is `any` (auto-detect, no assertion).
+
+### Usage
+```
+# NoAgent vs the current checkout:
+./run-petclinic.sh
+
+# A version sweep for the historic curve — each token is a jar path, `current`
+# (builds this checkout's shadow jar), or a release version (downloaded from Maven Central):
+./run-petclinic.sh 1.54.0 1.58.0 1.62.0 current
+```
+Artifacts (throughput CSV + per-variant JFRs) are copied to `build/petclinic-results/results-<ts>/`.
+Analyze any recording on its own with `./analyze-jfr.sh <recording.jfr>`.
+
+#### Heap regime (ample vs tight)
+Server heap is a fixed `Xms=Xmx` knob (default 256m) — the regime where allocation overhead turns
+into throughput cost. Set one heap with `server_heap=` in the `.rc` (or `PERF_HEAP=`), or **sweep**
+several by passing a list — the full variant sweep runs once per heap:
+```
+PETCLINIC_HEAPS="64m 256m 512m" ./run-petclinic.sh 1.54.0 1.58.0 current
+```
+Results land in per-heap subdirs (`heap-64m/`, …) plus a combined `throughput-grid.csv` (leading
+`Heap` column). **Runtime is heaps × variants** — a heap list multiplies the wall clock, so size the
+version list accordingly.
+
+PetClinic is pinned to a specific commit in `fetch-petclinic.sh` (bump deliberately); JFR is opt-in in
+`run-perf-test.sh` via `PERF_JFR=1`, and endpoints/load live in `perf-test-petclinic-settings.rc`.
+
+### Reading the results honestly
+- **Allocation is the anchor; throughput is directional.** Allocation trends are stable across runs;
+  throughput on a shared dev box is noisy and swings with the CPU-contention regime — read it as a
+  direction, not a precise number.
+- Allocation here is **TLAB-sampled** (via JFR) — authoritative for *ratios* (which thread, which
+  package), with only an *estimated* total. That's sufficient for a macro trend.
+- Run on an otherwise-**quiet machine**, and respect the warmup (the harness warms each endpoint
+  before measuring). Results depend on the machine's core count / heap regime.
+
+### Known limitations (follow-ups)
+- No containerization yet — no pinned cores/heap, so cross-machine comparisons are approximate.
+- `run-perf-test.sh` is still lightly macOS-flavored (`/usr/bin/time`, `lsof`, `nc`); there is a basic
+  Linux guard but not full cross-platform hardening.

dd-java-agent/benchmark-integration/analyze-jfr.sh

@@ -0,0 +1,112 @@
+#!/usr/bin/env bash
+#
+# Analyze a perf-test JFR recording and print the tracer-overhead split that our methodology cares
+# about: self-time (and sampled allocation) bucketed by thread role (foreground request threads vs
+# background agent threads) and by code owner (instrumentation vs tracer-core vs app/JDK).
+#
+# This encodes the analysis so the numbers are trustworthy to someone running it blindly -- the tool
+# emits the meaningful split, not a raw JFR to hand-parse.
+#
+# Usage:   ./analyze-jfr.sh /tmp/perf_<label>.jfr
+# Requires: the JDK 'jfr' CLI (ships with the JDK) and awk.
+
+set -euo pipefail
+
+jfr_file="${1:-}"
+if [ -z "$jfr_file" ] || [ ! -f "$jfr_file" ]; then
+  echo "usage: ./analyze-jfr.sh <recording.jfr>" >&2
+  exit 1
+fi
+# The jfr CLI ships with JDK 11+. If it is not already on PATH, try the JDK env vars.
+if ! command -v jfr >/dev/null 2>&1; then
+  for jh in "${BENCH_JAVA_HOME:-}" "${JAVA_17_HOME:-}" "${JAVA_21_HOME:-}" "${JAVA_HOME:-}"; do
+    if [ -n "$jh" ] && [ -x "$jh/bin/jfr" ]; then PATH="$jh/bin:$PATH"; break; fi
+  done
+fi
+if ! command -v jfr >/dev/null 2>&1; then
+  echo "ERROR: the 'jfr' CLI was not found (ships with JDK 11+); set BENCH_JAVA_HOME to a JDK 17+." >&2
+  exit 1
+fi
+
+# Shared awk classifier. Reads a stream of "THREAD<TAB>TOPFRAME" lines and prints the bucketed
+# breakdown. Foreground = servlet-container request threads; background = agent + JFR/GC threads.
+# Owner: instrumentation (datadog.trace.instrumentation.*) vs core (other datadog.*) vs app/other.
+classify_awk='
+function role(t) {
+  if (t ~ /^http-nio/ || t ~ /^http-bio/ || t ~ /exec-/ || t ~ /^tomcat/ || t ~ /^XNIO/ || t ~ /^reactor-http/ || t ~ /^qtp/)
+    return "foreground";
+  if (t ~ /^dd-/ || t ~ /^dd\./ || t ~ /datadog/ || t ~ /^JFR/ || t ~ /Flight Recorder/ || t ~ /^GC / || t ~ /G1 / || t ~ /Reference Handl/)
+    return "background";
+  return "other";
+}
+function owner(f) {
+  if (f ~ /^datadog\.trace\.instrumentation\./) return "instrumentation";
+  if (f ~ /^datadog\./) return "core";
+  return "app";
+}
+{
+  total++;
+  r = role($1); o = owner($2);
+  role_tot[r]++;
+  if (o == "instrumentation" || o == "core") {
+    tracer_tot++;
+    split_ct[r SUBSEP o]++;
+  }
+}
+END {
+  printf "  samples total: %d   tracer (instrumentation+core): %d (%.1f%% of all)\n",
+    total, tracer_tot, total ? 100.0*tracer_tot/total : 0;
+  printf "  by thread role:  foreground=%d  background=%d  other=%d\n",
+    role_tot["foreground"]+0, role_tot["background"]+0, role_tot["other"]+0;
+  printf "\n  tracer self-%s split (share of tracer, and of all samples):\n", METRIC;
+  printf "    %-28s %10s %10s\n", "bucket", "of-tracer", "of-all";
+  emit(tracer_tot, total, "foreground", "instrumentation");
+  emit(tracer_tot, total, "foreground", "core");
+  emit(tracer_tot, total, "background", "instrumentation");
+  emit(tracer_tot, total, "background", "core");
+  emit(tracer_tot, total, "other",      "instrumentation");
+  emit(tracer_tot, total, "other",      "core");
+}
+function emit(tt, all, r, o,   c) {
+  c = split_ct[r SUBSEP o] + 0;
+  if (c == 0) return;
+  printf "    %-28s %9.1f%% %9.1f%%\n", r"/"o, tt ? 100.0*c/tt : 0, all ? 100.0*c/all : 0;
+}
+'
+
+# jfr print emits an event block per sample; the first stackTrace frame line after the event is the
+# leaf. We reduce each ExecutionSample to "sampledThread<TAB>leafFrame". The sampledThread field and
+# the stackTrace appear as fields; parse defensively across JDK jfr output formatting.
+extract_stream() {
+  local event="$1"
+  jfr print --events "$event" "$jfr_file" 2>/dev/null | awk '
+    /sampledThread = / { thr=$0; sub(/.*sampledThread = /,"",thr); sub(/ .*/,"",thr); gsub(/"/,"",thr); next }
+    /eventThread = /   { if (thr=="") { thr=$0; sub(/.*eventThread = /,"",thr); sub(/ .*/,"",thr); gsub(/"/,"",thr) } next }
+    /stackTrace = \[/  { want=1; next }
+    want==1 {
+      leaf=$1; sub(/\(.*/,"",leaf);           # strip "(params)"
+      gsub(/^[ \t]+/,"",leaf);
+      if (leaf != "" && leaf != "...") { print thr "\t" leaf }
+      want=0; thr="";
+    }
+  '
+}
+
+echo "== JFR analysis: $jfr_file =="
+echo
+echo "[CPU] jdk.ExecutionSample"
+cpu_stream=$(extract_stream jdk.ExecutionSample)
+if [ -n "$cpu_stream" ]; then
+  echo "$cpu_stream" | awk -F'\t' -v METRIC="cpu" "$classify_awk"
+else
+  echo "  (no ExecutionSample events found)"
+fi
+
+echo
+echo "[ALLOC] jdk.ObjectAllocationSample (TLAB-sampled -- ratios authoritative, total estimated)"
+alloc_stream=$(extract_stream jdk.ObjectAllocationSample)
+if [ -n "$alloc_stream" ]; then
+  echo "$alloc_stream" | awk -F'\t' -v METRIC="alloc" "$classify_awk"
+else
+  echo "  (no ObjectAllocationSample events found)"
+fi

dd-java-agent/benchmark-integration/fetch-petclinic.sh

@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+#
+# Clone + build Spring PetClinic at a pinned commit, and print the path to the runnable fat jar.
+#
+# Idempotent: if the jar already exists it is reused (no clone, no rebuild). All progress output
+# goes to stderr so that stdout is ONLY the jar path -- callers capture it with:
+#     petclinic_jar=$(./fetch-petclinic.sh)
+#
+# Requires: git, a JDK (for the Maven wrapper), and network access on the first run.
+
+set -euo pipefail
+
+# Pinned so results are reproducible across machines and over time. Bump deliberately.
+PETCLINIC_REPO="https://github.com/spring-projects/spring-petclinic.git"
+PETCLINIC_SHA="51045d1648dad955df586150c1a1a6e22ef400c2"
+
+script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# build/ is already git-ignored by the module, so the checkout is never committed.
+src_dir="$script_dir/build/spring-petclinic"
+
+log() { echo "[fetch-petclinic] $*" >&2; }
+
+# Reuse an existing build if present.
+existing_jar=$(ls "$src_dir"/target/spring-petclinic-*.jar 2>/dev/null \
+  | grep -v -- '-sources.jar$' | grep -v -- '.original$' | head -1 || true)
+if [ -n "$existing_jar" ]; then
+  log "reusing existing jar: $existing_jar"
+  echo "$existing_jar"
+  exit 0
+fi
+
+# Fetch exactly the pinned commit (shallow) -- avoids pulling the whole history.
+if [ ! -d "$src_dir/.git" ]; then
+  log "cloning $PETCLINIC_REPO @ $PETCLINIC_SHA"
+  mkdir -p "$src_dir"
+  git -C "$src_dir" init -q
+  git -C "$src_dir" remote add origin "$PETCLINIC_REPO" 2>/dev/null || true
+fi
+log "fetching pinned commit"
+git -C "$src_dir" fetch -q --depth 1 origin "$PETCLINIC_SHA"
+git -C "$src_dir" checkout -q FETCH_HEAD
+
+log "building PetClinic (this can take a few minutes on first run)"
+( cd "$src_dir" && ./mvnw -q -B -DskipTests package )
+
+jar=$(ls "$src_dir"/target/spring-petclinic-*.jar 2>/dev/null \
+  | grep -v -- '-sources.jar$' | grep -v -- '.original$' | head -1 || true)
+if [ -z "$jar" ]; then
+  log "ERROR: build finished but no runnable jar found under $src_dir/target"
+  exit 1
+fi
+log "built jar: $jar"
+echo "$jar"

dd-java-agent/benchmark-integration/perf-test-petclinic-settings.rc

@@ -0,0 +1,27 @@
+# Settings for running the perf test against Spring PetClinic.
+# Sourced by run-perf-test.sh (copy to perf-test-settings.rc, or point PERF_SETTINGS at it).
+
+# wrk settings
+test_warmup_seconds=60     # reach C2 steady state before measuring
+test_time_seconds=120      # stable throughput + ample JFR samples per endpoint
+test_num_connections=8
+test_num_threads=4
+
+# Quiet gap between version runs, so thermals/GC/OS settle and one run's tail does not
+# bleed into the next JVM. Only honored by run-perf-test.sh when set (>0).
+test_cooldown_seconds=60
+
+# Server heap, fixed Xms=Xmx (default 256m). A single run's regime. For a heap SWEEP use
+# run-petclinic.sh with PETCLINIC_HEAPS="64m 256m 512m"; PERF_HEAP overrides this per run.
+server_heap=256m
+
+# PetClinic endpoints -- a read-heavy mix spanning view rendering, a form, a DB-backed query,
+# a detail page, and the error path. PetClinic's default profile is in-memory H2 (no external DB).
+declare -A endpoints
+endpoints['home']='http://localhost:8080/'
+endpoints['vets']='http://localhost:8080/vets.html'
+endpoints['find-owners']='http://localhost:8080/owners/find'
+endpoints['owners-query']='http://localhost:8080/owners?lastName='
+endpoints['owner-detail']='http://localhost:8080/owners/1'
+endpoints['error']='http://localhost:8080/oups'
+test_order=( 'home' 'vets' 'find-owners' 'owners-query' 'owner-detail' 'error' )