Only warn about trace buffer overflow when a kept trace is lost

[dougqh]

What Does This Do

Splits the trace-writer buffer-overflow log so that only genuine data loss reaches the user as a warning.

RemoteWriter logged Dropped due to a buffer overflow: [N spans] at WARN for every overflow, regardless of whether the lost trace was one the tracer intended to send. Prioritization routes sampled-out traces (SAMPLER_DROP / USER_DROP) to a separate queue from kept traces, so it already knows which case occurred — but that distinction was discarded by the time it reached the log site.

• DROPPED_BUFFER_OVERFLOW — a kept trace was lost. Still WARN, and the message now names dd.writer.queue.size so it is actionable.
• DROPPED_BUFFER_OVERFLOW_SAMPLED_OUT (new) — an already sampled-out trace was lost. Now DEBUG.

Health metrics and drop accounting are unchanged: both cases still go through handleDroppedTrace, so samplerDropDroppedTraces / samplerKeepDroppedTraces continue to report exactly as before.

Motivation

This warning has generated repeated support tickets and escalations for what is usually a non-event. Investigating one of them took five tracer flares to establish that no trace data was being lost at all — the overflow was landing entirely on traffic that trace sampling had already discarded.

The signal that distinguishes the two cases (samplerKeepDroppedTraces being zero or not) is not discoverable from the log line, which prints only a span count. Users reasonably read "Dropped" as "you are losing traces," and the WARN level endorses that reading.

Two details worth noting for review:

Classification happens at the source, not at the log site. Re-deriving severity from the sampling priority in RemoteWriter would misclassify force-keep traces: FastLaneStrategy.publish sends them to the primary queue regardless of their priority byte, so losing one is real data loss even though its priority reads as drop. Returning the distinction from publish avoids that.

The rate limiter has one shared budget. RatelimitedLogger holds a single nextLogNanos per instance. Had both cases stayed on the warning path, a flood of benign overflow could consume the budget and suppress the warning that matters for up to a minute — inverting the priority. Routing the benign case to log.debug avoids needing a second limiter. There is a test for this.

Trade-off

A service with a genuinely undersized queue under heavy sampled-out load loses an early warning signal. That seems right: the health metrics still carry it, and the current behaviour costs real support time for a non-event. But "no warnings" will no longer imply "queue is comfortably sized."

Additional Notes

The message text change means anyone alerting on the exact old string will need to update. Given the string was the source of the confusion, that seems acceptable, but flagging it.

Contributor Checklist

• Format code with ./gradlew spotlessApply
• New tests are JUnit 5 + Java
• Both new assertions verified to fail against the pre-fix behaviour
• :dd-trace-core:test --tests "datadog.trace.common.writer.*" passes

Jira ticket: APMS-20414

🤖 Generated with Claude Code

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.79 s	14.72 s	[-0.4%; +1.2%] (no difference)
startup:insecure-bank:tracing:Agent	13.62 s	13.72 s	[-1.6%; +0.2%] (no difference)
startup:petclinic:appsec:Agent	17.54 s	17.27 s	[+0.6%; +2.5%] (maybe worse)
startup:petclinic:iast:Agent	17.45 s	17.46 s	[-0.9%; +0.7%] (no difference)
startup:petclinic:profiling:Agent	17.44 s	17.35 s	[-0.6%; +1.7%] (no difference)
startup:petclinic:sca:Agent	17.45 s	17.18 s	[+0.6%; +2.5%] (maybe worse)
startup:petclinic:tracing:Agent	16.51 s	16.67 s	[-2.0%; +0.0%] (no difference)

Commit: 88ae28f8 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a3213514c1

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[datadog-prod-us1-4]

A full single-span sampling queue can drop spans that sampling rules can keep, but the change hides this warning. The kept-trace warning also names a setting that the tracer does not read.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit a321351 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}