Skip to content

[SLES-2907] chore(deps): align libdatadog rev, dedupe libdd-* crates#1303

Open
litianningdatadog wants to merge 5 commits into
mainfrom
tianning.li/bump-dd-trace-rs-v0.5.0-full-update
Open

[SLES-2907] chore(deps): align libdatadog rev, dedupe libdd-* crates#1303
litianningdatadog wants to merge 5 commits into
mainfrom
tianning.li/bump-dd-trace-rs-v0.5.0-full-update

Conversation

@litianningdatadog

@litianningdatadog litianningdatadog commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Overview

Follows up on the dd-trace-rs v0.5.0 bump (#1302): bumps the shared libdatadog git rev (libdd-capabilities, libdd-common, libdd-trace-protobuf, libdd-trace-utils, libdd-trace-normalization, libdd-trace-obfuscation, libdd-trace-stats) from a820699 to 85ce322a, matching the exact versions datadog-opentelemetry v0.5.0 pulls from crates.io. Adds a [patch.crates-io] block redirecting those (and the other transitively-pulled libdd-* crates) to the same rev, collapsing what would otherwise be two compiled copies of each.

This addresses the duplication Copilot flagged on serverless-components#144 — see bottlecap/LIBDATADOG_VERSION_ALIGNMENT.md for the full writeup of options considered and why this one (patch + shared rev bump) was chosen over switching to crates.io versions directly (tried first, reverted — broke type identity for ReplaceRule/Endpoint/Span/etc. across the git-vs-registry boundary throughout the trace pipeline) or leaving the duplication as-is.

Why not just add this to #1302?

#1302 skipped this on purpose. The duplication only cost +256 bytes on the release binary (LTO strips the unreachable copy), while actually fixing it means an untested major-version bump across six libdatadog crates, including libdd-trace-obfuscation — the code that redacts sensitive data from spans. Not something to risk on the same PR as an urgent customer fix (SLES-2907, B3 propagation errors). This PR does that bump on its own, with its own tests.

JIRA: SLES-2907

Code changes at the new rev

Most of the API changes in SpanConcentrator are just things the compiler forces on you, with no behavior change:

  • new() gained an obfuscation-config param. We pass None — bottlecap has never done client-side stats obfuscation.
  • flush() now returns FlushResult { obfuscated_buckets, unobfuscated_buckets } instead of a flat Vec. We concatenate both — obfuscation is off, so everything ends up in unobfuscated_buckets anyway.
  • pb::TracerPayload gained container_debug, pb::ClientGroupedStats gained additional_metric_tags. Added the missing fields to 3 struct literals so things compile; both are left at their defaults.

One change actually changes behavior. SpanConcentrator::new also gained a cardinality-override param, and the new rev defaults to capping trace-stats aggregation at 7,000 distinct keys per 10-second bucket, added in DataDog/libdatadog#2158 to bound SpanConcentrator's own memory use — an APM/trace-stats decision, unrelated to Datadog's custom-metrics cardinality limits. #2158 has the memory math behind the 7,000 default (~45.6 MB typical, ~700 MB worst case across 3 buckets, with the limit on); #2159 followed up with a collapsed_spans counter for when it's hit.

Passing None here like everywhere else would silently collapse any Lambda workload's stats past 7,000 distinct keys into a generic tracer_blocked_value bucket. We pass Some(usize::MAX) instead, restoring the pre-bump behavior — there was no limit before simply because the mechanism didn't exist yet, not because it was proven safe at scale, so this brings back the old unbounded-memory risk rather than adding a new one. test_no_cardinality_limit_applied covers the unbounded behavior; we don't wire up collapsed_spans (moot while the limit's off, worth revisiting if that changes).

Testing

  • cargo build --all-targets — clean
  • cargo test --all-targets — 548/548 pass across all test binaries
  • cargo clippy --all-targets -- -D warnings — clean
  • cargo fmt --all -- --check — clean
  • cargo tree --duplicates | grep ^libdd- — empty (zero libdd-* duplicates)
  • dd-rust-license-tool check — passes
  • Verified against the real serverless-components#145 commit (not a local file:// override used during earlier iteration)

@datadog-datadog-prod-us1

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aligns Bottlecap’s libdatadog-family dependencies to a single git revision and adds a [patch.crates-io] override so that datadog-opentelemetry’s crates.io libdd-* dependencies resolve to the same source, eliminating duplicate compiled copies. It also updates Bottlecap code/tests for upstream API changes introduced by the new libdatadog rev.

Changes:

  • Bump libdd-* git rev and add a [patch.crates-io] section to dedupe transitive libdd-* crates.
  • Adapt trace stats concentrator flushing logic to the new FlushResult { obfuscated_buckets, unobfuscated_buckets } return type.
  • Update trace/stats payload struct literals in unit/integration tests for newly added protobuf fields.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
bottlecap/Cargo.toml Bumps libdd-* git rev and adds [patch.crates-io] overrides to collapse duplicate libdd-* builds.
bottlecap/Cargo.lock Reflects dependency graph changes from the rev bump + patching crates.io libdd-* crates to the git source.
bottlecap/src/traces/stats_concentrator_service.rs Adjusts concentrator initialization args and handles new flush result bucket split.
bottlecap/src/traces/trace_processor.rs Updates test payload construction for new TracerPayload field(s).
bottlecap/tests/apm_integration_test.rs Updates integration test payload construction for new protobuf field(s).
bottlecap/LICENSE-3rdparty.csv Updates third-party license inventory for newly introduced transitive crates.

Comment thread bottlecap/src/traces/stats_concentrator_service.rs Outdated
Base automatically changed from tianning.li/bump-dd-trace-rs-v0.5.0 to main July 17, 2026 21:08
Follows up on the dd-trace-rs v0.5.0 bump: bumps the shared libdatadog
git rev (libdd-capabilities, libdd-common, libdd-trace-protobuf,
libdd-trace-utils, libdd-trace-normalization, libdd-trace-obfuscation,
libdd-trace-stats) from a820699 to 85ce322a, matching the exact
versions datadog-opentelemetry v0.5.0 pulls from crates.io. Adds a
[patch.crates-io] block redirecting those (and the other transitively
pulled libdd-* crates) to the same rev, collapsing what would
otherwise be two compiled copies of each.

This addresses the duplication Copilot flagged on
serverless-components#144 - see LIBDATADOG_VERSION_ALIGNMENT.md for
the full writeup of options considered and why this one (patch +
shared rev bump) was chosen over switching to crates.io versions
directly (tried first, reverted - broke type identity for
ReplaceRule/Endpoint/Span/etc. across the git-vs-registry boundary
throughout the trace pipeline) or leaving the duplication as-is.

Code changes forced by real upstream API changes at the new rev (not
duplication artifacts):
- src/traces/stats_concentrator_service.rs: SpanConcentrator::new
  gained a cardinality-override and an obfuscation-config parameter
  (both None - bottlecap doesn't use either); flush() now returns a
  FlushResult{obfuscated_buckets, unobfuscated_buckets} split instead
  of a flat Vec (combined both, since obfuscation is disabled).
- src/traces/trace_processor.rs, tests/apm_integration_test.rs:
  pb::TracerPayload gained container_debug and pb::ClientGroupedStats
  gained additional_metric_tags; added the missing fields to 3 struct
  literals (2 in the integration test, which fails to compile without
  the change).

Depends on DataDog/serverless-components#145 (which itself builds on
must stay in sync with whatever commit that PR lands at.

JIRA: https://datadoghq.atlassian.net/browse/SLES-2907

Verified: cargo build/test/clippy/fmt all clean against the real
serverless-components#145 commit (not a local override), 548/548
tests pass across all test binaries, cargo tree --duplicates shows
zero libdd-* duplicates, dd-rust-license-tool check passes.
@litianningdatadog
litianningdatadog force-pushed the tianning.li/bump-dd-trace-rs-v0.5.0-full-update branch from 08b3996 to ad794bb Compare July 17, 2026 21:17
litianningdatadog and others added 3 commits July 17, 2026 17:38
Start from unobfuscated_buckets (normally the only non-empty one since
obfuscation is disabled) instead of obfuscated_buckets, avoiding an
allocation and copy to grow the usually-empty vec.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
DataDog/serverless-components#145 merged as d0c7f44191445e20d309734675d5e8b91d2a5d51.
Re-pin dogstatsd/datadog-fips/datadog-agent-config to that commit instead
of the pre-merge draft SHA, and regenerate Cargo.lock. Content is
identical to the draft SHA for the affected file.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@litianningdatadog
litianningdatadog marked this pull request as ready for review July 22, 2026 14:37
@litianningdatadog
litianningdatadog requested review from a team as code owners July 22, 2026 14:37

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 39dfc70f0d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread bottlecap/src/traces/stats_concentrator_service.rs Outdated
The dd-trace-rs v0.5.0 / libdatadog rev bump introduced a per-bucket
cardinality limit (default 7,000 distinct aggregation keys) in
SpanConcentrator that didn't exist before. Bottlecap opted into the
new default, so Lambda workloads exceeding that cardinality in a 10s
window would have stats silently collapsed into a generic
tracer_blocked_value bucket with no observability into it happening.

Pass Some(usize::MAX) instead of None to disable the limit and keep
the previous unbounded behavior.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@litianningdatadog
litianningdatadog removed the request for review from duncanista July 22, 2026 18:01

@purple4reina purple4reina left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few things I'd like to understand better before I can approve.

First, on the PR description: I'd love a full rewrite. Right now it reads like deep implementation notes — accurate, but pitched at someone already inside the code, not at a reviewer trying to understand intent and risk. For each change, could you write a couple of plain-language sentences on what it does, why it's needed, and what the impact/risk is? That's the level I need to actually evaluate this, and it'll help whoever reads this PR's history down the line too.

What I understand this PR to be doing: bumping dependencies and then aligning dep versions between here and datadog-opentelemetry, so these packages are only compiled into the binary once. That part makes sense to me.

What I don't understand, and would like explained:

The other changes in the PR don't seem related to the dependency change — or if they are, that hasn't been made clear to me yet.

The first of those changes disables the cardinality limit on metrics. This seems like a huge and impactful change. Why are we doing this? What impact does it have? How do we know this is safe? I know Datadog just announced some level of unlimited cardinality on metrics for our users — is this related to that? Has it been approved by the metrics team? I'm not comfortable approving this specific change without answers to these questions. As is, the release notes won't say anything about turning off cardinality.

The second change has something to do with obfuscation. I don't understand what this is doing or why. Is it related to the dependency change? If not, it needs a clearer explanation.

@litianningdatadog

Copy link
Copy Markdown
Contributor Author

A few things I'd like to understand better before I can approve.

First, on the PR description: I'd love a full rewrite. Right now it reads like deep implementation notes — accurate, but pitched at someone already inside the code, not at a reviewer trying to understand intent and risk. For each change, could you write a couple of plain-language sentences on what it does, why it's needed, and what the impact/risk is? That's the level I need to actually evaluate this, and it'll help whoever reads this PR's history down the line too.

What I understand this PR to be doing: bumping dependencies and then aligning dep versions between here and datadog-opentelemetry, so these packages are only compiled into the binary once. That part makes sense to me.

What I don't understand, and would like explained:

The other changes in the PR don't seem related to the dependency change — or if they are, that hasn't been made clear to me yet.

The first of those changes disables the cardinality limit on metrics. This seems like a huge and impactful change. Why are we doing this? What impact does it have? How do we know this is safe? I know Datadog just announced some level of unlimited cardinality on metrics for our users — is this related to that? Has it been approved by the metrics team? I'm not comfortable approving this specific change without answers to these questions. As is, the release notes won't say anything about turning off cardinality.

The second change has something to do with obfuscation. I don't understand what this is doing or why. Is it related to the dependency change? If not, it needs a clearer explanation.

@purple4reina The PR message has been revised to address your concerns. Aside from the dependency and required API updates, the remaining changes were triggered by Codex/Copilot code review.

.collect(),
// Disable the cardinality limit to match pre-existing (unbounded) behavior.
Some(usize::MAX),
// Bottlecap does not perform client-side stats obfuscation.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Bottlecap does not perform client-side stats obfuscation.
// Bottlecap does not perform agent-side stats obfuscation.

@purple4reina purple4reina left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the explanation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants