[SLES-2907] chore(deps): align libdatadog rev, dedupe libdd-* crates#1303
[SLES-2907] chore(deps): align libdatadog rev, dedupe libdd-* crates#1303litianningdatadog wants to merge 5 commits into
Conversation
This comment has been minimized.
This comment has been minimized.
b4123c4 to
08b3996
Compare
There was a problem hiding this comment.
Pull request overview
This PR aligns Bottlecap’s libdatadog-family dependencies to a single git revision and adds a [patch.crates-io] override so that datadog-opentelemetry’s crates.io libdd-* dependencies resolve to the same source, eliminating duplicate compiled copies. It also updates Bottlecap code/tests for upstream API changes introduced by the new libdatadog rev.
Changes:
- Bump
libdd-*git rev and add a[patch.crates-io]section to dedupe transitivelibdd-*crates. - Adapt trace stats concentrator flushing logic to the new
FlushResult { obfuscated_buckets, unobfuscated_buckets }return type. - Update trace/stats payload struct literals in unit/integration tests for newly added protobuf fields.
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| bottlecap/Cargo.toml | Bumps libdd-* git rev and adds [patch.crates-io] overrides to collapse duplicate libdd-* builds. |
| bottlecap/Cargo.lock | Reflects dependency graph changes from the rev bump + patching crates.io libdd-* crates to the git source. |
| bottlecap/src/traces/stats_concentrator_service.rs | Adjusts concentrator initialization args and handles new flush result bucket split. |
| bottlecap/src/traces/trace_processor.rs | Updates test payload construction for new TracerPayload field(s). |
| bottlecap/tests/apm_integration_test.rs | Updates integration test payload construction for new protobuf field(s). |
| bottlecap/LICENSE-3rdparty.csv | Updates third-party license inventory for newly introduced transitive crates. |
Follows up on the dd-trace-rs v0.5.0 bump: bumps the shared libdatadog
git rev (libdd-capabilities, libdd-common, libdd-trace-protobuf,
libdd-trace-utils, libdd-trace-normalization, libdd-trace-obfuscation,
libdd-trace-stats) from a820699 to 85ce322a, matching the exact
versions datadog-opentelemetry v0.5.0 pulls from crates.io. Adds a
[patch.crates-io] block redirecting those (and the other transitively
pulled libdd-* crates) to the same rev, collapsing what would
otherwise be two compiled copies of each.
This addresses the duplication Copilot flagged on
serverless-components#144 - see LIBDATADOG_VERSION_ALIGNMENT.md for
the full writeup of options considered and why this one (patch +
shared rev bump) was chosen over switching to crates.io versions
directly (tried first, reverted - broke type identity for
ReplaceRule/Endpoint/Span/etc. across the git-vs-registry boundary
throughout the trace pipeline) or leaving the duplication as-is.
Code changes forced by real upstream API changes at the new rev (not
duplication artifacts):
- src/traces/stats_concentrator_service.rs: SpanConcentrator::new
gained a cardinality-override and an obfuscation-config parameter
(both None - bottlecap doesn't use either); flush() now returns a
FlushResult{obfuscated_buckets, unobfuscated_buckets} split instead
of a flat Vec (combined both, since obfuscation is disabled).
- src/traces/trace_processor.rs, tests/apm_integration_test.rs:
pb::TracerPayload gained container_debug and pb::ClientGroupedStats
gained additional_metric_tags; added the missing fields to 3 struct
literals (2 in the integration test, which fails to compile without
the change).
Depends on DataDog/serverless-components#145 (which itself builds on
must stay in sync with whatever commit that PR lands at.
JIRA: https://datadoghq.atlassian.net/browse/SLES-2907
Verified: cargo build/test/clippy/fmt all clean against the real
serverless-components#145 commit (not a local override), 548/548
tests pass across all test binaries, cargo tree --duplicates shows
zero libdd-* duplicates, dd-rust-license-tool check passes.
08b3996 to
ad794bb
Compare
Start from unobfuscated_buckets (normally the only non-empty one since obfuscation is disabled) instead of obfuscated_buckets, avoiding an allocation and copy to grow the usually-empty vec. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
DataDog/serverless-components#145 merged as d0c7f44191445e20d309734675d5e8b91d2a5d51. Re-pin dogstatsd/datadog-fips/datadog-agent-config to that commit instead of the pre-merge draft SHA, and regenerate Cargo.lock. Content is identical to the draft SHA for the affected file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 39dfc70f0d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The dd-trace-rs v0.5.0 / libdatadog rev bump introduced a per-bucket cardinality limit (default 7,000 distinct aggregation keys) in SpanConcentrator that didn't exist before. Bottlecap opted into the new default, so Lambda workloads exceeding that cardinality in a 10s window would have stats silently collapsed into a generic tracer_blocked_value bucket with no observability into it happening. Pass Some(usize::MAX) instead of None to disable the limit and keep the previous unbounded behavior. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
purple4reina
left a comment
There was a problem hiding this comment.
A few things I'd like to understand better before I can approve.
First, on the PR description: I'd love a full rewrite. Right now it reads like deep implementation notes — accurate, but pitched at someone already inside the code, not at a reviewer trying to understand intent and risk. For each change, could you write a couple of plain-language sentences on what it does, why it's needed, and what the impact/risk is? That's the level I need to actually evaluate this, and it'll help whoever reads this PR's history down the line too.
What I understand this PR to be doing: bumping dependencies and then aligning dep versions between here and datadog-opentelemetry, so these packages are only compiled into the binary once. That part makes sense to me.
What I don't understand, and would like explained:
The other changes in the PR don't seem related to the dependency change — or if they are, that hasn't been made clear to me yet.
The first of those changes disables the cardinality limit on metrics. This seems like a huge and impactful change. Why are we doing this? What impact does it have? How do we know this is safe? I know Datadog just announced some level of unlimited cardinality on metrics for our users — is this related to that? Has it been approved by the metrics team? I'm not comfortable approving this specific change without answers to these questions. As is, the release notes won't say anything about turning off cardinality.
The second change has something to do with obfuscation. I don't understand what this is doing or why. Is it related to the dependency change? If not, it needs a clearer explanation.
@purple4reina The PR message has been revised to address your concerns. Aside from the dependency and required API updates, the remaining changes were triggered by Codex/Copilot code review. |
| .collect(), | ||
| // Disable the cardinality limit to match pre-existing (unbounded) behavior. | ||
| Some(usize::MAX), | ||
| // Bottlecap does not perform client-side stats obfuscation. |
There was a problem hiding this comment.
| // Bottlecap does not perform client-side stats obfuscation. | |
| // Bottlecap does not perform agent-side stats obfuscation. |
purple4reina
left a comment
There was a problem hiding this comment.
Thanks for the explanation.
Overview
Follows up on the
dd-trace-rsv0.5.0 bump (#1302): bumps the sharedlibdatadoggit rev (libdd-capabilities,libdd-common,libdd-trace-protobuf,libdd-trace-utils,libdd-trace-normalization,libdd-trace-obfuscation,libdd-trace-stats) froma820699to85ce322a, matching the exact versionsdatadog-opentelemetryv0.5.0 pulls from crates.io. Adds a[patch.crates-io]block redirecting those (and the other transitively-pulledlibdd-*crates) to the same rev, collapsing what would otherwise be two compiled copies of each.This addresses the duplication Copilot flagged on serverless-components#144 — see
bottlecap/LIBDATADOG_VERSION_ALIGNMENT.mdfor the full writeup of options considered and why this one (patch + shared rev bump) was chosen over switching to crates.io versions directly (tried first, reverted — broke type identity forReplaceRule/Endpoint/Span/etc. across the git-vs-registry boundary throughout the trace pipeline) or leaving the duplication as-is.Why not just add this to #1302?
#1302 skipped this on purpose. The duplication only cost +256 bytes on the release binary (LTO strips the unreachable copy), while actually fixing it means an untested major-version bump across six libdatadog crates, including
libdd-trace-obfuscation— the code that redacts sensitive data from spans. Not something to risk on the same PR as an urgent customer fix (SLES-2907, B3 propagation errors). This PR does that bump on its own, with its own tests.JIRA: SLES-2907
Code changes at the new rev
Most of the API changes in
SpanConcentratorare just things the compiler forces on you, with no behavior change:new()gained an obfuscation-config param. We passNone— bottlecap has never done client-side stats obfuscation.flush()now returnsFlushResult { obfuscated_buckets, unobfuscated_buckets }instead of a flatVec. We concatenate both — obfuscation is off, so everything ends up inunobfuscated_bucketsanyway.pb::TracerPayloadgainedcontainer_debug,pb::ClientGroupedStatsgainedadditional_metric_tags. Added the missing fields to 3 struct literals so things compile; both are left at their defaults.One change actually changes behavior.
SpanConcentrator::newalso gained a cardinality-override param, and the new rev defaults to capping trace-stats aggregation at 7,000 distinct keys per 10-second bucket, added in DataDog/libdatadog#2158 to boundSpanConcentrator's own memory use — an APM/trace-stats decision, unrelated to Datadog's custom-metrics cardinality limits. #2158 has the memory math behind the 7,000 default (~45.6 MB typical, ~700 MB worst case across 3 buckets, with the limit on); #2159 followed up with acollapsed_spanscounter for when it's hit.Passing
Nonehere like everywhere else would silently collapse any Lambda workload's stats past 7,000 distinct keys into a generictracer_blocked_valuebucket. We passSome(usize::MAX)instead, restoring the pre-bump behavior — there was no limit before simply because the mechanism didn't exist yet, not because it was proven safe at scale, so this brings back the old unbounded-memory risk rather than adding a new one.test_no_cardinality_limit_appliedcovers the unbounded behavior; we don't wire upcollapsed_spans(moot while the limit's off, worth revisiting if that changes).Testing
cargo build --all-targets— cleancargo test --all-targets— 548/548 pass across all test binariescargo clippy --all-targets -- -D warnings— cleancargo fmt --all -- --check— cleancargo tree --duplicates | grep ^libdd-— empty (zerolibdd-*duplicates)dd-rust-license-tool check— passesserverless-components#145commit (not a localfile://override used during earlier iteration)