Skip to content

feat(config): add DD_SERVERLESS_APM_ONLY traces-only mode#1293

Closed
zarirhamza wants to merge 4 commits into
mainfrom
zarir.hamza/serverless-apm-only-v2
Closed

feat(config): add DD_SERVERLESS_APM_ONLY traces-only mode#1293
zarirhamza wants to merge 4 commits into
mainfrom
zarir.hamza/serverless-apm-only-v2

Conversation

@zarirhamza

@zarirhamza zarirhamza commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a single master switch — DD_SERVERLESS_APM_ONLY — that puts the extension in APM-only ("traces only") mode. When enabled, every billable metrics and logs egress path is suppressed so the customer incurs no infrastructure-monitoring or log-ingestion charges, while traces and APM trace stats are unaffected.

This closes the gap that made a true "traces only" configuration impossible before: there was no toggle to disable custom DogStatsD metrics, so even with logs and enhanced metrics turned off, custom metrics could still egress. A single, defensive flag now guarantees the behavior.

  • Default is false — existing behavior is completely preserved.
  • Readable from both environment (DD_SERVERLESS_APM_ONLY) and YAML.

Behavior

Signal APM-only mode
Traces / APM trace stats sent (unchanged)
Enhanced metrics suppressed
Process (proc) enhanced metrics suppressed
Custom DogStatsD metrics suppressed
OTLP metrics suppressed
Logs (incl. OTLP logs) suppressed

Implementation (on the Config model)

  • LambdaConfig / LambdaConfigSource gain a serverless_apm_only field, merged from both env (DD_SERVERLESS_APM_ONLY) and YAML via merge_fields!. Defaults to false.
  • get_config() applies a post-merge override (apply_serverless_apm_only) that forces serverless_logs_enabled, enhanced_metrics, lambda_proc_enhanced_metrics (on .ext) and the core otlp_config_metrics_enabled / otlp_config_logs_enabled toggles off — overriding any individually set values so the guarantee holds regardless of other configuration.

Enforcement (defense in depth)

  1. Config (config/mod.rs::apply_serverless_apm_only) — forces the logs/metrics/OTLP toggles off after the shared env/yaml merge.
  2. Metrics wiring (main.rs::start_metrics_flushers) — returns no flushers, so anything drained from the DogStatsD aggregator (custom / enhanced / process) is discarded rather than sent.
  3. Logs flusher (logs/flusher.rs::flush) — short-circuits to an empty request set, guaranteeing no log egress even if something is queued or redriven.

Testing

  • cargo fmt --all -- --check — passes.
  • cargo clippy --workspace --all-targets --features default — passes (no new warnings).
  • cargo test — passes, including two new tests:
    • serverless_apm_only_defaults_off — flag defaults off; other toggles unchanged.
    • serverless_apm_only_forces_metrics_and_logs_off — flag overrides explicitly-enabled metrics/logs/OTLP toggles.

Live validation on AWS Lambda

Built the layer (arm64) and deployed two fully-distinct Node.js 22 functions (Datadog Node layer + this extension), one with the flag off and one on, then invoked each and queried the Datadog API:

Signal flag OFF (serverless-apm-full) flag ON (serverless-apm-standalone)
Spans / traces present present
Logs present 0
Custom metric present absent (never ingested)

The extension's Lambda Telemetry API subscription also drops from [Platform, Extension, Function] to [Platform] when the flag is on, confirming logs are cut off at the source.

Risk / rollback

Low. Behavior is gated entirely behind a new flag that defaults to false; when unset the code paths are unchanged. Rollback is leaving DD_SERVERLESS_APM_ONLY unset.

Notes for reviewers

  • The FIPS-feature clippy variant could not be run locally (macOS is missing the aws-lc FIPS crypto dylib); it exercises the same feature-agnostic config/logs code and runs on the Ubuntu CI runners.

@datadog-prod-us1-3

This comment has been minimized.

Introduce a single master switch, DD_SERVERLESS_APM_ONLY, that puts the
extension in APM-only ("traces only") mode. When enabled, every billable
metrics and logs egress path is suppressed so the customer incurs no
infrastructure-monitoring or log-ingestion charges, while traces and APM
trace stats are unaffected.

Implemented on the datadog-agent-config Config<LambdaConfig> model:
- LambdaConfig / LambdaConfigSource gain a serverless_apm_only field, merged
  from both env (DD_SERVERLESS_APM_ONLY) and YAML. Defaults to false.
- get_config() applies a post-merge override that forces
  serverless_logs_enabled, enhanced_metrics, lambda_proc_enhanced_metrics and
  the core OTLP metrics/logs toggles off, overriding any individually set
  values so the guarantee holds regardless of other configuration.

Enforced defensively at the egress points as well:
- main: start_metrics_flushers returns no flushers, so custom DogStatsD /
  enhanced / process metrics drained from the aggregator are discarded.
- logs flusher: flush() short-circuits to guarantee no log egress even if
  something is queued.

Adds unit tests covering the default-off case and the forced-override case.
@zarirhamza
zarirhamza force-pushed the zarir.hamza/serverless-apm-only-v2 branch from 827c848 to 3f4006a Compare July 15, 2026 18:59
@zarirhamza
zarirhamza marked this pull request as ready for review July 21, 2026 15:22
@zarirhamza
zarirhamza requested a review from a team as a code owner July 21, 2026 15:22
@zarirhamza
zarirhamza requested review from Copilot and duncanista July 21, 2026 15:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new master configuration switch, DD_SERVERLESS_APM_ONLY, to run Bottlecap in an APM-only (“traces only”) mode by suppressing logs and all metrics egress paths while leaving traces and trace stats unaffected.

Changes:

  • Extends LambdaConfig/LambdaConfigSource with serverless_apm_only and applies a post-merge override in get_config() to force-disable logs/metrics/OTLP logs+metrics toggles.
  • Prevents metrics egress by returning no metrics flushers when APM-only mode is enabled.
  • Prevents logs egress by short-circuiting the logs flusher in APM-only mode; adds config-level tests for default/override behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
bottlecap/src/config/mod.rs Adds serverless_apm_only to config + applies override to force-disable logs/metrics/OTLP signals; includes new tests.
bottlecap/src/bin/bottlecap/main.rs Disables metrics shipping by not starting metrics flushers in APM-only mode.
bottlecap/src/logs/flusher.rs Disables log shipping by returning no requests from the logs flusher in APM-only mode.

Comment thread bottlecap/src/logs/flusher.rs Outdated
Comment thread bottlecap/src/config/mod.rs
Address Copilot review feedback on the DD_SERVERLESS_APM_ONLY PR:
- Drain and discard the logs aggregator in APM-only mode instead of
  returning early, keeping memory bounded and mirroring the metrics path
  (which drains its aggregator and discards the data). No log egress
  either way.
- Add serverless_apm_only_from_yaml to guard the YAML key mapping,
  complementing the existing env-var override coverage.
Comment thread bottlecap/src/bin/bottlecap/main.rs Outdated
Comment thread bottlecap/src/config/mod.rs Outdated

@duncanista duncanista left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some integration testing would be good here, overall, not sure about the naming of the env var, is this going to be applied to every serverless product? does this type of env var exist outside of serverless?

…gration suite

Address review feedback on the APM standalone (traces-only) PR:
- Rename DD_SERVERLESS_APM_ONLY -> DD_APM_STANDALONE_ENABLED (field
  apm_standalone_enabled, fn apply_apm_standalone) to align with the
  "APM Standalone" product terminology.
- Rescope the metrics-flusher comment so it only describes the metrics
  egress path (logs/OTLP suppression lives in apply_apm_standalone and the
  logs flusher guard), and drop the redundant debug line there — the single
  authoritative debug now lives in apply_apm_standalone.
- Add an apm-standalone integration suite: a baseline function and a
  standalone (DD_APM_STANDALONE_ENABLED=true) function that both emit a
  trace, a custom metric, and logs. Asserts traces survive for both while
  metrics and logs are suppressed for the standalone function, even though
  DD_SERVERLESS_LOGS_ENABLED=true is inherited from the default env.
@zarirhamza

Copy link
Copy Markdown
Contributor Author

@duncanista thanks for the review — addressed your points in a1b1b89:

Naming. Renamed DD_SERVERLESS_APM_ONLYDD_APM_STANDALONE_ENABLED (field apm_standalone_enabled, fn apply_apm_standalone) to line up with the "APM Standalone" product terminology (the brief is "APM Standalone on Lambda"). The closest prior art is the tracers' DD_APM_TRACING_ENABLED (Standalone ASM/DSM), which is the inverse direction; DD_APM_STANDALONE_ENABLED reads as an explicit opt-in here.

Applied to every serverless product? That's the intent — the extension is serverless-only, so I dropped the SERVERLESS_ prefix but kept it product-agnostic so Cloud Run / Azure Container Apps can adopt the same flag. Only the Lambda egress paths (metrics flushers + logs flusher) are wired today; other products can hook into the same apm_standalone_enabled override as they land.

Outside serverless? Not currently — this flag lives in the Lambda ConfigExtension, so it's scoped to the extension. If we ever want a core-agent equivalent we can revisit aligning names.

Integration testing. Added an apm-standalone suite (integration-tests/tests/apm-standalone.test.ts + stack + test-suites.yaml). It deploys a baseline function and a standalone (DD_APM_STANDALONE_ENABLED=true) function that both emit a trace, a custom DogStatsD metric, and logs, then asserts:

  • traces (incl. the aws.lambda root span) survive for both
  • aws.lambda.enhanced.invocations and the custom metric are emitted for baseline, suppressed for standalone
  • logs are forwarded for baseline, not for standalone — even though DD_SERVERLESS_LOGS_ENABLED=true is inherited from the default env, which the mode overrides.

// also dropped upstream (serverless_logs_enabled is forced off, so the
// processor never queues them), but this guard guarantees no log egress
// regardless of aggregator or redrive state. See DD_APM_STANDALONE_ENABLED.
if self.config.ext.apm_standalone_enabled {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't there like an "enabled" section where we'd just add this?

@duncanista duncanista left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM – i'd cut down the amount of comments, they're too long, can be done with fewer words, overall, left another comment, would test it on actual resources/integration tests before merging!

@zarirhamza
zarirhamza marked this pull request as draft July 21, 2026 18:16
@zarirhamza zarirhamza closed this Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants