From 897ebdbc0b546a6eb491492bc356e52f78500e80 Mon Sep 17 00:00:00 2001 From: Yiming Luo <10097700+lym953@users.noreply.github.com> Date: Thu, 16 Jul 2026 22:17:58 -0400 Subject: [PATCH 1/2] docs(agents): warn against adding logs inside the logs pipeline The extension re-ingests its own stdout/stderr as logs, so logging while handling a log feeds back through the same pipeline and can amplify into a flood (worst at DD_LOG_LEVEL=debug). Document this gotcha for agents. Co-Authored-By: Claude Opus 4.8 (1M context) --- AGENTS.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 2e27f380a..3eca056fe 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -71,3 +71,21 @@ Event sources | `appsec/` | Application Security Monitoring | | `tags/` | Tag extraction and propagation | | `fips/` | FIPS mode cryptography support | + +## Gotchas + +### Be careful adding logs inside the logs pipeline + +Avoid adding log statements inside the logs pipeline (`logs/` — the processor, agent, and +flusher) unless you really need them, and never in a per-log-line hot path. + +The extension ingests its own stdout/stderr as logs (it subscribes to the Lambda Telemetry +API `extension` and `function` log types) and runs them back through this same pipeline. So a +line you log *while handling a log* becomes another log that is processed — and logged — +the same way. In a hot path this self-amplifies into a feedback loop that floods both +CloudWatch and the Datadog logs intake. It is worst at `DD_LOG_LEVEL=debug`, where a single +invocation can balloon into thousands of log lines and each emitted line can spawn more. + +If you need diagnostics in this area, prefer a temporary instrumented build, keep the logging +off per-log-line paths, or surface the signal through something that is not re-ingested as a +log (e.g. a metric or a one-shot line outside the loop). From 6a5e8f1925931c0dd96544c70e188f84244aaafa Mon Sep 17 00:00:00 2001 From: Yiming Luo <10097700+lym953@users.noreply.github.com> Date: Fri, 17 Jul 2026 15:20:50 -0400 Subject: [PATCH 2/2] docs(agents): tighten logs-pipeline gotcha wording Condense the warning about re-ingesting logs while logging inside the logs pipeline, keeping the same guidance more concise. Co-Authored-By: Claude Sonnet 5 --- AGENTS.md | 20 +++++++------------- 1 file changed, 7 insertions(+), 13 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 3eca056fe..f18a00881 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -76,16 +76,10 @@ Event sources ### Be careful adding logs inside the logs pipeline -Avoid adding log statements inside the logs pipeline (`logs/` — the processor, agent, and -flusher) unless you really need them, and never in a per-log-line hot path. - -The extension ingests its own stdout/stderr as logs (it subscribes to the Lambda Telemetry -API `extension` and `function` log types) and runs them back through this same pipeline. So a -line you log *while handling a log* becomes another log that is processed — and logged — -the same way. In a hot path this self-amplifies into a feedback loop that floods both -CloudWatch and the Datadog logs intake. It is worst at `DD_LOG_LEVEL=debug`, where a single -invocation can balloon into thousands of log lines and each emitted line can spawn more. - -If you need diagnostics in this area, prefer a temporary instrumented build, keep the logging -off per-log-line paths, or surface the signal through something that is not re-ingested as a -log (e.g. a metric or a one-shot line outside the loop). +The extension re-ingests its own stdout/stderr as logs, so a line logged while handling a log +(in `logs/` — processor, agent, flusher) feeds back through the same pipeline. In a +per-log-line hot path this self-amplifies into a flood of both CloudWatch and Datadog logs, +worst at `DD_LOG_LEVEL=debug`. + +Avoid logging in this area unless necessary; prefer a temporary instrumented build, or surface +signal via a metric instead.