From c06cfa4c2043d42a9b5abf369bb867b502f0481c Mon Sep 17 00:00:00 2001 From: Eric Law <39393654+acn-ericlaw@users.noreply.github.com> Date: Tue, 22 Sep 2026 10:44:06 -0700 Subject: [PATCH] Test report: OpenTelemetry certification, four runtimes - this host's forwarder and llm.stream in the 2026-09-22 drive The Java and Rust Playground edges rendered real Gemini tokens progressively through this host's llm.stream (traces c90af9e3... and a9686f1f...) and the Node.js twin, every application forwarding its spans under its own service name: the cross-application lineage read from both sides' datasets, zero export failures, and the provider observations. Backend confirmation by the maintainer remains. Co-Authored-By: Claude Fable 5.1 --- .../otel-dynatrace-certification.md | 87 +++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 88 insertions(+) create mode 100644 docs/test-reports/otel-dynatrace-certification.md diff --git a/docs/test-reports/otel-dynatrace-certification.md b/docs/test-reports/otel-dynatrace-certification.md new file mode 100644 index 0000000..829408f --- /dev/null +++ b/docs/test-reports/otel-dynatrace-certification.md @@ -0,0 +1,87 @@ +--- +title: Test Report — OpenTelemetry Certification, four runtimes +summary: Permanent record of the live four-runtime OpenTelemetry drive of 2026-09-22 - the + Java and Rust engines rendering Gemini tokens progressively through this host's llm.stream + and its Node.js twin, every application forwarding its spans to Dynatrace, one trace per request. +layer: reference +audience: [developer, architect] +keywords: [opentelemetry, otlp, dynatrace, distributed trace, llm.stream, test report] +--- + +# Test Report — OpenTelemetry Certification, four runtimes + +*The live drive of 2026-09-22 (UTC) that certified this host's +[OpenTelemetry forwarder](../guides/config-logging-actuators.md#distributed-tracing-the-opentelemetry-forwarder-opt-in) +in company: the Java and Rust engines' Playground edges rendering real Gemini tokens progressively, +the tokens produced by this host's `llm.stream` AI node (and by the Node.js host's twin), every +application forwarding its spans to the same Dynatrace tenant under its own service name — one trace +per request across an engine and a polyglot function host. The twin record on the engines is +Scenario 8 of their `otel-dynatrace-certification` reports.* + +## What was driven + +The forwarder exists on all four runtimes: the Java `opentelemetry-forwarder` module, the Rust port's +`mercury-opentelemetry-forwarder`, and the two zero-dependency ports of the Rust OTLP encoder merged +that day — this host (PR #33) and the Node.js host (mercury-nodejs #101). The maintainer's scenario is +the agent-orchestration experiment E0 stretched across them: `POST /api/llm/stream` on an engine's +Playground relays its reply lane into the event-over-http mapped `llm.stream` on a host, and the +provider's token batches re-render progressively out the engine's edge; `POST +/api/graph/support-triage` runs the E0 graph whose `llm.chat` node is a `graph.task` on the host. + +This host ran `mercury-serve examples/demo_app.py` on `:8086` with `-Dotel.forwarding=true`, +`-Dotel.service.name=mercury-otel-cert-python`, `-Dllm.provider=gemini -Dllm.model=gemini-3.6-flash`, +the OTLP endpoint and credential from the environment (`OTLP_API_ENDPOINT`, `OTLP_AUTH_HEADER`, +`OTLP_TOKEN` — the demo `application.yml` wiring) and `GEMINI_API_KEY`. The Java Playground (4.12.14, +`:8085`) and the Rust Playground (its E0 twin, `:8090`) forwarded as `mercury-otel-cert-java` and +`mercury-otel-cert-rust`; the Node.js host (`:8087`) as `mercury-otel-cert-node`. Every request +carried a caller-set `traceparent`. + +## The traces through this host + +| Edge → this host | Trace | `llm.stream` span start (UTC) | Token frames | Outcome | +|------------------|-------|-------------------------------|--------------|---------| +| **Java → Python** | `c90af9e36d8dbd3c2390db240b406d3a` | 17:36:37.175Z | 2 + `done` | `STOP`, 34 output tokens, 5.8 s | +| **Rust → Python** | `a9686f1f87327466e46cc451ff34b319` | 17:32:57.146Z | 2 + `done` | `STOP`, 21.2 s | + +The graph verdict through this host: Java → Python `372b040e245495158330fdb09b412ada` (17:32:29Z, +label `bug`). The `done` frame of each stream carried the model, `stop_reason`, usage and the trace +and business correlation ids — the continuity is self-documenting in the edge's output. (The Node.js +twin carried the other two pairings: Java → Node `888a3f721907d31a9b0ec9836b2e580a`, Rust → Node +`1232ab83511f3402a519e65a80e1a144`.) + +**The lineage, read from both sides' own datasets.** The engine's relay span is the parent of this +host's `llm.stream` span, and this host's span is the parent of the engine's reply-lane deliveries — +the same ids in two applications' logs: + +```text +Rust → Python a9686f1f… rust llm.stream.relay b05fccf48da0d67a + python llm.stream 02a46cfc25802ae5 (parent b05f…, 21.2 s) + rust async.http.response.stream.0 9f9206c70ec1e37e (parent 02a4…) + rust async.http.response.stream.0 ad15a089c5a72849 (parent 02a4…) +Java → Python c90af9e3… java llm.stream.relay babc1831e6f0d231 + python llm.stream 8bd6892d99b278d8 (parent babc…, 5.8 s) + java async.http.response.stream.0 813208e56d985f29 (parent 8bd6…) + java async.http.response.stream.0 902070e822a557d6 (parent 8bd6…) +``` + +**Exports.** Zero export failures in this host (and in every other application) in every run — five +drives, 24 LLM calls. This host exported 1–2 spans per round: its `llm.stream` executions. The +graph's `llm.chat` is an RPC leg, which folds into the caller's span on every runtime, so no span is +exported for it here — the engines' own rule, and the same one the `distributed.tracing` log follows. + +## Observations + +- **The provider, not the pipeline, decided which calls succeeded.** Gemini answered `503 This model + is currently experiencing high demand` on roughly half the calls across the drives, `429 + RESOURCE_EXHAUSTED` once, and `gemini-2.5-flash` proved retired (`no longer available to new users`, + its 404 text recommending `gemini-3.6-flash`, which then answered); the stable alias + `gemini-flash-latest` — now the demo's default — was the one under demand, so the drives pinned + `gemini-3.6-flash` with `-Dllm.model`. Every failure was itself a trace: this host rendered the + provider's status through the portable error contract (`LLM provider error - 503 ...`), the edge + returned it, and the forwarders exported those spans too. +- **The current flash models think before they answer.** A 200-token budget was spent entirely on + reasoning (`stop_reason: MAX_TOKENS`, `output_tokens: 0`, an empty stream); 1000 tokens rendered two + token frames and a `STOP`. A streaming AI node's budget is a certification setting, not a default. +- **What remains:** the backend's view — the maintainer's Dynatrace lookup of the traces above, each + expected to show two services with the parentage the datasets assert, this host's spans under the + instrumentation scope `mercury-composable-python`. diff --git a/mkdocs.yml b/mkdocs.yml index c01bc7d..0fa8072 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -125,5 +125,6 @@ nav: - Configuration Reference: guides/configuration-reference.md - HTTP Surface Reference: guides/http-surface-reference.md - Interop Test Report — Progressive Rendering: test-reports/progressive-rendering-interop.md + - Test Report — OpenTelemetry Certification: test-reports/otel-dynatrace-certification.md - Release Notes: https://github.com/Accenture/mercury-python/blob/main/CHANGELOG.md - Contributing: https://github.com/Accenture/mercury-python/blob/main/CONTRIBUTING.md