Skip to content

fix(strands): normalize reasoning and Agent output boundaries - #283

Merged
ralf0131 merged 1 commit into
mainfrom
liuyu/fix-strands-reasoning-boundaries
Oct 29, 2026
Merged

ralf0131 merged 1 commit into
mainfrom
liuyu/fix-strands-reasoning-boundaries

Conversation

@sipercai

Copy link
Copy Markdown
Collaborator

Description

Strands reasoning blocks currently reach GenAI messages as raw SDK dictionaries without a type, and Agent output includes the final model's reasoning alongside its answer. Normalize provider-public reasoning through the shared Reasoning type in model inputs/outputs and filter intermediate reasoning/tool parts from Agent output.

Signatures, redacted/encrypted reasoning and unavailable text are omitted from telemetry. Final multimodal parts are retained, and SDK messages/results are left unchanged. README and CHANGELOG document the behavior.

Fixes #282

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • This change requires a documentation update

How Has This Been Tested?

  • tox -c tox-loongsuite.ini -e py312-test-loongsuite-instrumentation-strands-oldest — Strands 1.50.2 / OTel 1.37.0: 46 passed, 2 skipped (framework has no invocation-baggage feature).
  • tox -c tox-loongsuite.ini -e py312-test-loongsuite-instrumentation-strands-latest — Strands 1.62.0 / OTel 1.45.1: 48 passed.
  • PRE_COMMIT_HOME="$(mktemp -d /tmp/loongsuite-precommit.XXXXXX)" tox -e precommit
  • git diff --check

Validation Evidence

Area Result and evidence
Target / runtime Base c27eb2a6d8f2ece92b0f315025697b5771a30fd5, head 97a68ffb053b3fdc58cccd405934213862420a5b. Rebased over #278/#280; the entire Strands tree is identical to the runtime-tested baseline/patch respectively. Python 3.12.15; real SDK workload uses Strands 1.60.0, OpenAI 1.55.3 and OTel 1.45.1. Dependency boundary tests additionally cover Strands 1.50.2 and 1.62.0.
Regression discriminator Added streaming/non-streaming reasoning assertions fail before normalization (2 failed, 2 passed). Applying only the reasoning mapping leaves the separate Agent boundary assertions failing. Full patch passes all 48 tests.
Native complex application Real Strands Agent loop and OpenAI SDK JSON/SSE parsing; 2 user turns per session, 3 model decisions and 3 actual tools per turn. Stock and price tools execute in parallel before multiplication. Non-streaming plus 2 concurrent streaming sessions produce correct final answers and preserve prior-turn history. Only the provider HTTP/SSE boundary is deterministic; scheduling, tool dispatch, instrumentation and exporter are real.
Reasoning / boundaries Same normal workload: raw LLM reasoning 18 → 0, canonical reasoning 0 → 18, Agent outputs containing intermediate reasoning 6 → 0. All 18 tool call IDs link to executed tools. Original framework reasoning/signatures remain in business results. Invalid, signature-only and redacted reasoning are omitted; final image URI/text parts are retained.
Fail-open / lifecycle Injected one failure in the new Agent-output mapper during concurrent streaming: all business requests complete, siblings remain isolated and context restores. Early stream close from another task, cancellation and subsequent clean calls preserve behavior with no active state or detach errors. Existing provider-error and probe-fault regressions also pass.
Content / volume SPAN_ONLY and NO_CONTENT compared before/after. Each normal workload remains 63 spans and 0 span events; LLM usage remains 20 input + 10 output = 30 total, Agent totals 90. SPAN_ONLY serialized span payload changes from 123,567 to 121,940 bytes; NO_CONTENT is unchanged at 56,693 bytes and omits message/tool argument/result content. No additional spans or token counting are introduced.
Structured events SPAN_AND_EVENT supplementary run: 24 local log events on each revision, correlated with spans and matching their input/output messages. These log events were verified locally; cloud readback below covers spans.
ARMS / cloud readback Compared with the ARMS LLM field specification. Real OTLP span upload followed by run-scoped cloud readback through the existing arms CLI profile: 9 datasets, 493 spans; exact span IDs, parents, trace IDs, kinds and every source attribute match after normalizing storage representations. Backend-added derived attributes are accounted for separately.
Design references Reused the repository's shared GenAI types/handler. OpenInference Strands processor at ca3ed46053701c1af48d857864523915ce6bf637 confirms extraction of public reasoning text. A bounded OpenLLMetry tree search at 8d9c93f37b352131f5871dcb6ecf5e79c19aeb40 found no corresponding Strands implementation.
Privacy / scope Synthetic prompts, reasoning, signatures and tool values only. No new recordings, credentials or provider billing. No shared util, SDK lifecycle, dependency or packaging changes.
Static / CI Static readiness and review-matrix checks pass for the focused diff. CI status is reported by the checks on this PR.

The committed regression can be reproduced with the focused tox commands above. The additional SDK/exporter/cloud workload and readback artifacts are retained with the review; they use a deterministic provider boundary and do not claim validation of a remote model service or cloud log ingestion.

Does This PR Require a Core Repo Change?

  • Yes. - Link to PR:
  • No.

Checklist:

  • Followed the style guidelines of this project
  • Changelogs have been updated
  • Unit tests have been added
  • Documentation has been updated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No outstanding review issues are identified, and the change is scoped to the documented reasoning normalization and final-response telemetry behavior.

No open issues 🎉

What changed in this PR

This pull request normalizes Strands reasoning content into shared GenAI Reasoning parts and limits Agent output telemetry to final response content without mutating SDK results.

Changes:

  • Convert public reasoning text while omitting signatures, redacted content, and unavailable text.
  • Filter reasoning, tool calls, and tool results from Agent output while preserving final multimodal parts.
  • Add regression tests and document the updated telemetry behavior.
File Summary
instrumentation-loongsuite/​loongsuite-instrumentation-strands/​tests/​test_instrumentor.py Adds coverage for reasoning normalization, streaming, privacy filtering, Agent output boundaries, SDK-result preservation, and multimodal content.
instrumentation-loongsuite/​loongsuite-instrumentation-strands/​src/​opentelemetry/​instrumentation/​strands/​_hooks.py Implements reasoning conversion and final-only Agent output mapping.
instrumentation-loongsuite/​loongsuite-instrumentation-strands/​README.md Documents reasoning normalization and Agent output semantics.
instrumentation-loongsuite/​loongsuite-instrumentation-strands/​CHANGELOG.md Records the telemetry behavior fixes.

Review effort: Lite

@ralf0131
ralf0131 merged commit d70e600 into main Oct 29, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(strands): normalize public reasoning and keep Agent output final-only

5 participants