Repository navigation
feat: add loongsuite-instrumentation-agentuniverse - #275
Draft
RichardoMrMu wants to merge 39 commits into
Draft
RichardoMrMu wants to merge 39 commits into
RichardoMrMu wants to merge 39 commits into
Conversation
github-actions
Bot
requested review from
123liuziming,
Cirilla-zmh and
ralf0131
September 28, 2026 07:03
RichardoMrMu
force-pushed
the
feat/agentuniverse-instrumentation
branch
from
September 28, 2026 07:05
48a3dac to
8e0486f
Compare
RichardoMrMu
marked this pull request as draft
September 28, 2026 07:11
The previous implementation wrapped Agent.run / Agent.async_run directly,
which produced a duplicate span whenever agentUniverse's own
AgentInstrumentor (0.0.18+) was enabled as well -- which is the normal
setup.
Delegate span creation to the native instrumentor and add the LoongSuite
gen_ai.* conventions to the very same span:
- reuse an already-active native AgentInstrumentor, or create and
instrument one only when none is active (tracked as _native_owned)
- patch AgentSpanAttributesSetter.set_input_attributes,
set_success_attributes and set_error_attributes so each calls the native
original first and then writes gen_ai.* on the same span
- the shared content-capture switch is read once, at instrument() time:
only SPAN_ONLY and SPAN_AND_EVENT write gen_ai.input.messages
- sentinel-guarded, idempotent instrument/uninstrument; uninstrument
restores the original setters and only tears down a native instrumentor
that this bridge created
No span is created here and neither run method is wrapped, so Agent.run /
Agent.async_run stay untouched. Native code keeps ownership of the span
name (au.agent.{source}), span kind, span status, metrics, streaming
first-token timing and conversation-memory recording.
The bridge deliberately never constructs AgentInstrumentor() while a
native instrumentor is active: AgentInstrumentor is a BaseInstrumentor
singleton whose __init__ runs on every construction and would reset the
saved wrapper originals, so a later uninstrument() through that fresh
handle would blank the trace-module globals and break every subsequent
Agent.run() with TypeError: 'NoneType' object is not callable.
Tests now run against real agentUniverse 0.0.19.1 with no stand-in,
initializing the ApplicationConfigManager that the native
ConversationMemoryModule requires. They cover the native-only,
LoongSuite-only and both-enabled paths -- asserting exactly one span per
agent call in every case, with both attribute namespaces coexisting on the
same span when both are enabled -- plus content capture on/off, the error
path, async_run, double instrument and uninstrument restore.
…ics proof Follow-up to the compatibility-bridge refactor, addressing independent audit findings. Content privacy: the native AgentSpanAttributesSetter writes the raw prompt to au.agent.input and the result to au.agent.output unconditionally, so declining gen_ai.input.messages alone did not actually keep user content off the span. When this bridge activated the native instrumentor and content capture is off, the two content-bearing setters are now handed a redacting span view that drops exactly those two attributes -- au.span.kind, au.agent.name, the caller attributes, pair_id, duration, status, usage.* and error.* are all still recorded. A native instrumentor the application activated itself is never filtered: its au.agent.input/output behaviour stays governed by the application's own configuration, and that asymmetry is documented in the README and CHANGELOG. Session propagation is now explicitly out of scope in the docs: au.trace.session.id and AUSessionPropagator come from agentUniverse's TelemetryManager.init_from_config(), the bridge activates only the Agent instrumentor, and a test asserts the global propagator is unchanged across instrument()/uninstrument(). Metrics, streaming first-token timing and token usage are now asserted directly rather than assumed: an InMemoryMetricReader checks that every native metric is recorded exactly once, that agent_calls_total reads 1 on the both-enabled path (which is what proves the native instrumentor was not instrumented a second time), that a real queue.Queue output_stream produces the first-token data point with au_agent_streaming=True, and that au.agent.usage.* keeps its native shape. Package metadata: wrapt is dropped -- the bridge patches the native static setters with setattr and never used it, and opentelemetry-instrumentation already depends on it. requires-python is bounded to >=3.10,<3.13 and the 3.13 classifier is gone, matching the tox matrix, because agentUniverse's pins (numpy<2, grpcio==1.63.0, pyarrow<17) have no cp313 wheels. Mutable evidence for the new behaviour: dropping the privacy filter fails 7 tests, neutering the setter bridge fails 18, and making the bridge always create its own native instrumentor fails 4. 27 tests pass on Python 3.12 against a real agentUniverse 0.0.19.1.
The initial scope covered only the Agent layer, which left the LLM and tool
spans produced by the same agentUniverse call tree without the LoongSuite
GenAI conventions -- and, more importantly, left the package's coverage
incomplete rather than wrong. agentUniverse ships one native instrumentor per
layer (agentUniverse 0.0.19.1, upstream 254ecd280f54c2d2de654bbe5373e9f947d2dedf):
* agentuniverse/base/tracing/otel/instrumentation/agent/agent_instrumentor.py
* agentuniverse/base/tracing/otel/instrumentation/llm/llm_instrumentor.py
* agentuniverse/base/tracing/otel/instrumentation/tool/tool_instrumentor.py
each of which patches the module-level wrapper globals in
agentuniverse/base/annotation/trace.py that the @trace_agent, @trace_llm and
@trace_tool decorators read at call time. The bridge now treats those three as
interchangeable layers and never creates a span itself.
Implementation. A frozen _LayerSpec describes each layer (native instrumentor
path, setter path, wrapper globals, private content attributes, applier and the
list of bridged setter methods) and a _LayerBridge installs or uninstalls one
layer: it reuses a live native instrumentor found through the wrapper globals,
or constructs and instruments one and marks it owned. Each layer gets its own
applier that writes the LoongSuite attributes on the native span after the
native setter has run:
* agent -- gen_ai.span.kind=AGENT, gen_ai.operation.name=invoke_agent,
gen_ai.agent.name; content on gen_ai.input.messages
* llm -- gen_ai.span.kind=LLM, gen_ai.operation.name=chat,
gen_ai.request.temperature (the native -1 sentinel is skipped), content on
gen_ai.input.messages / gen_ai.output.messages, and
gen_ai.response.time_to_first_token via a bridged
set_first_token_attributes
* tool -- gen_ai.span.kind=TOOL, gen_ai.operation.name=execute_tool,
gen_ai.tool.name, gen_ai.tool.type, gen_ai.tool.call.id from the native
au.tool.pair_id, content on gen_ai.tool.call.arguments /
gen_ai.tool.call.result
and all three also mirror their own au.*.usage.* counters into gen_ai.usage.*
(non-zero only) and record error.type from the exception class. Ownership and
the privacy policy are per layer: a bridge-owned layer with capture off is
handed a redacting span view that drops exactly that layer's native content
attributes, while a layer the application activated itself is never modified.
Two latent bugs surfaced while wiring the LLM and tool layers and are fixed
here: _unwrap_params treated every mapping whose values were all mappings as a
kwargs payload, which turned the LLM call's bound {'prompt': ..., 'kwargs': {}}
into a JSON blob for gen_ai.input.messages -- it now unwraps only the single
'kwargs' key the agent entry point produces. And the tool applier read the
pair id from the wrong argument index (args[4] instead of args[3] of
set_input_attributes), so gen_ai.tool.call.id was always missing.
Tests. 52 tests now run against a real agentUniverse 0.0.19.1 on Python 3.12,
with no stand-in, and cover native-only, LoongSuite-only, both-enabled and
partially-owned setups across all three layers, asserting exactly one span per
layer -- a typical call tree is one agent + one LLM + one tool span, never two.
Sync and async paths, agent and LLM streaming first-token (positive duration on
the span and in the histogram), error status and error metrics, the content
capture matrix in both ownership modes, the instrument/uninstrument lifecycle
including setter and wrapper-global restoration, and token usage are all
asserted directly:
* a real @trace_llm child returning TokenUsage(text_in=3, text_out=5)
aggregates into au.agent.usage.total_tokens=8, prompt_tokens=3,
completion_tokens=5 on the parent span, into agent_total_tokens=8,
agent_prompt_tokens=3, agent_completion_tokens=5 on the parent metrics and
into gen_ai.usage.total_tokens=8 on the same span (TokenUsage is a
pydantic-v1 model whose prompt_tokens/completion_tokens/total_tokens are
derived properties, so TokenUsage(prompt=3, completion=5, total=8) sets
nothing at all)
* every native metric is recorded exactly once through an
InMemoryMetricReader, *_calls_total reads 1 per layer, and the *_tokens
histograms carry the real non-zero totals
* the session path is exercised for real in a separate process:
TelemetryManager.init_from_config() registers SessionSpanProcessor and
AUSessionPropagator, the agent, LLM and tool spans all carry
au.trace.session.id, and the propagator injects and extracts both
AU-SessionId and auSessionId
Mutation checks keep the tests honest: neutering the agent, LLM or tool bridge,
disabling the privacy filter, disabling the gen_ai usage mirror, or removing
the session id from the session probe fails 16 / 16 / 12 / 14 / 4 / 1 tests
respectively, and the suite is green again once the files are restored.
Two agentUniverse 0.0.19.1 behaviours the bridge does not change are pinned by
tests and documented: streamed LLM usage is counted twice into the parent agent
span and metrics (the LLM span keeps the real numbers), and a native
instrumentor's __init__ resets its own saved state, so it must not be
constructed while it is already active. README and CHANGELOG are rewritten for
the three-layer scope, and the README covers the ownership/privacy matrix, the
conditional session story and the requirements range.
…mentation Replace the compatibility bridge with an independent implementation: the instrumentor takes over the six module-level wrapper extension points of agentuniverse.base.annotation.trace and produces the Agent, LLM and Tool telemetry itself -- spans, the au.* attribute set, all 26 metrics, streaming first-token timing, conversation memory, the invocation chain, token aggregation and error handling -- and adds the LoongSuite gen_ai.* conventions to those same spans through the shared ExtendedTelemetryHandler. There is no runtime import, delegation or patching of the framework's instrumentors, span-attribute setters, span managers or metric recorders. Installing is transactional: the six extension points are snapshotted, replaced, and rolled back on failure; uninstrument restores each saved object unless something else replaced it in the meantime. Two differences from the framework's own instrumentation are intentional and pinned by baseline tests: content capture is honoured (NO_CONTENT writes no content carrier on any layer), and streamed LLM usage is aggregated once instead of twice. Tests: 74 passed, 1 skipped against a real agentUniverse 0.0.19.1 on Python 3.12, including a native-vs-LoongSuite baseline matrix. Mutation checks for the three layer wrappers, the takeover, rollback/restore, privacy, stream finalization, token aggregation and the session probe all turn red as expected.
…he framework An errored tool span now also carries the `au.tool.usage.*` attributes, so its `au.*` contract matches the framework's own instrumentation attribute for attribute -- the framework records the zero usage of a failed call too. The span status was already ERROR on both sides. Pin it with a baseline comparison of the tool error path: the framework's run and this package's run of the same failing tool are compared on span kind, status, parent and the `au.*` key set, plus the `tool_errors_total` point. The only differences left on that path are the documented ones: the error-message text under the privacy policy, and the framework writing content unconditionally. Refresh the documented evidence while here: 75 passing tests, and the per-mutation failure counts.
…n the metric sets Declare the package's metrics in its packaging metadata (`supports_metrics` in pyproject.toml, `_supports_metrics` in package.py) so the generated instrumentation-loongsuite README lists it with metrics support instead of claiming none. The key stays out of the wheel METADATA and the build is unaffected. Strengthen the baseline matrix: the metric families are compared as exact sets on both runs -- nine agent, nine LLM and eight tool families, none missing and none extra, which is 23 exported families per successful run plus the shared handler's two `gen_ai.client.*` families on this package's spans. The tool call carriers are asserted directly in both capture modes instead of being covered only by the generic content list. Refresh the documented evidence: 77 passing tests, and the per-mutation failure counts.
main added two loongsuite extension packages that land in the same generated files this branch edits: a2a (alibaba#274) and llama-index (alibaba#273). Those two commits are the whole diff from the merge base, so the conflicts are the three list files (tox-loongsuite.ini, pyproject.toml, instrumentation-loongsuite/README.md) and the two generated workflow files. Resolved as a union: every entry from both sides is kept, with agentuniverse placed next to a2a and llama-index. The two workflow files are not hand-edited; they are the output of the repo's own generator run over the merged tox-loongsuite.ini (python .github/workflows/generate_workflows_loongsuite.py), so they match `tox -e generate-workflows` byte for byte.
RichardoMrMu
marked this pull request as ready for review
October 12, 2026 06:37
[testenv]deps carries `agentuniverse: {[testenv]test_deps}` and `agentuniverse:
-r .../loongsuite-instrumentation-agentuniverse/test-requirements.txt`. tox
splits an env name on `-` into factors, so the factor `agentuniverse` also
matches `lint-loongsuite-instrumentation-agentuniverse`, and that env installs
the package's test-requirements on top of `-r dev-requirements.txt`.
Measured on bc14430 (runs/38213319242/job/114692312074): the job fails during
install_deps, before ruff runs.
install_deps> uv pip install ... -r dev-requirements.txt
-r .../loongsuite-instrumentation-agentuniverse/test-requirements.txt
x No solution found when resolving dependencies:
... And because you require sphinx-rtd-theme==2.0.0rc4 and
agentuniverse>=0.0.19, we can conclude that your requirements are unsatisfiable.
dev-requirements.txt pins sphinx-rtd-theme==2.0.0rc4 while agentUniverse>=0.0.19
needs sphinx-rtd-theme>=2.0.0,<3.0.0, and the package declares requires-python
<3.13 while this env runs 3.13.
Give the env its own deps: the list the lint factor produces for every other
package, without the framework. Verified with tox 4.65.3 that `tox -c
tox-loongsuite.ini c -e lint-loongsuite-instrumentation-agentuniverse` no longer
resolves test-requirements.txt, and that the six generated workflow files are
byte-identical before and after this change.
Alternative considered: raising dev-requirements.txt to sphinx-rtd-theme>=2.0.0.
Not taken -- it changes every lint env in the repo for one package's dependency.
Collaborator
Author
|
Root cause of the red For this env, The contradiction is inside agentUniverse's own metadata:
Measured with
The CI run reaches the same conclusion in its own words: So no constraint layering, no Python marker and no pinned version inside agentUniverse's declared range makes this env installable. The two tweaks that would make it green both change what is tested: Kept in the PR, unchanged by the above: the matrix stops at 3.12 and The env becomes installable again when agentUniverse ships a release that drops |
sipercai
reviewed
Oct 13, 2026
sipercai
left a comment
sipercai
left a comment
Collaborator
There was a problem hiding this comment.
Agent / LLM / Tool 三层接入有价值，wrapper 的接管、回滚和恢复也考虑得比较完整。
目前 GenAI util 主要参与属性写入与结束处理，Span 创建、Context、流生命周期和指标仍由插件自行维护。建议优先复用 util 能承接的生命周期及共享 advice，减少重复实现，便于后续统一维护。
在 agentUniverse 0.0.19.1 + 真实模型调用中复现了下面几个问题，建议先修复并补充回归测试。模型/provider 映射、ReAct STEP 和 Agent 最终输出边界也需要进一步对齐。
Review follow-ups on the LLM layer; every behaviour here has a test that fails on the previous code. _llm.py: process_async_stream is a plain function again, returning a ClaimedAsyncStream. As an async generator its body -- and with it mark_streaming(), which hands the caller's context token over -- only ran on the caller's first __anext__. A stream that was awaited and then closed without being read therefore kept the caller inside the LLM span and left that span recording, and a caller resuming after the await started its next span as a child of a span that never ended. The wrapper claims the stream when it is created and finalizes the span on an aclose() that never read a chunk, which is what the sync twin's early close already did. _llm.py: usage_of() reads the usage the framework left in LLMOutput.raw through TokenUsage.from_openai/from_anthropic when LLMOutput.usage is empty, so au.*.usage.* and the shared handler's gen_ai.usage.* are populated for a provider whose usage payload does not reach the field the framework reads. Streamed chunks go through the same reader. _llm.py: chunk_text() keeps the empty text of a role-only or usage-only chunk empty instead of falling back to str(chunk); the async loop used to write a whole LLMOutput repr into gen_ai.output.messages. _common.py: LayerSpan.attach detaches its context token and ends its span when the token bookkeeping that follows the attach raises, so a failed attach cannot leave the trace filing later spans under a span that never ends. Tests: tests/test_llm_review_followups.py, 9 cases. 5 fail before this commit and pass after it; the rest are guards that pass on both.
The review body asked for the LLM layer's span/context lifecycle to be the
shared handler's responsibility, and for the GenAI identity to match the other
plugins. Two behaviour changes, both measured on the current head before the
fix:
teardown abandon_llm was never called: finalize detached the leftover
context token itself and ended the span itself
request.model gen_ai.request.model carried the component name (stub_llm)
rather than the model the call asks for
finalize now hands the invocation to the handler's own best-effort cleanup
(abandon_llm) after stop_llm/fail_llm, so the plugin no longer detaches a token
or ends an LLM span. The token hand-over before finalizing stays, because that
is what stop_llm detaches. gen_ai.request.model reads LLM.model_name and falls
back to the component name, which au.llm.name keeps reporting either way;
gen_ai.provider.name stays the channel name, the only provider identity the
framework exposes (LLMChannel has channel_model_name and no provider field).
Span creation and the context attach stay in the plugin: start_llm creates the
span and names it "{operation_name} {request_model}", so delegating it would
rename the exported au.llm.<name> spans.
tests/test_llm_lifecycle_alignment.py pins all of it, including start_span
raising before attach (the call stays uninstrumented and the next call of the
trace is a root span) and the component-name fallback.
Verified: the two modules 15 passed; package suite 1 failed / 92 passed, the
single failure being the pre-existing
test_intentional_difference_streaming_token_count baseline drift on
agentUniverse 0.1.0a2; ruff format --check and ruff check clean.
Collaborator
Author
|
Re-verified against the review body's three points, on (a) Lifecycle reuse - delegated where the handler owns it, not where it does not.
What stays in the plugin, and why:
(b) Semantic alignment.
(c) Verification on |
Collaborator
Author
|
Unrelated red light in this run's CI: every |
sipercai
reviewed
Oct 13, 2026
sipercai
left a comment
sipercai
left a comment
Collaborator
There was a problem hiding this comment.
异步流返回前 detach、未消费即 aclose、attach 后初始化失败清理已通过复验；LLM 的 abandon_llm 清理委托也有改进。
版本范围需要同时考虑当前稳定版与预发布版：PyPI 默认稳定版仍为 agentUniverse 0.0.19.1，0.1.0a2 是新 API 的 alpha。稳定版复验中，真实同步/异步流被新增 usage fallback 的 AttributeError 中断；在 0.1.0a2 上，真实流式业务和 Token 正常，但 LLMStreamEvent 的输出内容没有采集。下面分别给出兼容、fail-open 和新版流式映射的行内意见。
依赖与测试矩阵也需要对齐实际支持范围。本插件的 Python 3.10/3.11/3.12 CI 在 install_deps 阶段因框架与 OTel 依赖矛盾失败，不能仅以其他包的无关失败解释。本次稳定版运行使用受控依赖环境隔离运行行为，默认完整依赖解析另行检查，仍为 unsatisfiable；应解决正式安装路径，而不是依靠 --no-deps 作为安装验收。新版测试也需按原生行为更新，不能继续假设 native streamed usage 翻倍。
0.1.0a2 的原生工具循环已跑通两轮，每轮 3 次模型调用、2 次工具，最终返回 60/30；仍缺 STEP，工具规划 LLM 的 output_messages 缺失，Agent output 包含整份输入与框架元数据。逐调用 model 覆盖也仍未映射：实际 call(model='qwen-plus') 时 Span 记录的仍是实例配置 qwen-turbo。建议一并补齐真实 adapter、原生工具循环与内容边界测试，并继续收敛共享 advice 和 util 生命周期的责任边界。
| payload = raw.get("usage") if isinstance(raw, dict) else None | ||
| if not isinstance(payload, dict): | ||
| return None | ||
| for convert in (TokenUsage.from_openai, TokenUsage.from_anthropic): |
Collaborator
There was a problem hiding this comment.
[P1] 兼容当前稳定版的 usage 转换 API
这里在构造 tuple 时就访问 TokenUsage.from_anthropic，而当前稳定版 agentUniverse 0.0.19.1 没有这个方法。即使 raw payload 是 OpenAI usage，也会在第一次转换前抛 AttributeError。真实同步/异步流因此中断，非流式虽保留业务结果，但结束映射提前失败，输出和 Token 仍缺失。请避免无条件访问可选转换 API，分别覆盖稳定版与 0.1.0a2 的真实 adapter/stream 返回结构。
Collaborator
Author
There was a problem hiding this comment.
按能力取转换器了：raw_usage_converter() 逐个 getattr 再检查 callable，0.0.19.1 没有 from_anthropic 也能正常转换。我拿 stable 的 wheel（0.0.19.1）里的 TokenUsage 直接跑 usage_of，返回的 usage 正确；测试里也补了对应用例（缺 from_anthropic、以及两个转换器都没有这两种情况）。
| first = False | ||
| run_safely(self._finish_first_token) | ||
| output.append(chunk_text(chunk)) | ||
| chunk_usage = usage_of(chunk) |
Collaborator
There was a problem hiding this comment.
[P1] 隔离逐 chunk 的采集异常
chunk_text / usage_of 的采集处理直接位于业务迭代内，发生异常会走下面的 re-raise。在稳定版和 0.1.0a2 上，实测将 usage_of 注入为采集故障后，消费者收到同一 probe 异常，原本应交付的 3 个 chunk 一个也没收到；下一次干净调用正常。请用共享 advice 隔离纯采集步骤，即使丢弃遥测也继续按原顺序交付同一 chunk 对象；provider 迭代本身的业务异常仍保持原对象透传。同步循环也需覆盖相同回归。
Collaborator
Author
There was a problem hiding this comment.
逐 chunk 的读取（文本、usage、tool call 累加）都收进 run_safely 了，采集异常只留在遥测层。新增「坏 chunk」测试：3 个 chunk 原对象、原顺序交付给调用方，span 依然正常结束。
| without a ``text`` field is not a framework payload and is rendered as it | ||
| is. | ||
| """ | ||
| text = getattr(chunk, "text", None) |
Collaborator
There was a problem hiding this comment.
[P2] 采集新版 LLMStreamEvent 的 text_delta
agentUniverse 0.1.0a2 的真实流返回 LLMStreamEvent，文本位于 text_delta，text 是构造 classmethod。这里将其视为“存在但非字符串”并返回空串，因此业务完整收到答复、Token 正确，但同步/异步 LLM Span 的 gen_ai.output.messages 均缺失。请按实际事件类型采集文本增量、工具调用与终态，并用真实 adapter 的流式输出补回归测试。
Collaborator
Author
There was a problem hiding this comment.
chunk_record() 现在先读 text、再读 text_delta。用真实的 LLMStreamEvent.text("part-1") 实测：修复前 span 输出是空串，修复后是 part-1part-2。
Nine review findings, each with a red-then-green check: - read streamed text from LLMStreamEvent.text_delta as well as from text; - resolve the raw-usage converter by capability, so a release without TokenUsage.from_anthropic (0.0.19.1) still converts; - keep a per-chunk collector failure inside the telemetry layer, so the chunks reach the caller unchanged and in order; - report a tool-planning round's tool calls in gen_ai.output.messages; - publish the agent's answer instead of the framework's call envelope; - map a per-call model= override onto gen_ai.request.model; - record one ReAct STEP span per tool-calling round, with the round's usage promoted onto the agent span and an open step closed when the round aborts; - follow the installed release's streamed-usage accounting in the matrix. Tests: pytest tests -q -> 108 passed; ruff check and ruff format --check clean. Red baseline with the sources reverted and the new tests kept: 12 failed. Requires-Python: >=3.10,<3.13.
Collaborator
Author
|
总评这 6 点都处理了，命令与结果放在最后。
命令与结果： |
sipercai
marked this pull request as draft
October 22, 2026 08:16
…vs core proto-grpc agentuniverse 0.0.19.1 pins grpcio==1.63.0 exactly, while the pinned opentelemetry-exporter-otlp-proto-grpc (core SHA 37dea4b) requires grpcio>=1.63.2 on Python <3.13, making uv dependency resolution fail before tests run. Add a uv override file pinning grpcio==1.63.0 (the agentuniverse-supported patch release) and wire it via UV_OVERRIDE only for the agentuniverse tox envs. ruff check passes.
0.0.19.1 pins grpcio==1.63.0 exactly, conflicting with the pinned core otlp-proto-grpc dev0 which requires >=1.63.2. The 0.1.0a2 prerelease drops the exact grpcio pin entirely and requires opentelemetry-api/sdk >=1.30.0 (satisfied by core HEAD). All conftest import paths (Agent, Tool, AgentModel, trace_llm, AppConfiger, native instrumentors, LLMOutput) exist in both versions. Remove the UV_OVERRIDE shim and overrides.txt.
…ride" This reverts commit 8c58cee.
….0.19.1 vs core proto-grpc" This reverts commit 06f3e2d.
agentUniverse 0.0.19.1 hard-pins grpcio==1.63.0, conflicting with the pinned core otlp-proto-grpc dev0 which requires grpcio>=1.63.2. 0.1.0a2 drops the exact grpcio pin and requires opentelemetry-api/sdk >=1.30.0 (satisfied by core HEAD). The instrumentation remains compatible with both lines; this selects the installable prerelease for the formal install path rather than --no-deps or a forced grpcio override.
…1 (was hardcoded == 0.0.19.1)
…ool_call message The 0.1.0a2 ToolCallingLoopAgent.run_tool_calling_loop detects tool calls via LLMOutput.message.has_tool_calls(), not the raw payload. The ToolPlanningLLM stub only set raw, so the loop returned after round 1 and the ReAct step/usage tests recorded one round instead of two. Return an assistant Message carrying a real ToolCall (built only where the SDK exports the ToolCall/Message surface), and register the real stub_tool so the native execute_tool_call dispatcher runs a successful tool rather than a not-found error. Verified exact same input on agentUniverse 0.0.19.1 (116 passed/7 skipped, unchanged) and 0.1.0a2 (the two TestReactSteps loop tests now pass; remaining 9 failures are pre-existing stable-protocol fixtures tracked separately).
… extra blank line)
…DK-version aware On alpha 0.1.0a2 Agent ships native on_before/on_after_llm_round and the LLM channel yields LLMStreamEvent (typed usage, no raw dict). Keep the strict install/uninstall and usage-bearing-chunk assertions but read the SDK-appropriate surface instead of assuming the stable 0.0.19.1 layout. Verified: old 0.0.19.1 116 passed/7 skipped unchanged; alpha 0.1.0a2 7 failed/115 passed (was 9 failed/113 passed).
Fix ruff I001 (unsorted-imports) flagged by precommit on the round-hook / streaming-usage test edit.
…n-claimed Detect when Agent.on_before/on_after_llm_round has been replaced by a non-framework owner (qualname != Agent.*) and raise before wrapping, so the install rolls back atomically instead of overwriting a foreign hook.
|
|
A prior on_before/on_after_llm_round owner (plain passthrough, functools.wraps or name metadata) is a legitimate composition point, not a conflict. Remove the qualname gate that refused it. Replace the B2 atomic test with a real partial install-failure seam: force the second hook assignment to raise and assert the whole install rolls back, restoring core wrappers and native hooks.
Replace the invalid staticmethod metaclass patch (which raised TypeError on the first assignment) with a bound metaclass __setattr__ that raises exactly once on the second hook assignment, lets restore through, and asserts the before->after-fail->restore event sequence plus 6 core wrappers and native hooks identity restoration.
Snapshot hooks after any synthetic monkeypatch so the test does not crash on stable 0.0.19.1 (which ships no round hooks); on stable the synthetic seam lambdas are restored by rollback and dropped by monkeypatch teardown.
…c OPENAI_API_KEY - runtime_app.py: InventoryAgent (SDK 0.0.19.1) has no max_iterations field; setting it raises pydantic ValueError. Remove the assignment. - test_llm_review_followups: monkeypatch OPENAI_API_KEY so the SDK's default OpenAI client construction path does not raise OpenAIError in CI.
The stable 0.0.19.1 Agent.invoke_llm resolves the LLM by name through LLMManager; a process_llm override is never consulted. This left llm=None in judge_stream, failing every native ReAct conversation test with AttributeError: 'NoneType' object has no attribute '_channel_instance'. Register the channel-backed ChannelLLM instance so the framework finds it by name, and unregister after the workload to avoid leaking into other tests. The MockTransport clients hold no real sockets.
… on alpha - test_usage_of_reads: pre-populate OpenAIStyleLLM._client_pool[api_base] on agentUniverse 0.1.0a2, where _get_client() bypasses self.client. - test_stable_native_react_conversations (4 params) and test_concurrent_collector_fault: skip on alpha >=0.1.0 because the stable text ReAct protocol (messages[-1] text Action format) differs from alpha structured tool_calls (empty messages on first round). - Verified 118 passed, 5 skipped, 0 failed on agentUniverse 0.1.0a2 / Python 3.12.14 / pytest 9.1.1.
…sertions Remove the pytest.mark.skipif that skipped stable ReAct tests on alpha. Instead, make the runtime contract and lifecycle tests detect alpha 0.1.x vs stable 0.0.x at import time and assert protocol-appropriate values: - alpha structured tool_calls: 3 LLM rounds/turn (18 calls total), parent span kind AGENT or STEP, reasoning_effort high|none, no prior-conversation retention in messages (chat_history carries it). - stable text ReAct: 4 rounds/turn (24 calls), AGENT parent, reasoning_effort high, prior conversation retained in messages. runtime_app.py: on alpha use StructuredProvider with profile instruction template and skip LLMManager channel close (shared channel). runtime_provider.py: validate_body is now an instance method that skips the prior-conversation check on alpha and accepts reasoning_effort high|none. All 123 tests pass on agentUniverse 0.1.0a2 (Python 3.12, mock transport).
Move importlib.metadata import to stdlib group at top of files. Remove unused _stable_react skip decorator definition. Fixes CI failures: ruff check I001 (4 errors) and precommit ruff-format (4 files reformatted).
Auto-applied ruff check --fix (3 I001 import errors) and ruff format (4 files). Verified locally: ruff check clean, ruff format --check clean, 123 tests pass.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds independent Agent, LLM and Tool instrumentation for stable agentUniverse
0.0.19.1 APIs, using the framework's six wrapper extension points and the
shared GenAI handler. Stable native ReAct uses LangChain's real AgentExecutor;
alpha-only round hooks and provider/dispatcher APIs are capability-conditional
and do not prevent the stable core layers from enabling.
Draft: normal installation is blocked by the published framework package.
agentUniverse 0.0.19.1 pins
grpcio==1.63.0; the aligned OpenTelemetry gRPCexporter requires
grpcio>=1.63.2. The framework unconditionally depends onboth HTTP and gRPC exporter distributions, so selecting HTTP does not remove
this conflict. An upstream dependency correction and a new compatible stable
package are required before this PR is ready to merge. The version constraint
will need to follow that actual verified release.
Runtime diagnostic tests use a visibly versioned local framework wheel that
changes only dependency metadata. All 576 framework files are byte-identical
to the official stable wheel. These runs establish API behavior; they are
not acceptance of an unmodified published-wheel installation. Neither
--no-depsnorskip_dep_checkis an advertised installation path.The instrumentation preserves framework result objects and exceptions, reads
stable public reasoning/response IDs/finish reasons from retained SDK payloads,
corrects inclusive provider Token totals, captures native ReAct JSON arguments,
keeps Agent output limited to the final answer, and preserves sessions and
stream cleanup. Text ReAct supplies no provider tool-call ID; none is invented.
Type of change
How Has This Been Tested?
diagnostic environment: 117 passed, six alpha-only API checks skipped.
The OpenTelemetry SDK/exporters use the repository-pinned core revision.
tox -e precommitwith a fresh hook cache, including full repositoryRuff, formatting, uv-lock and rstcheck hooks. No unrelated lockfile changes.
tox -e generate-workflows; generated workflows remain current.parsing: three sessions, each with two user turns and four model decisions
per turn; actual stock, price and multiplication tools; synchronous calls
and two concurrent full-stream reasoning conversations.
tails, real stable-channel raw tool-call JSON and fragmented SSE arguments,
SPAN_ONLY/NO_CONTENT, fault isolation, a clean sibling, cancellation and
cross-task/unread stream cleanup.
tox environment's resolver rejects its exact gRPC pin before tests run.
the declared CI matrix remains blocked on the same installation problem.
Validation Evidence
The representative stable application uses the framework's original executor,
tool conversion/dispatch and actual SDK JSON/SSE decoding. Only the synthetic
provider HTTP boundary is deterministic. Each workload performs 24 model
requests and 18 real tool executions, retaining second-turn history and
isolating concurrent sessions. Separate native-instrumentation and
uninstrumented executions serve as controls.
Compared original PR head
870fff6with the stable-compatible implementation.The original accepts the stable version but raises while installing missing
alpha round hooks and leaves wrappers partially installed; this failed
activation is explicitly recorded in the diagnostic baseline. Under that
observed state, the same workload produces 51 spans before and after. Its LLM
usage is corrected from 600 input + 312 output = 912 to provider-reported
480 input + 240 output = 720. Nonempty public reasoning is recorded separately
on all 24 model outputs when enabled. Native tool arguments and final Agent
answers are verified against actual business execution.
Seven bounded before/after/control datasets plus a separate final-commit
healthy-path verification were exported through the actual OTLP exporter and
read back from ARMS: 361 spans, with no differences in source span IDs,
parents, kinds or attributes. Content-disabled runs omit message and tool
argument/result carriers. Cloud validation covers traces; event-log and metric
delivery were not part of this stable-version cloud acceptance.
The span count remains 51 on the healthy representative path. Serialized span
payload is approximately 148 KB before and 150 KB after; corrected metadata
and public reasoning account for the changed content. These are payload
comparisons, not billing estimates. Fixture privacy scanning is clean.
The stable framework also emits an existing StreamOutPutCallbackHandler
AgentFinish warning in uninstrumented/native control runs. Final Agent results
are correct; this PR does not change that framework callback or claim its
application-level stream completion event has been repaired.
Does This PR Require a Core Repo Change?
required before installation and merge acceptance.
Checklist: