refactor: support telemetry in ai agent related extensions - #9841
refactor: support telemetry in ai agent related extensions#9841JerryYang (JerryYangKai) wants to merge 3 commits into
Conversation
|
Azure Pipelines: Successfully started running 8 pipeline(s). 14 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
🟡 Changes recommended
Unused telemetry scaffolding unnecessarily raises four extensions’ host requirements, while duplicated reporters and a self-referential contract test need correction.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds a consistent, best-effort telemetry layer across Foundry-related extensions.
Changes:
- Adds timeout-bound telemetry reporters and tests.
- Introduces typed events for Agents and Inspector.
- Updates documentation, SDK dependencies, and host-version constraints.
File summaries
| File | Description |
|---|---|
cli/azd/extensions/azure.ai.toolboxes/README.md |
Documents telemetry usage. |
cli/azd/extensions/azure.ai.toolboxes/internal/telemetry/reporter.go |
Adds telemetry reporter. |
cli/azd/extensions/azure.ai.toolboxes/internal/telemetry/reporter_test.go |
Tests reporter behavior. |
cli/azd/extensions/azure.ai.toolboxes/internal/telemetry/events.go |
Adds event-builder placeholder. |
cli/azd/extensions/azure.ai.toolboxes/go.sum |
Updates SDK checksums. |
cli/azd/extensions/azure.ai.toolboxes/go.mod |
Updates azd SDK. |
cli/azd/extensions/azure.ai.toolboxes/extension.yaml |
Raises host requirement. |
cli/azd/extensions/azure.ai.skills/README.md |
Documents telemetry usage. |
cli/azd/extensions/azure.ai.skills/internal/telemetry/reporter.go |
Adds telemetry reporter. |
cli/azd/extensions/azure.ai.skills/internal/telemetry/reporter_test.go |
Tests reporter behavior. |
cli/azd/extensions/azure.ai.skills/internal/telemetry/events.go |
Adds event-builder placeholder. |
cli/azd/extensions/azure.ai.skills/go.sum |
Updates SDK checksums. |
cli/azd/extensions/azure.ai.skills/go.mod |
Updates telemetry dependencies. |
cli/azd/extensions/azure.ai.skills/extension.yaml |
Raises host requirement. |
cli/azd/extensions/azure.ai.routines/README.md |
Documents telemetry usage. |
cli/azd/extensions/azure.ai.routines/internal/telemetry/reporter.go |
Adds telemetry reporter. |
cli/azd/extensions/azure.ai.routines/internal/telemetry/reporter_test.go |
Tests reporter behavior. |
cli/azd/extensions/azure.ai.routines/internal/telemetry/events.go |
Adds event-builder placeholder. |
cli/azd/extensions/azure.ai.routines/go.sum |
Updates SDK checksums. |
cli/azd/extensions/azure.ai.routines/go.mod |
Updates azd SDK. |
cli/azd/extensions/azure.ai.routines/extension.yaml |
Raises host requirement. |
cli/azd/extensions/azure.ai.projects/README.md |
Documents telemetry usage. |
cli/azd/extensions/azure.ai.projects/internal/telemetry/reporter.go |
Adds telemetry reporter. |
cli/azd/extensions/azure.ai.projects/internal/telemetry/reporter_test.go |
Tests reporter behavior. |
cli/azd/extensions/azure.ai.projects/internal/telemetry/events.go |
Adds event-builder placeholder. |
cli/azd/extensions/azure.ai.inspector/README.md |
Documents Inspector telemetry. |
cli/azd/extensions/azure.ai.inspector/internal/telemetry/reporter.go |
Adds telemetry reporter. |
cli/azd/extensions/azure.ai.inspector/internal/telemetry/reporter_test.go |
Tests reporter behavior. |
cli/azd/extensions/azure.ai.inspector/internal/telemetry/events.go |
Defines UI-ready event. |
cli/azd/extensions/azure.ai.inspector/internal/inspector/telemetry.go |
Uses typed telemetry events. |
cli/azd/extensions/azure.ai.inspector/internal/inspector/telemetry_test.go |
Updates event assertions. |
cli/azd/extensions/azure.ai.inspector/internal/cmd/inspector.go |
Integrates the reporter. |
cli/azd/extensions/azure.ai.inspector/internal/cmd/inspector_test.go |
Tests reporter integration. |
cli/azd/extensions/azure.ai.connections/README.md |
Documents telemetry usage. |
cli/azd/extensions/azure.ai.connections/internal/telemetry/reporter.go |
Adds telemetry reporter. |
cli/azd/extensions/azure.ai.connections/internal/telemetry/reporter_test.go |
Tests reporter behavior. |
cli/azd/extensions/azure.ai.connections/internal/telemetry/events.go |
Adds event-builder placeholder. |
cli/azd/extensions/azure.ai.connections/go.sum |
Updates SDK checksums. |
cli/azd/extensions/azure.ai.connections/go.mod |
Updates azd SDK. |
cli/azd/extensions/azure.ai.connections/extension.yaml |
Raises host requirement. |
cli/azd/extensions/azure.ai.agents/README.md |
Documents Agents telemetry. |
cli/azd/extensions/azure.ai.agents/internal/telemetry/reporter.go |
Adds telemetry reporter. |
cli/azd/extensions/azure.ai.agents/internal/telemetry/reporter_test.go |
Tests reporter behavior. |
cli/azd/extensions/azure.ai.agents/internal/telemetry/events.go |
Defines route-selection event. |
cli/azd/extensions/azure.ai.agents/internal/cmd/run.go |
Integrates typed reporting. |
cli/azd/extensions/azure.ai.agents/internal/cmd/run_test.go |
Updates route telemetry tests. |
Review details
- Files reviewed: 42/46 changed files
- Comments generated: 6
- Review effort level: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
📋 Prioritization NoteThanks for the contribution! The linked issue isn't in the current milestone yet. |
There was a problem hiding this comment.
🔵 Needs a closer look
The Agents test no longer independently protects the telemetry wire schema from regressions.
Review details
Suppressed comments (1)
cli/azd/extensions/azure.ai.agents/internal/cmd/run_test.go:288
- This expected value is produced by the same builder as the production path, so changes to the event name, key, or enum values make both sides change and the schema-preservation test still passes. Assert the promised wire schema independently, as the Inspector test does.
wantEvent := telemetry.LocalClientRouteSelected(tt.wantRoute)
if reporter.event.Name != wantEvent.Name || !maps.Equal(reporter.event.Attributes, wantEvent.Attributes) {
t.Fatalf("event = %#v, want %#v", reporter.event, wantEvent)
- Files reviewed: 38/42 changed files
- Comments generated: 0 new
- Review effort level: Balanced
There was a problem hiding this comment.
🔵 Needs a closer look
The Agents test derives its expected schema from the production builder and therefore cannot detect an accidental schema change.
Review details
Suppressed comments (1)
cli/azd/extensions/azure.ai.agents/internal/cmd/run_test.go:286
- This expectation is built with the same event builder used by production, so the test still passes if that builder accidentally changes the established event name or attribute key. Assert the documented schema literals here, as the Inspector migration does, so this test protects the “schema unchanged” requirement.
wantEvent := telemetry.LocalClientRouteSelected(tt.wantRoute)
- Files reviewed: 38/42 changed files
- Comments generated: 0 new
- Review effort level: Balanced
azure.ai.agents PR buildNote This is an unsigned development build. Install it only if you trust this PR. Install the extension: azd ext install "https://azuresdkartifacts.z5.web.core.windows.net/azd/extensions/pr/9841/azure-ai-agents.zip"
|
azure.ai.skills PR buildNote This is an unsigned development build. Install it only if you trust this PR. Install the extension: azd ext install "https://azuresdkartifacts.z5.web.core.windows.net/azd/extensions/pr/9841/azure-ai-skills.zip"
|
azure.ai.connections PR buildNote This is an unsigned development build. Install it only if you trust this PR. Install the extension: azd ext install "https://azuresdkartifacts.z5.web.core.windows.net/azd/extensions/pr/9841/azure-ai-connections.zip"
|
azure.ai.projects PR buildNote This is an unsigned development build. Install it only if you trust this PR. Install the extension: azd ext install "https://azuresdkartifacts.z5.web.core.windows.net/azd/extensions/pr/9841/azure-ai-projects.zip"
|
azure.ai.inspector PR buildNote This is an unsigned development build. Install it only if you trust this PR. Install the extension: azd ext install "https://azuresdkartifacts.z5.web.core.windows.net/azd/extensions/pr/9841/azure-ai-inspector.zip"
|
azure.ai.toolboxes PR buildNote This is an unsigned development build. Install it only if you trust this PR. Install the extension: azd ext install "https://azuresdkartifacts.z5.web.core.windows.net/azd/extensions/pr/9841/azure-ai-toolboxes.zip"
|
azure.ai.routines PR buildNote This is an unsigned development build. Install it only if you trust this PR. Install the extension: azd ext install "https://azuresdkartifacts.z5.web.core.windows.net/azd/extensions/pr/9841/azure-ai-routines.zip"
|
|
Overall looks good to me |
| @@ -0,0 +1,101 @@ | |||
| // Copyright (c) Microsoft Corporation. All rights reserved. | |||
There was a problem hiding this comment.
Can this be moved somewhere common, like the azd/pkg/foundry package? We've been talking about pulling that package out of core and into something we can manage independently, but even without that being done yet I'd much rather have it somewhere that is shared instead of copied to every extension and that has to be updated 7+ times
Adds a consistent, best-effort telemetry reporting layer to the seven Foundry
extensions owned by the team and migrates the existing Agents and Inspector
telemetry producers to it.
This PR consumes the host-controlled
TelemetryService.ReportUsageAPI deliveredby #9174. It does not modify the
gRPC API or azd's telemetry pipeline.
Why this is needed
The Foundry extensions are separate Go modules and currently have inconsistent
telemetry integration:
extension-level reporting boundary.
This leaves command and product code responsible for protobuf request creation,
timeouts, error handling, logging, and test doubles. It also makes it easier for
telemetry failures to accidentally affect command behavior or for sensitive
attribute values to appear in diagnostics.
A narrow extension-local interface keeps those transport concerns out of product
code while preserving the host's existing identity, privacy, admission, and
export controls.
Approach
internal/telemetrypackage to:Event, containing an extension-owned event name and attributes;Reporter, exposingReport(context.Context, Event)with no error return;NewReporter, adapting the generated azd telemetry gRPC client.Accepted: false;internal/telemetry/events.go.extension.event = local_client.route.selectedext.route = inspector | playground | suppressedextension.event = inspector.funnel.stageext.stage = ui_readyext.outcome = succeededcommand and Inspector business-logic tests.
product events.
customer content in all seven READMEs.
The reporter is intentionally duplicated across the seven modules. These
extensions are independently built and released, cannot import each other's
internalpackages, and this team's scope does not include moving the abstractioninto
pkg/azdext.Telemetry and privacy
This PR adds no new product usage signal.
Agents and Inspector retain their previously documented events and values.
Connections, Projects, Routines, Skills, and Toolboxes only receive the reporting
interface and emit no events through it.
The azd host continues to own:
ext.*attribute namespacing;Extension telemetry must remain low cardinality and must never include prompts,
responses, resource or service names, IDs, endpoints, URLs, paths, credentials,
definitions, or other customer content.
Validation