Skip to content

refactor: support telemetry in ai agent related extensions - #9841

Open
JerryYang (JerryYangKai) wants to merge 3 commits into
mainfrom
kaiyan/tel
Open

refactor: support telemetry in ai agent related extensions#9841
JerryYang (JerryYangKai) wants to merge 3 commits into
mainfrom
kaiyan/tel

Conversation

@JerryYangKai

@JerryYangKai JerryYang (JerryYangKai) commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Adds a consistent, best-effort telemetry reporting layer to the seven Foundry
extensions owned by the team and migrates the existing Agents and Inspector
telemetry producers to it.

This PR consumes the host-controlled TelemetryService.ReportUsage API delivered
by #9174. It does not modify the
gRPC API or azd's telemetry pipeline.

Why this is needed

The Foundry extensions are separate Go modules and currently have inconsistent
telemetry integration:

  • Agents calls the generated telemetry gRPC client directly.
  • Inspector maintains its own callback adapter around the generated client.
  • Connections, Projects, Routines, Skills, and Toolboxes have no shared
    extension-level reporting boundary.

This leaves command and product code responsible for protobuf request creation,
timeouts, error handling, logging, and test doubles. It also makes it easier for
telemetry failures to accidentally affect command behavior or for sensitive
attribute values to appear in diagnostics.

A narrow extension-local interface keeps those transport concerns out of product
code while preserving the host's existing identity, privacy, admission, and
export controls.

Approach

  • Add the same internal/telemetry package to:
    • Agents
    • Connections
    • Inspector
    • Projects
    • Routines
    • Skills
    • Toolboxes
  • Introduce:
    • Event, containing an extension-owned event name and attributes;
    • Reporter, exposing Report(context.Context, Event) with no error return;
    • NewReporter, adapting the generated azd telemetry gRPC client.
  • Make reporting strictly best effort:
    • use a one-second timeout;
    • never retry;
    • ignore Accepted: false;
    • tolerate nil clients and nil responses;
    • log only event names and gRPC status codes in debug mode;
    • never log attributes or raw transport error details.
  • Clone the attribute map before sending it to the host.
  • Keep extension-specific event builders and bounded value types in
    internal/telemetry/events.go.
  • Migrate the existing Agents signal without changing its schema:
    • extension.event = local_client.route.selected
    • ext.route = inspector | playground | suppressed
  • Migrate the existing Inspector signal without changing its schema:
    • extension.event = inspector.funnel.stage
    • ext.stage = ui_ready
    • ext.outcome = succeeded
  • Remove generated telemetry protobuf/gRPC dependencies from the corresponding
    command and Inspector business-logic tests.
  • Add the telemetry interface to the other five extensions without emitting new
    product events.
  • Upgrade the SDK dependency used to compile the telemetry adapters while preserving each extension's existing runtime compatibility floor until it has a production telemetry call site.
  • Document usage, failure semantics, official-registry behavior, and prohibited
    customer content in all seven READMEs.

The reporter is intentionally duplicated across the seven modules. These
extensions are independently built and released, cannot import each other's
internal packages, and this team's scope does not include moving the abstraction
into pkg/azdext.

Telemetry and privacy

This PR adds no new product usage signal.

Agents and Inspector retain their previously documented events and values.
Connections, Projects, Routines, Skills, and Toolboxes only receive the reporting
interface and emit no events through it.

The azd host continues to own:

  • extension identity;
  • ext.* attribute namespacing;
  • official-registry admission;
  • event and attribute bounds;
  • the 100-event per-invocation budget;
  • trace correlation;
  • consent, transport, and export.

Extension telemetry must remain low cardinality and must never include prompts,
responses, resource or service names, IDs, endpoints, URLs, paths, credentials,
definitions, or other customer content.

Validation

Seven extension telemetry package test suites       PASS
Seven complete `go test d:.` suites               PASS
Seven `golangci-lint run --timeout 10m0s` runs      PASS — 0 issues
Seven `go fix -diff d:.` checks                    PASS — clean
Seven `azd x build --skip-install` builds            PASS
Seven README cspell 8.13.1 checks                    PASS — 0 issues
VS Code diagnostics                                 PASS — 0 errors
git diff --check                                    PASS
Seven-folder scope check                            PASS

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 8 pipeline(s).
14 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@github-actions github-actions Bot added ext-agents azure.ai.agents extension ext-connections azure.ai.connections extension ext-foundry azure.ai.{agents,connections,inspector,projects,routines,skills,toolboxes}, microsoft.foundry ext-inspector azure.ai.inspector extension ext-projects azure.ai.projects extension ext-routines azure.ai.routines extension ext-skills azure.ai.skills extension ext-toolboxes azure.ai.toolboxes extension labels Sep 2, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unused telemetry scaffolding unnecessarily raises four extensions’ host requirements, while duplicated reporters and a self-referential contract test need correction.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a consistent, best-effort telemetry layer across Foundry-related extensions.

Changes:

  • Adds timeout-bound telemetry reporters and tests.
  • Introduces typed events for Agents and Inspector.
  • Updates documentation, SDK dependencies, and host-version constraints.
File summaries
File Description
cli/azd/extensions/azure.ai.toolboxes/README.md Documents telemetry usage.
cli/azd/extensions/azure.ai.toolboxes/internal/telemetry/reporter.go Adds telemetry reporter.
cli/azd/extensions/azure.ai.toolboxes/internal/telemetry/reporter_test.go Tests reporter behavior.
cli/azd/extensions/azure.ai.toolboxes/internal/telemetry/events.go Adds event-builder placeholder.
cli/azd/extensions/azure.ai.toolboxes/go.sum Updates SDK checksums.
cli/azd/extensions/azure.ai.toolboxes/go.mod Updates azd SDK.
cli/azd/extensions/azure.ai.toolboxes/extension.yaml Raises host requirement.
cli/azd/extensions/azure.ai.skills/README.md Documents telemetry usage.
cli/azd/extensions/azure.ai.skills/internal/telemetry/reporter.go Adds telemetry reporter.
cli/azd/extensions/azure.ai.skills/internal/telemetry/reporter_test.go Tests reporter behavior.
cli/azd/extensions/azure.ai.skills/internal/telemetry/events.go Adds event-builder placeholder.
cli/azd/extensions/azure.ai.skills/go.sum Updates SDK checksums.
cli/azd/extensions/azure.ai.skills/go.mod Updates telemetry dependencies.
cli/azd/extensions/azure.ai.skills/extension.yaml Raises host requirement.
cli/azd/extensions/azure.ai.routines/README.md Documents telemetry usage.
cli/azd/extensions/azure.ai.routines/internal/telemetry/reporter.go Adds telemetry reporter.
cli/azd/extensions/azure.ai.routines/internal/telemetry/reporter_test.go Tests reporter behavior.
cli/azd/extensions/azure.ai.routines/internal/telemetry/events.go Adds event-builder placeholder.
cli/azd/extensions/azure.ai.routines/go.sum Updates SDK checksums.
cli/azd/extensions/azure.ai.routines/go.mod Updates azd SDK.
cli/azd/extensions/azure.ai.routines/extension.yaml Raises host requirement.
cli/azd/extensions/azure.ai.projects/README.md Documents telemetry usage.
cli/azd/extensions/azure.ai.projects/internal/telemetry/reporter.go Adds telemetry reporter.
cli/azd/extensions/azure.ai.projects/internal/telemetry/reporter_test.go Tests reporter behavior.
cli/azd/extensions/azure.ai.projects/internal/telemetry/events.go Adds event-builder placeholder.
cli/azd/extensions/azure.ai.inspector/README.md Documents Inspector telemetry.
cli/azd/extensions/azure.ai.inspector/internal/telemetry/reporter.go Adds telemetry reporter.
cli/azd/extensions/azure.ai.inspector/internal/telemetry/reporter_test.go Tests reporter behavior.
cli/azd/extensions/azure.ai.inspector/internal/telemetry/events.go Defines UI-ready event.
cli/azd/extensions/azure.ai.inspector/internal/inspector/telemetry.go Uses typed telemetry events.
cli/azd/extensions/azure.ai.inspector/internal/inspector/telemetry_test.go Updates event assertions.
cli/azd/extensions/azure.ai.inspector/internal/cmd/inspector.go Integrates the reporter.
cli/azd/extensions/azure.ai.inspector/internal/cmd/inspector_test.go Tests reporter integration.
cli/azd/extensions/azure.ai.connections/README.md Documents telemetry usage.
cli/azd/extensions/azure.ai.connections/internal/telemetry/reporter.go Adds telemetry reporter.
cli/azd/extensions/azure.ai.connections/internal/telemetry/reporter_test.go Tests reporter behavior.
cli/azd/extensions/azure.ai.connections/internal/telemetry/events.go Adds event-builder placeholder.
cli/azd/extensions/azure.ai.connections/go.sum Updates SDK checksums.
cli/azd/extensions/azure.ai.connections/go.mod Updates azd SDK.
cli/azd/extensions/azure.ai.connections/extension.yaml Raises host requirement.
cli/azd/extensions/azure.ai.agents/README.md Documents Agents telemetry.
cli/azd/extensions/azure.ai.agents/internal/telemetry/reporter.go Adds telemetry reporter.
cli/azd/extensions/azure.ai.agents/internal/telemetry/reporter_test.go Tests reporter behavior.
cli/azd/extensions/azure.ai.agents/internal/telemetry/events.go Defines route-selection event.
cli/azd/extensions/azure.ai.agents/internal/cmd/run.go Integrates typed reporting.
cli/azd/extensions/azure.ai.agents/internal/cmd/run_test.go Updates route telemetry tests.
Review details
  • Files reviewed: 42/46 changed files
  • Comments generated: 6
  • Review effort level: Balanced

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread cli/azd/extensions/azure.ai.agents/internal/cmd/run_test.go
Comment thread cli/azd/extensions/azure.ai.connections/extension.yaml Outdated
Comment thread cli/azd/extensions/azure.ai.routines/extension.yaml Outdated
Comment thread cli/azd/extensions/azure.ai.skills/extension.yaml Outdated
Comment thread cli/azd/extensions/azure.ai.toolboxes/extension.yaml Outdated
Comment thread cli/azd/extensions/azure.ai.projects/internal/telemetry/reporter.go
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

📋 Prioritization Note

Thanks for the contribution! The linked issue isn't in the current milestone yet.
Thank you for logging this issue; our team is reviewing it. If you need urgent prioritization, tag Rick Winter (@RickWinter) and Kristen Womack (@kristenwomack) to let us know.

Copilot AI review requested due to automatic review settings September 2, 2026 06:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The Agents test no longer independently protects the telemetry wire schema from regressions.

Review details

Suppressed comments (1)

cli/azd/extensions/azure.ai.agents/internal/cmd/run_test.go:288

  • This expected value is produced by the same builder as the production path, so changes to the event name, key, or enum values make both sides change and the schema-preservation test still passes. Assert the promised wire schema independently, as the Inspector test does.
			wantEvent := telemetry.LocalClientRouteSelected(tt.wantRoute)
			if reporter.event.Name != wantEvent.Name || !maps.Equal(reporter.event.Attributes, wantEvent.Attributes) {
				t.Fatalf("event = %#v, want %#v", reporter.event, wantEvent)
  • Files reviewed: 38/42 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

Copilot AI review requested due to automatic review settings September 2, 2026 06:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The Agents test derives its expected schema from the production builder and therefore cannot detect an accidental schema change.

Review details

Suppressed comments (1)

cli/azd/extensions/azure.ai.agents/internal/cmd/run_test.go:286

  • This expectation is built with the same event builder used by production, so the test still passes if that builder accidentally changes the established event name or attribute key. Assert the documented schema literals here, as the Inspector migration does, so this test protects the “schema unchanged” requirement.
			wantEvent := telemetry.LocalClientRouteSelected(tt.wantRoute)
  • Files reviewed: 38/42 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

@azure-sdk-automation

Copy link
Copy Markdown
Contributor

azure.ai.agents PR build

Note

This is an unsigned development build. Install it only if you trust this PR.

Install the extension:

azd ext install "https://azuresdkartifacts.z5.web.core.windows.net/azd/extensions/pr/9841/azure-ai-agents.zip"
  • Version: 1.0.0-beta.13.pr.9841.6776708
  • Merge commit: d626220

@azure-sdk-automation

Copy link
Copy Markdown
Contributor

azure.ai.skills PR build

Note

This is an unsigned development build. Install it only if you trust this PR.

Install the extension:

azd ext install "https://azuresdkartifacts.z5.web.core.windows.net/azd/extensions/pr/9841/azure-ai-skills.zip"
  • Version: 1.0.0-beta.4.pr.9841.6776741
  • Merge commit: bd76a37

@azure-sdk-automation

Copy link
Copy Markdown
Contributor

azure.ai.connections PR build

Note

This is an unsigned development build. Install it only if you trust this PR.

Install the extension:

azd ext install "https://azuresdkartifacts.z5.web.core.windows.net/azd/extensions/pr/9841/azure-ai-connections.zip"
  • Version: 1.0.0-beta.5.pr.9841.6776733
  • Merge commit: bd76a37

@azure-sdk-automation

Copy link
Copy Markdown
Contributor

azure.ai.projects PR build

Note

This is an unsigned development build. Install it only if you trust this PR.

Install the extension:

azd ext install "https://azuresdkartifacts.z5.web.core.windows.net/azd/extensions/pr/9841/azure-ai-projects.zip"
  • Version: 1.0.0-beta.8.pr.9841.6776737
  • Merge commit: bd76a37

@azure-sdk-automation

Copy link
Copy Markdown
Contributor

azure.ai.inspector PR build

Note

This is an unsigned development build. Install it only if you trust this PR.

Install the extension:

azd ext install "https://azuresdkartifacts.z5.web.core.windows.net/azd/extensions/pr/9841/azure-ai-inspector.zip"
  • Version: 1.0.0-beta.5.pr.9841.6776735
  • Merge commit: bd76a37

@azure-sdk-automation

Copy link
Copy Markdown
Contributor

azure.ai.toolboxes PR build

Note

This is an unsigned development build. Install it only if you trust this PR.

Install the extension:

azd ext install "https://azuresdkartifacts.z5.web.core.windows.net/azd/extensions/pr/9841/azure-ai-toolboxes.zip"
  • Version: 1.0.0-beta.5.pr.9841.6776742
  • Merge commit: bd76a37

@azure-sdk-automation

Copy link
Copy Markdown
Contributor

azure.ai.routines PR build

Note

This is an unsigned development build. Install it only if you trust this PR.

Install the extension:

azd ext install "https://azuresdkartifacts.z5.web.core.windows.net/azd/extensions/pr/9841/azure-ai-routines.zip"
  • Version: 1.0.0-beta.4.pr.9841.6776739
  • Merge commit: bd76a37

@huimiu

Copy link
Copy Markdown
Member

Overall looks good to me

@@ -0,0 +1,101 @@
// Copyright (c) Microsoft Corporation. All rights reserved.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be moved somewhere common, like the azd/pkg/foundry package? We've been talking about pulling that package out of core and into something we can manage independently, but even without that being done yet I'd much rather have it somewhere that is shared instead of copied to every extension and that has to be updated 7+ times

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ext-agents azure.ai.agents extension ext-connections azure.ai.connections extension ext-foundry azure.ai.{agents,connections,inspector,projects,routines,skills,toolboxes}, microsoft.foundry ext-inspector azure.ai.inspector extension ext-projects azure.ai.projects extension ext-routines azure.ai.routines extension ext-skills azure.ai.skills extension ext-toolboxes azure.ai.toolboxes extension

Projects

None yet

Development

Successfully merging this pull request may close these issues.

refactor: standardize telemetry reporting across Foundry extensions

4 participants