You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Adds diagnostics to the symbol publishing path so that the next failure explains itself from the build log, rather than requiring an investigation to reconstruct what happened.
This started as an investigation into symbol publishing failing against the PPE server. That is now resolved and no change in this repository was needed to fix it — see Root cause below. What remains here is the instrumentation, kept deliberately: the failure took far longer to diagnose than it should have, largely because the publish script discarded the information that would have identified it.
Root cause
Symbol publishing to symbolrequestppe.trafficmanager.net began failing on 2026-09-17 with a bare Permission denied, while Production publishing continued to work.
The cause was outside this repository. The 1ES Linux network isolation policy moved from Permissive to DefaultDeny:
The change landed between builds 26251.1 (2026-09-08, last Permissive) and 26252.1 (2026-09-09, first DefaultDeny). symbolrequestppe.trafficmanager.net is not in the 1ES endpoint allowlist, so under default-deny the TCP connection is refused with EACCES before any HTTP exchange occurs. symbolrequestprod.trafficmanager.net is allowlisted and continued to work.
Supporting observations:
The task is 1ESNetworkIsolation-**linux**-x64, so the Windows container is unaffected. The SNI pipeline published to the same PPE endpoint from windows_build_container 27 seconds before one of our Linux failures (build 176140 vs 175832).
The agent image version is identical across passing and failing runs (20260824.1.170484886), and the same pools and regions appear on both sides. Identity, token audience and symbol project were unchanged throughout.
The first PPE attempt after the policy change was 2026-09-17, which is why the failure date and the policy date differ.
A request to add the PPE endpoint to the 1ES allowlist is being raised separately with 1ES/CloudBuild.
Why these changes, then
Permission denied is the EACCES errno text. It is indistinguishable, in a log, from an application-level authorization denial — and it was misread as exactly that for some time. The publish script made this worse: Invoke-RestMethod surfaces only a message, and the script discarded the status code, the response body, the correlation identifiers and the exception type. There was no way to tell "the service rejected us" from "we never reached the service".
These changes make that distinction visible, and are intended as defensive instrumentation for the next incident rather than a fix for this one.
What was added
Always emitted:
Executed commands — the az token acquisition and each Invoke-RestMethod call, with method, URI, headers and body. Secrets are redacted by default.
Token identity — SHA-256 fingerprint plus allow-listed JWT header and payload claims (aud, iss, tid, appid, oid, idtyp, roles, exp, …). These identify the calling principal and audience without exposing the credential. The signature segment is never decoded.
HTTP failure detail — status code, response body, and correlation identifiers such as mise-correlation-id, plus the exception and inner-exception types. When no response was received it says so explicitly:
Exception: System.Net.Http.HttpRequestException
Inner exception: System.Net.Sockets.SocketException: Permission denied
HTTP status: <none - no HTTP response was received>
That single line is what finally separated an egress block from a service rejection.
Under the existing debug parameter:
Unredacted commands — -LogUnredactedCommands, so the real token and Authorization header can be inspected. Emits a warning that the log contains secrets.
Reachability probe — test-endpoint-reachability.ps1 runs before the publish request and probes both symbol servers from inside the build container, separating DNS resolution from TCP connect and reporting socket error codes verbatim:
sqlclient-non-official 26266.6 exercises this branch on the ordinary publish path — publishSymbols=true, debug=true, no special mode. The PDBs upload successfully and only the publish request is refused, so the symbols end up staged but unpublishable.
The probe, from inside the publishing job:
=== symbolrequestppe.trafficmanager.net:443 ===
DNS : 20.119.8.23
TCP : by name -> FAILED in 117 ms - AccessDenied (errno 13): Permission denied
=== symbolrequestprod.trafficmanager.net:443 ===
DNS : 40.112.243.108
TCP : by name -> connected in 66 ms - connected
=== Summary ===
BLOCKED symbolrequestppe.trafficmanager.net - AccessDenied (errno 13): Permission denied
REACHABLE symbolrequestprod.trafficmanager.net - connected
Before this change the same failure read, in its entirety, Error: Permission denied.
For contrast, 26266.4 shows the token diagnostics that are emitted on every run, including the JWT claims identifying the calling principal and audience.
Notes for reviewers
The publish path is unchanged. No change to how requests are authenticated, constructed or sent. The template diffs are additive; the only behavioural change is the new debug-gated probe step.
debug now also exposes the bearer token. Its label is updated to Enable debug output (may expose secrets) with a warning comment. Agent-side scrubbing of the token is best-effort and should not be relied on.
This root cause was already in our logs. Every failing run's 🔒 Stop Network Isolation step reported Blocked connections: symbolrequestppe.trafficmanager.net and Policy: Default Deny, from the first failure onward. Reading sibling steps in a failing job, not just the failing step, would have found it immediately. The diagnostics added here are still worthwhile, but the more valuable lesson was cheaper than the instrumentation.
Checklist
Tests added or updated (122 Pester tests pass)
Public API changes documented (n/a — pipeline-only change)
Verified against customer repro (build 177611 on the ordinary publish path; 177314, 177588, 177593 during investigation)
Symbol publishing to the PPE server began failing with a bare "Permission
denied". That text is the EACCES errno, and in a log it is indistinguishable
from an application-level authorization denial. The publish script made the
ambiguity worse: Invoke-RestMethod surfaces only a message, and the script
discarded the status code, response body, correlation identifiers and
exception type, so there was no way to tell "the service rejected us" from
"we never reached the service".
The cause turned out to be outside this repository. The 1ES Linux network
isolation policy moved from Permissive to DefaultDeny between builds 26251.1
and 26252.1, and symbolrequestppe.trafficmanager.net is not in the endpoint
allowlist, so the connection is refused before any HTTP exchange occurs. The
Production endpoint is allowlisted and was unaffected, as was the Windows
container, since the task is linux-only. That is being raised with 1ES
separately; nothing here caused it and nothing here can fix it.
These diagnostics are kept anyway, for the next time symbol publishing
misbehaves.
Always emitted:
- the commands executed, covering the token acquisition and each REST
call, with secrets redacted;
- the token's SHA-256 fingerprint and its allow-listed JWT header and
payload claims, which identify the calling principal and audience
without exposing the credential. The signature segment is never decoded;
- the HTTP status, response body, correlation identifiers and exception
types on failure, stating explicitly when no response was received. That
distinction is what separates an egress block from a service rejection.
Under the existing debug parameter:
- the same commands with the bearer token and Authorization header
unredacted, preceded by a warning that the log contains secrets;
- a reachability probe that tests both symbol servers from inside the
build container, separating DNS resolution from TCP connect and
reporting socket error codes verbatim.
The publish path itself is unchanged: how requests are authenticated,
constructed and sent is the same, and the template changes are additive
aside from the new debug-gated probe step.
Worth recording: every failing run already reported this in its Stop Network
Isolation step, naming the blocked domain and the Default Deny policy, from
the first failure onward. Reading the sibling steps of a failing job, rather
than only the failing step, would have identified it immediately.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The new response-body extraction is not covered by the added Pester cases: the HTTP mock verifies status and a correlation header, but no test supplies ErrorDetails.Message and asserts the Response body: diagnostic. Because this exception shape varies by PowerShell version and the response body is a stated goal of the change, add a focused failure test for it.
Clarify never-throws contract for endpoint failures versus invalid input
The “never throws” contract contradicts the explicit throw for an empty parsed host list at line 64 (and parameter binding can also fail). Clarify that endpoint failures are non-terminating while invalid invocation still fails.
# A port on loopback that nothing is listening on.
$script:closedPort = 9
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Area\EngineeringUse this for issues that are targeted for changes in the 'eng' folder or build systems.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
paulmedynski
added
the
Area\Engineering
Summary
Adds diagnostics to the symbol publishing path so that the next failure explains itself from the build log, rather than requiring an investigation to reconstruct what happened.
This started as an investigation into symbol publishing failing against the PPE server. That is now resolved and no change in this repository was needed to fix it — see Root cause below. What remains here is the instrumentation, kept deliberately: the failure took far longer to diagnose than it should have, largely because the publish script discarded the information that would have identified it.
Root cause
Symbol publishing to
symbolrequestppe.trafficmanager.netbegan failing on 2026-09-17 with a barePermission denied, while Production publishing continued to work.The cause was outside this repository. The 1ES Linux network isolation policy moved from
PermissivetoDefaultDeny:The change landed between builds 26251.1 (2026-09-08, last
Permissive) and 26252.1 (2026-09-09, firstDefaultDeny).symbolrequestppe.trafficmanager.netis not in the 1ES endpoint allowlist, so under default-deny the TCP connection is refused withEACCESbefore any HTTP exchange occurs.symbolrequestprod.trafficmanager.netis allowlisted and continued to work.Supporting observations:
1ESNetworkIsolation-**linux**-x64, so the Windows container is unaffected. The SNI pipeline published to the same PPE endpoint fromwindows_build_container27 seconds before one of our Linux failures (build 176140 vs 175832).20260824.1.170484886), and the same pools and regions appear on both sides. Identity, token audience and symbol project were unchanged throughout.A request to add the PPE endpoint to the 1ES allowlist is being raised separately with 1ES/CloudBuild.
Why these changes, then
Permission deniedis theEACCESerrno text. It is indistinguishable, in a log, from an application-level authorization denial — and it was misread as exactly that for some time. The publish script made this worse:Invoke-RestMethodsurfaces only a message, and the script discarded the status code, the response body, the correlation identifiers and the exception type. There was no way to tell "the service rejected us" from "we never reached the service".These changes make that distinction visible, and are intended as defensive instrumentation for the next incident rather than a fix for this one.
What was added
Always emitted:
Executed commands — the
aztoken acquisition and eachInvoke-RestMethodcall, with method, URI, headers and body. Secrets are redacted by default.Token identity — SHA-256 fingerprint plus allow-listed JWT header and payload claims (
aud,iss,tid,appid,oid,idtyp,roles,exp, …). These identify the calling principal and audience without exposing the credential. The signature segment is never decoded.HTTP failure detail — status code, response body, and correlation identifiers such as
mise-correlation-id, plus the exception and inner-exception types. When no response was received it says so explicitly:That single line is what finally separated an egress block from a service rejection.
Under the existing
debugparameter:Unredacted commands —
-LogUnredactedCommands, so the real token andAuthorizationheader can be inspected. Emits a warning that the log contains secrets.Reachability probe —
test-endpoint-reachability.ps1runs before the publish request and probes both symbol servers from inside the build container, separating DNS resolution from TCP connect and reporting socket error codes verbatim:Example: these diagnostics in a real run
sqlclient-non-official 26266.6 exercises this branch on the ordinary publish path —
publishSymbols=true,debug=true, no special mode. The PDBs upload successfully and only the publish request is refused, so the symbols end up staged but unpublishable.The probe, from inside the publishing job:
Then the publish request itself:
Before this change the same failure read, in its entirety,
Error: Permission denied.For contrast, 26266.4 shows the token diagnostics that are emitted on every run, including the JWT claims identifying the calling principal and audience.
Notes for reviewers
debug-gated probe step.debugnow also exposes the bearer token. Its label is updated toEnable debug output (may expose secrets)with a warning comment. Agent-side scrubbing of the token is best-effort and should not be relied on.🔒 Stop Network Isolationstep reportedBlocked connections: symbolrequestppe.trafficmanager.netandPolicy: Default Deny, from the first failure onward. Reading sibling steps in a failing job, not just the failing step, would have found it immediately. The diagnostics added here are still worthwhile, but the more valuable lesson was cheaper than the instrumentation.Checklist