Optimize SPFx release pages for agentic upgrade workflows#10921
Optimize SPFx release pages for agentic upgrade workflows#10921garrytrinder wants to merge 1 commit into
Conversation
|
Learn Build status updates of commit 90e3802: ❌ Validation status: errorsPlease follow instructions here which may help to resolve issue.
For more details, please refer to the build report. Note: Your PR may contain errors or warnings or suggestions unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them. |
|
Learn Build status updates of commit 90e3802: ❌ Validation status: errorsPlease follow instructions here which may help to resolve issue.
For more details, please refer to the build report. Note: Your PR may contain errors or warnings or suggestions unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them. |
garrytrinder
force-pushed
the
garrytrinder-replace-cli-upgrade-tips-with-warning
branch
from
90e3802 to
1dd128e
Compare
PoliCheck Scan ReportThe following report lists PoliCheck issues in PR files. Before you merge the PR, you must fix all severity-1 and severity-2 issues. The AI Review Details column lists suggestions for either removing or replacing the terms. If you find a false positive result, mention it in a PR comment and include this text: #policheck-false-positive. This feedback helps reduce false positives in future scans. ✅ No issues foundMore information about PoliCheckInformation: PoliCheck | Severity Guidance | Term |
|
Learn Build status updates of commit 1dd128e:
|
PoliCheck Scan ReportThe following report lists PoliCheck issues in PR files. Before you merge the PR, you must fix all severity-1 and severity-2 issues. The AI Review Details column lists suggestions for either removing or replacing the terms. If you find a false positive result, mention it in a PR comment and include this text: #policheck-false-positive. This feedback helps reduce false positives in future scans. ✅ No issues foundMore information about PoliCheckInformation: PoliCheck | Severity Guidance | Term |
|
Learn Build status updates of commit f2ddca8:
|
PoliCheck Scan ReportThe following report lists PoliCheck issues in PR files. Before you merge the PR, you must fix all severity-1 and severity-2 issues. The AI Review Details column lists suggestions for either removing or replacing the terms. If you find a false positive result, mention it in a PR comment and include this text: #policheck-false-positive. This feedback helps reduce false positives in future scans. ✅ No issues foundMore information about PoliCheckInformation: PoliCheck | Severity Guidance | Term |
PoliCheck Scan ReportThe following report lists PoliCheck issues in PR files. Before you merge the PR, you must fix all severity-1 and severity-2 issues. The AI Review Details column lists suggestions for either removing or replacing the terms. If you find a false positive result, mention it in a PR comment and include this text: #policheck-false-positive. This feedback helps reduce false positives in future scans. ✅ No issues foundMore information about PoliCheckInformation: PoliCheck | Severity Guidance | Term |
|
Learn Build status updates of commit c9853a9: 💡 Validation status: suggestions
This comment lists only the first 25 files in the pull request. docs/spfx/release-1.22.0.md
For more details, please refer to the build report. Note: Your PR may contain errors or warnings or suggestions unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them. |
|
Set to draft, found some inconsistencies that I need to investigate. |
PoliCheck Scan ReportThe following report lists PoliCheck issues in PR files. Before you merge the PR, you must fix all severity-1 and severity-2 issues. The AI Review Details column lists suggestions for either removing or replacing the terms. If you find a false positive result, mention it in a PR comment and include this text: #policheck-false-positive. This feedback helps reduce false positives in future scans. ✅ No issues foundMore information about PoliCheckInformation: PoliCheck | Severity Guidance | Term |
|
Learn Build status updates of commit 520e4c6: 💡 Validation status: suggestions
This comment lists only the first 25 files in the pull request. docs/spfx/release-1.22.0.md
For more details, please refer to the build report. Note: Your PR may contain errors or warnings or suggestions unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them. |
PoliCheck Scan ReportThe following report lists PoliCheck issues in PR files. Before you merge the PR, you must fix all severity-1 and severity-2 issues. The AI Review Details column lists suggestions for either removing or replacing the terms. If you find a false positive result, mention it in a PR comment and include this text: #policheck-false-positive. This feedback helps reduce false positives in future scans. ✅ No issues foundMore information about PoliCheckInformation: PoliCheck | Severity Guidance | Term |
|
Learn Build status updates of commit 1d58183: ✅ Validation status: passed
This comment lists only the first 25 files in the pull request. |
garrytrinder
changed the title
andrewconnell
left a comment
andrewconnell
left a comment
There was a problem hiding this comment.
WRT one file in this PR: docs/spfx/toolchain/migrate-gulptoolchain-hefttoolchain.md ...
While there is value in providing the CLI upgrade steps, but want to get approval from @VesaJuvonen about removing all detailed steps & documentation on what changed.
Some companies can't use open source/community tools, while others can't unless it goes through a rigorous security review. Only providing instructions for a tool that some people might not be allowed to use removes a resource. IMHO, both options should be present, not just the CLI option.
Flagging & requesting @VesaJuvonen review to make the decsision.
|
|
||
| ## Upgrading projects from the SPFx v1.21.1 to v1.22.0 version | ||
|
|
||
| The upgrade steps required to convert a [gulp-based toolchain](toolchain/sharepoint-framework-toolchain.md) SPFx project to the [Heft-based toolchain](toolchain/sharepoint-framework-toolchain-rushstack-heft.md) are detailed in the following article: [Migrate from the Gulp Toolchain to Heft Toolchain](toolchain/migrate-gulptoolchain-hefttoolchain.md). |
There was a problem hiding this comment.
Pending peer review by @VesaJuvonen, if we keep the detailed steps in the migration doc, linking to the migration doc with a callout that it covers detailed steps should remain.
There was a problem hiding this comment.
These are good to get to the release notes - based on the evals on the agentic testing scenarios. It does cause duplicate work but the challenge has been for agents to find the required guidance and based on testing - this would resolve the issue.
| [!INCLUDE [pnp-o365cli](../../includes/snippets/open-source/pnp-o365cli.md)] | ||
|
|
||
| Starting with SPFx v1.22.0, new projects use Heft as the build task orchestrator instead of a [gulp-based toolchain](./toolchain/sharepoint-framework-toolchain.md) used in SPFx v1.0 - v1.21.1 releases. This change represents a fundamental shift in how SPFx projects are built, configured, and customized, though the underlying bundling technology (webpack) remains the same. | ||
| For an overview of what changes during this migration from the [Gulp-based toolchain](toolchain/sharepoint-framework-toolchain.md) to the [Heft-based toolchain](toolchain/sharepoint-framework-toolchain-rushstack-heft.md), see: [Migrate from the Gulp Toolchain to Heft Toolchain](toolchain/migrate-gulptoolchain-hefttoolchain.md). |
There was a problem hiding this comment.
Pending peer review by @VesaJuvonen for the big change in the migration doc, if we keep the detailed steps, this should be reverted as the previous edit in this file will make this repetitive.
andrewconnell
added
pr:awaiting-peer-reviews
|
@andrewconnell - badly communicated from my side. This is for improving agents in the upgrade scenarios and based on detailed testing with multiple models we currently have a challenge with the upgrades and agent(s) start doing wrong things. Calling out the challenge of the CLI as open-source is valid and we might want to adjust that embedded message a bit to be calling that out in same ways as we always callout PnP tooling to be officially not supported by Microsoft, even though it's mentioned. We do not really have good upgrade guidance otherwise except the CLI tooling - and when it's used - upgrade results were significantly better with agents. So - on the TIP - let's be specific that it's an option, but not the only way - as that would be wrong as well. Include has the open-source note already nicely. |
PoliCheck Scan ReportThe following report lists PoliCheck issues in PR files. Before you merge the PR, you must fix all severity-1 and severity-2 issues. The AI Review Details column lists suggestions for either removing or replacing the terms. If you find a false positive result, mention it in a PR comment and include this text: #policheck-false-positive. This feedback helps reduce false positives in future scans. ✅ No issues foundMore information about PoliCheckInformation: PoliCheck | Severity Guidance | Term |
|
Learn Build status updates of commit 71c2153: ❌ Validation status: errorsPlease follow instructions here which may help to resolve issue.
docs/spfx/release-1.23.2.md
For more details, please refer to the build report. Note: Your PR may contain errors or warnings or suggestions unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them. |
garrytrinder
force-pushed
the
garrytrinder-replace-cli-upgrade-tips-with-warning
branch
from
71c2153 to
f41ca2e
Compare
|
Learn Build status updates of commit f41ca2e: ✅ Validation status: passed
This comment lists only the first 25 files in the pull request. |
PoliCheck Scan ReportThe following report lists PoliCheck issues in PR files. Before you merge the PR, you must fix all severity-1 and severity-2 issues. The AI Review Details column lists suggestions for either removing or replacing the terms. If you find a false positive result, mention it in a PR comment and include this text: #policheck-false-positive. This feedback helps reduce false positives in future scans. ✅ No issues foundMore information about PoliCheckInformation: PoliCheck | Severity Guidance | Term |
- Replace shared TIP with WARNING about upgrade complexity - Add version-specific TIP with npx upgrade command to each release page - Add CLI for Microsoft 365 community disclaimer (pnp-o365cli snippet) - Rename snippet from spfx-release-upgrade-tip to spfx-release-upgrade-warning - Rewrite migrate-gulptoolchain-hefttoolchain.md for CLI-driven approach - Restructure release-1.22.0 upgrade section - Fix broken bookmark in release-1.15.2 (azure-ad -> entra-id) - Fix absolute link in release-1.22.0 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
garrytrinder
force-pushed
the
garrytrinder-replace-cli-upgrade-tips-with-warning
branch
from
f41ca2e to
2937ceb
Compare
PoliCheck Scan ReportThe following report lists PoliCheck issues in PR files. Before you merge the PR, you must fix all severity-1 and severity-2 issues. The AI Review Details column lists suggestions for either removing or replacing the terms. If you find a false positive result, mention it in a PR comment and include this text: #policheck-false-positive. This feedback helps reduce false positives in future scans. ✅ No issues foundMore information about PoliCheckInformation: PoliCheck | Severity Guidance | Term |
|
Learn Build status updates of commit 2937ceb:
|
| File | Status | Preview URL | Details |
|---|---|---|---|
| docs/spfx/release-1.15.2.md | View | Details | |
| docs/spfx/release-1.22.0.md | 💡Suggestion | View | Details |
| docs/spfx/release-0.0.0.md.template | ✅Succeeded | ||
| docs/spfx/release-1.0.0.md | ✅Succeeded | View | |
| docs/spfx/release-1.1.0.md | ✅Succeeded | View | |
| docs/spfx/release-1.10.0.md | ✅Succeeded | View | |
| docs/spfx/release-1.11.0.md | ✅Succeeded | View | |
| docs/spfx/release-1.12.1.md | ✅Succeeded | View | |
| docs/spfx/release-1.13.0.md | ✅Succeeded | View | |
| docs/spfx/release-1.13.1.md | ✅Succeeded | View | |
| docs/spfx/release-1.14.0.md | ✅Succeeded | View | |
| docs/spfx/release-1.15.0.md | ✅Succeeded | View | |
| docs/spfx/release-1.16.0.md | ✅Succeeded | View | |
| docs/spfx/release-1.16.1.md | ✅Succeeded | View | |
| docs/spfx/release-1.17.0.md | ✅Succeeded | View | |
| docs/spfx/release-1.17.1.md | ✅Succeeded | View | |
| docs/spfx/release-1.17.2.md | ✅Succeeded | View | |
| docs/spfx/release-1.17.3.md | ✅Succeeded | View | |
| docs/spfx/release-1.17.4.md | ✅Succeeded | View | |
| docs/spfx/release-1.18.0.md | ✅Succeeded | View | |
| docs/spfx/release-1.18.1.md | ✅Succeeded | View | |
| docs/spfx/release-1.18.2.md | ✅Succeeded | View | |
| docs/spfx/release-1.19.0.md | ✅Succeeded | View | |
| docs/spfx/release-1.20.0.md | ✅Succeeded | View | |
| docs/spfx/release-1.21.0.md | ✅Succeeded | View |
This comment lists only the first 25 files in the pull request.
docs/spfx/release-1.15.2.md
- Line 64, Column 44: [Warning: bookmark-not-found - See documentation]
Cannot find bookmark '#requesting-permissions-to-azure-ad-applications-in-another-tenant' in 'spfx/use-aadhttpclient.md', did you mean '#requesting-permissions-to-entra-id-applications-in-another-tenant'?
docs/spfx/release-1.22.0.md
- Line 12, Column 3: [Suggestion: docs-link-absolute - See documentation]
Absolute link 'https://learn.microsoft.com/javascript/api/overview/sharepoint?view=sp-typescript-latest' will be broken in isolated environments. Replace with a relative link. - Line 12, Column 3: [Suggestion: preserve-view-not-set - See documentation]
You've pinned this link to a specific version of content with the view parameter. It's recommended not to pin a version unless that version is A) not the default view and B) the context is about that version specifically. To proceed with pinning a version add the &preserve-view=true to the URL. Otherwise, remove the view parameter. URL: https://learn.microsoft.com/javascript/api/overview/sharepoint?view=sp-typescript-latest
For more details, please refer to the build report.
Note: Your PR may contain errors or warnings or suggestions unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
|
Thanks @andrewconnell and @VesaJuvonen for the review. Findings from testingBefore discussing next steps, I want to share the reasoning behind the changes in this PR, because none of them were arbitrary. How agents approach upgradesFrom our trajectory analysis, agents form their upgrade plan before they read any documentation. By the time the agent fetches the release notes or migration guide, it's already decided on a manual approach — read package.json, work out what needs changing, apply edits. It fetches docs to confirm versions, not to discover an approach. The 1.22.0 release notes page links to the Gulp-to-Heft migration guide. When an agent reads the 1.22.0 page, it follows that link because a detailed migration guide aligns with its pre-formed manual plan. Once on the migration page, the detailed step-by-step instructions confirm the manual approach and the agent ignores the CLI recommendation from the release page — we tested this and upgrade quality was significantly worse than when the agent used the CLI. This is why the changes in this PR work as a combined intervention. The WARNING challenges the agent's pre-formed manual plan. The TIP gives it a concrete alternative. And the conceptual migration page reinforces the CLI recommendation when the agent follows the link, instead of providing manual steps that let it revert to its original plan. The TIP wording matters too. Framing the CLI as optional ("You can use...") scored 0/5 across 7 scenarios (35 runs). When the agent sees two optional approaches, it picks the one that aligns with its pre-formed manual plan — every time. Why the CLI produces better upgrade outcomesThe CLI computes the full delta for an upgrade: it detects the source version, calculates every change across each intermediate version, and produces a complete, project-specific report. Given the prompt "upgrade to v1.22.2", agents consistently fetch two release notes pages — DecisionThe core question is: should SPFx upgrades performed by agents be driven by manual steps from documentation, or by the CLI? Based on our results:
ActionsDone
Needs decisionIf the decision is CLI-driven upgrades: the PR as-is is the right shape. @andrewconnell's concern about companies that can't use open-source tools is valid — the manual steps could live on a separate reference page rather than being removed entirely, so they're available without competing with the CLI recommendation on the migration page. If the decision is documentation-driven upgrades: the PR changes to the migration page and TIP wording should be reverted, and the investment shifts to making the docs comprehensive enough to cover multi-version upgrade paths — covering all incremental changes and guiding agents through chaining version-specific docs together. |
3 participants
Summary
Updates all SPFx release notes pages and the Gulp-to-Heft migration guide to improve upgrade guidance by directing developers to use CLI for Microsoft 365.
Changes
All release pages (40 files)
package.jsonalone will cause build failures.npxcommand to runm365 spfx project upgrade --toVersion <version>for each release page.pnp-o365clisnippet) clarifying CLI for Microsoft 365 is open-source with no Microsoft SLA.spfx-release-upgrade-tip.md→spfx-release-upgrade-warning.mdto reflect its new content.docs/spfx/release-1.22.0.mddocs/spfx/toolchain/migrate-gulptoolchain-hefttoolchain.mdm365 spfx project upgradeto generate the full set of changes automatically.