Add soft-gate PR risk scan automation for agentic PRs#1969
Open
aaronpowell wants to merge 9 commits into
Open
Conversation
Introduce a PR risk scanner script plus two workflows: one to scan changed files and upload findings, and one to upsert a sticky PR comment with a summary table and findings. This adds non-blocking supply-chain risk visibility for agentic contributions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
github-actions
Bot
added
new-submission
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new “soft-gate” PR automation that scans agentic contribution paths for supply-chain / unsafe-execution patterns, uploads a structured results artifact, and posts (or updates) an idempotent PR comment summarizing findings.
Changes:
- Introduces
eng/pr-risk-scan.mjsto scan changed files for a small set of risk heuristics and emitresults.json+report.md. - Adds a PR gate workflow to run the scanner and upload the results artifact.
- Adds a follow-on
workflow_runworkflow to download the artifact and upsert a marker-based PR comment.
Show a summary per file
| File | Description |
|---|---|
eng/pr-risk-scan.mjs |
Implements the risk scan rules, report generation, and JSON output. |
.github/workflows/pr-risk-scan.yml |
Runs the scanner on relevant PRs and uploads the scan artifact. |
.github/workflows/pr-risk-scan-comment.yml |
Downloads the artifact and upserts a single PR comment with the report. |
Copilot's findings
- Files reviewed: 3/3 changed files
- Comments generated: 2
Reject absolute paths, enforce repo-root containment after resolution, and tighten unpinned-version detection to dependency/version contexts to avoid markdown noise. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Comment on lines
+208
to
+217
| function scanSkillScriptPath(filePath, findings) { | ||
| const normalized = filePath.replace(/\\/g, "/"); | ||
| if (!normalized.startsWith("skills/")) { | ||
| return; | ||
| } | ||
|
|
||
| const extension = path.extname(normalized).toLowerCase(); | ||
| if (!SCRIPT_EXTENSIONS.has(extension)) { | ||
| return; | ||
| } |
Comment on lines
+46
to
+51
| - name: Upload scan artifact | ||
| uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7 | ||
| with: | ||
| name: pr-risk-scan-results | ||
| path: pr-risk-results/ | ||
| retention-days: 1 |
Comment on lines
+15
to
+18
| permissions: | ||
| contents: read | ||
| actions: read | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Checklist
npm startand verified thatREADME.mdis up to date.stagedbranch for this pull request.Description
This adds a non-blocking supply-chain risk signal for PRs that touch agentic contribution paths. The goal is to surface risky patterns early (guardrail bypass language, remote shell execution, dynamic package execution, and skill script additions) without blocking contributor flow.
The change introduces a scanner script (
eng/pr-risk-scan.mjs) plus two GitHub workflows. The gate workflow scans changed PR files and uploads a structured artifact (results.json+report.md). A follow-on comment workflow runs with write permissions, reads only that artifact, and upserts a single marker-based PR comment containing severity counts and a findings table.The workflows are intentionally soft-gate in v1: findings are visible and repeatable, but do not fail the PR. This keeps false-positive tuning practical before any stricter enforcement.
Type of Contribution
eng/.Additional Notes
.github/workflows/pr-risk-scan.yml.github/workflows/pr-risk-scan-comment.ymleng/pr-risk-scan.mjs<!-- pr-risk-scan-results -->to ensure idempotent updates.skill-script-addedsignal and one noisy unpinned-version match, which we can tighten in a follow-up.By submitting this pull request, I confirm that my contribution abides by the Code of Conduct and will be licensed under the MIT License.