Feature/4 implement skill pr review by miroslavpojer · Pull Request #6 · AbsaOSS/agentic-toolkit

miroslavpojer · 2026-05-21T11:39:06Z

Add `pr-review` skill

Adds a unified pull request review skill that applies structured, risk-aware code review based on the files a PR touches. Produces concise Blocker / Important / Nit comments grouped by severity.

Updated README.md — added pr-review to the Skill Catalog table
Updated docs/README.md — added pr-review to the Skill Guides table
Added docs/pr-review.md — end-user guide covering sections, trigger phrases, helpers, and install

Release Notes

Added skills/pr-review/SKILL.md — full review instructions covering standard, API contracts, dependency bumps, CI/CD, infrastructure, DB migrations, and elevated-risk sections
Added skills/pr-review/references/output-template.md — canonical output format examples for reviews
Added skills/pr-review/references/security-antipatterns.md — security patterns the skill actively scans for
Added skills/pr-review/scripts/fetch_pr.sh — fetches PR diff and file list via gh
Added skills/pr-review/scripts/classify_sections.py — determines which review sections apply from a file list
Added skills/pr-review/evals/ — trigger eval, fixture map, eval fixtures, and results summary

Closes #4

…DME and documentation for clarity

…nes and skill overview

…pts for PR review - Created `results-summary.md` to document evaluation results for the pr-review skill, detailing iterations, fixes, and overall trajectory. - Added `trigger-eval.json` containing a set of queries for evaluating trigger conditions, including both triggering and non-triggering examples. - Introduced `output-template.md` for standardized PR review output examples, outlining formatting rules and section guidelines. - Developed `security-antipatterns.md` as a reference for identifying security issues during PR reviews, including common patterns and severity guidance. - Implemented `classify_sections.py` to classify changed files into relevant PR review sections based on defined patterns. - Created `fetch_pr.sh` script to fetch PR details and changed files using the GitHub CLI, facilitating integration with the classification script.

…l definitions

…ew skill definition evals

lsulak · 2026-05-26T08:18:16Z

@@ -0,0 +1,36 @@
+# Sprint 23 — User Service Changelog


why also the token saving skill in the PR?

This PR is build on top of previous as I am doing also folder moves.
Let's review.merge token-saving skill first, pls.

…-skill-pr-review

Copilot

Pull request overview

Adds a new pr-review skill to the toolkit, including structured review instructions, helper scripts to fetch/classify PR context, reference templates, and an eval suite; also updates documentation indexes and introduces a workflow to enforce release-notes presence in PR descriptions.

Changes:

Introduces skills/pr-review/ with SKILL definition, references, scripts, and eval fixtures/results.
Adds end-user documentation for the new skill and updates README indexes/catalogs.
Adds a GitHub Actions workflow to check for “Release Notes” in PR descriptions.

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
`.github/copilot-instructions.md`	Points PR review flow at the new `pr-review` skill (removes duplicated inline checks).
`.github/workflows/check_pr_release_notes.yml`	Adds a workflow to require release notes presence in PR descriptions.
`README.md`	Adds `pr-review` to the Skill Catalog.
`docs/README.md`	Adds `pr-review` to the docs index tables and normalizes table formatting.
`docs/getting-started.md`	Minor formatting/line-wrapping cleanup.
`docs/pr-review.md`	New user guide explaining purpose, sections, triggers, helpers, and installation.
`skills/pr-review/SKILL.md`	Core PR review skill definition and structured review checklist/format rules.
`skills/pr-review/evals/evals.json`	Adds eval suite definitions for the skill (fixtures + expectations).
`skills/pr-review/evals/fixture-map.md`	Maps eval IDs to fixture diffs and summarizes coverage.
`skills/pr-review/evals/results-summary.md`	Documents evaluation iterations and outcomes.
`skills/pr-review/evals/trigger-eval.json`	Adds trigger/negative-trigger queries to validate skill activation boundaries.
`skills/pr-review/evals/files/api-rename.diff`	Eval fixture for API contract rename scenario.
`skills/pr-review/evals/files/ci-gate-bypass.diff`	Eval fixture for CI gate bypass + secrets in workflow scenario.
`skills/pr-review/evals/files/db-migration-risks.diff`	Eval fixture for migration rollback/data-loss risk scenario.
`skills/pr-review/evals/files/dependency-bump-risk.diff`	Eval fixture for dependency bump risk scenario.
`skills/pr-review/evals/files/docs-release-notes.diff`	Eval fixture for “release notes” (non-review) negative case.
`skills/pr-review/evals/files/elevated-risk-auth-refactor.diff`	Eval fixture for elevated-risk auth change scenario.
`skills/pr-review/evals/files/iac-wildcard-iam.diff`	Eval fixture for IaC wildcard IAM scenario.
`skills/pr-review/evals/files/large-pr-and-vague-desc.diff`	Eval fixture for “large PR + vague description” scenario.
`skills/pr-review/evals/files/multi-section-risks.diff`	Eval fixture combining DB/API/CI risks in one PR.
`skills/pr-review/evals/files/skill-definition-clean.diff`	Eval fixture for a clean SKILL.md addition.
`skills/pr-review/evals/files/skill-definition-violations.diff`	Eval fixture for SKILL.md convention violations.
`skills/pr-review/evals/files/standard-clean-pr.diff`	Eval fixture for a clean “LGTM” utility PR.
`skills/pr-review/references/output-template.md`	Canonical output formatting examples for reviews.
`skills/pr-review/references/security-antipatterns.md`	Security anti-pattern reference for elevated-risk/security reviews.
`skills/pr-review/scripts/classify_sections.py`	Helper to infer which review sections apply based on changed file paths.
`skills/pr-review/scripts/fetch_pr.sh`	Helper to fetch PR description/files/diff via `gh` CLI.

Copilot

Pull request overview

Copilot reviewed 27 out of 28 changed files in this pull request and generated 15 comments.

…ile paths in evaluation documentation and scripts for consistency.

Copilot

Pull request overview

Copilot reviewed 26 out of 27 changed files in this pull request and generated 9 comments.

+Output example:
+    Sections to apply:
+      [x] Standard review       (always)
+      [x] API contracts         (router.py, schemas.py)
+      [ ] Elevated risk
+      [ ] Dependency bumps
+      [ ] CI/CD
+      [ ] Infrastructure
+"""


+        "CI/CD: skipping test_user_api tests (-k flag) flagged as quality gate bypass (Blocker)",
+        "CI/CD: deploy trigger widened from branch-scoped to all pushes flagged",
+        "Elevated-risk overlay is NOT applied \u2014 PR has no auth/security/infra/wide-refactor touches despite touching multiple sections"
+      ]    },


+        "Missing trigger keywords in description is flagged",
+        "Hardcoded absolute path is flagged",
+        "Unreferenced script is flagged"
+      ]    }


miroslavpojer added 5 commits May 20, 2026 15:41

Implement token-saving skill and related evaluation files; update REA…

677b52d

…DME and documentation for clarity

Add title parameter to release notes presence check in PR workflow

e709f12

Update documentation for token-saving skill; add contribution guideli…

f653b27

…nes and skill overview

Add token-saving skill documentation with response formatting rules

fd9f957

miroslavpojer self-assigned this May 21, 2026

miroslavpojer requested review from lsulak, oto-macenauer-absa and tmikula-dev as code owners May 21, 2026 11:39

miroslavpojer added the work in progress Work on this item is not yet finished (mainly intended for PRs) label May 21, 2026

Enhance PR review skill documentation with additional checks for skil…

3dce2b5

…l definitions

miroslavpojer added work in progress Work on this item is not yet finished (mainly intended for PRs) and removed work in progress Work on this item is not yet finished (mainly intended for PRs) labels May 22, 2026

Refactor PR review skill documentation and evaluation criteria; add n…

b808333

…ew skill definition evals

miroslavpojer removed the work in progress Work on this item is not yet finished (mainly intended for PRs) label May 22, 2026

lsulak reviewed May 26, 2026

View reviewed changes

Merge remote-tracking branch 'origin/master' into feature/4-implement…

a73b8f9

…-skill-pr-review

miroslavpojer requested a review from Copilot June 1, 2026 12:05

Copilot started reviewing on behalf of miroslavpojer June 1, 2026 12:06 View session

Copilot AI reviewed Jun 1, 2026

View reviewed changes

Fix review comments.

1a1ec7a

miroslavpojer requested a review from Copilot June 1, 2026 12:34

Copilot started reviewing on behalf of miroslavpojer June 1, 2026 12:34 View session

Copilot AI reviewed Jun 1, 2026

View reviewed changes

miroslavpojer requested review from Copilot June 1, 2026 13:54

Copilot started reviewing on behalf of miroslavpojer June 1, 2026 13:55 View session

Remove unused GitHub workflow for checking PR release notes; update f…

97a5932

…ile paths in evaluation documentation and scripts for consistency.

Copilot AI reviewed Jun 1, 2026

View reviewed changes

Conversation

miroslavpojer commented May 21, 2026

Add pr-review skill

Release Notes

Uh oh!

lsulak May 26, 2026

Choose a reason for hiding this comment

Uh oh!

miroslavpojer May 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add `pr-review` skill