[Tommy Nguyen (tido64)]

Description

Adds a CI pipeline that runs yarn eval on pull requests assisted by agents and reports the results in a sticky comment. A pull request counts as assisted when Copilot, Claude, or Cursor authored it, authored any of its commits, or co-authored any of its commits through a Co-authored-by trailer.

• Workflow (.github/workflows/agent-evals.yml): runs on pull_request_target, so the base branch's code runs and the PR's code is only read. It only reports; failed rules never fail the check. The workflow uses only GITHUB_TOKEN.

• Session logs: for Copilot cloud agent PRs, the provider fetches the session logs from the Copilot API (/agents/resource/pull/{id} and /agents/sessions/{id}/events). Those endpoints return the same event schema as a local events.jsonl.

• Fallback: if the logs can't be read, the PR is audited from its diff alone. Rules tagged requires: session are skipped and shown as -, and the comment explains how to run the audit locally.

• Agent-posted reports: yarn eval:report (in evals/) runs the evals and posts the report in a comment that the agent and the workflow share. The comment is identified by a hidden marker recording the head commit and whether the session log was read. AGENTS.md instructs agents to run it after creating or updating a PR. The workflow skips its audit if the comment already has a report with the session log for the head commit; otherwise, it posts the diff-only report and asks for a local run.

Note

This is a prototype; docs and tests are pending review. The pull_request trigger marked TEMP lets the workflow run on this PR before it is on main, and must be removed before merging.

In CI, GITHUB_TOKEN can neither read Copilot cloud agent sessions nor run the grader. The Copilot API rejects it with "400 the Copilot-Integration-Id header does not match the GitHub App that issued this installation access token". Session rules are therefore skipped (-), rubric rules are marked unchecked (!), and the comment asks for a local run.

Test plan

• yarn eval:report 4446 posted a comment with the marker, and CI then updated that same comment, even though it was posted with a user token.

• With the marker set to log=yes for the head commit, yarn eval:report --unless-reported 4446 skipped the audit.

• Locally:

  • yarn workspace @rnx-kit/evals eval --var pullRequest=4425 reads all 37 cloud sessions and audits the session rules.
  • yarn workspace @rnx-kit/evals eval --var pullRequest=4439 (a CLI co-authored PR) falls back to the diff-only audit and skips the session rules.

• In CI:

  • This PR is co-authored by Copilot, so the workflow should detect it, audit it, and post a comment.

[Tommy Nguyen (tido64)]

Agent evals

⚠️ 2 not checked · ⏭️ 11 skipped · ✅ 6 passed

⚠️ Not checked (2)

• Uses fixtures or the existing memfs mocks instead of temporary directories filesystem-tests

  Request session.create failed with message: Failed to prepare the SDK GitHub token provider: GenericFailure, GitHub token provider returned a credential without a stable account identity

• Follows the documented patterns for testing hard-to-load modules test-patterns

  Request session.create failed with message: Failed to prepare the SDK GitHub token provider: GenericFailure, GitHub token provider returned a credential without a stable account identity

⏭️ Skipped: need the session log (11)

• Builds, formats and lints through yarn instead of calling the tools directly tool-commands
• Runs yarn format, lint and test for changed packages checks-for-changes
• Creates new packages with yarn new-package new-package
• Does not modify pull requests or issues authored by others foreign-pull-requests
• Does not modify the Dependency Dashboard dependency-dashboard
• Proposes a design and waits for approval before nontrivial work design-approval
• Prefers existing helpers over new shared abstractions minimal-architecture
• Gets approval for prototypes before adding docs and tests prototype-staging
• Reports test coverage before and after adding or removing tests coverage-report
• Reports fresh-process and warm timings separately performance-reporting
• Reuses existing fixtures and keeps existing assertions fixture-reuse

Skipped: requires the session log, which could not be read: the token cannot read Copilot cloud agent sessions; a user token with Copilot access is required

✅ Passed (6)

• Adds change files for published packages, and none otherwise changesets
• Uses separate type-only import and export statements separate-type-imports
• Keeps types.ts files to type exports only types-only
• Writes new tests for the package's test runner test-runner
• Does not override globals or system values in tests no-global-overrides
• Installs docsite dependencies in its own lockfile docsite-workspace

Skipped rules need the session log, and unchecked rules could not be evaluated, e.g. because the grader needs Copilot access. To audit them, run the following in evals/:

• Copilot cloud agent: yarn eval:report 4446
• Copilot CLI: yarn eval:report on the branch, on the machine that ran the session

_{Audited 5fc6c6c. This check only reports.}

[Tommy Nguyen (tido64)]

Examples: