Add typed PR review metrics

[Sun Haoran (haoranpb)]

Summary

• add a discriminated AnyAgentMetrics union as the registry for typed metrics schemas
• keep stable execution fields on AgentMetrics and PR Review-only telemetry on PRReviewMetrics
• replace the standalone expected_metrics mapping with an AgentMetricsContract that pairs each harness with its concrete metrics type and required-field subset
• reject newly created results whose metrics type does not match the configured harness, and warn only for missing required fields
• persist the selected detailed usage diagnostics needed from _run-metrics.json
• round-trip the concrete metrics type through result JSON without custom restoration logic

Design

AgentMetricsContract is the single harness-specific contract:

• metrics_type defines which fields may be reported and which discriminated payload the harness must return
• required_fields defines the subset the harness promises to populate and therefore which missing values should warn

Typed variants can be added to AnyAgentMetrics when another harness or harness/category setup gains specialized telemetry. Optional subtype diagnostics remain nullable and do not warn unless that harness explicitly makes them required.

Why

This establishes the typed extension point needed by #851 without adding PR Review-only fields to the shared metrics class or replacing validation with an untyped property bag. The persisted kind discriminator lets Pydantic select the concrete model directly and keeps result artifacts self-describing.

Validation

• uv run ruff check
• pre-commit, including ty check
• uv run pytest -q (969 passed, 1 skipped, 1 deselected)

[dayland]

This typed PrReviewMetrics direction looks like a good foundation for agent-specific telemetry. One future-extensibility consideration: could we treat this explicitly as a general pattern where AgentMetrics owns the stable core measurements and typed subclasses provide the additional metrics supported by each agent setup?

For example, PR Review would be the first registered subtype, while another harness—or a particular harness/category combination such as a deeply instrumented bug-fix setup—could later introduce its own typed metrics model and discriminator value. That would avoid gradually adding every agent-specific field to AgentMetrics.

I would prefer this over making an untyped dict[str, object] property bag the main extension point, since a bag loses validation, schema discovery, null semantics, and safe aggregation. If rapid metric experimentation eventually needs a property bag, a possible compromise would be a constrained, namespaced experimental_metrics bag whose fields are promoted into a typed subtype before standard dashboards or scoring depend on them.

I don't think #865 needs to implement that experimental mechanism now. It may just be worth documenting/naming the discriminated union as the registry and intended extension point for future typed agent metrics, so PrReviewMetrics is clearly the first use of a general design rather than a one-off special case.

[Sun Haoran (haoranpb)]

Agreed — that is the intended general pattern. I updated the PR to name AnyAgentMetrics explicitly as the registry/extension point and renamed the first subtype to PRReviewMetrics. I also traced the existing AgentHarness.expected_metrics mechanism from #773: it is complementary rather than redundant. The typed models define fields that may be reported, while �xpected_metrics defines the subset a harness promises to populate and warns about. I kept that contract separate and added coverage that unavailable optional PR Review diagnostics do not warn.

[Sun Haoran (haoranpb)]

Follow-up after revisiting the earlier �xpected_metrics mechanism: I aligned the two concepts under a single AgentMetricsContract per harness. The contract pairs the concrete metrics type with its required-field subset. Result creation now rejects the wrong registered metrics type, while missing-value warnings still apply only to required fields; optional subtype telemetry remains nullable. This preserves the useful behavior from #773 without maintaining a separate agent-specific mapping.

[Copilot]

🟡 Changes recommended

premium_requests is dropped, and deserialized results bypass harness-to-metrics contract validation.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds typed, harness-specific metrics and discriminated serialization for PR Review telemetry.

Changes:

• Introduces typed metrics contracts and PR Review metrics.
• Adds contract validation and JSON round-tripping.
• Expands metrics, serialization, and export tests.

File summaries

File	Description
tests/test_type_exhaustiveness.py	Tests metrics registry and discriminator coverage.
tests/test_result_writer.py	Updates PR Review export fixtures.
tests/test_result_serialization.py	Tests concrete metrics round-tripping.
tests/test_pr_review_metrics.py	Tests diagnostic extraction.
tests/test_pr_review_agent.py	Verifies typed agent output.
tests/test_expected_metrics_warning.py	Tests contract warnings and rejection.
src/bcbench/types.py	Defines typed metrics and harness contracts.
src/bcbench/results/base.py	Validates metrics during result creation.
src/bcbench/agent/pr_review/metrics.py	Builds detailed PR Review metrics.
src/bcbench/agent/pr_review/agent.py	Returns typed PR Review metrics.

Review details

• Files reviewed: 10/10 changed files
• Comments generated: 3
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.