Skip to content

feat(rubrics_verifier): grade a multi-turn run as of turn k - #222

Merged
kaihackney-scale merged 2 commits into
mainfrom
kai/rubrics-verifier-grade-up-to-turn
Oct 15, 2026
Merged

kaihackney-scale merged 2 commits into
mainfrom
kai/rubrics-verifier-grade-up-to-turn

Conversation

@kaihackney-scale

@kaihackney-scale kaihackney-scale commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

To chart how a simulated-user conversation's score climbs, each turn's env snapshot (prompt_agent's snapshot_config.per_turn) has to be graded on what the agent had done by then. The rubric verifier always loads every turn, so a criterion about what the agent said would be credited from a later turn. grade_up_to_turn (1-based, also settable per run through step_params) makes the judge see only turns 1..k: the per-turn trajectories, the turn-k trajectory as the single-trajectory fallback, and the user's per-turn messages are cut off there. The stored response and structured_output are the conversation's last reply, so the judge instead gets a note that the reply at this point ends turn k's trajectory. The verification entry records graded_up_to_turn. The run's stored context is never modified, and with it unset, or k at or past the last turn, grading is unchanged.

Validation: new unit tests run the verifier on a three-turn run (turns 1..2, turn 1 alone, the per-run override, k at the last turn, unset, invalid values, round-trip); all but the unset case fail without the change. All verifier unit tests and ruff pass. The full unit suite's other failures (test_mcp_cli_builder, a parallel-only flake in task_run_teardown_test) also occur on main locally.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; both previous findings are fixed.

Summary

Adds grade_up_to_turn so rubric grading can use only the conversation through a chosen turn.

  • Rubric verification can score a conversation as of turn k.

Both previous findings are addressed. No new actionable issues were found.

Reviews (2) · Last reviewed commit: "fix(rubrics_verifier): grade_up_to_turn ..." · Reviewed by Greptile

grade_up_to_turn (1-based, also a per-run step_params override) makes the
judge see only turns 1..k: the per-turn trajectories, the turn-k trajectory
as the single-trajectory fallback, and the per-turn prompt parts are cut off
there. The stored response and structured output are the conversation's
last reply, so the cut copy replaces them with a pointer to the end of turn
k's trajectory. This lets a per-turn env snapshot (prompt_agent's
snapshot_config.per_turn) be scored on what the agent had done by then. The
verification entry records graded_up_to_turn; unset, nothing changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kaihackney-scale
kaihackney-scale requested a review from a team as a code owner October 15, 2026 17:23
Comment thread src/agent_env/task_step/task_steps/verifiers/rubrics_verifier.py Outdated
Comment thread src/agent_env/task_step/task_steps/verifiers/rubrics_verifier.py
…ce; a later failure no longer skips an earlier turn

A cut copy kept the last turn's error_type, so a run whose final turn failed
skipped every earlier turn as prompt_error. Turns before the last always
completed (a failed turn ends the conversation), so a cut drops the error.

A cut also replaced the reply with a pointer to turn k's trajectory, which
use_trajectory=False never loads, and a run without per-turn trajectories
silently ignored the cutoff. Both now raise; the turn count falls back to the
per-turn prompt parts, which are recorded every turn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kaihackney-scale
kaihackney-scale merged commit 62087ce into main Oct 15, 2026
14 checks passed
@kaihackney-scale
kaihackney-scale deleted the kai/rubrics-verifier-grade-up-to-turn branch October 15, 2026 19:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants