Repository navigation
feat(rubrics_verifier): grade a multi-turn run as of turn k - #222
Merged
Merged
Conversation
grade_up_to_turn (1-based, also a per-run step_params override) makes the judge see only turns 1..k: the per-turn trajectories, the turn-k trajectory as the single-trajectory fallback, and the per-turn prompt parts are cut off there. The stored response and structured output are the conversation's last reply, so the cut copy replaces them with a pointer to the end of turn k's trajectory. This lets a per-turn env snapshot (prompt_agent's snapshot_config.per_turn) be scored on what the agent had done by then. The verification entry records graded_up_to_turn; unset, nothing changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ce; a later failure no longer skips an earlier turn A cut copy kept the last turn's error_type, so a run whose final turn failed skipped every earlier turn as prompt_error. Turns before the last always completed (a failed turn ends the conversation), so a cut drops the error. A cut also replaced the reply with a pointer to turn k's trajectory, which use_trajectory=False never loads, and a run without per-turn trajectories silently ignored the cutoff. Both now raise; the turn count falls back to the per-turn prompt parts, which are recorded every turn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AlexS-scale
approved these changes
Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
To chart how a simulated-user conversation's score climbs, each turn's env snapshot (
prompt_agent'ssnapshot_config.per_turn) has to be graded on what the agent had done by then. The rubric verifier always loads every turn, so a criterion about what the agent said would be credited from a later turn.grade_up_to_turn(1-based, also settable per run throughstep_params) makes the judge see only turns 1..k: the per-turn trajectories, the turn-k trajectory as the single-trajectory fallback, and the user's per-turn messages are cut off there. The storedresponseandstructured_outputare the conversation's last reply, so the judge instead gets a note that the reply at this point ends turn k's trajectory. The verification entry recordsgraded_up_to_turn. The run's stored context is never modified, and with it unset, or k at or past the last turn, grading is unchanged.Validation: new unit tests run the verifier on a three-turn run (turns 1..2, turn 1 alone, the per-run override, k at the last turn, unset, invalid values, round-trip); all but the unset case fail without the change. All verifier unit tests and
ruffpass. The full unit suite's other failures (test_mcp_cli_builder, a parallel-only flake intask_run_teardown_test) also occur onmainlocally.🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge; both previous findings are fixed.Summary
Adds
grade_up_to_turnso rubric grading can use only the conversation through a chosen turn.Both previous findings are addressed. No new actionable issues were found.
Reviews (2) · Last reviewed commit: "fix(rubrics_verifier): grade_up_to_turn ..." · Reviewed by Greptile