Skip to content

feat(verify): honour a disabled tester role so the LLM tester is no longer the fallback for a failed verification (AGT-4679) - #820

Merged
unohee merged 1 commit into
mainfrom
fix/agt-4679-honour-disabled-tester
Oct 3, 2026
Merged

unohee merged 1 commit into
mainfrom
fix/agt-4679-honour-disabled-tester

Conversation

@unohee

@unohee unohee commented Oct 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • runTesterWithVerification takes llmFallback. With false, a deterministic infrastructure failure is rethrown (the attempt ends as infra_error) and a repository with no deterministic command gets a passing result that says nothing was verified. The LLM tester is never asked.
  • PairPipeline.runTester passes llmFallback: roles.tester.enabled !== false, so only an explicit enabled: false turns it off. Default behaviour is unchanged.
  • The stage log line says LLM tester is disabled, no verdict instead of falling back to LLM tester in that case.

Why

tester.enabled: false was already configured, but every verification that hit the 300 s pytest cap still fell back to the LLM tester (tester stages of 764 s and 814 s observed). The disabled role was not honoured.

Tests

  • deterministicTester.test.ts: no command + llmFallback: false returns the unverified result and never calls the fallback; the default still calls it.
  • pairPipeline.verify.test.ts: infra failure with the tester role disabled ends as infra_error without runTester; no command with the role disabled passes without runTester.
  • Mutation checks: removing the rethrow or the unverified return makes the matching tests fail.

…onger the fallback for a failed verification (AGT-4679)
@unohee
unohee merged commit 180177f into main Oct 3, 2026
7 checks passed
@unohee
unohee deleted the fix/agt-4679-honour-disabled-tester branch October 3, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant