Skip to content

fix(verify): let the timeout of a discovered test command be configured (AGT-4678) - #819

Merged
unohee merged 1 commit into
mainfrom
fix/agt-4678-verify-timeout
Oct 3, 2026
Merged

unohee merged 1 commit into
mainfrom
fix/agt-4678-verify-timeout

Conversation

@unohee

@unohee unohee commented Oct 3, 2026

Copy link
Copy Markdown
Collaborator

TL;DR

A test command the verifier discovers itself (discover.ts) gets a fixed 300 s timeout. cgf-portal's apps/pipelines suite takes 272 s with the machine calm and 295 s with two verifications at once, so the cap is inside the suite's natural duration and the verdict is lost whenever the machine carries any load; the attempt then falls back to the LLM tester (about 16 minutes).

Classification of the bound (before relaxing it)

  • Product requirement or slack: slack. 300 s is the generic default of discover.ts (DEFAULT_TIMEOUT_MS), not a figure derived from this repository. A repository that declares its own manifest already sets it per command (manifest.ts, up to 900 s).
  • Natural duration, measured: 272.08 s (1 failed, 5946 passed, 812 skipped in 272.08s, calm), 294.91 s (skipped, 1 warning in 294.91s, two verifications at once), and 87% at the cap for another run. The cap is below the duration, so it cannot pass.
  • What the cap is for (stopping a hung test) still holds at 600 s, about 2.2 times the calm duration.

Evidence (daemon log, 22:20 restart, after the AGT-4676 slot limit)

Two verifications finished and both ended timeout after 300000ms: one at [ 87%], one with pytest reporting in 294.91s. Before the slot limit: 7 of 7 at 3%.

Change

  • autonomous.verify.commandTimeoutMs (optional, 60 s to 900 s; unset keeps 300 s).
  • withDiscoveredTimeout applies it in loadTrustedVerifyPlan to discovered commands only. A command a repository declared in its own manifest keeps the timeout it declared.

Tests

  • Discovered command takes the configured timeout; unset keeps 300 s; a manifest command keeps its own; withDiscoveredTimeout does not mutate its input.
  • loadConfig carries the field through and leaves it unset by default; values below 60 s, above 900 s, 0 and negative are rejected. (A schema field loadConfig does not carry never reaches the verifier.)
  • Mutations: not applying the timeout, overriding manifest commands as well, and removing the schema field each fail the matching tests.
  • vitest deterministic tester, config, verify and pipeline verify suites: 286 of 287 pass. The one failure (runs npm package scripts against head-installed node_modules at base) fails the same way on a checkout without this change (it runs real npm under machine load).

Risk

A verification that really hangs now holds its slot for up to 600 s instead of 300 s; with two slots and the 20 minute slot wait bound that is acceptable.

After deploy

Set autonomous.verify.commandTimeoutMs: 600000 in the machine's config.yaml; the share of verifications that end in the timeout should fall.

…ed (AGT-4678)

A test command the verifier discovers itself gets the generic 300 s. cgf-portal's
apps/pipelines suite takes 272 s calm and 295 s with two verifications at once, so the
cap sits inside the suite's natural duration and both verdicts after the AGT-4676
slot limit were lost to it: one at 87%, one with pytest reporting 294.91 s.

Add autonomous.verify.commandTimeoutMs (60 s to 900 s, unset keeps 300 s) and apply it
to discovered commands only; a command a repository declares in its own manifest keeps
the timeout it declared. The bound is generic slack, not a product requirement, and
what it exists for (stopping a hung test) still holds at 600 s.
@unohee
unohee merged commit 875600c into main Oct 3, 2026
7 checks passed
@unohee
unohee deleted the fix/agt-4678-verify-timeout branch October 3, 2026 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant