fix(verify): bound concurrent deterministic verifications to two (AGT-4676)

[unohee]

TL;DR

Every attempt that reaches verification runs the target repository's full suite in its own sandbox, and nothing limited how many do so at once. With eight slots that is up to eight full cgf-portal suites together, on a machine other sessions also load. Under that load the pytest run hits its 300 s cap at about 3% progress, the deterministic verdict is lost, and the attempt falls back to the LLM tester (about 16 minutes).

Evidence (daemon log since the 21:01 restart, 2026-10-03)

• 7 of 7 deterministic verifications ended Deterministic verify unavailable; falling back to LLM tester ... timeout after 300000ms, each at 3% of the suite.
• The same suite finishes in 272 s when the machine is calm (1 failed, 5946 passed, 812 skipped in 272.08s): about 100 times slower under load.
• Stage averages since restart: worker 482 s, reviewer 201 s, tester 989 s.
• The daemon's own python children were 4 of about 35; three other sessions were running the full suite with 10 xdist workers each, plus a GitHub Actions build and a DAW. The daemon is one contributor, and the one this repository can bound.

Change

• New verifySlots.ts: a FIFO counting semaphore, sized once at startup (OPENSWARM_VERIFY_MAX_CONCURRENT, default 2), with a bounded wait (OPENSWARM_VERIFY_SLOT_WAIT_MS, default 20 min). A waiter that gives up leaves the queue, so a slot is never handed to someone who is gone.
• runVerify takes a slot before any sandbox exists, so the 300 s cap counts only the run itself. Its two callers (deterministicTester, qualityHarness) do not nest. On a wait timeout runVerify throws, which runTesterWithVerification already turns into the LLM fallback: the worst case is today's behaviour.

Tests

• Never more than the limit at once, FIFO order, slot released when the work throws, a waiter that times out is rejected and does not strand the slot, the onWait report, runVerify goes through verifySlots, default limit 2.
• Mutations: bypassing the slots in runVerify, keeping a gave-up waiter in the queue, and ignoring the limit each fail the matching tests.
• vitest verify, deterministic tester and pipeline verify suites: 217 of 218 pass. The one failure, runs npm package scripts against head-installed node_modules at base, fails the same way on a checkout without this change (load average 215 while running); it executes real npm.

Not in this PR

Selecting tests by changed files, sharing a baseline run across attempts, and a machine-wide lock for other sessions' full runs.

After deploy

At most two verification suites from the daemon at once; the log shows [Verify] waiting for a verification slot (...) and fewer 300 s timeouts.