You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The help-conformance bench and skillgym are local-only and paid; regression detection depends on someone remembering to run them. The deterministic gates (#1411) now carry the drift risk, so the paid layer can be small and scheduled.
Deliverables
Weekly workflow: node scripts/help-conformance-bench.mjs --repeat 3 on cheap runners (haiku + mini tier), publishing the aggregate summary JSON as an artifact.
Soft comparison against the previous run's artifact: report per-case pass-rate deltas in the job summary; open/ping an issue when a case drops by more than a threshold across two consecutive weeks. NEVER a merge gate.
Budget guard: hard cap on runner x case x repeat count in the workflow so a case-list growth cannot silently multiply spend.
Part of #1412 (Track D).
Context
The help-conformance bench and skillgym are local-only and paid; regression detection depends on someone remembering to run them. The deterministic gates (#1411) now carry the drift risk, so the paid layer can be small and scheduled.
Deliverables
node scripts/help-conformance-bench.mjs --repeat 3on cheap runners (haiku + mini tier), publishing the aggregate summary JSON as an artifact.Labeled ready-for-human: needs CI secrets for model API access and a spend decision.
Acceptance