Skip to content

ci: weekly scheduled model-backed help-bench run with pass-rate deltas #1427

Description

@thymikee

Part of #1412 (Track D).

Context

The help-conformance bench and skillgym are local-only and paid; regression detection depends on someone remembering to run them. The deterministic gates (#1411) now carry the drift risk, so the paid layer can be small and scheduled.

Deliverables

  • Weekly workflow: node scripts/help-conformance-bench.mjs --repeat 3 on cheap runners (haiku + mini tier), publishing the aggregate summary JSON as an artifact.
  • Soft comparison against the previous run's artifact: report per-case pass-rate deltas in the job summary; open/ping an issue when a case drops by more than a threshold across two consecutive weeks. NEVER a merge gate.
  • Budget guard: hard cap on runner x case x repeat count in the workflow so a case-list growth cannot silently multiply spend.
  • First manual run validates the 8 new case families' matchers (tv/web regexes flagged in feat(bench): renderer-pinned samples, topic-coverage gate, error-recovery quizzes; trim skillgym to agentic checks #1411 as most likely to need loosening) before the schedule goes live.

Labeled ready-for-human: needs CI secrets for model API access and a spend decision.

Acceptance

  • Two consecutive scheduled runs complete within budget; deltas visible in job summaries.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestready-for-humanValid work that needs human implementation, judgment, or maintainer merge

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions