Repository navigation
harness-config: outcome eval cases and probes for audit-automation-gaps, audit-prompting-postures, unhobble (rollout slice C) #11563
Description
Parent
Depends on
- none: on the frontier now
Outcome
harness-config gains frozen outcome eval cases and probe files for audit-automation-gaps, audit-prompting-postures, unhobble, plus a recorded before-state, so rewrite slice C can be measured against them.
Demo: /evals:validate passes on the new cases, and the smoke-pass record shows each case run against the current (before) skill text.
Decisions
- Q93/Q94: the harness-config rollout (Concise skills rollout: harness-config (10 skills, 9 model-invoked) #6555) is split into sub-issues with blocked-by edges before any run: outcome eval cases first, then rewrite slices of a few skills each.
- Eval passes run on the Claude subscription; that usage is a capacity limit, not owner cost, so no spend approval gates them.
Key interfaces
Steps 1-3 of the per-plugin runbook (/playbooks:skill-authoring, reference/rewrite-a-plugin.md). Case directories carry a short per-skill tag prefix; shared fixtures live under the plugin's eval fixtures directory and are named by a scaffold or seed script.
Done when
- Each of
audit-automation-gaps,audit-prompting-postures,unhobblehas outcome cases written by a fresh agent from the before contract via/evals:design, and/evals:validatepasses on them. - Each
llmgrader is calibrated as/evals:plugin-eval"Calibrating a judge" says. - One smoke pass ran on the before version; each failure was classified (real, grader defect, case defect) by a fresh agent and grader or case defects fixed.
- Each skill has a frozen probe set in the plugin's
probes/<skill>.json, checked with/skill-quality:check measure-invocation. - The PR records a digest of these case directories (frozen; unchanged after the first after-version run) and the
/skill-quality:checkbefore-state output for these skills. - No skill body or description changes in this PR.
Out of scope
- Rewriting any skill (rewrite slice C).
- Cases for skills outside this slice.
Metadata
Metadata
Assignees
Labels
Type
Fields
No fields configured for Task.
Activity
github-actions commentedon Oct 31, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Known
Flags: none
Blocked by: none
Understood: add frozen outcome eval cases, calibrated llm graders and probe files for
audit-automation-gaps,audit-prompting-posturesandunhobblein harness-config. Also record a before-state smoke pass and digest, with no skill body or description changes. The outcome, done-when items and out-of-scope list are exact, and the issue states it has no dependencies. Eval passes use the subscription, so no cost flag applies. No likely duplicates found among the open issues.Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Oct 31, 2026
Not admitted to the issue lane: it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (
.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Oct 31, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: confirmed; its comment says why.kyle-sexton commentedon Nov 1, 2026