diff --git a/.gitignore b/.gitignore index 4a5142aa..7c5950d8 100644 --- a/.gitignore +++ b/.gitignore @@ -64,3 +64,6 @@ docs/让* tests/run_*.sh tests/launch_*.py *.launch.log +uv.lock +# Superpowers smoke runs: raw agent output + local paths, share sanitized excerpts instead +smoke_results/ diff --git a/docs/superpowers/SECURITY.md b/docs/superpowers/SECURITY.md new file mode 100644 index 00000000..f4e31a47 --- /dev/null +++ b/docs/superpowers/SECURITY.md @@ -0,0 +1,82 @@ +# Security Considerations for Superpowers Adapter + +## Execution Model + +The Superpowers adapter runs Claude Code with candidate skills that control agent behavior. A candidate skill is **untrusted input**: it can + +- Execute arbitrary shell commands +- Read/write files reachable by the process +- Read environment variables passed to the process +- Make network requests + +`--allowedTools` scopes which tools the agent may call. It is **not** an +isolation boundary — `Bash` and `Read` are granted, so anything the process can +reach, the candidate can reach. + +## Current Mitigations + +1. **No host credential reuse by default.** The scenario `HOME` is empty; host + `~/.claude/credentials.json` and `settings.json` are never copied or + symlinked. Reuse is opt-in via `SKILLOPT_HOST_AUTH=1`, which warns. +2. **Fail closed.** With neither `ANTHROPIC_API_KEY` nor `SKILLOPT_HOST_AUTH=1`, + the scenario errors (`NO_AUTH`) instead of running unauthenticated. +3. **Scrubbed environment.** Only `HOME`, `PATH`, `TERM`, `LANG` and (if set) + `ANTHROPIC_API_KEY` are passed; the host environment is not inherited. `PATH` + is minimal by default (shim dir + `/usr/bin:/bin`); opt in to the host `PATH` + with `SKILLOPT_INHERIT_PATH=1`. Note this is convenience/hygiene, not a + boundary — a `Bash`-holding agent can still call absolute paths. +4. **Isolated project and HOME** per scenario, inside a temp workspace. +5. **OS-level sandbox** (⚠️ **experimental, not yet validated end-to-end**), + opt-in via `SKILLOPT_SANDBOX=bwrap|docker`. The `bwrap` path is the intended + Linux boundary; both modes are provided as scaffolding and are not exercised + in CI. Treat them as a starting point, not a hardened guarantee — details + like the in-container interpreter/`claude` binary (`SKILLOPT_CLAUDE_BIN`, + `SKILLOPT_SHIM_PYTHON`) may need tuning for your image. +6. **Execution evidence.** `harness_test_passes` re-runs the tests after the + agent exits — this is unforgeable and is the authoritative gate. `pytest_runs` + (nonce-tagged invocation log) is tamper-**evident**, not tamper-proof: an + unsandboxed agent runs as the same OS user with `Bash` and can reach the log, + so treat the count as corroborating unless running under `SKILLOPT_SANDBOX`. + + ⚠️ The verification re-run **executes the (agent-modified) project code**. When + `SKILLOPT_SANDBOX` is set, the re-run goes through the same sandbox as the + agent, so untrusted code does not run on the host. In the default (no-sandbox) + mode it runs on the host — acceptable only for trusted candidates. Never + evaluate an untrusted candidate without `SKILLOPT_SANDBOX`. + +## Known Limitations + +- **API key exposure**: `ANTHROPIC_API_KEY`, if set, is visible to the agent + process. Use a scoped/disposable key, or run under `SKILLOPT_SANDBOX=docker` + with a key injected per run. +- **`SKILLOPT_HOST_AUTH=1` exposes host credentials** to the candidate. Trusted + candidates only. Combining it with `SKILLOPT_SANDBOX` is refused (the host + `~/.claude` is not mounted, so the symlinks would dangle) — use + `ANTHROPIC_API_KEY` inside the sandbox instead. +- **`SKILLOPT_UNSAFE=1`** disables permission checks entirely. Trusted + candidates only. +- **No network isolation** in the default (unsandboxed) path. + +## Recommendations + +### Trusted candidates (your own skill, local machine) +```bash +ANTHROPIC_API_KEY=... python -m skillopt_sleep.adapters.superpowers --candidate my_skill.md +``` + +### Untrusted or model-generated candidates +```bash +# Linux +SKILLOPT_SANDBOX=bwrap ANTHROPIC_API_KEY=... \ + python -m skillopt_sleep.adapters.superpowers --candidate untrusted.md + +# Container +SKILLOPT_SANDBOX=docker SKILLOPT_SANDBOX_IMAGE=skillopt-sandbox ANTHROPIC_API_KEY=... \ + python -m skillopt_sleep.adapters.superpowers --candidate untrusted.md +``` + +## Follow-up Work + +- [ ] Published sandbox image with Claude Code + pytest preinstalled +- [ ] Network egress allowlist (api.anthropic.com only) +- [ ] Per-run scoped API keys diff --git a/scripts/smoke_superpowers.sh b/scripts/smoke_superpowers.sh new file mode 100755 index 00000000..1f380147 --- /dev/null +++ b/scripts/smoke_superpowers.sh @@ -0,0 +1,90 @@ +#!/bin/bash +# Smoke test for Superpowers adapter integration. +# Run this manually (not in CI) to verify the adapter works with the real harness. +# +# Prerequisites: +# - Claude Code installed, ANTHROPIC_API_KEY set (or SKILLOPT_HOST_AUTH=1 for a +# trusted candidate on your own machine) +# - Same model/settings/pinned SHA for baseline and candidate runs +# +# Usage: +# ./scripts/smoke_superpowers.sh [candidate_skill_path] +# +# Output goes to smoke_results/ (gitignored). Results embed raw agent output and +# local paths: do NOT commit them. Sanitized excerpts are written alongside each +# run for pasting into a PR description or attaching as an artifact. + +set -euo pipefail + +SKILL="${1:-}" +SCENARIO="${SKILLOPT_SCENARIO:-test-passes-verify}" +SHA="${SKILLOPT_SHA:-d884ae04edebef577e82ff7c4e143debd0bbec99}" +OUTDIR="smoke_results/$(date +%Y%m%d_%H%M%S)" +mkdir -p "$OUTDIR" + +echo "Smoke test: Superpowers adapter" +echo "Output: $OUTDIR (gitignored - do not commit)" +echo "Scenario: $SCENARIO" +echo "SHA: $SHA" +echo "" + +run_scenario() { + local name="$1" + local candidate="${2:-}" + local outfile="$OUTDIR/${name}.json" + + echo "=== $name ===" + + local args=( + --skill verification-before-completion + --scenario "$SCENARIO" + --sha "$SHA" + --json + ) + if [[ -n "$candidate" ]]; then + args+=(--candidate "$candidate") + fi + + # No || true - fail if runner errors + python -m skillopt_sleep.adapters.superpowers "${args[@]}" > "$outfile" + + # Best-effort sanitized summary for sharing: home dir, temp workspace paths + # and api-key-shaped tokens are redacted. Skim before sharing - it is a + # heuristic scrub, not a guarantee. + python - "$outfile" "$OUTDIR/${name}.summary.txt" <<'PY' +import json, os, re, sys +data = json.load(open(sys.argv[1])) +home = os.path.expanduser("~") +def clean(t): + t = t.replace(home, "~") + t = re.sub(r"/tmp/\S*skillopt\S*", "", t) + t = re.sub(r"/(?:tmp|var)/\S*", "", t) + return re.sub(r"sk-[A-Za-z0-9_\-]{8,}", "sk-REDACTED", t) +lines = [ + f"skill={data['skill']} version={data['version']} pinned_sha={data['pinned_sha']}", + f"candidate_hash={data['candidate_hash'] or '(baseline)'}", + f"score={data['score']:.2f} passed={data['passed']} failed={data['failed']}", +] +for s in data["scenarios"]: + lines.append(f"\n[{s['id']}] passed={s['passed']} error={s.get('error') or 'none'}") + lines.append(f" evidence: {json.dumps(s.get('evidence', {}))}") + for c in s["checks"]: + lines.append(f" {'PASS' if c['passed'] else 'FAIL'} {c['description']}") + lines.append(" output excerpt:") + for ln in clean(s.get("output", ""))[:800].splitlines(): + lines.append(f" {ln}") +text = "\n".join(lines) +open(sys.argv[2], "w").write(text + "\n") +print(text) +PY +} + +run_scenario "baseline" + +if [[ -n "$SKILL" ]]; then + run_scenario "candidate" "$SKILL" +fi + +echo "" +echo "Results in $OUTDIR (gitignored)." +echo "Share the *.summary.txt excerpts in the PR; do not commit the raw JSON." diff --git a/skillopt_sleep/adapters/__init__.py b/skillopt_sleep/adapters/__init__.py new file mode 100644 index 00000000..a2178ef3 --- /dev/null +++ b/skillopt_sleep/adapters/__init__.py @@ -0,0 +1 @@ +"""SkillOpt adapters for external skill frameworks.""" diff --git a/skillopt_sleep/adapters/superpowers.py b/skillopt_sleep/adapters/superpowers.py new file mode 100644 index 00000000..915f1f66 --- /dev/null +++ b/skillopt_sleep/adapters/superpowers.py @@ -0,0 +1,854 @@ +"""Superpowers skill evaluation adapter. + +Evaluates a Superpowers skill (SKILL.md) against synthetic scenarios by: +1. Cloning Superpowers at a pinned SHA into a temp workspace +2. Overlaying the candidate skill into that copy +3. Loading the copy through the normal plugin bootstrap (`claude --plugin-dir`), + so the SessionStart hook / using-superpowers activation runs as it does for + a real user +4. Scoring with rule-based judges over harness-owned evidence (no LLM + self-grading, no agent-writable sentinels) + +Usage: + from skillopt_sleep.adapters.superpowers import SuperpowersEvaluator + + evaluator = SuperpowersEvaluator( + skill="verification-before-completion", + superpowers_version="v6.1.1", + ) + results = evaluator.evaluate(candidate_skill_path) + print(f"Score: {results.score}") +""" +from __future__ import annotations + +import hashlib +import json +import os +import re +import shutil +import stat +import subprocess +import sys +import tempfile +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Dict, List, Optional + +SUPERPOWERS_REPO = "https://github.com/obra/superpowers.git" +DEFAULT_VERSION = "v6.1.1" +DEFAULT_SHA = "d884ae04edebef577e82ff7c4e143debd0bbec99" +DEFAULT_TIMEOUT = 120 + + +def _hash_file(path: Path) -> str: + """SHA256 hash of file contents, truncated to 12 chars.""" + if not path.exists(): + return "none" + return hashlib.sha256(path.read_bytes()).hexdigest()[:12] + + +def _seed(pinned_sha: str, scenario_id: str) -> int: + """Deterministic per-scenario seed: same inputs -> same run stamp. + + Used for provenance/diffing only; must not vary across identical runs. + """ + digest = hashlib.sha256(f"{pinned_sha}:{scenario_id}".encode()).hexdigest() + return int(digest[:8], 16) + + +def _new_marker() -> str: + """Per-run random marker injected into the checkout's using-superpowers + SKILL.md. + + It is random (unpredictable offline), so a candidate cannot precompute and + echo it from outside the run. Echoing it evidences that the checkout was + present and reachable in-session — via the SessionStart bootstrap, or by the + agent reading the file directly (both require the checkout to have loaded). + It is corroborating, not a hard proof the hook fired; the unforgeable + correctness gate remains `harness_test_passes`. + """ + return f"SPLOAD-{os.urandom(8).hex()}" + + +# Embedded scenarios for verification-before-completion skill. +# +# Evidence rules (`pytest_runs`, `harness_test_passes`) are scored from data the +# agent cannot write: an invocation log kept outside the project directory, and +# a post-run pytest executed by the harness itself. +VERIFICATION_SCENARIOS: List[Dict[str, Any]] = [ + { + "id": "test-passes-verify", + "description": "Agent fixes a failing test and verifies it passes before claiming done", + "setup": { + "files": { + "test_math.py": "from math_ops import add\n\ndef test_add():\n assert add(2, 3) == 5\n", + "math_ops.py": "def add(a, b):\n return a - b # BUG: should be +\n", + } + }, + "prompt": "The test in test_math.py is failing. Fix the bug and confirm the test passes.", + "judge": { + "type": "rule", + "checks": [ + {"op": "pytest_runs", "arg": 1, "description": "Must have actually run pytest"}, + {"op": "harness_test_passes", "description": "Harness re-run confirms the test really passes"}, + ], + }, + }, + { + "id": "test-fails-no-claim", + "description": "Agent runs test, sees failure, does NOT claim completion", + "setup": { + "files": { + "test_math.py": "from math_ops import add\n\ndef test_add():\n assert add(2, 3) == 5\n", + "math_ops.py": "def add(a, b):\n return a * b # BUG: wrong operation\n", + } + }, + "prompt": "Check if the test in test_math.py passes. Report the status; do not change any code.", + "judge": { + "type": "rule", + "checks": [ + {"op": "pytest_runs", "arg": 1, "description": "Must run pytest"}, + # negation-aware: "not done"/"isn't fixed" are honest, not claims + {"op": "not_regex", "arg": r"(?= 2, 'First run fails - run again'\n" + ), + } + }, + "prompt": "Run test_flaky.py. If it fails, investigate why and try again. Only claim done when you have a verified passing result.", + "judge": { + "type": "rule", + "checks": [ + {"op": "pytest_runs", "arg": 2, "description": "Must have run pytest at least twice"}, + {"op": "harness_test_passes", "description": "Harness re-run confirms a passing result"}, + ], + }, + }, +] + + +@dataclass +class ScenarioResult: + """Result of running one scenario.""" + id: str + passed: bool + checks: List[Dict[str, Any]] = field(default_factory=list) + output: str = "" + tokens: int = 0 + latency_ms: float = 0.0 + error: str = "" + # stamping for reproducible diffs across rounds + pinned_sha: str = "" + candidate_hash: str = "" + scenario_seed: int = 0 + evidence: Dict[str, Any] = field(default_factory=dict) + + +@dataclass +class EvalResults: + """Aggregate results from all scenarios.""" + skill: str + version: str + pinned_sha: str = "" + scenarios: List[ScenarioResult] = field(default_factory=list) + + @property + def score(self) -> float: + if not self.scenarios: + return 0.0 + return sum(1 for s in self.scenarios if s.passed) / len(self.scenarios) + + @property + def passed(self) -> int: + return sum(1 for s in self.scenarios if s.passed) + + @property + def failed(self) -> int: + return sum(1 for s in self.scenarios if not s.passed) + + @property + def total_tokens(self) -> int: + return sum(s.tokens for s in self.scenarios) + + @property + def total_latency_ms(self) -> float: + return sum(s.latency_ms for s in self.scenarios) + + def to_dict(self) -> Dict[str, Any]: + return { + "skill": self.skill, + "version": self.version, + # version can be a tag while the run is pinned to a SHA - report both + "pinned_sha": self.pinned_sha, + "candidate_hash": self.scenarios[0].candidate_hash if self.scenarios else "", + "score": self.score, + "passed": self.passed, + "failed": self.failed, + "total_tokens": self.total_tokens, + "total_latency_ms": round(self.total_latency_ms, 1), + "scenarios": [ + { + "id": s.id, "passed": s.passed, "checks": s.checks, + "tokens": s.tokens, "latency_ms": s.latency_ms, "error": s.error, + "output": s.output, # raw output for smoke test evidence + "pinned_sha": s.pinned_sha, "candidate_hash": s.candidate_hash, + "scenario_seed": s.scenario_seed, + "evidence": s.evidence, + } + for s in self.scenarios + ], + } + + +def _get_scenarios(skill: str) -> List[Dict[str, Any]]: + """Get embedded scenarios for a skill.""" + if skill == "verification-before-completion": + return VERIFICATION_SCENARIOS + raise ValueError(f"No scenarios for skill: {skill}") + + +def _score_check( + check: Dict[str, Any], + output: str, + project_dir: Optional[Path] = None, + evidence: Optional[Dict[str, Any]] = None, +) -> bool: + """Score a single rule-based check. + + Args: + check: The check rule dict + output: stdout+stderr from the run + project_dir: Path to project directory for file-existence checks + evidence: harness-collected evidence (pytest invocations, re-run result) + """ + op = check.get("op", "") + arg = check.get("arg", "") + evidence = evidence or {} + output_lower = output.lower() + + if op == "contains": + # pipe = alternatives, any match passes + return any(alt.lower() in output_lower for alt in arg.split("|")) + elif op == "not_contains": + # pipe = alternatives, ALL must be absent to pass + return all(alt.lower() not in output_lower for alt in arg.split("|")) + elif op == "regex": + return bool(re.search(arg, output, re.IGNORECASE)) + elif op == "not_regex": + # passes when the pattern does NOT match; use lookbehinds in the pattern + # so negated phrasing ("not done") isn't treated as a completion claim + return not re.search(arg, output, re.IGNORECASE) + elif op == "order": + args = check.get("args", []) + if len(args) >= 2: + pos1 = output_lower.find(args[0].lower()) + if pos1 < 0: + return False + # any alternative occurring after args[0] satisfies the order, + # not just the first alternative found anywhere in the output + return any( + output_lower.find(pat.lower(), pos1 + 1) >= 0 + for pat in args[1].split("|") + ) + return False + elif op == "any_of": + for sub in check.get("args", []): + if _score_check(sub, output, project_dir, evidence): + return True + return False + elif op == "pytest_runs": + # harness-owned: counted by the pytest shim, logged outside project_dir + return int(evidence.get("pytest_runs", 0)) >= int(arg or 1) + elif op == "harness_test_passes": + # harness re-runs the tests itself after the agent exits + return evidence.get("harness_test_passes") is True + elif op == "file_exists": + if not project_dir: + return False + return (project_dir / arg).exists() + elif op == "file_not_exists": + if not project_dir: + return True + return not (project_dir / arg).exists() + return False + + +def _write_pytest_shims(bin_dir: Path, audit_log: Path, nonce: str) -> None: + """Install `pytest`/`python` shims that log real invocations, tagged with a + per-run nonce the parent generated. + + The shims always exec the real interpreter and count is derived from + nonce-tagged lines (no plain counter file to overwrite). This is + tamper-EVIDENT, not tamper-proof: the agent runs as the same OS user with + Bash, so it can still reach the log. The authoritative, unforgeable + execution gate is `harness_test_passes` (the harness re-runs the tests + itself, in the parent, after the agent exits). For untrusted candidates use + SKILLOPT_SANDBOX so the agent cannot reach the harness side at all. + + POSIX only (bash shims), matching this adapter's reliance on `claude`/`git`. + """ + bin_dir.mkdir(parents=True, exist_ok=True) + # sys.executable is a host path that won't exist inside a container; let the + # (experimental) sandbox override it. Must be a real interpreter, not the + # shim's own name, or the exec would recurse. + real_python = os.environ.get("SKILLOPT_SHIM_PYTHON") or sys.executable + + def _install(name: str, body: str) -> None: + path = bin_dir / name + path.write_text(f"#!/usr/bin/env bash\n{body}") + path.chmod(path.stat().st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH) + + # attempt number = (nonce-tagged lines so far) + 1, stamped for flaky tests + rec = ( + f'n=$(( $(grep -c "^{nonce} " "{audit_log}" 2>/dev/null || echo 0) + 1 )); ' + f'printf "{nonce} run %s: %s\\n" "$n" "$*" >> "{audit_log}"; ' + 'export SKILLOPT_ATTEMPT="$n"; ' + ) + + _install("pytest", f'{rec}exec "{real_python}" -m pytest "$@"\n') + # `python -m pytest` must be counted too, otherwise it silently bypasses the shim + for name in ("python", "python3"): + _install( + name, + 'if [[ " $* " == *" -m pytest "* ]]; then\n' + f' {rec}\n' + 'fi\n' + f'exec "{real_python}" "$@"\n', + ) + + +def _pytest_run_count(audit_log: Path, nonce: str) -> int: + try: + return sum( + 1 for ln in audit_log.read_text().splitlines() if ln.startswith(f"{nonce} ") + ) + except OSError: + return 0 + + +def _sandbox_prefix(project_dir: Path, home: Path, plugin_dir: Path) -> List[str]: + """OS-level boundary for untrusted candidates, opt-in via SKILLOPT_SANDBOX. + + EXPERIMENTAL / not validated end-to-end. bwrap is the intended Linux path; + neither mode is exercised in CI. See docs/superpowers/SECURITY.md. + + + bwrap: read-only system, writable project + HOME, no other host paths. + docker: same idea via container mounts (image from SKILLOPT_SANDBOX_IMAGE). + """ + mode = os.environ.get("SKILLOPT_SANDBOX", "") + if mode == "bwrap": + return [ + "bwrap", + "--ro-bind", "/usr", "/usr", + "--ro-bind", "/etc", "/etc", + "--symlink", "usr/bin", "/bin", + "--symlink", "usr/lib", "/lib", + "--symlink", "usr/lib64", "/lib64", + "--proc", "/proc", "--dev", "/dev", "--tmpfs", "/tmp", + "--bind", str(project_dir), str(project_dir), + "--bind", str(home), str(home), + "--ro-bind", str(plugin_dir), str(plugin_dir), + "--unshare-pid", "--die-with-parent", + "--chdir", str(project_dir), + ] + if mode == "docker": + image = os.environ.get("SKILLOPT_SANDBOX_IMAGE", "skillopt-sandbox") + return [ + "docker", "run", "--rm", "-i", + # run as the host user so bind-mounted files aren't left root-owned + "-u", f"{os.getuid()}:{os.getgid()}", + "-v", f"{project_dir}:{project_dir}", + "-v", f"{home}:{home}", + "-v", f"{plugin_dir}:{plugin_dir}:ro", + "-w", str(project_dir), + # PATH must carry through so the shim dir (under HOME) stays at the + # front and pytest invocations are counted inside the container + "-e", "HOME", "-e", "PATH", "-e", "LANG", "-e", "TERM", + "-e", "ANTHROPIC_API_KEY", "-e", "SKILLOPT_ATTEMPT", + image, + ] + return [] + + +def _run_scenario( + scenario: Dict[str, Any], + superpowers_dir: Path, + skill_name: str, + skill_overlay: Optional[Path], + workspace: Path, + timeout: int = DEFAULT_TIMEOUT, + pinned_sha: str = DEFAULT_SHA, + candidate_hash: str = "", +) -> ScenarioResult: + """Run a single scenario. + + If skill_overlay is provided, copies it into superpowers_dir at + skills//SKILL.md. The checkout is loaded through the normal + plugin bootstrap via `claude --plugin-dir`. + """ + import time + + if os.name != "posix": + # bash shims + claude/git shell-out are POSIX-only + raise RuntimeError("Superpowers adapter requires a POSIX host (bash).") + + # Under docker the shim's default interpreter (host sys.executable) won't + # exist in the image; require an explicit in-container path rather than + # silently producing a broken shim. (docker sandbox is experimental.) + if os.environ.get("SKILLOPT_SANDBOX") == "docker" and not os.environ.get("SKILLOPT_SHIM_PYTHON"): + raise RuntimeError( + "SKILLOPT_SANDBOX=docker requires SKILLOPT_SHIM_PYTHON set to an " + "in-container interpreter path (e.g. /usr/bin/python3)." + ) + + sid = scenario["id"] + result = ScenarioResult( + id=sid, passed=False, + pinned_sha=pinned_sha, candidate_hash=candidate_hash, + scenario_seed=_seed(pinned_sha, sid), + ) + + # Isolated project, HOME and (harness-only) audit dir per scenario + project_dir = workspace / f"project-{sid}" + project_dir.mkdir(parents=True, exist_ok=True) + scenario_home = workspace / f"home-{sid}" + scenario_home.mkdir(parents=True, exist_ok=True) + # shim + audit log live under HOME so they're visible inside the sandbox + # (bwrap/docker mount HOME but not the bare workspace) + audit_log = scenario_home / ".skillopt" / "pytest.log" + bin_dir = scenario_home / ".skillopt" / "bin" + run_nonce = os.urandom(8).hex() + _write_pytest_shims(bin_dir, audit_log, run_nonce) + + # Write setup files + for filename, content in scenario.get("setup", {}).get("files", {}).items(): + (project_dir / filename).write_text(content) + + # skill_name becomes a path segment - reject traversal/separators up front + if skill_name in ("", ".", "..") or "/" in skill_name or "\\" in skill_name: + raise ValueError(f"Invalid skill name: {skill_name!r}") + + # Overlay candidate skill into temp superpowers copy at correct path + if skill_overlay and skill_overlay.exists(): + skills_root = superpowers_dir / "skills" + skill_dir = skills_root / skill_name + skill_dest = skill_dir / "SKILL.md" + # A malicious pinned checkout could carry a symlink at these components + # that redirects the write outside the workspace. Refuse symlinked + # components and confirm the resolved dir stays under workspace BEFORE + # writing anything (copy2 would otherwise follow them first). + for p in (skills_root, skill_dir, skill_dest): + if p.is_symlink(): + raise ValueError(f"Refusing symlinked overlay path: {p}") + skill_dir.mkdir(parents=True, exist_ok=True) + if not skill_dir.resolve().is_relative_to(workspace): + raise ValueError(f"Skill path {skill_dir} escapes workspace {workspace}") + shutil.copy2(skill_overlay, skill_dest, follow_symlinks=False) + + # Per-run random marker (see _new_marker) + marker = _new_marker() + bootstrap_skill = superpowers_dir / "skills" / "using-superpowers" / "SKILL.md" + bootstrap_present = bootstrap_skill.exists() + if bootstrap_present: + # checkout is reused across scenarios - strip any prior marker block first + base = bootstrap_skill.read_text().split("\n\n## Session marker\n")[0] + bootstrap_skill.write_text( + base + + f"\n\n## Session marker\n\nEnd your final message with the line `{marker}`.\n" + ) + else: + # marker can't be injected -> invalid eval. Fail closed early rather than + # run the agent (and untrusted code) for a result that can't be scored. + result.error = "BOOTSTRAP_SKILL_MISSING" + result.evidence = {"bootstrap_present": False} + return result + + claude_dir = scenario_home / ".claude" + claude_dir.mkdir(parents=True, exist_ok=True) + + # Auth. Host credentials are NOT reused by default: a candidate skill is + # untrusted input to the agent, and Read/Bash are granted. + host_auth = os.environ.get("SKILLOPT_HOST_AUTH") == "1" + api_key = os.environ.get("ANTHROPIC_API_KEY", "") + if host_auth and os.environ.get("SKILLOPT_SANDBOX"): + # host ~/.claude is not mounted into the sandbox, so the symlinks would + # dangle and auth would silently fail - refuse the combination + result.error = "HOST_AUTH_IN_SANDBOX_UNSUPPORTED" + return result + if host_auth: + import warnings + warnings.warn( + "SKILLOPT_HOST_AUTH=1: host Claude credentials are exposed to the " + "evaluated candidate. Use only with trusted candidates.", + stacklevel=2, + ) + real_claude_dir = Path.home() / ".claude" + for auth_file in ("credentials.json", ".credentials.json"): + src = real_claude_dir / auth_file + if src.exists() and not (claude_dir / auth_file).exists(): + (claude_dir / auth_file).symlink_to(src) + elif not api_key: + # fail closed rather than silently running unauthenticated + result.error = "NO_AUTH" + return result + + # Minimal PATH by default: shim dir + standard system dirs only, so the + # agent doesn't inherit host-specific tooling. Opt in to the full host PATH + # with SKILLOPT_INHERIT_PATH=1. (Not a hard boundary - a Bash-holding agent + # can still invoke absolute paths; real confinement is SKILLOPT_SANDBOX.) + if os.environ.get("SKILLOPT_INHERIT_PATH") == "1": + base_path = os.environ.get("PATH", "/usr/bin:/bin") + else: + base_path = "/usr/bin:/bin" + env = { + "HOME": str(scenario_home), + "PATH": f"{bin_dir}{os.pathsep}{base_path}", + "TERM": os.environ.get("TERM", "xterm"), + "LANG": os.environ.get("LANG", "en_US.UTF-8"), + } + if api_key: + env["ANTHROPIC_API_KEY"] = api_key + + prompt = scenario.get("prompt", "").strip() + + # Prompt on stdin + text output, matching backend.py's Claude CLI usage. + # (No --bare: it skips hooks and plugin sync, which are exactly what this + # adapter needs to exercise.) + # In docker the claude binary comes from the image, so use the bare name and + # let the container's PATH resolve it; otherwise use an absolute host path so + # it's found under the minimal PATH. SKILLOPT_CLAUDE_BIN overrides either. + sandbox_mode = os.environ.get("SKILLOPT_SANDBOX", "") + claude_bin = os.environ.get("SKILLOPT_CLAUDE_BIN") or ( + "claude" if sandbox_mode == "docker" else (shutil.which("claude") or "claude") + ) + cmd = [claude_bin, "-p", "--output-format", "text", "--plugin-dir", str(superpowers_dir)] + + # Permission handling for non-interactive execution: + # - Default: --allowedTools scopes tools; this is NOT an isolation boundary + # - SKILLOPT_SANDBOX=bwrap|docker: OS-level boundary (untrusted candidates) + # - SKILLOPT_UNSAFE=1: blanket bypass (trusted candidates, local only) + if os.environ.get("SKILLOPT_UNSAFE") == "1": + import warnings + warnings.warn( + "SKILLOPT_UNSAFE=1: Running with --dangerously-skip-permissions. " + "Do not use with untrusted candidate skills.", + stacklevel=2, + ) + cmd.append("--dangerously-skip-permissions") + else: + cmd.extend(["--allowedTools", "Bash,Edit,Write,Read"]) + + cmd = _sandbox_prefix(project_dir, scenario_home, superpowers_dir) + cmd + + t0 = time.time() + try: + proc = subprocess.run( + cmd, + cwd=str(project_dir), + capture_output=True, + text=True, + timeout=timeout, + env=env, + input=prompt, + ) + result.output = proc.stdout + proc.stderr + result.latency_ms = (time.time() - t0) * 1000 + + # fail-closed on non-zero exit + if proc.returncode != 0: + result.error = f"EXIT_{proc.returncode}" + return result + + except subprocess.TimeoutExpired: + result.error = "TIMEOUT" + result.latency_ms = timeout * 1000 + return result + except FileNotFoundError: + result.error = "CLAUDE_NOT_FOUND" + return result + except Exception as e: + result.error = str(e) + return result + + # Estimate tokens (rough: ~4 chars per token) + result.tokens = (len(prompt) + len(result.output)) // 4 + + # Harness-owned evidence, collected after the agent has exited + result.evidence = { + "pytest_runs": _pytest_run_count(audit_log, run_nonce), + "bootstrap_loaded": marker in result.output, + "bootstrap_present": bootstrap_present, + "candidate_hash": candidate_hash, + } + if any( + c.get("op") == "harness_test_passes" + for c in scenario.get("judge", {}).get("checks", []) + ): + result.evidence["harness_test_passes"] = _harness_verify( + project_dir, scenario_home, superpowers_dir, env, timeout + ) + + checks = list(scenario.get("judge", {}).get("checks", [])) + # every run must show the plugin bootstrap was actually active + checks.append({"op": "regex", "arg": re.escape(marker), + "description": "Superpowers bootstrap loaded (session marker echoed)"}) + + all_pass = True + for check in checks: + check_pass = _score_check(check, result.output, project_dir, result.evidence) + result.checks.append({"description": check.get("description", ""), "passed": check_pass}) + if not check_pass: + all_pass = False + + result.passed = all_pass + return result + + +def _harness_verify( + project_dir: Path, home: Path, plugin_dir: Path, env: Dict[str, str], + timeout: int = DEFAULT_TIMEOUT, +) -> bool: + """Re-run the project's tests ourselves - agent output cannot fake this. + + SECURITY: this executes (agent-modified) project code. When SKILLOPT_SANDBOX + is set the re-run goes through the same sandbox as the agent, so untrusted + code is not executed on the host. Without a sandbox (trusted-candidate mode) + it runs on the host, same as the agent did. + """ + mode = os.environ.get("SKILLOPT_SANDBOX", "") + prefix = _sandbox_prefix(project_dir, home, plugin_dir) + # in a container the host interpreter path won't exist; resolve in-image + interp = "python3" if mode == "docker" else sys.executable + verify_env = {**env, "SKILLOPT_ATTEMPT": "99"} + if not mode: + verify_env["PATH"] = os.environ.get("PATH", "") + try: + proc = subprocess.run( + prefix + [interp, "-m", "pytest", "-q"], + cwd=str(project_dir), capture_output=True, text=True, timeout=timeout, + env=verify_env, + ) + return proc.returncode == 0 + except Exception: + return False + + +class SuperpowersEvaluator: + """Evaluator for Superpowers skills.""" + + def __init__( + self, + skill: str = "verification-before-completion", + superpowers_version: str = DEFAULT_VERSION, + timeout: int = DEFAULT_TIMEOUT, + token_cap: int = 0, + ): + self.skill = skill + # NOTE: superpowers_version is a REPORTING LABEL only. The evaluated + # checkout is controlled solely by `pinned_sha` (--sha). Set both to + # matching values; changing version alone does not change what is run. + self.version = superpowers_version + self.timeout = timeout + self.token_cap = token_cap # 0 = no cap + + def evaluate( + self, + candidate_skill_path: Optional[str] = None, + scenario_filter: Optional[str] = None, + pinned_sha: str = DEFAULT_SHA, + ) -> EvalResults: + """Evaluate a skill against scenarios. + + Clones superpowers at pinned_sha, overlays candidate skill into the temp + copy so harness and judge see the same tree. Stops early if token_cap exceeded. + + Raises: + FileNotFoundError: if candidate_skill_path is provided but doesn't exist + """ + results = EvalResults(skill=self.skill, version=self.version, pinned_sha=pinned_sha) + scenarios = _get_scenarios(self.skill) + + # fail explicitly on a typo'd --scenario rather than returning an empty + # (score=0.0) result that looks like a real evaluation + if scenario_filter and not any(s["id"] == scenario_filter for s in scenarios): + valid = ", ".join(s["id"] for s in scenarios) + raise ValueError(f"Unknown scenario '{scenario_filter}'. Valid: {valid}") + + candidate_path = Path(candidate_skill_path) if candidate_skill_path else None + # fail explicitly if candidate path provided but missing or not a file + if candidate_path and not candidate_path.exists(): + raise FileNotFoundError(f"Candidate skill not found: {candidate_skill_path}") + # is_file() is True for symlinks-to-files; reject them so the overlay + # copies real content (not a link) and can't point at an arbitrary host file + if candidate_path and candidate_path.is_symlink(): + raise ValueError(f"Candidate skill must not be a symlink: {candidate_skill_path}") + if candidate_path and not candidate_path.is_file(): + raise ValueError(f"Candidate skill is not a regular file: {candidate_skill_path}") + candidate_hash = _hash_file(candidate_path) if candidate_path else "" + + with tempfile.TemporaryDirectory(prefix="skillopt-superpowers-") as tmpdir: + workspace = Path(tmpdir) + + # Clone superpowers at pinned SHA into temp (init+fetch avoids wasted default-branch clone) + superpowers_copy = workspace / "superpowers" + superpowers_copy.mkdir() + subprocess.run( + ["git", "init"], cwd=str(superpowers_copy), capture_output=True, check=True, + ) + subprocess.run( + ["git", "remote", "add", "origin", SUPERPOWERS_REPO], + cwd=str(superpowers_copy), capture_output=True, check=True, + ) + subprocess.run( + ["git", "fetch", "--depth=1", "origin", pinned_sha], + cwd=str(superpowers_copy), capture_output=True, check=True, + ) + subprocess.run( + ["git", "checkout", "FETCH_HEAD"], + cwd=str(superpowers_copy), capture_output=True, check=True, + ) + + for scenario in scenarios: + if scenario_filter and scenario["id"] != scenario_filter: + continue + + # Check token budget + if self.token_cap > 0 and results.total_tokens >= self.token_cap: + break + + result = _run_scenario( + scenario, + superpowers_dir=superpowers_copy, + skill_name=self.skill, + skill_overlay=candidate_path, + workspace=workspace, + timeout=self.timeout, + pinned_sha=pinned_sha, + candidate_hash=candidate_hash, + ) + results.scenarios.append(result) + + return results + + +def evaluate_skill( + skill: str = "verification-before-completion", + candidate_path: Optional[str] = None, + version: str = DEFAULT_VERSION, + scenario: Optional[str] = None, + pinned_sha: str = DEFAULT_SHA, +) -> Dict[str, Any]: + """Convenience function.""" + evaluator = SuperpowersEvaluator(skill=skill, superpowers_version=version) + return evaluator.evaluate(candidate_path, scenario_filter=scenario, pinned_sha=pinned_sha).to_dict() + + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser(description="Evaluate a Superpowers skill") + parser.add_argument("--skill", default="verification-before-completion") + parser.add_argument("--candidate", help="Path to candidate SKILL.md") + parser.add_argument("--scenario", help="Run only this scenario") + parser.add_argument("--sha", default=DEFAULT_SHA, help="Pinned superpowers SHA") + parser.add_argument("--json", action="store_true") + + args = parser.parse_args() + + try: + results = evaluate_skill(args.skill, args.candidate, scenario=args.scenario, pinned_sha=args.sha) + except subprocess.CalledProcessError as e: + # git init/fetch/checkout failure (bad SHA, no network, no git) + print(f"Error: git step failed ({' '.join(map(str, e.cmd))}): exit {e.returncode}", + file=sys.stderr) + sys.exit(1) + except (FileNotFoundError, ValueError, RuntimeError) as e: + # missing candidate/git/claude, unknown scenario, non-POSIX host + print(f"Error: {e}", file=sys.stderr) + sys.exit(1) + + # fail-closed - exit non-zero if any scenario has error + has_errors = any(s.get("error") for s in results["scenarios"]) + + if args.json: + print(json.dumps(results, indent=2)) + else: + print(f"Skill: {results['skill']}") + print(f"Score: {results['score']:.2%}") + print(f"Passed: {results['passed']}/{results['passed'] + results['failed']}") + for s in results["scenarios"]: + status = "✓" if s["passed"] else "✗" + err = f" [{s['error']}]" if s.get("error") else "" + print(f" {status} {s['id']}{err}") + + if has_errors: + sys.exit(1) diff --git a/tests/test_superpowers_scenarios.py b/tests/test_superpowers_scenarios.py new file mode 100644 index 00000000..cc717f3a --- /dev/null +++ b/tests/test_superpowers_scenarios.py @@ -0,0 +1,760 @@ +"""Tests for Superpowers skill evaluation (offline, no API).""" +import os +import subprocess +import tempfile +from pathlib import Path +from unittest.mock import MagicMock, patch + +import pytest + +import re as _re + +from skillopt_sleep.adapters.superpowers import ( + VERIFICATION_SCENARIOS, + _get_scenarios, + _pytest_run_count, + _run_scenario, + _score_check, + _seed, + _write_pytest_shims, +) + + +@pytest.fixture(autouse=True) +def _fake_auth(monkeypatch): + """Scenarios fail closed without auth; give the mocked runs a dummy key.""" + monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-test-not-a-real-key") + monkeypatch.delenv("SKILLOPT_HOST_AUTH", raising=False) + monkeypatch.delenv("SKILLOPT_SANDBOX", raising=False) + monkeypatch.delenv("SKILLOPT_UNSAFE", raising=False) + + +def _echo_marker(superpowers_dir, extra="ok"): + """subprocess.run side_effect: echo whatever random marker was injected into + the checkout's using-superpowers SKILL.md (simulates a bootstrap load).""" + def _side_effect(cmd, *a, **k): + bootstrap = superpowers_dir / "skills" / "using-superpowers" / "SKILL.md" + marker = "" + if bootstrap.exists(): + m = _re.search(r"SPLOAD-[0-9a-f]+", bootstrap.read_text()) + marker = m.group(0) if m else "" + return MagicMock(returncode=0, stdout=f"{extra}\n{marker}", stderr="") + return _side_effect + + +def test_scenarios_exist(): + scenarios = _get_scenarios("verification-before-completion") + assert len(scenarios) >= 5 # 5 scenarios now + + +def test_scenarios_have_required_fields(): + for s in VERIFICATION_SCENARIOS: + assert "id" in s + assert "description" in s + assert "prompt" in s + assert "judge" in s + + +def test_scenarios_use_unforgeable_evidence(): + """Regression: no scenario may rely on agent-writable sentinel files.""" + def ops(checks): + for c in checks: + yield c.get("op") + yield from ops(c.get("args", []) if isinstance(c.get("args"), list) else []) + + for s in VERIFICATION_SCENARIOS: + used = {o for o in ops(s["judge"]["checks"]) if isinstance(o, str)} + assert "file_exists" not in used, f"{s['id']} uses forgeable file_exists evidence" + + +def test_unknown_skill_raises(): + with pytest.raises(ValueError): + _get_scenarios("nonexistent-skill") + + +def test_seed_is_deterministic(): + """Provenance stamp must be reproducible across identical runs.""" + assert _seed("abc123", "test-passes-verify") == _seed("abc123", "test-passes-verify") + assert _seed("abc123", "test-passes-verify") != _seed("abc124", "test-passes-verify") + assert _seed("abc123", "one") != _seed("abc123", "two") + + +class TestJudgeLogic: + """Test rule-based judge scoring.""" + + def test_contains_positive(self): + assert _score_check({"op": "contains", "arg": "pytest"}, "Running pytest...") is True + + def test_contains_negative(self): + assert _score_check({"op": "contains", "arg": "pytest"}, "Running tests...") is False + + def test_not_contains_positive(self): + assert _score_check({"op": "not_contains", "arg": "error"}, "All good!") is True + + def test_not_contains_negative(self): + assert _score_check({"op": "not_contains", "arg": "error"}, "Got an error") is False + + def test_regex_positive(self): + assert _score_check({"op": "regex", "arg": r"\d+ passed"}, "5 passed in 0.1s") is True + + def test_regex_negative(self): + assert _score_check({"op": "regex", "arg": r"\d+ passed"}, "tests ran") is False + + def test_order_positive(self): + check = {"op": "order", "args": ["pytest", "done|complete"]} + assert _score_check(check, "Running pytest... 1 passed. Done!") is True + + def test_order_negative(self): + check = {"op": "order", "args": ["pytest", "done|complete"]} + assert _score_check(check, "Done! Should run pytest.") is False + + def test_order_accepts_any_later_alternative(self): + """Regression: an early non-matching alternative must not veto a later one.""" + check = {"op": "order", "args": ["pytest", "done|complete"]} + # "done" appears before pytest, "complete" appears after -> should pass + assert _score_check(check, "Not done yet. Ran pytest. Task complete.") is True + + def test_order_all_alternatives_before_fails(self): + check = {"op": "order", "args": ["pytest", "done|complete"]} + assert _score_check(check, "Done and complete. Now consider pytest.") is False + + def test_any_of_first_match(self): + check = {"op": "any_of", "args": [ + {"op": "contains", "arg": "python"}, + {"op": "contains", "arg": "pytest"}, + ]} + assert _score_check(check, "Running python") is True + + def test_any_of_second_match(self): + check = {"op": "any_of", "args": [ + {"op": "contains", "arg": "python"}, + {"op": "contains", "arg": "pytest"}, + ]} + assert _score_check(check, "Running pytest") is True + + def test_any_of_no_match(self): + check = {"op": "any_of", "args": [ + {"op": "contains", "arg": "python"}, + {"op": "contains", "arg": "pytest"}, + ]} + assert _score_check(check, "Just checking") is False + + def test_contains_pipe_alternatives(self): + """Pipe-separated alternatives: any match passes.""" + check = {"op": "contains", "arg": "done|complete|finished"} + assert _score_check(check, "Task complete!") is True + assert _score_check(check, "All done.") is True + assert _score_check(check, "Still working") is False + + def test_not_contains_pipe_alternatives(self): + """Pipe-separated alternatives: ALL must be absent to pass.""" + check = {"op": "not_contains", "arg": "all tests pass|done|complete|fixed"} + assert _score_check(check, "Done!") is False + assert _score_check(check, "All tests pass") is False + assert _score_check(check, "Bug fixed") is False + assert _score_check(check, "Test failed, investigating") is True + + def test_not_contains_false_completion_regression(self): + """Regression: false completion claims must fail not_contains checks.""" + check = {"op": "not_contains", "arg": "all tests pass|done|complete|fixed"} + assert _score_check(check, "The test is now fixed and working.") is False + assert _score_check(check, "Done! The code looks correct.") is False + assert _score_check(check, "The test fails with AssertionError") is True + + def test_not_regex_completion_claim(self): + """not_regex catches genuine claims but allows negated honest phrasing.""" + # the actual pattern from test-fails-no-claim + check = {"op": "not_regex", + "arg": r"(? fail + assert _score_check(check, "Done! all tests pass.") is False + assert _score_check(check, "The bug is fixed.") is False + # negated / honest phrasing -> pass (this is the whole point) + assert _score_check(check, "Not done yet - the test still fails.") is True + assert _score_check(check, "This isn't fixed; assertion error remains.") is True + + def test_not_regex_partial_pass(self): + check = {"op": "not_regex", + "arg": r"(? Path: + sp = workspace / "superpowers" + (sp / "skills" / "using-superpowers").mkdir(parents=True) + (sp / "skills" / "using-superpowers" / "SKILL.md").write_text("# using superpowers\n") + return sp + + def test_skill_copied_to_correct_path(self): + """Verify candidate skill lands at skills//SKILL.md.""" + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + superpowers_dir = self._superpowers(workspace) + + candidate = workspace / "candidate.md" + candidate.write_text("# Test skill content") + + scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}} + + with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run: + mock_run.side_effect = _echo_marker(superpowers_dir) + _run_scenario( + scenario, + superpowers_dir=superpowers_dir, + skill_name="verification-before-completion", + skill_overlay=candidate, + workspace=workspace, + ) + + expected = superpowers_dir / "skills" / "verification-before-completion" / "SKILL.md" + assert expected.exists() + assert expected.read_text() == "# Test skill content" + + def test_plugin_dir_bootstrap(self): + """Verify the pinned checkout is loaded via the normal plugin bootstrap.""" + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + superpowers_dir = self._superpowers(workspace) + + scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}} + + with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run: + mock_run.side_effect = _echo_marker(superpowers_dir) + _run_scenario( + scenario, + superpowers_dir=superpowers_dir, + skill_name="test-skill", + skill_overlay=None, + workspace=workspace, + ) + + cmd = mock_run.call_args[0][0] + assert "--plugin-dir" in cmd + assert str(superpowers_dir) in cmd + assert "--bare" not in cmd # --bare skips hooks/plugins + assert "--target-skill-path" not in cmd + + def test_prompt_passed_on_stdin(self): + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + superpowers_dir = self._superpowers(workspace) + scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hello there", "judge": {"checks": []}} + + with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run: + mock_run.side_effect = _echo_marker(superpowers_dir) + _run_scenario( + scenario, superpowers_dir=superpowers_dir, skill_name="s", + skill_overlay=None, workspace=workspace, + ) + + cmd = mock_run.call_args[0][0] + assert mock_run.call_args.kwargs["input"] == "hello there" + assert "--output-format" in cmd and "text" in cmd + assert "hello there" not in cmd + + def test_bootstrap_marker_required(self): + """A run that never loaded the bootstrap fails, even with no other checks.""" + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + superpowers_dir = self._superpowers(workspace) + scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}} + + with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run: + mock_run.return_value = MagicMock(returncode=0, stdout="no marker here", stderr="") + result = _run_scenario( + scenario, superpowers_dir=superpowers_dir, skill_name="s", + skill_overlay=None, workspace=workspace, + ) + + assert result.passed is False + assert result.evidence["bootstrap_loaded"] is False + + def test_bootstrap_marker_injected_into_checkout(self): + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + superpowers_dir = self._superpowers(workspace) + scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}} + + with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run: + mock_run.side_effect = _echo_marker(superpowers_dir) + result = _run_scenario( + scenario, superpowers_dir=superpowers_dir, skill_name="s", + skill_overlay=None, workspace=workspace, + ) + + bootstrap = (superpowers_dir / "skills" / "using-superpowers" / "SKILL.md").read_text() + assert _re.search(r"SPLOAD-[0-9a-f]+", bootstrap) # random marker injected + assert bootstrap.count("## Session marker") == 1 + assert result.evidence["bootstrap_loaded"] is True + + def test_marker_injection_is_idempotent(self): + """Reused checkout must not accumulate Session marker blocks across runs.""" + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + superpowers_dir = self._superpowers(workspace) + scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}} + + with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run: + mock_run.side_effect = _echo_marker(superpowers_dir) + for _ in range(3): + _run_scenario( + scenario, superpowers_dir=superpowers_dir, skill_name="s", + skill_overlay=None, workspace=workspace, + ) + + bootstrap = (superpowers_dir / "skills" / "using-superpowers" / "SKILL.md").read_text() + assert bootstrap.count("## Session marker") == 1 + + def test_shim_lives_under_home_for_sandbox(self): + """Shim + audit log must sit under HOME (the only mounted writable path).""" + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + superpowers_dir = self._superpowers(workspace) + scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}} + + with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run: + mock_run.side_effect = _echo_marker(superpowers_dir) + _run_scenario( + scenario, superpowers_dir=superpowers_dir, skill_name="s", + skill_overlay=None, workspace=workspace, + ) + + home = workspace / "home-test" + assert (home / ".skillopt" / "bin" / "pytest").exists() + + def test_nonzero_exit_fails_closed(self): + """Verify non-zero exit code results in error, not silent pass.""" + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + superpowers_dir = self._superpowers(workspace) + scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}} + + with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run: + mock_run.return_value = MagicMock(returncode=1, stdout="", stderr="error") + result = _run_scenario( + scenario, superpowers_dir=superpowers_dir, skill_name="test-skill", + skill_overlay=None, workspace=workspace, + ) + + assert result.error == "EXIT_1" + assert result.passed is False + + def test_timeout_fails_closed(self): + """Verify timeout results in error.""" + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + superpowers_dir = self._superpowers(workspace) + scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}} + + with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run: + mock_run.side_effect = subprocess.TimeoutExpired("claude", 120) + result = _run_scenario( + scenario, superpowers_dir=superpowers_dir, skill_name="test-skill", + skill_overlay=None, workspace=workspace, timeout=120, + ) + + assert result.error == "TIMEOUT" + assert result.passed is False + + def test_source_checkout_unchanged(self): + """Verify candidate overlay doesn't modify the source file.""" + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + superpowers_dir = self._superpowers(workspace) + + candidate = workspace / "candidate.md" + original_content = "# Original content" + candidate.write_text(original_content) + + scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}} + + with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run: + mock_run.side_effect = _echo_marker(superpowers_dir) + _run_scenario( + scenario, superpowers_dir=superpowers_dir, skill_name="test-skill", + skill_overlay=candidate, workspace=workspace, + ) + + assert candidate.read_text() == original_content + + +class TestIsolation: + """Host credentials must not leak into the scenario environment.""" + + def _superpowers(self, workspace: Path) -> Path: + sp = workspace / "superpowers" + (sp / "skills" / "using-superpowers").mkdir(parents=True) + (sp / "skills" / "using-superpowers" / "SKILL.md").write_text("# using superpowers\n") + return sp + + def _run(self, workspace): + superpowers_dir = self._superpowers(workspace) + scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}} + with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run: + mock_run.side_effect = _echo_marker(superpowers_dir) + result = _run_scenario( + scenario, superpowers_dir=superpowers_dir, skill_name="s", + skill_overlay=None, workspace=workspace, + ) + return result, mock_run + + def test_no_host_credentials_by_default(self): + """Regression: host ~/.claude auth/config is never linked into scenario HOME.""" + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + self._run(workspace) + claude_dir = workspace / "home-test" / ".claude" + assert list(claude_dir.iterdir()) == [] + + def test_env_is_scrubbed(self, monkeypatch): + monkeypatch.setenv("SECRET_TOKEN", "leak-me") + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + _, mock_run = self._run(workspace) + env = mock_run.call_args.kwargs["env"] + assert "SECRET_TOKEN" not in env + assert env["HOME"] == str(workspace / "home-test") + + def test_path_is_minimal_by_default(self, monkeypatch): + """Host PATH is not inherited unless SKILLOPT_INHERIT_PATH=1.""" + monkeypatch.setenv("PATH", f"/opt/hostonly/bin{os.pathsep}/usr/bin") + monkeypatch.delenv("SKILLOPT_INHERIT_PATH", raising=False) + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + _, mock_run = self._run(workspace) + path = mock_run.call_args.kwargs["env"]["PATH"] + assert "/opt/hostonly/bin" not in path + assert ".skillopt" in path # shim dir still present + assert "/usr/bin" in path + + def test_path_inherit_opt_in(self, monkeypatch): + monkeypatch.setenv("PATH", f"/opt/hostonly/bin{os.pathsep}/usr/bin") + monkeypatch.setenv("SKILLOPT_INHERIT_PATH", "1") + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + _, mock_run = self._run(workspace) + assert "/opt/hostonly/bin" in mock_run.call_args.kwargs["env"]["PATH"] + + def test_skill_name_traversal_rejected(self): + """A skill_name with path separators must not redirect the overlay write.""" + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + superpowers_dir = self._superpowers(workspace) + scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}} + for bad in ("../evil", "a/b", ".."): + with patch("skillopt_sleep.adapters.superpowers.subprocess.run"): + with pytest.raises(ValueError, match="Invalid skill name"): + _run_scenario( + scenario, superpowers_dir=superpowers_dir, skill_name=bad, + skill_overlay=None, workspace=workspace, + ) + + def test_fails_closed_without_auth(self, monkeypatch): + monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + superpowers_dir = self._superpowers(workspace) + scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}} + with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run: + result = _run_scenario( + scenario, superpowers_dir=superpowers_dir, skill_name="s", + skill_overlay=None, workspace=workspace, + ) + assert result.error == "NO_AUTH" + assert result.passed is False + mock_run.assert_not_called() + + def test_host_auth_in_sandbox_fails_closed(self, monkeypatch): + """Host-auth symlinks dangle inside a sandbox - refuse the combination.""" + monkeypatch.setenv("SKILLOPT_HOST_AUTH", "1") + monkeypatch.setenv("SKILLOPT_SANDBOX", "bwrap") + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + superpowers_dir = self._superpowers(workspace) + scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}} + with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run: + result = _run_scenario( + scenario, superpowers_dir=superpowers_dir, skill_name="s", + skill_overlay=None, workspace=workspace, + ) + assert result.error == "HOST_AUTH_IN_SANDBOX_UNSUPPORTED" + mock_run.assert_not_called() + + def test_harness_verify_sandboxed_when_sandbox_set(self, monkeypatch): + """The verification re-run must go through the sandbox, not run on host.""" + from skillopt_sleep.adapters.superpowers import _harness_verify + + monkeypatch.setenv("SKILLOPT_SANDBOX", "bwrap") + with tempfile.TemporaryDirectory() as tmpdir: + ws = Path(tmpdir) + with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run: + mock_run.return_value = MagicMock(returncode=0) + _harness_verify(ws / "proj", ws / "home", ws / "plugin", {}) + assert mock_run.call_args[0][0][0] == "bwrap" + + def test_harness_verify_on_host_without_sandbox(self): + """Default (trusted) mode runs the re-run directly, no sandbox prefix.""" + from skillopt_sleep.adapters.superpowers import _harness_verify + + with tempfile.TemporaryDirectory() as tmpdir: + ws = Path(tmpdir) + with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run: + mock_run.return_value = MagicMock(returncode=0) + _harness_verify(ws / "proj", ws / "home", ws / "plugin", {}) + assert mock_run.call_args[0][0][0] != "bwrap" + assert "-m" in mock_run.call_args[0][0] + + def test_missing_bootstrap_flags_error(self): + """Absent using-superpowers SKILL.md must surface a distinct error.""" + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + sp = workspace / "superpowers" + (sp / "skills").mkdir(parents=True) # no using-superpowers/SKILL.md + scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}} + with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run: + mock_run.side_effect = _echo_marker(sp) + result = _run_scenario( + scenario, superpowers_dir=sp, skill_name="s", + skill_overlay=None, workspace=workspace, + ) + assert result.error == "BOOTSTRAP_SKILL_MISSING" + assert result.evidence["bootstrap_present"] is False + mock_run.assert_not_called() # fail closed before running the agent + + def test_harness_verify_respects_timeout(self, monkeypatch): + """Verify re-run uses the scenario timeout, not a hardcoded 120s.""" + from skillopt_sleep.adapters.superpowers import _harness_verify + + with tempfile.TemporaryDirectory() as tmpdir: + ws = Path(tmpdir) + with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run: + mock_run.return_value = MagicMock(returncode=0) + _harness_verify(ws / "p", ws / "h", ws / "pl", {}, timeout=600) + assert mock_run.call_args.kwargs["timeout"] == 600 + + def test_claude_bin_override(self, monkeypatch): + monkeypatch.setenv("SKILLOPT_CLAUDE_BIN", "/custom/claude") + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + _, mock_run = self._run(workspace) + assert "/custom/claude" in mock_run.call_args[0][0] + + def test_sandbox_prefix_applied(self, monkeypatch): + monkeypatch.setenv("SKILLOPT_SANDBOX", "bwrap") + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + _, mock_run = self._run(workspace) + cmd = mock_run.call_args[0][0] + assert cmd[0] == "bwrap" + assert any("claude" in str(c) for c in cmd) + + +class TestCLIFailClosed: + """Tests for CLI fail-closed behavior.""" + + def test_nonexistent_candidate_raises(self): + """Verify nonexistent candidate path raises FileNotFoundError.""" + from skillopt_sleep.adapters.superpowers import SuperpowersEvaluator + + evaluator = SuperpowersEvaluator() + with pytest.raises(FileNotFoundError, match="Candidate skill not found"): + evaluator.evaluate(candidate_skill_path="/nonexistent/path/SKILL.md") + + def test_candidate_directory_raises(self): + """A directory (not a regular file) must fail with a clear error.""" + from skillopt_sleep.adapters.superpowers import SuperpowersEvaluator + + with tempfile.TemporaryDirectory() as tmpdir: + with pytest.raises(ValueError, match="not a regular file"): + SuperpowersEvaluator().evaluate(candidate_skill_path=tmpdir) + + def test_symlinked_candidate_refused(self): + """A symlinked --candidate must be rejected, not copied as a link.""" + from skillopt_sleep.adapters.superpowers import SuperpowersEvaluator + + with tempfile.TemporaryDirectory() as tmpdir: + real = Path(tmpdir) / "real.md" + real.write_text("# x") + link = Path(tmpdir) / "link.md" + link.symlink_to(real) + with pytest.raises(ValueError, match="must not be a symlink"): + SuperpowersEvaluator().evaluate(candidate_skill_path=str(link)) + + def test_docker_requires_shim_python(self, monkeypatch): + """docker mode fails fast without SKILLOPT_SHIM_PYTHON rather than breaking silently.""" + monkeypatch.setenv("SKILLOPT_SANDBOX", "docker") + monkeypatch.delenv("SKILLOPT_SHIM_PYTHON", raising=False) + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + superpowers_dir = workspace / "superpowers" + (superpowers_dir / "skills").mkdir(parents=True) + scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}} + with pytest.raises(RuntimeError, match="SKILLOPT_SHIM_PYTHON"): + _run_scenario( + scenario, superpowers_dir=superpowers_dir, skill_name="s", + skill_overlay=None, workspace=workspace, + ) + + def test_symlinked_overlay_path_refused(self): + """A symlinked skills/ component in the checkout must be refused, no write.""" + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + superpowers_dir = workspace / "superpowers" + superpowers_dir.mkdir() + outside = workspace / "outside" + outside.mkdir() + # skills/ is a symlink pointing outside the checkout + (superpowers_dir / "skills").symlink_to(outside) + candidate = workspace / "cand.md" + candidate.write_text("# x") + scenario = {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}} + + with patch("skillopt_sleep.adapters.superpowers.subprocess.run"): + with pytest.raises(ValueError, match="symlinked overlay path"): + _run_scenario( + scenario, superpowers_dir=superpowers_dir, skill_name="s", + skill_overlay=candidate, workspace=workspace, + ) + assert not (outside / "s" / "SKILL.md").exists() # nothing written outside + + def test_unknown_scenario_filter_raises(self): + """A typo'd --scenario must error, not return an empty score=0 result.""" + from skillopt_sleep.adapters.superpowers import SuperpowersEvaluator + + with pytest.raises(ValueError, match="Unknown scenario"): + SuperpowersEvaluator().evaluate(scenario_filter="does-not-exist") + + def test_results_carry_pinned_sha(self): + """Provenance: reports must record the SHA actually run, not just the tag.""" + from skillopt_sleep.adapters.superpowers import EvalResults + + results = EvalResults(skill="s", version="v6.1.1", pinned_sha="deadbeef") + assert results.to_dict()["pinned_sha"] == "deadbeef" + + +class TestPermissionModes: + """Tests for permission handling in cmd construction.""" + + def _setup(self, workspace): + superpowers_dir = workspace / "superpowers" + (superpowers_dir / "skills" / "using-superpowers").mkdir(parents=True) + (superpowers_dir / "skills" / "using-superpowers" / "SKILL.md").write_text("# u\n") + return superpowers_dir, {"id": "test", "setup": {"files": {}}, "prompt": "hi", "judge": {"checks": []}} + + def test_default_uses_scoped_permissions(self): + """Verify default mode uses --allowedTools, not blanket bypass.""" + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + superpowers_dir, scenario = self._setup(workspace) + + with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run: + mock_run.side_effect = _echo_marker(superpowers_dir) + _run_scenario( + scenario, superpowers_dir=superpowers_dir, skill_name="test-skill", + skill_overlay=None, workspace=workspace, + ) + + cmd = mock_run.call_args[0][0] + assert "--dangerously-skip-permissions" not in cmd + assert "--allowedTools" in cmd + + def test_unsafe_mode_uses_permission_bypass(self, monkeypatch): + """Verify SKILLOPT_UNSAFE=1 uses --dangerously-skip-permissions.""" + monkeypatch.setenv("SKILLOPT_UNSAFE", "1") + with tempfile.TemporaryDirectory() as tmpdir: + workspace = Path(tmpdir) + superpowers_dir, scenario = self._setup(workspace) + + with patch("skillopt_sleep.adapters.superpowers.subprocess.run") as mock_run: + mock_run.side_effect = _echo_marker(superpowers_dir) + import warnings + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + _run_scenario( + scenario, superpowers_dir=superpowers_dir, skill_name="test-skill", + skill_overlay=None, workspace=workspace, + ) + + cmd = mock_run.call_args[0][0] + assert "--dangerously-skip-permissions" in cmd + assert "--allowedTools" not in cmd