You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Adds four developer-experience improvements to make EvalMonkey feel native to the agent development workflow:
Regression Guard — evalmonkey guard exits with code 1 if your agent's score drops vs the last baseline (CI/CD gate); auto-warns on every run-benchmark run.
Agent Card — evalmonkey report generates a shareable Markdown file with a shields.io badge and per-scenario score table, ready to paste into your README.
Smart Recommend — evalmonkey recommend reads agent_type from evalmonkey.yaml and shows only the relevant benchmark subset (e.g. research_agent → hotpotqa, drop, gaia-benchmark) instead of all 22.
External & Private Datasets — bring your own data via --dataset my_cases.jsonl, hf::org/dataset (any HuggingFace dataset), confident-ai::id / braintrust::ref / langsmith::id prefixes (harness on top of your existing eval platform datasets), or a Generic REST endpoint configured in evalmonkey.yaml
Activity
himmi-01 commentedon May 28, 2026
Completed by #14