Guide · Agent skill
How to use the Eval Harness agent skill: setup, triggers and examples
Updated · written with AI from the project's own documentation and checked against our catalogue data
The Eval Harness skill (eval-harness) gives Claude Code a structured way to test AI-assisted work. Before a change is made, the agent writes down what success looks like, runs checks, and produces a pass/fail report. This guide explains what the skill does, how to install it, how to trigger it, and where it does not fit, so you can decide whether it is worth adding to your workflow.
Eval Harnessaffaan-m/ECC · MIT Download the skill Quick start
- 1Download and unzip. Download the ZIP from this page and unzip it so you have the eval-harness folder with SKILL.md and agents/openai.yaml.
- 2Place the skill folder. Move it into ~/.claude/skills/ for personal use or into your project's .claude/skills/ folder. Alternatively, upload it under Settings → Capabilities → Skills in the Claude apps.
- 3Confirm your test tooling. Make sure your project has commands such as npm test and npm run build, or equivalents for your stack, since the example checks depend on them.
- 4Define evals before coding. Ask the agent to write capability and regression checklists for the change, with explicit success criteria, before any implementation.
- 5Run the checks. Use grep, test suite and build commands as code graders, recording PASS or FAIL for each check.
- 6Generate the report. Ask for an eval report showing capability and regression results, pass@k figures and a status line.
- 7Review manually. Have a human review the result, especially for security-related changes, before relying on it.
What the Eval Harness skill makes the agent do
The skill implements eval-driven development (EDD), which treats evals as the unit tests of AI development. You define expected behavior before implementation, run evals during development, and track regressions with each change.
There are two main kinds of eval. Capability evals test whether Claude can do something it could not do before. Regression evals check that existing functionality still works after a change.
- Capability evals: new behavior, written as a checklist of success criteria.
- Regression evals: things that must keep working, recorded as PASS or FAIL against a baseline.
- Graders: code-based checks (grep, test suites, build commands), model-based grading of open-ended output, or flags for human review.
- Metrics: pass@k (at least one success in k attempts) and pass^k (all k trials succeed).
Install the Eval Harness skill
Download the ZIP from this page and unzip it. The skill folder contains agents/openai.yaml and SKILL.md. Choose one of the following locations based on how you want to use it.
For a personal install across projects, unzip into ~/.claude/skills/. For a single project, unzip into that project's .claude/skills/ folder.
If you use the Claude apps, you can instead upload the skill under Settings → Capabilities → Skills.
- Personal install: ~/.claude/skills/
- Project install: .claude/skills/ inside your project
- Claude apps: Settings → Capabilities → Skills, then upload
Trigger the skill with natural requests
You do not need a special command to start. The skill is meant to activate when a workflow needs a formal eval before it is trusted or changed. Describe the goal in plain language.
The source also lists slash-command patterns that the skill documents for integration: define, check and report. These create and read eval files under .claude/evals/.
- Ask for an eval before a change: "Set up evals for adding password reset before you write any code."
- Ask for a regression check: "Create a regression eval so existing login behavior is protected."
- Ask for reliability measurement: "Measure pass@3 for this refactor across the test suite."
- Ask for a report: "Generate an eval report for the feature we just built."
/eval define feature-name
/eval check feature-name
/eval report feature-nameExample: adding authentication with evals
The skill's own example follows four phases: define, implement, evaluate, and report. Evals are written first, as checklists, and the implementation is judged against them.
The Define phase lists capability items such as user registration, login with valid credentials, and rejected invalid credentials. It also lists regression items such as public routes and unchanged API responses. The Report phase summarizes results as counts and a status line.
- Define: write capability and regression checklists before coding.
- Implement: write code to pass the defined evals.
- Evaluate: run /eval check on the feature.
- Report: summarize capability and regression results with pass@k and a status.
EVAL REPORT: add-authentication
==============================
Capability: 5/5 passed (pass@3: 100%)
Regression: 3/3 passed (pass^3: 100%)
Status: SHIP ITWrite code-based checks
Code-based graders are the most reliable option, because they are deterministic. The skill's examples use grep to confirm an expected pattern exists, npm test to run a filtered test suite, and npm run build to confirm the build succeeds. Adapt these to your own stack.
Each check prints PASS or FAIL, which the eval report can record.
- Check a pattern exists with grep.
- Run a filtered test suite with npm test.
- Confirm the project builds with npm run build.
grep -q "export function handleAuth" src/auth.ts && echo "PASS" || echo "FAIL"
npm test -- --testPathPattern="auth" && echo "PASS" || echo "FAIL"
npm run build && echo "PASS" || echo "FAIL"Know what the skill needs and its safety limits
The skill requires Claude Code or a compatible agent with the Read, Write, Edit, Bash, Grep and Glob tools. Most example checks assume npm and code files, so you need a project with test tooling, such as npm test and npm run build, or an equivalent for your stack.
The skill runs scripts, because the checks are shell commands run through Bash. It does not use the network. The skill itself says security checks need human review, so treat it as one input, not the only safeguard for security-sensitive changes.
- Runs scripts: yes (shell checks via Bash)
- Uses network: no
- Security: human review is required; do not rely on the skill alone
- Licence: MIT
Tips for getting useful results
Keep evals fast, because slow evals do not get run. Prefer code graders over model graders when you can, since deterministic checks are more repeatable. Store evals with the code so they are versioned alongside it.
The skill suggests a typical target of pass@3 above 90% for capability evals, and pass^3 at 100% for regression evals on critical paths. These are the skill's example targets; set your own for your project.
- Define evals before coding, not after.
- Run evals frequently to catch regressions early.
- Track pass@k over time to see reliability trends.
- Store files in .claude/evals/, including feature definitions, run logs and baseline.json.
What people use it for
- Setting up regression tests before the agent changes login or session code on an important project.
- Checking that a refactor still passes the existing test suite and build.
- Comparing how consistently an agent completes the same task across model versions using pass@k.
- Creating a checklist-based review for a prompt or agent workflow that a team relies on.
Frequently asked questions
How do I install the Eval Harness skill?
Download the ZIP from this page and unzip it into ~/.claude/skills/ for personal use, or into a project's .claude/skills/ folder. In the Claude apps, you can upload it under Settings → Capabilities → Skills.
Is the Eval Harness skill free?
The skill is published under the MIT licence, as stated in its SKILL.md file. This page does not list any price, so check the licence terms in the source repository for details.
What do I need to use it?
You need Claude Code or a compatible agent with Read, Write, Edit, Bash, Grep and Glob tools. Most example checks also assume a project with test tooling such as npm test and npm run build.
Does the skill write my code for me?
No. It is a method and template set for defining and running evals and producing reports. It judges whether an AI workflow is reliable and does not do the coding itself.
Does Eval Harness use the internet or run scripts?
It runs scripts, since its checks are shell commands through Bash, and it does not use the network. Human review is still needed for security checks, so do not rely on the skill as the only safeguard.
Is it a good fit for small scripts?
Probably not. The skill is intended for workflows where a formal eval is needed before trust or change, and a formal process adds overhead for quick one-off scripts.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use the Eval Harness skill if you work with Claude Code on important projects or maintain regression tests for AI-assisted workflows, and want evidence before trusting changes. Skip it for quick one-off scripts, for security checks that need more than a human-reviewed checklist, or if you have no test suite or build setup to run its examples.
Download the skill