How to use the Skill Eval Harness agent skill

What is an agent skill, and what does this one make Claude Code do?

An agent skill is a folder containing a SKILL.md file that gives Claude Code extra instructions for a specific kind of task. When the request matches the skill, the agent follows its workflow.

This skill applies evals-driven development (EDD). Evals are treated as the unit tests of AI work: you define the expected behaviour before implementation, run the evals continuously, track regressions with each change, and measure reliability with pass@k metrics.

• Capability evals: check whether Claude can do something it could not do before, with a list of success criteria.
• Regression evals: check that changes do not break existing functionality, using a baseline such as a git SHA or checkpoint name.
• Three grader types: code-based checks (grep, npm test, npm run build), model-based grading with a rubric, and human review for risky changes.
• Reporting: an EVAL REPORT summarising pass results and pass@1 / pass@3 metrics.

How do you install the eval-harness skill?

Download the ZIP file from the skill's page on this directory, then unzip it. The folder should contain SKILL.md. Put the folder in one of these locations:

• Personal use across projects: ~/.claude/skills/
• Single project: the .claude/skills/ folder inside your project
• Claude apps: upload the ZIP under Settings → Capabilities → Skills

How do you trigger the skill with natural requests?

You do not need a special command to start. Describe the goal in plain language and mention evals, pass/fail criteria, reliability or regression. The skill's own list of activation situations includes setting up EDD for AI-assisted workflows, defining pass/fail criteria for task completion, measuring reliability with pass@k, creating regression suites for prompt or agent changes, and comparing agent performance across model versions.

• "Before you implement this login change, write capability and regression evals for it."
• "Set up a regression suite for our agent prompt and record a baseline."
• "Measure how reliable this task is over three attempts using pass@3."
• "Compare how the agent performs on this workflow across two model versions."

Example: a test-first workflow for a feature

The skill's workflow has four stages: define, implement, evaluate, report. In the define stage the agent writes an eval definition with capability evals, regression evals and success metrics such as pass@3 above 90% for capability evals and pass^3 = 100% for regression evals.

The skill also suggests slash-command patterns for each stage. These are described in the skill text as patterns; check that your Claude Code setup provides them before relying on them.

• Before coding: /eval define feature-name creates .claude/evals/feature-name.md.
• During coding: /eval check feature-name runs current evals and reports status.
• After coding: /eval report feature-name generates the full eval report.

What do you need before you start?

The skill assumes a working development setup. Most examples are written around npm, so check these requirements before you install:

• Claude Code.
• A project with a test runner and a build script. The examples use npm test and npm run build.
• Shell access through the Bash tool.
• Git is implied for SHA baselines but is not strictly required.
• The skill runs scripts, but the source does not say it uses the network.

Where are evals stored, and how are results tracked?

Evals live inside your project so they can be versioned with the code, which the skill recommends as a best practice. The suggested layout keeps a definition file, an execution log, and a baseline file for regressions.

• feature-xyz.md: the eval definition.
• feature-xyz.log: the execution history.
• baseline.json: regression baselines.

What do pass@k and pass^k mean?

pass@k means at least one success in k attempts, so pass@1 is first-attempt reliability and pass@3 is success within three tries. pass^k means all k trials succeed, which is a stricter test of stability. The skill suggests pass@3 of at least 0.90 for capability evals, and pass^3 of 1.00 for critical release paths in regression evals.

Use these metrics as guidance for your own thresholds. The skill names them as recommended values, not guarantees.

Safety notes: scripts, human review and graders

The skill runs shell commands and writes files in your project, so review the commands before letting the agent run them, especially in a project you care about. Its tools list includes Read, Write, Edit, Bash, Grep and Glob.

The skill says security checks should keep a human in the loop and should never be fully automated. Model-based grading can be inconsistent, so do not allow unstable graders to gate releases.

• Keep human review for security-related changes.
• Prefer code-based graders because they are deterministic.
• Keep evals fast, because slow evals are not run.