Skip to content
Get packs

Eval Harness

Lets an agent define, run and report pass/fail tests for Claude Code workflows before trusting or changing them.

Source Download skill
Repository
affaan-m/ECC
Stars
295,127
Licence
MIT
Files
2 files

What it does

When installed, the agent can set up a structured testing process for AI-assisted coding work. Before a change is made, it writes down what success looks like as checklists (capability tests for new behavior, regression tests for things that must keep working), then runs checks such as grep, test suites or build commands and produces a pass/fail report with reliability numbers like pass@k. It is a method and template set for judging whether an AI workflow is reliable, not a product that does the coding itself.

Great for

  • Developers who use Claude Code on important projects and want evidence before trusting its changes
  • Teams setting up regression tests for prompts, agents or AI-assisted workflows
  • People comparing how reliable an agent is across model versions

Skip it if

  • Quick one-off scripts where a formal eval process adds overhead with no benefit
  • Security checks: the skill itself says human review is needed for security, so it should not be the only safeguard
  • Non-technical users without a test suite or build setup, since most example checks assume npm and code files

Try asking

  • “Set up an eval definition for adding a login feature before we write any code”
  • “Run the regression evals for the payment module and report which ones fail”
  • “Create a pass@3 reliability report for the code-review prompt we changed last week”
  • “Write a baseline.json of current test results so we can detect regressions later”
Step-by-step guide: how to use Eval HarnessInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 2 original files, the MIT licence and an attribution note.

    eval-harness.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip eval-harness.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip eval-harness.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Write eval definitions before coding so success criteria are fixed in advance
  • Prefer code-based graders (tests, grep, build) over model-based grading where possible, and review any shell commands the agent runs before approving them
  • Keep eval files in the project under .claude/evals and version them with the code

What's inside

  • agents/openai.yaml
  • SKILL.md

Redistributed unmodified from affaan-m/ECC under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.