How to use the Agent Evaluation Methods agent skill

What does the evaluation skill make the agent do?

Agents are harder to test than ordinary software because they make dynamic decisions, can produce different results between runs, and often have no single correct answer. The skill tells the agent to judge whether the final outcome is correct via a reasonable process, rather than checking for an exact sequence of steps. Two agents can take different routes to a valid answer, and both should pass.

Once the skill is active, the agent is directed to run cheap deterministic checks, such as schema validity, required evidence paths and rubric math, before any model-based judging. It scores results across separate dimensions such as factual accuracy, completeness, citation accuracy, source quality and tool efficiency, then computes weighted aggregates. It also helps set up regression suites, baseline comparisons and continuous production monitoring.

• Focus on outcomes, not execution paths.
• Score each dimension separately and store the breakdown with the total.
• Use numeric scores from 0.0 to 1.0, with thresholds around 0.7 for general use and 0.9 for high-stakes work.
• Stratify test sets from simple to very complex so easy cases do not inflate results.

How do you install the evaluation skill?

Download the ZIP for this skill from the page where you found this guide. Then either unzip it into your user skills folder, ~/.claude/skills/, or into a project's .claude/skills/ folder so that it applies only to that project.

If you use the Claude apps instead of a local setup, upload the ZIP under Settings → Capabilities → Skills.

• Personal install: unzip into ~/.claude/skills/.
• Project install: unzip into the project's .claude/skills/ folder.
• Claude apps: upload the ZIP under Settings → Capabilities → Skills.

What files come with the skill?

The package contains SKILL.md, which holds the main instructions and when to activate, a references/metrics.md file with metric definitions and scoring levels, and a scripts/evaluator.py file. The Python script provides starter classes for rubrics, test sets, a runner and a production monitor.

You need Python 3 to run evaluator.py. The source excerpt does not list any third-party packages for it.

• SKILL.md: main instructions and activation rules.
• references/metrics.md: metric definitions with levels from Excellent (1.0) to Failed (0.0).
• scripts/evaluator.py: starter classes for rubrics, test sets, runner and production monitor.

How do you trigger the skill with natural requests?

You do not need a special command. Describe the evaluation task in ordinary language and the agent should pick up the skill when the request matches its activation rules. Name the agent or pipeline you want to test and say what you want to measure.

Good requests state the goal, the artifact and the decision you need to make. Requests about judge design or autonomous control belong to other skills, so the agent may point you elsewhere.

• "Set up a regression suite for my support agent so I can catch regressions before I deploy a prompt change."
• "Compare these two agent configurations using one consistent scoring method."
• "Design quality gates for my research agent pipeline."
• "Plan production monitoring for an agent that answers customer questions."
• "Build a 30-case test set for my agent, stratified from simple to very complex."

What does a typical evaluation workflow look like?

A practical workflow starts small. Begin with roughly 20 to 30 representative cases during early development, sampled from real usage and including known edge cases. Run deterministic checks first, then score the outputs against a multi-dimensional rubric. Store per-dimension scores, then compare against a baseline before accepting a change.

As the system matures, scale toward 50 or more cases and report scores per complexity stratum. For stateful agents that change files, databases or configurations, check the final state against expected assertions after each run.

What safety notes should you know?

The skill is marked as running scripts and it includes evaluator.py, so read the script before running it on your system. The source states that the skill does not use the network.

Evaluation alone does not give you rollback or approval controls. If an autonomous agent changes files or systems without human review, use the separate harness-engineering skill for those controls.

• Review scripts/evaluator.py before executing it.
• The skill is documented as not using the network.
• Do not rely on evaluation alone to stop an autonomous agent from making unreviewed changes.

Tips for getting reliable results

Keep LLM-judge work separate from your core scoring. The skill says to use model-judged evaluation only after deterministic checks and rubrics are stable. When you do use an LLM judge, the skill recommends a different model family from the agent being evaluated to reduce self-enhancement bias.

Route edge cases and a random sample of production traffic to human reviewers, and feed the patterns they find back into your automated criteria.

• Fail fast on machine-checkable problems before spending tokens on judging.
• Use production-realistic token limits when you run evaluations.
• Compare a multi-agent design against a single-agent baseline before adopting it.
• Keep a baseline so every change is compared against the same reference.