How to use the Advanced LLM Evaluation agent skill

What does the advanced-evaluation skill make the agent do?

An agent skill is a folder of instructions and reference files that an AI agent loads when a task matches it. This one covers LLM-as-judge work: scoring a single response against a rubric (direct scoring) and comparing two responses head to head (pairwise comparison).

The skill also gives the agent known countermeasures for judge biases such as position bias (favoring whichever answer appears first), length bias (favoring longer answers), and self-enhancement bias. It includes guidance on choosing metrics and checking agreement with human reviewers.

Install the skill from the ZIP

Download the ZIP from this page and unzip it. To make the skill available to Claude Code in all your projects, place the skill folder in ~/.claude/skills/. To make it available only inside one project, place it in that project's .claude/skills/ folder.

In the Claude apps, you can instead upload the ZIP under Settings → Capabilities → Skills.

• Personal (all projects): ~/.claude/skills/
• Single project: .claude/skills/ inside the project folder
• Claude apps: Settings → Capabilities → Skills, then upload the ZIP

Trigger the skill with natural requests

You do not need a special command. The skill is meant to activate when your request is about building or debugging LLM evaluation. Phrases that match its stated scope include building an LLM-as-judge system, comparing several model responses to pick the best one, designing an A/B test for a prompt or model change, creating a rubric for an LLM judge, and analysing how automated judgments correlate with human ones.

Name the task and the data you have. The more specific you are about criteria and what counts as quality, the better the rubric the agent will write.

• Help me build an LLM-as-judge that scores chatbot answers from 1 to 5 for factual accuracy.
• Compare these two prompt variants on tone and persuasiveness and tell me which wins, with a confidence level.
• My automated judge disagrees with human raters on the 'helpfulness' criterion. Help me debug the rubric.

What the agent does with your request

For subjective comparisons, the skill has the agent run deterministic pre-checks first, then judge the pair in both orders. If the two passes disagree, the result is a TIE with reduced confidence. For direct scoring, the prompt requires the judge to find evidence in the response before emitting a score.

The bundled example script, scripts/evaluation_example.py, is a Python 3 example that uses only standard typing features and no third-party packages. Reference files cover bias mitigation, the evaluation pipeline, implementation patterns, and metrics.

• Pairwise: pre-checks, pass with A first, pass with B first, consistency check, then a verdict
• Direct scoring: criteria, calibrated scale, evidence before score, structured JSON output
• Metrics: matched to task type, such as F1 for pass/fail, Spearman's rho for 1-5 scales, agreement rate for pairwise

What you need before running it

The skill itself only provides instructions and reference material. To run real evaluations you need an LLM API or the host agent to act as the judge. To run the bundled example script you need Python 3. The script uses no third-party packages.

The skill does not use the network itself. It does run scripts, so read the script before you run it on your machine.

• Python 3 for scripts/evaluation_example.py
• An LLM API or the host agent to act as the judge, if you run real evaluations
• No third-party Python packages are required by the example script

Safety notes on scripts and network use

According to the skill listing, it runs scripts and does not use the network. Review scripts/evaluation_example.py before executing it, as you would with any script an agent may run.

Any real judge calls you make go through whatever LLM API or host agent you connect, so those calls are governed by that provider's terms and your own data handling choices.

Tips for reliable judgments

Do not treat LLM judgments as ground truth. The skill itself notes that judges can carry systematic biases and should be validated against human ratings. Prioritise systematic disagreement patterns on specific criteria over overall agreement rates, because a judge that consistently disagrees on one criterion is more harmful than one with random noise.

Use a different model for generation and evaluation to reduce self-enhancement bias, and use criteria-specific rubrics that penalise irrelevant detail.

• Always swap positions in pairwise tests
• Pick the scale to match rubric detail: 1-3 or 1-5 for most cases, 1-10 only with detailed per-level rubrics
• Use the harness-engineering or evaluation skills for autonomous loops and deterministic regression tests