How to use the Advanced LLM Evaluation agent skill

What the skill makes the agent do

Once installed, the agent knows how to design the parts of an automated evaluation: a rubric that defines what good output means, a direct score that rates one response on a scale, or a pairwise comparison that picks the better of two responses. The skill's core idea is that LLM-as-a-Judge is a family of approaches, and choosing the right one for the context is the main task.

The skill also covers the known biases judge models show, such as position bias (favouring the first-listed response), length bias (favouring longer responses) and self-enhancement bias (rating their own outputs higher). For each one it lists mitigations, for example evaluating twice with swapped positions. It also covers choosing metrics, such as agreement rate for pairwise preference or Cohen's κ and Spearman's ρ for ordinal scales, and checking how automated judgments line up with human labels.

• Write scoring rubrics with level descriptions, characteristics, edge cases and scoring guidelines
• Pick between direct scoring and pairwise comparison for a given task
• Guard against position, length and other judge biases
• Choose metrics that match the task type
• Check judge agreement against human-labelled cases

Install the advanced-evaluation skill

Download the skill's ZIP file from this page. Then choose one of the two install options below.

For a personal install that works across your projects, unzip the file into ~/.claude/skills/. To limit the skill to a single project, unzip it into that project's .claude/skills/ folder instead.

If you use the Claude apps, you can upload the skill instead. Go to Settings → Capabilities → Skills and upload it there.

• Personal install: unzip into ~/.claude/skills/
• Project install: unzip into the project's .claude/skills/ folder
• Claude apps: Settings → Capabilities → Skills, then upload

Trigger the skill with natural requests

You do not need a special command. The skill is meant to activate when your request matches its description, which covers judge models, evals, position bias and calibration against human labels. Describe the evaluation problem you have in your own words.

The skill's own list of activation cases includes building automated evaluation pipelines, comparing multiple model responses to pick the best one, designing A/B tests for prompt or model changes, debugging evaluation systems with inconsistent results, creating rubrics, and analysing correlation between automated and human judgments.

• Help me write a rubric for scoring our support replies on accuracy and tone.
• Compare these two answers and tell me how to test which one a judge prefers without position bias.
• Our LLM judge gives different scores on reruns. Help me debug the evaluation pipeline.
• Design an A/B test for two prompt versions using a pairwise judge.
• How do I check how well my AI judge agrees with our human reviewers?

Example: a pairwise judge with position-bias checks

Suppose you are comparing two prompt versions. Ask the agent to set up a pairwise judge. The skill describes a protocol: run the first pass with Response A in first position, run a second pass with Response B in first position, and if the passes disagree, return TIE with reduced confidence. Only a consistent winner is kept.

The skill also includes a prompt structure that tells the judge not to prefer longer answers or based on position, and to output JSON with per-criterion comparisons, an overall winner, a confidence from 0 to 1 and reasoning. Expect the agent to adapt these templates to your criteria and data.

What you need before you start

The skill's metadata lists no required tools, packages or accounts. It does not run scripts and does not use the network. The package contains a single file, SKILL.md, which the agent reads to guide its work.

Running a judge model against your outputs is a separate step. The skill tells the agent how to design that system, but you will still need access to whichever model you use as the judge, and human-labelled cases if you want to measure agreement.

• No listed requirements or dependencies
• Package contents: SKILL.md only
• Does not run scripts; does not use the network
• You supply your own judge model access and evaluation cases

Safety notes and limits

Because the skill runs no scripts and makes no network calls, it adds guidance rather than executing code. Read the evaluation designs it produces before using them on real data, especially if the results will inform release decisions.

The skill is explicit that its guidance is not a guaranteed fix. Measure rubric and position-bias effects on your own cases, because no universal improvement figure exists. Also note that it is not meant for grading a single document yourself, where no pipeline is needed.

• Test every rubric and bias mitigation on your own representative cases
• Do not treat the skill's guidance as a guaranteed fix for bias
• Do not request hidden reasoning from the judge; ask for short, observable justifications instead

Tips for better results

Define the intended decision, rubric and acceptable disagreement before you look at judge outputs. The skill recommends calibrating the judge before trusting it, and running the pipeline as input, criteria, direct score or pairwise comparison, bias checks, human calibration, then report.

Use domain-specific terms in rubrics. The skill notes that a code-readability rubric should mention variables, functions and comments, while a medical rubric should reference clinical terminology and evidence standards.

• Choose a scale carefully: 1-3 for lowest cognitive load, 1-5 as a standard Likert scale, 1-10 only with detailed rubrics
• Look for systematic disagreement with humans on specific criteria, not just overall agreement
• Use a judge model different from the generator where possible, to reduce self-enhancement bias