Advanced LLM Evaluation
Design LLM-as-judge rubrics, scoring and pairwise comparisons with bias mitigation
- Repository
- sickn33/agentic-awesome-skills
- Stars
- 49,023
- Licence
- MIT
- Files
- 1 file
What it does
This skill teaches an AI agent how to build and debug systems where one AI model grades the outputs of another. When installed, the agent knows how to write scoring rubrics, choose between rating a single answer on a scale or comparing two answers head to head, and guard against known judge biases such as favoring longer or first-listed responses. It also covers choosing metrics and checking how well automated judgments agree with human labels.
Great for
- Engineers building automated evaluation pipelines for LLM outputs
- Teams designing A/B tests for prompt or model changes
- Researchers calibrating an AI judge against human-labelled data
Skip it if
- Simply grading a single document or essay yourself, since no tooling or pipeline is needed for that (the skill is aimed at building evaluation systems)
- Treating its position-bias and metric guidance as a guaranteed fix: the file itself says results must be measured on your own cases and no universal improvement figure exists
Try asking
- “Design a pairwise comparison prompt that controls for position bias for choosing between two chatbot replies”
- “Write a 1-5 scoring rubric for factual accuracy and instruction following, with criteria and output format”
- “Help me measure how well our LLM judge agrees with human reviewers and which metric to use”
- “Debug why our evaluation pipeline gives inconsistent scores across runs”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 1 original files, the MIT licence and an attribution note.
advanced-evaluation.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip advanced-evaluation.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip advanced-evaluation.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Give the agent your actual task type (binary, ordinal, pairwise, multi-label) so it picks the right metrics from the table.
- Ask for a small set of human-reviewed cases to validate any judge before trusting its scores.
- Treat the example prompts as templates and adapt the criteria to your own domain.
What's inside
- SKILL.md
Redistributed unmodified from sickn33/agentic-awesome-skills under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.