LLM Benchmark Harness
Run standard academic benchmarks (MMLU, GSM8K, HumanEval and more) on language models and compare results.
- Repository
- Orchestra-Research/AI-Research-SKILLs
- Stars
- 14,195
- Licence
- MIT
- Files
- 5 files
What it does
Once installed, the agent can set up and run EleutherAI's lm-evaluation-harness, a widely used tool that tests language models on 60+ standard academic benchmarks such as MMLU, GSM8K, HellaSwag and HumanEval. It writes the correct command-line invocations, picks suitable tasks and settings, saves results to files, and helps compare models or track scores across training checkpoints. It is a guide for running an existing tool, not a new evaluation method.
Great for
- ML engineers who need to report standard benchmark scores for a model release
- Researchers comparing several open models on the same academic tests
- Teams tracking model quality across training checkpoints
Skip it if
- Judging how well a model performs on your own real-world tasks, since public benchmarks may not reflect them and may be contaminated in training data
- Anyone without a GPU or cloud budget, since larger models and tasks like MMLU can take hours to run
- Running untrusted code-generation tasks without a sandbox, because HumanEval-style tasks execute generated code
Try asking
- “Run MMLU, GSM8K and HellaSwag on my local Llama checkpoint and save the results to JSON”
- “Compare Mistral-7B and Phi-2 on TruthfulQA and GSM8K and build a comparison table”
- “Set up a quick 0-shot evaluation to run after every 1,000 training steps”
- “List the available lm-eval tasks that cover code generation”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 5 original files, the MIT licence and an attribution note.
lm-evaluation-harness.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip lm-evaluation-harness.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip lm-evaluation-harness.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Use the same few-shot setting and batch size across models so the scores are comparable, and record the config in the output file.
- Prefer quick tasks like HellaSwag, GSM8K or PIQA for frequent checkpoint evaluation and reserve MMLU for final reports.
- Review any shell or Python scripts the agent generates before running them, especially those that save checkpoints or call os.system.
What's inside
- references/api-evaluation.md
- references/benchmark-guide.md
- references/custom-tasks.md
- references/distributed-eval.md
- SKILL.md
Redistributed unmodified from Orchestra-Research/AI-Research-SKILLs under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.