Skip to content
Get packs

lm-evaluation-harness

A Python framework that runs standard benchmark tests on language models and reports scores.

GitHub Download v0.4.13
Stars
14.4k
Forks
3.7k
Licence
MIT
Language
Python
Last update
2 months ago
Type
Dev tool

What it is

This is a command-line framework from EleutherAI for testing large language models on a wide set of standard benchmarks. You point it at a model (from Hugging Face, vLLM, or a commercial API) and choose tasks, and it reports how the model scores. Researchers use it to compare models with the same prompts so results are reproducible.

  • Run the hellaswag benchmark on a Hugging Face model with a single lm_eval command
  • List available tasks with lm-eval ls tasks
  • Evaluate a model served by vLLM or an OpenAI-compatible API using the matching model type
  • Define a task or evaluation in a YAML config file and run it with --config
  • Evaluate a local model path or a GGUF model through the hf backend

Great for

  • Researchers who need reproducible scores on standard academic benchmarks
  • Engineers comparing several language models on the same tasks
  • Teams evaluating a locally hosted or Hugging Face model
  • People who want to add their own custom tasks or metrics

Not ideal for

  • People who just want to chat with a model; this only runs evaluations
  • Anyone without Python experience, since setup and configuration assume it
  • Users needing a GUI; the README describes a command-line interface
  • Multimodal evaluation at scale, since the README describes multimodal support as a prototype and points to lmms-eval for broader coverage
Step-by-step guide: how to use lm-evaluation-harnessInstall, first run, examples, tips and FAQs
lm-evaluation-harness vs promptfoo

How to get it

  1. 1

    Clone and build from source

    You'll need: Python · pip · CUDA-compatible GPU (for the HuggingFace example in the README)

    Clone the repository

    git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness

    Enter the directory

    cd lm-evaluation-harness

    Install the package

    pip install -e .

    Install the HuggingFace backend

    pip install "lm_eval[hf]"
  2. 2

    Run it

    • Install the package and the hf backend extra
    • List available tasks with lm-eval ls tasks
    • Run an evaluation on a Hugging Face model: lm_eval --model hf --model_args pretrained=EleutherAI/gpt-j-6B --tasks hellaswag --device cuda:0 --batch_size 8
    • Review the scores printed by the run
  3. 3

    Get the most out of it

    • Use --batch_size auto to let the tool pick the largest batch that fits your device
    • Use YAML config files via --config for repeatable evaluations
    • Register custom backends, filters, or metrics through the plugin system described in the Plugin Guide
    • Use the Python API (simple_evaluate) to run evaluations from your own code
    • Pass revision= in --model_args to evaluate partially trained checkpoints

Downloads

Source code (.zip)main branch · straight from GitHubRelease v0.4.13Release notes and sourceOfficial website & docswww.eleuther.ai

Strengths

  • Covers a large number of standard benchmarks (the README says over 60) with many subtasks
  • Supports many backends, including Hugging Face transformers, vLLM, SGLang, and commercial APIs
  • Permissive MIT license and an active project with a DOI for citation

Trade-offs

  • The base install omits model backends, so you must add extras such as lm_eval[hf] before running
  • Some benchmarks and backends need a GPU and significant setup, and the README's examples assume CUDA
  • The README is truncated here, so details on the full set of options and extras are not confirmed

How it compares

ProjectStarsTypeLevelUpdatedBest for
lm-evaluation-harness (this)14.4kDev toolintermediate2 months agoResearchers who need reproducible scores on standard academic benchmarks
promptfoo26.7kDev toolintermediate4 days agoDevelopers building apps on top of LLM APIs who want repeatable prompt tests
whisper.cpp54.9kLibraryintermediate7 days agoDevelopers who want to embed offline speech-to-text in a C, C++, or other native app
nano-vllm16.3kLibraryintermediate6 months agoDevelopers who want to understand how LLM inference engines work internally
LoRA13.9kLibraryintermediate1.9 years agoMachine learning engineers adapting RoBERTa, DeBERTa or GPT-2 style models in PyTorch
generative-ai-for-beginners123kLearning resourcebeginner5 days agoDevelopers who know some Python or TypeScript and want a guided introduction to generative AI

Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.