Guide · Open source

How to use lm-evaluation-harness: install and run benchmarks

Updated 2026-10-29 · written with AI from the project's own documentation and checked against our catalogue data

lm-evaluation-harness is an open-source, command-line framework from EleutherAI for scoring language models on a large set of standard benchmarks. This guide is for people comfortable with Python and a terminal who want reproducible scores for a model. By the end you will know how to install it, run a Hugging Face model on a task, and find the tasks you can choose from.

lm-evaluation-harnessEleutherAI/lm-evaluation-harness · ★ 14.4k · MIT Install options & downloads

Quick start

1Clone the repository. Run git clone --depth 1 on the EleutherAI repository and enter the lm-evaluation-harness folder.
2Install the package. Run pip install -e . inside the folder to install the core framework.
3Install a model backend. Install the hf extra with pip install "lm_eval[hf]" for Hugging Face models.
4List tasks. Run lm-eval ls tasks to see the benchmarks available.
5Run an evaluation. Run lm_eval with --model hf, --model_args pretrained set to your model, --tasks, --device and --batch_size.

Install lm-evaluation-harness from the repository

The package is installed by cloning the GitHub repository and building it with pip. The base install is the core framework only, so you also need to add a model backend.

Model backends are installed separately with optional extras. For a Hugging Face transformers model, install the hf extra.

Prerequisites named in the README: Python, pip, and a CUDA-compatible GPU for the Hugging Face example.

git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .
pip install "lm_eval[hf]"

Run a Hugging Face model on a benchmark

To evaluate a model hosted on the Hugging Face Hub, pass the hf model type, the model name in --model_args, and the task in --tasks. The README's example uses EleutherAI/gpt-j-6B on hellaswag with a CUDA device.

You can pass extra constructor arguments through --model_args, such as a revision or a datatype. Setting --batch_size to auto detects the largest batch that fits on your device.

--model hf selects the Hugging Face backend.
--device cuda:0 selects the GPU.
--batch_size 8 sets a fixed batch size; auto or auto:N lets the harness pick it.

lm_eval --model hf \
    --model_args pretrained=EleutherAI/gpt-j-6B \
    --tasks hellaswag \
    --device cuda:0 \
    --batch_size 8

Find the tasks you can run

The framework includes over 60 standard academic benchmarks with hundreds of subtasks and variants. Use the ls subcommand to list them before you choose what to run.

Run lm-eval -h for top-level options, or lm-eval run -h for evaluation options. The CLI reference and configuration guide in the docs folder describe the rest.

lm-eval ls tasks

Evaluate a local path, a GGUF model or a commercial API

A local path can be passed to pretrained in the same way as a Hub name. The README also says the hf backend supports GGUF models for evaluation.

For API-based models such as OpenAI, install the api extra. The README lists OpenAI and TextSynth among supported commercial APIs, and notes that vLLM is supported for fast inference. Install those backends with their extras as needed.

Local path: pretrained=/path/to/model
Other backends: vllm and api extras, installed with pip install "lm_eval[vllm]" or "lm_eval[api]"

Add your own tasks, metrics or plugins

The README describes config-based task creation, Jinja2 prompt design, and custom metrics. Recent updates also let you register model backends, filters, metrics and aggregations from your own package, or point --plugins at a local module.

Read the Task Guide and Plugin Guide linked from the README before writing custom code, as the exact configuration format is documented there.

Config-based task YAML files
Custom prompts and evaluation metrics
Plugins via an lm_eval.* entry point or --plugins

What people use it for

Compare several language models on hellaswag or other standard benchmarks using the same prompts.
Check a revision or partially trained checkpoint of a Hugging Face model by passing revision in --model_args.
Evaluate a locally stored model by giving its path as pretrained.
Run evaluations against a hosted model through a vLLM server using the local-completions model type, as the README recommends for Llama 405B.

Frequently asked questions

Is lm-evaluation-harness free to use?

Yes, the project is released under the MIT licence, according to the listing. The README does not state any pricing, but any model API you call may charge its own fees.

What do I need to install lm-evaluation-harness?

The README names Python, pip, and a CUDA-compatible GPU for the Hugging Face example. You clone the repository and run pip install -e, then add a backend such as the hf extra.

How do I run lm-evaluation-harness on a Hugging Face model?

Run lm_eval with --model hf, set --model_args pretrained to the model name, and pass a task to --tasks. The README example uses EleutherAI/gpt-j-6B on hellaswag.

Does lm-evaluation-harness have a GUI?

No. The README describes a command-line interface, and this tool only runs evaluations rather than chatting with a model.

Can lm-evaluation-harness evaluate multimodal models?

Multimodal support is described in the README as a prototype. For broader multimodal coverage, the README points to lmms-eval.

How does lm-evaluation-harness compare to HELM or OpenCompass?

This guide does not compare benchmark results or features across these projects. The listing names HELM, OpenCompass and lmms-eval as comparable projects, so check each project's documentation for the details that matter to you.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use lm-evaluation-harness if you need reproducible scores on standard academic benchmarks, or want to compare several models on the same tasks. Skip it if you want to chat with a model, need a graphical interface, or have no Python experience, and consider lmms-eval for broader multimodal work.

Install options & downloads