Complete LM Evaluation Harness Tutorial: The Standard Framework for Evaluating Large Language Models
Evaluating large language models (LLMs) is a critical step in the AI development lifecycle. Without systematic evaluation, we cannot measure model performance, compare models against each other, or verify that fine-tuning efforts yield real improvements. LM Evaluation Harness (lm-eval) by EleutherAI has become the de facto standard for LLM evaluation, used by researchers, practitioners, and major organizations worldwide.
This tutorial provides an in-depth guide to using lm-eval for evaluating various language models, from installation to creating custom benchmarks.
What Is LM Evaluation Harness?
LM Evaluation Harness is an open-source framework developed by EleutherAI for evaluating generative language models in a consistent and reproducible manner. The framework provides hundreds of standard benchmarks including MMLU, HellaSwag, GSM8K, ARC, and many more.
Key advantages of lm-eval include:
- Standardization: Hundreds of integrated benchmarks with consistent implementations
- Reproducibility: Evaluation results can be reproduced with the same configuration
- Flexibility: Supports models from HuggingFace, OpenAI API, Anthropic, vLLM, and more
- Extensibility: Easy to add new benchmarks and model backends
- Community: Widely adopted across the global AI research community
Installation and Setup
Basic Installation
The quickest way to install lm-eval is via pip:
pip install lm-eval
Installation from Source
To get the latest version with all features:
git clone https://github.com/EleutherAI/lm-evaluation-harness.git
cd lm-evaluation-harness
pip install -e ".[all]"
Installation with Additional Dependencies
If you want to use models from specific providers:
# For HuggingFace models with quantization
pip install lm-eval[gptq,awq]
For API-based models
pip install lm-eval[api]
For vLLM backend
pip install lm-eval[vllm]
For all dependencies
pip install lm-eval[all]
Verify Installation
lmeval --help
If the command above displays a list of options, the installation was successful.
Basic Usage
Evaluating a HuggingFace Model
The simplest example is evaluating a model from the HuggingFace Hub:
lmeval --model hf \
--modelargs pretrained=microsoft/phi-2 \
--tasks hellaswag \
--batchsize 8 \
--device cuda:0
Parameters explained:
--model hf: Use the HuggingFace Transformers backend--modelargs: Model-specific arguments, including the Hub model name--tasks: Benchmarks to run--batchsize: Batch size for inference--device: Device to use (cuda:0, cpu, etc.)
Evaluating Multiple Benchmarks at Once
You can evaluate a model on several benchmarks simultaneously:
lmeval --model hf \
--model
args pretrained=meta-llama/Llama-3.1-8B \
--tasks hellaswag,mmlu,arceasy,arcchallenge,winogrande \
--batchsize auto \
--device cuda:0 \
--outputpath results/llama3-8b/
The --batchsize auto parameter automatically determines the optimal batch size based on available GPU memory.
Saving Evaluation Results
Evaluation results can be saved in JSON format:
lmeval --model hf \
--modelargs pretrained=Qwen/Qwen2.5-7B \
--tasks mmlu \
--batchsize auto \
--outputpath results/qwen25-7b/ \
--logsamples
The --logsamples flag saves details for each evaluation sample, useful for error analysis.
Popular Benchmarks
MMLU (Massive Multitask Language Understanding)
MMLU tests model knowledge across 57 domains, from mathematics to law:
lmeval --model hf \
--modelargs pretrained=mistralai/Mistral-7B-v0.3 \
--tasks mmlu \
--numfewshot 5 \
--batchsize auto
The --numfewshot 5 parameter provides 5 examples before the actual question (standard for MMLU).
HellaSwag
A benchmark for testing common-sense reasoning ability:
lmeval --model hf \
--model
args pretrained=microsoft/phi-2 \
--tasks hellaswag \
--numfewshot 10 \
--batchsize auto
GSM8K (Grade School Math)
Tests mathematical reasoning capabilities:
lmeval --model hf \
--model
args pretrained=meta-llama/Llama-3.1-8B \
--tasks gsm8k \
--numfewshot 5 \
--batchsize auto
ARC (AI2 Reasoning Challenge)
Elementary and middle school science questions:
lmeval --model hf \
--model
args pretrained=google/gemma-2-9b \
--tasks arceasy,arcchallenge \
--numfewshot 25 \
--batchsize auto
TruthfulQA
Tests whether models produce correct and non-misleading answers:
lmeval --model hf \
--model
args pretrained=meta-llama/Llama-3.1-8B \
--tasks truthfulqamc2 \
--numfewshot 0 \
--batchsize auto
Running a Complete Benchmark Suite
For comprehensive evaluation, run multiple benchmarks at once:
lmeval --model hf \
--modelargs pretrained=meta-llama/Llama-3.1-8B \
--tasks mmlu,hellaswag,arceasy,arcchallenge,winogrande,truthfulqamc2,gsm8k \
--numfewshot 5 \
--batchsize auto \
--outputpath results/llama3-8b-full/
Evaluating Models with Different Backends
Evaluating GGUF Models with llama.cpp
For models quantized in GGUF format:
lmeval --model gguf \
--modelargs baseurl=http://localhost:8080 \
--tasks hellaswag,mmlu \
--batchsize 1
Make sure the llama.cpp server is running first.
Evaluating Models via vLLM
vLLM provides faster inference:
lmeval --model vllm \
--modelargs pretrained=meta-llama/Llama-3.1-8B,tensorparallelsize=2 \
--tasks mmlu,hellaswag \
--batchsize auto
Evaluating Models via OpenAI API
To evaluate GPT models:
export OPENAIAPIKEY="sk-your-api-key"
lmeval --model openai-completions \
--modelargs model=gpt-3.5-turbo \
--tasks hellaswag,mmlu \
--batchsize 1
For chat models:
lmeval --model openai-chat-completions \
--modelargs model=gpt-4 \
--tasks mmlu \
--batchsize 1 \
--applychattemplate
Evaluating Quantized Models (GPTQ, AWQ)
# GPTQ model
lmeval --model hf \
--modelargs pretrained=TheBloke/Llama-2-7B-GPTQ,gptq=True \
--tasks hellaswag \
--batchsize auto
AWQ model
lmeval --model hf \
--modelargs pretrained=TheBloke/Llama-2-7B-AWQ,awq=True \
--tasks hellaswag \
--batchsize auto
Python API Usage
Beyond the CLI, lm-eval also provides a flexible Python API:
import lmeval
results = lm
eval.simpleevaluate(
model="hf",
model
args="pretrained=microsoft/phi-2",
tasks=["hellaswag", "mmlu"],
numfewshot=5,
batchsize="auto",
device="cuda:0",
)
Access results
for taskname, taskresult in results["results"].items():
print(f"\n{taskname}:")
for metric, value in taskresult.items():
if isinstance(value, (int, float)):
print(f" {metric}: {value:.4f}")
Advanced Configuration Evaluation
import lmeval
from lm
eval.models.huggingface import HFLM
Initialize model with custom configuration
model = HFLM(
pretrained="meta-llama/Llama-3.1-8B",
dtype="float16",
batchsize="auto",
device="cuda:0",
maxlength=4096,
)
Run evaluation
results = lmeval.simpleevaluate(
model=model,
tasks=["mmlu", "hellaswag", "gsm8k"],
numfewshot=5,
logsamples=True,
)
Save results
import json
with open("evaluationresults.json", "w") as f:
json.dump(results["results"], f, indent=2)
Comparing Multiple Models
import lmeval
import json
modelstocompare = [
"microsoft/phi-2",
"google/gemma-2-2b",
"Qwen/Qwen2.5-3B",
]
tasks = ["hellaswag", "arceasy", "mmlu"]
allresults = {}
for modelname in modelstocompare:
print(f"\nEvaluating {modelname}...")
results = lmeval.simpleevaluate(
model="hf",
modelargs=f"pretrained={modelname}",
tasks=tasks,
numfewshot=5,
batchsize="auto",
device="cuda:0",
)
allresults[modelname] = results["results"]
Print comparison table
print("\n" + "=" 80)
print(f"{'Model':<30} {'HellaSwag':<15} {'ARC-Easy':<15} {'MMLU':<15}")
print("=" 80)
for modelname, result in allresults.items():
hellaswag = result.get("hellaswag", {}).get("accnorm,none", 0)
arc = result.get("arceasy", {}).get("accnorm,none", 0)
mmlu = result.get("mmlu", {}).get("acc,none", 0)
shortname = modelname.split("/")[-1]
print(f"{shortname:<30} {hellaswag:<15.4f} {arc:<15.4f} {mmlu:<15.4f}")
Creating Custom Benchmarks
YAML Task Structure
You can create custom benchmarks using the YAML format:
# mycustomtask.yaml
task: mycustombenchmark
datasetpath: json
datasetkwargs:
datafiles: mytestdata.jsonl
outputtype: multiplechoice
trainingsplit: null
validationsplit: null
testsplit: train
doctotext: "Question: {{question}}\nAnswer:"
doctotarget: "{{answer}}"
metriclist:
- metric: acc
aggregation: mean
higherisbetter: true
numfewshot: 0
Creating an Evaluation Dataset
Prepare data in JSONL format:
{"question": "What is the capital of France?", "choices": ["London", "Paris", "Berlin", "Madrid"], "answer": 1}
{"question": "Who invented the telephone?", "choices": ["Edison", "Bell", "Tesla", "Marconi"], "answer": 1}
{"question": "Largest planet in the solar system?", "choices": ["Mars", "Venus", "Jupiter", "Saturn"], "answer": 2}
Generative Format Tasks
For generative text evaluation (not multiple choice):
# generativetask.yaml
task: my
generativetask
dataset
path: json
datasetkwargs:
datafiles: gentestdata.jsonl
outputtype: generateuntil
trainingsplit: null
testsplit: train
doctotext: "Translate to French: {{english}}\nTranslation:"
doctotarget: "{{french}}"
generationkwargs:
maxgentoks: 256
temperature: 0.0
dosample: false
metriclist:
- metric: bleu
aggregation: mean
higherisbetter: true
- metric: exactmatch
aggregation: mean
higherisbetter: true
numfewshot: 3
Running Custom Tasks
lmeval --model hf \
--modelargs pretrained=meta-llama/Llama-3.1-8B \
--tasks mycustombenchmark \
--includepath /path/to/custom/tasks/ \
--batchsize auto
Filters and Post-Processing
Using Built-in Filters
lm-eval provides several filters for processing model output:
task: filteredtask
datasetpath: json
datasetkwargs:
datafiles: testdata.jsonl
outputtype: generateuntil
doctotext: "{{question}}"
doctotarget: "{{answer}}"
generationkwargs:
maxgentoks: 512
until:
- "\n\n"
- "Question:"
filterlist:
- name: getanswer
filter:
- function: regex
regexpattern: "The answer is (\\w+)"
- function: takefirst
metriclist:
- metric: exactmatch
aggregation: mean
higherisbetter: true
Evaluating Fine-tuned Models
One of the most common uses of lm-eval is comparing models before and after fine-tuning:
import lmeval
tasks = ["mmlu", "hellaswag", "gsm8k"]
Evaluate base model
print("Evaluating base model...")
baseresults = lmeval.simpleevaluate(
model="hf",
modelargs="pretrained=meta-llama/Llama-3.1-8B",
tasks=tasks,
numfewshot=5,
batchsize="auto",
device="cuda:0",
)
Evaluate fine-tuned model
print("Evaluating fine-tuned model...")
ftresults = lmeval.simpleevaluate(
model="hf",
modelargs="pretrained=./my-finetuned-llama",
tasks=tasks,
numfewshot=5,
batchsize="auto",
device="cuda:0",
)
Compare results
print("\n" + "=" 60)
print(f"{'Task':<20} {'Base Model':<20} {'Fine-tuned':<20}")
print("=" 60)
for task in tasks:
baseacc = baseresults["results"].get(task, {}).get("acc,none", 0)
ftacc = ftresults["results"].get(task, {}).get("acc,none", 0)
delta = ftacc - baseacc
sign = "+" if delta > 0 else ""
print(f"{task:<20} {baseacc:<20.4f} {ftacc:.4f} ({sign}{delta:.4f})")
Performance Optimization Tips
Using auto batchsize
lmeval --model hf \
--modelargs pretrained=meta-llama/Llama-3.1-8B \
--tasks mmlu \
--batchsize auto:4
The number after auto: determines the maximum factor for auto-sizing.
Using Model Parallelism
For very large models:
lmeval --model hf \
--model
args pretrained=meta-llama/Llama-3.1-70B,parallelize=True \
--tasks hellaswag \
--batchsize 4
Using Quantization for Efficiency
lmeval --model hf \
--modelargs pretrained=meta-llama/Llama-3.1-8B,loadin4bit=True \
--tasks mmlu,hellaswag \
--batchsize auto
Limiting Sample Count
For quick testing:
lmeval --model hf \
--model
args pretrained=microsoft/phi-2 \
--tasks mmlu \
--limit 100 \
--batchsize auto
Best Practices
1. Consistency in Evaluation
Always use the same parameters when comparing models:
EVALCONFIG = {
"tasks": ["mmlu", "hellaswag", "arcchallenge", "winogrande", "gsm8k"],
"numfewshot": 5,
"batchsize": "auto",
}
resultsa = lmeval.simpleevaluate(
model="hf",
modelargs="pretrained=model-a",
EVALCONFIG,
)
resultsb = lmeval.simpleevaluate(
model="hf",
modelargs="pretrained=model-b",
EVALCONFIG,
)
2. Document Evaluation Configuration
Save the evaluation configuration alongside results:
import json
from datetime import datetime
evalrecord = {
"timestamp": datetime.now().isoformat(),
"model": "meta-llama/Llama-3.1-8B",
"lmevalversion": lmeval.version,
"tasks": ["mmlu", "hellaswag"],
"numfewshot": 5,
"results": results["results"],
}
with open("evalrecord.json", "w") as f:
json.dump(evalrecord, f, indent=2)
3. Choose the Right Benchmarks
Match benchmarks to your evaluation needs:
- General knowledge: MMLU, ARC
- Reasoning: HellaSwag, WinoGrande
- Mathematics: GSM8K, MATH
- Coding: HumanEval, MBPP
- Factual accuracy: TruthfulQA
- Domain-specific: Create custom tasks with your own dataset
4. Use Standard Few-shot Counts
Follow the standard few-shot counts for each benchmark:
| Benchmark | Standard Few-shot |
|-----------|------------------|
| MMLU | 5 |
| HellaSwag | 10 |
| ARC | 25 |
| WinoGrande | 5 |
| GSM8K | 5 |
| TruthfulQA | 0 |
5. Evaluate Regularly
Integrate evaluation into your CI/CD pipeline:
#!/bin/bash
evalpipeline.sh
MODELPATH=$1
OUTPUTDIR="evalresults/$(date +%Y%m%d%H%M%S)"
mkdir -p $OUTPUTDIR
lmeval --model hf \
--modelargs pretrained=$MODELPATH \
--tasks mmlu,hellaswag,arcchallenge,gsm8k \
--numfewshot 5 \
--batchsize auto \
--outputpath $OUTPUTDIR \
--logsamples
echo "Results saved to $OUTPUTDIR"
Integration with Other Tools
Integration with Weights & Biases
import lmeval
import wandb
wandb.init(project="llm-evaluation", name="llama3-8b-eval")
results = lm
eval.simpleevaluate(
model="hf",
model
args="pretrained=meta-llama/Llama-3.1-8B",
tasks=["mmlu", "hellaswag", "gsm8k"],
numfewshot=5,
batchsize="auto",
)
for taskname, taskresult in results["results"].items():
for metric, value in taskresult.items():
if isinstance(value, (int, float)):
wandb.log({f"{taskname}/{metric}": value})
wandb.finish()
Integration with MLflow
import lmeval
import mlflow
mlflow.set
experiment("llm-evaluation")
with mlflow.startrun(runname="llama3-8b-eval"):
results = lmeval.simpleevaluate(
model="hf",
modelargs="pretrained=meta-llama/Llama-3.1-8B",
tasks=["mmlu", "hellaswag"],
numfewshot=5,
batchsize="auto",
)
mlflow.logparam("model", "meta-llama/Llama-3.1-8B")
mlflow.logparam("tasks", "mmlu,hellaswag")
for taskname, taskresult in results["results"].items():
for metric, value in taskresult.items():
if isinstance(value, (int, float)):
mlflow.logmetric(f"{taskname}_{metric}", value)
Conclusion
LM Evaluation Harness is an essential tool in the toolkit of every AI/ML practitioner working with large language models. With hundreds of built-in benchmarks, support for various model backends, and the ability to create custom evaluations, lm-eval provides a solid foundation for systematic and reproducible LLM evaluation.
Key takeaways:
By mastering lm-eval, you can make more informed decisions about which model best fits your specific needs, and accurately measure the impact of fine-tuning and optimization efforts.