Complete LM Evaluation Harness Tutorial: The Standard Framework for Evaluating Large Language Models

# Tutorial Lengkap LM Evaluation Harness: Framework Standar untuk Evaluasi Model Bahasa Besar Evaluasi model bahasa besar (LLM) merupakan tahap krusial dalam siklus pengembangan AI. Tanpa evaluasi ya...

By Ruby Abdullah · · tutorial
LLM EvaluationLM EvalBenchmarkingEleutherAIMachine Learning

Complete LM Evaluation Harness Tutorial: The Standard Framework for Evaluating Large Language Models

Evaluating large language models (LLMs) is a critical step in the AI development lifecycle. Without systematic evaluation, we cannot measure model performance, compare models against each other, or verify that fine-tuning efforts yield real improvements. LM Evaluation Harness (lm-eval) by EleutherAI has become the de facto standard for LLM evaluation, used by researchers, practitioners, and major organizations worldwide.

This tutorial provides an in-depth guide to using lm-eval for evaluating various language models, from installation to creating custom benchmarks.

What Is LM Evaluation Harness?

LM Evaluation Harness is an open-source framework developed by EleutherAI for evaluating generative language models in a consistent and reproducible manner. The framework provides hundreds of standard benchmarks including MMLU, HellaSwag, GSM8K, ARC, and many more.

Key advantages of lm-eval include:

  • Standardization: Hundreds of integrated benchmarks with consistent implementations
  • Reproducibility: Evaluation results can be reproduced with the same configuration
  • Flexibility: Supports models from HuggingFace, OpenAI API, Anthropic, vLLM, and more
  • Extensibility: Easy to add new benchmarks and model backends
  • Community: Widely adopted across the global AI research community

Installation and Setup

Basic Installation

The quickest way to install lm-eval is via pip:

pip install lm-eval

Installation from Source

To get the latest version with all features:

git clone https://github.com/EleutherAI/lm-evaluation-harness.git

cd lm-evaluation-harness

pip install -e ".[all]"

Installation with Additional Dependencies

If you want to use models from specific providers:

# For HuggingFace models with quantization

pip install lm-eval[gptq,awq]

For API-based models

pip install lm-eval[api]

For vLLM backend

pip install lm-eval[vllm]

For all dependencies

pip install lm-eval[all]

Verify Installation

lmeval --help

If the command above displays a list of options, the installation was successful.

Basic Usage

Evaluating a HuggingFace Model

The simplest example is evaluating a model from the HuggingFace Hub:

lmeval --model hf \

--modelargs pretrained=microsoft/phi-2 \

--tasks hellaswag \

--batchsize 8 \

--device cuda:0

Parameters explained:

  • --model hf: Use the HuggingFace Transformers backend
  • --modelargs: Model-specific arguments, including the Hub model name
  • --tasks: Benchmarks to run
  • --batchsize: Batch size for inference
  • --device: Device to use (cuda:0, cpu, etc.)

Evaluating Multiple Benchmarks at Once

You can evaluate a model on several benchmarks simultaneously:

lmeval --model hf \

--modelargs pretrained=meta-llama/Llama-3.1-8B \

--tasks hellaswag,mmlu,arceasy,arcchallenge,winogrande \

--batchsize auto \

--device cuda:0 \

--outputpath results/llama3-8b/

The --batchsize auto parameter automatically determines the optimal batch size based on available GPU memory.

Saving Evaluation Results

Evaluation results can be saved in JSON format:

lmeval --model hf \

--modelargs pretrained=Qwen/Qwen2.5-7B \

--tasks mmlu \

--batchsize auto \

--outputpath results/qwen25-7b/ \

--logsamples

The --logsamples flag saves details for each evaluation sample, useful for error analysis.

MMLU (Massive Multitask Language Understanding)

MMLU tests model knowledge across 57 domains, from mathematics to law:

lmeval --model hf \

--modelargs pretrained=mistralai/Mistral-7B-v0.3 \

--tasks mmlu \

--numfewshot 5 \

--batchsize auto

The --numfewshot 5 parameter provides 5 examples before the actual question (standard for MMLU).

HellaSwag

A benchmark for testing common-sense reasoning ability:

lmeval --model hf \

--modelargs pretrained=microsoft/phi-2 \

--tasks hellaswag \

--numfewshot 10 \

--batchsize auto

GSM8K (Grade School Math)

Tests mathematical reasoning capabilities:

lmeval --model hf \

--modelargs pretrained=meta-llama/Llama-3.1-8B \

--tasks gsm8k \

--numfewshot 5 \

--batchsize auto

ARC (AI2 Reasoning Challenge)

Elementary and middle school science questions:

lmeval --model hf \

--modelargs pretrained=google/gemma-2-9b \

--tasks arceasy,arcchallenge \

--numfewshot 25 \

--batchsize auto

TruthfulQA

Tests whether models produce correct and non-misleading answers:

lmeval --model hf \

--modelargs pretrained=meta-llama/Llama-3.1-8B \

--tasks truthfulqamc2 \

--numfewshot 0 \

--batchsize auto

Running a Complete Benchmark Suite

For comprehensive evaluation, run multiple benchmarks at once:

lmeval --model hf \

--modelargs pretrained=meta-llama/Llama-3.1-8B \

--tasks mmlu,hellaswag,arceasy,arcchallenge,winogrande,truthfulqamc2,gsm8k \

--numfewshot 5 \

--batchsize auto \

--outputpath results/llama3-8b-full/

Evaluating Models with Different Backends

Evaluating GGUF Models with llama.cpp

For models quantized in GGUF format:

lmeval --model gguf \

--modelargs baseurl=http://localhost:8080 \

--tasks hellaswag,mmlu \

--batchsize 1

Make sure the llama.cpp server is running first.

Evaluating Models via vLLM

vLLM provides faster inference:

lmeval --model vllm \

--modelargs pretrained=meta-llama/Llama-3.1-8B,tensorparallelsize=2 \

--tasks mmlu,hellaswag \

--batchsize auto

Evaluating Models via OpenAI API

To evaluate GPT models:

export OPENAIAPIKEY="sk-your-api-key"

lmeval --model openai-completions \

--modelargs model=gpt-3.5-turbo \

--tasks hellaswag,mmlu \

--batchsize 1

For chat models:

lmeval --model openai-chat-completions \

--modelargs model=gpt-4 \

--tasks mmlu \

--batchsize 1 \

--applychattemplate

Evaluating Quantized Models (GPTQ, AWQ)

# GPTQ model

lmeval --model hf \

--modelargs pretrained=TheBloke/Llama-2-7B-GPTQ,gptq=True \

--tasks hellaswag \

--batchsize auto

AWQ model

lmeval --model hf \

--modelargs pretrained=TheBloke/Llama-2-7B-AWQ,awq=True \

--tasks hellaswag \

--batchsize auto

Python API Usage

Beyond the CLI, lm-eval also provides a flexible Python API:

import lmeval

results = lmeval.simpleevaluate(

model="hf",

modelargs="pretrained=microsoft/phi-2",

tasks=["hellaswag", "mmlu"],

numfewshot=5,

batchsize="auto",

device="cuda:0",

)

Access results

for taskname, taskresult in results["results"].items():

print(f"\n{taskname}:")

for metric, value in taskresult.items():

if isinstance(value, (int, float)):

print(f" {metric}: {value:.4f}")

Advanced Configuration Evaluation

import lmeval

from lmeval.models.huggingface import HFLM

Initialize model with custom configuration

model = HFLM(

pretrained="meta-llama/Llama-3.1-8B",

dtype="float16",

batchsize="auto",

device="cuda:0",

maxlength=4096,

)

Run evaluation

results = lmeval.simpleevaluate(

model=model,

tasks=["mmlu", "hellaswag", "gsm8k"],

numfewshot=5,

logsamples=True,

)

Save results

import json

with open("evaluationresults.json", "w") as f:

json.dump(results["results"], f, indent=2)

Comparing Multiple Models

import lmeval

import json

modelstocompare = [

"microsoft/phi-2",

"google/gemma-2-2b",

"Qwen/Qwen2.5-3B",

]

tasks = ["hellaswag", "arceasy", "mmlu"]

allresults = {}

for modelname in modelstocompare:

print(f"\nEvaluating {modelname}...")

results = lmeval.simpleevaluate(

model="hf",

modelargs=f"pretrained={modelname}",

tasks=tasks,

numfewshot=5,

batchsize="auto",

device="cuda:0",

)

allresults[modelname] = results["results"]

Print comparison table

print("\n" + "=" 80)

print(f"{'Model':<30} {'HellaSwag':<15} {'ARC-Easy':<15} {'MMLU':<15}")

print("=" 80)

for modelname, result in allresults.items():

hellaswag = result.get("hellaswag", {}).get("accnorm,none", 0)

arc = result.get("arceasy", {}).get("accnorm,none", 0)

mmlu = result.get("mmlu", {}).get("acc,none", 0)

shortname = modelname.split("/")[-1]

print(f"{shortname:<30} {hellaswag:<15.4f} {arc:<15.4f} {mmlu:<15.4f}")

Creating Custom Benchmarks

YAML Task Structure

You can create custom benchmarks using the YAML format:

# mycustomtask.yaml

task: mycustombenchmark

datasetpath: json

datasetkwargs:

datafiles: mytestdata.jsonl

outputtype: multiplechoice

trainingsplit: null

validationsplit: null

testsplit: train

doctotext: "Question: {{question}}\nAnswer:"

doctotarget: "{{answer}}"

metriclist:

  • metric: acc
aggregation: mean

higherisbetter: true

numfewshot: 0

Creating an Evaluation Dataset

Prepare data in JSONL format:

{"question": "What is the capital of France?", "choices": ["London", "Paris", "Berlin", "Madrid"], "answer": 1}

{"question": "Who invented the telephone?", "choices": ["Edison", "Bell", "Tesla", "Marconi"], "answer": 1}

{"question": "Largest planet in the solar system?", "choices": ["Mars", "Venus", "Jupiter", "Saturn"], "answer": 2}

Generative Format Tasks

For generative text evaluation (not multiple choice):

# generativetask.yaml

task: mygenerativetask

datasetpath: json

datasetkwargs:

datafiles: gentestdata.jsonl

outputtype: generateuntil

trainingsplit: null

testsplit: train

doctotext: "Translate to French: {{english}}\nTranslation:"

doctotarget: "{{french}}"

generationkwargs:

maxgentoks: 256

temperature: 0.0

dosample: false

metriclist:

  • metric: bleu
aggregation: mean

higherisbetter: true

  • metric: exactmatch
aggregation: mean

higherisbetter: true

numfewshot: 3

Running Custom Tasks

lmeval --model hf \

--modelargs pretrained=meta-llama/Llama-3.1-8B \

--tasks mycustombenchmark \

--includepath /path/to/custom/tasks/ \

--batchsize auto

Filters and Post-Processing

Using Built-in Filters

lm-eval provides several filters for processing model output:

task: filteredtask

datasetpath: json

datasetkwargs:

datafiles: testdata.jsonl

outputtype: generateuntil

doctotext: "{{question}}"

doctotarget: "{{answer}}"

generationkwargs:

maxgentoks: 512

until:

  • "\n\n"
  • "Question:"
filter
list:

  • name: getanswer
filter:

  • function: regex
regex
pattern: "The answer is (\\w+)"

  • function: takefirst
metriclist:

  • metric: exactmatch
aggregation: mean

higherisbetter: true

Evaluating Fine-tuned Models

One of the most common uses of lm-eval is comparing models before and after fine-tuning:

import lmeval

tasks = ["mmlu", "hellaswag", "gsm8k"]

Evaluate base model

print("Evaluating base model...")

baseresults = lmeval.simpleevaluate(

model="hf",

modelargs="pretrained=meta-llama/Llama-3.1-8B",

tasks=tasks,

numfewshot=5,

batchsize="auto",

device="cuda:0",

)

Evaluate fine-tuned model

print("Evaluating fine-tuned model...")

ftresults = lmeval.simpleevaluate(

model="hf",

modelargs="pretrained=./my-finetuned-llama",

tasks=tasks,

numfewshot=5,

batchsize="auto",

device="cuda:0",

)

Compare results

print("\n" + "=" 60)

print(f"{'Task':<20} {'Base Model':<20} {'Fine-tuned':<20}")

print("=" 60)

for task in tasks:

baseacc = baseresults["results"].get(task, {}).get("acc,none", 0)

ftacc = ftresults["results"].get(task, {}).get("acc,none", 0)

delta = ftacc - baseacc

sign = "+" if delta > 0 else ""

print(f"{task:<20} {baseacc:<20.4f} {ftacc:.4f} ({sign}{delta:.4f})")

Performance Optimization Tips

Using auto batchsize

lmeval --model hf \

--modelargs pretrained=meta-llama/Llama-3.1-8B \

--tasks mmlu \

--batchsize auto:4

The number after auto: determines the maximum factor for auto-sizing.

Using Model Parallelism

For very large models:

lmeval --model hf \

--modelargs pretrained=meta-llama/Llama-3.1-70B,parallelize=True \

--tasks hellaswag \

--batchsize 4

Using Quantization for Efficiency

lmeval --model hf \

--modelargs pretrained=meta-llama/Llama-3.1-8B,loadin4bit=True \

--tasks mmlu,hellaswag \

--batchsize auto

Limiting Sample Count

For quick testing:

lmeval --model hf \

--modelargs pretrained=microsoft/phi-2 \

--tasks mmlu \

--limit 100 \

--batchsize auto

Best Practices

1. Consistency in Evaluation

Always use the same parameters when comparing models:

EVALCONFIG = {

"tasks": ["mmlu", "hellaswag", "arcchallenge", "winogrande", "gsm8k"],

"numfewshot": 5,

"batchsize": "auto",

}

resultsa = lmeval.simpleevaluate(

model="hf",

modelargs="pretrained=model-a",

EVALCONFIG,

)

resultsb = lmeval.simpleevaluate(

model="hf",

modelargs="pretrained=model-b",

EVALCONFIG,

)

2. Document Evaluation Configuration

Save the evaluation configuration alongside results:

import json

from datetime import datetime

evalrecord = {

"timestamp": datetime.now().isoformat(),

"model": "meta-llama/Llama-3.1-8B",

"lmevalversion": lmeval.version,

"tasks": ["mmlu", "hellaswag"],

"numfewshot": 5,

"results": results["results"],

}

with open("evalrecord.json", "w") as f:

json.dump(evalrecord, f, indent=2)

3. Choose the Right Benchmarks

Match benchmarks to your evaluation needs:

  • General knowledge: MMLU, ARC
  • Reasoning: HellaSwag, WinoGrande
  • Mathematics: GSM8K, MATH
  • Coding: HumanEval, MBPP
  • Factual accuracy: TruthfulQA
  • Domain-specific: Create custom tasks with your own dataset

4. Use Standard Few-shot Counts

Follow the standard few-shot counts for each benchmark:

| Benchmark | Standard Few-shot |

|-----------|------------------|

| MMLU | 5 |

| HellaSwag | 10 |

| ARC | 25 |

| WinoGrande | 5 |

| GSM8K | 5 |

| TruthfulQA | 0 |

5. Evaluate Regularly

Integrate evaluation into your CI/CD pipeline:

#!/bin/bash

evalpipeline.sh

MODELPATH=$1

OUTPUTDIR="evalresults/$(date +%Y%m%d%H%M%S)"

mkdir -p $OUTPUTDIR

lmeval --model hf \

--modelargs pretrained=$MODELPATH \

--tasks mmlu,hellaswag,arcchallenge,gsm8k \

--numfewshot 5 \

--batchsize auto \

--outputpath $OUTPUTDIR \

--logsamples

echo "Results saved to $OUTPUTDIR"

Integration with Other Tools

Integration with Weights & Biases

import lmeval

import wandb

wandb.init(project="llm-evaluation", name="llama3-8b-eval")

results = lmeval.simpleevaluate(

model="hf",

modelargs="pretrained=meta-llama/Llama-3.1-8B",

tasks=["mmlu", "hellaswag", "gsm8k"],

numfewshot=5,

batchsize="auto",

)

for taskname, taskresult in results["results"].items():

for metric, value in taskresult.items():

if isinstance(value, (int, float)):

wandb.log({f"{taskname}/{metric}": value})

wandb.finish()

Integration with MLflow

import lmeval

import mlflow

mlflow.setexperiment("llm-evaluation")

with mlflow.startrun(runname="llama3-8b-eval"):

results = lmeval.simpleevaluate(

model="hf",

modelargs="pretrained=meta-llama/Llama-3.1-8B",

tasks=["mmlu", "hellaswag"],

numfewshot=5,

batchsize="auto",

)

mlflow.logparam("model", "meta-llama/Llama-3.1-8B")

mlflow.logparam("tasks", "mmlu,hellaswag")

for taskname, taskresult in results["results"].items():

for metric, value in taskresult.items():

if isinstance(value, (int, float)):

mlflow.logmetric(f"{taskname}_{metric}", value)

Conclusion

LM Evaluation Harness is an essential tool in the toolkit of every AI/ML practitioner working with large language models. With hundreds of built-in benchmarks, support for various model backends, and the ability to create custom evaluations, lm-eval provides a solid foundation for systematic and reproducible LLM evaluation.

Key takeaways:

  • Always use consistent evaluation configurations when comparing models
  • Choose benchmarks that align with your specific use case
  • Use standard few-shot counts for each benchmark to ensure comparable results
  • Document every evaluation session for reproducibility
  • Integrate evaluation into your model development pipeline
  • By mastering lm-eval, you can make more informed decisions about which model best fits your specific needs, and accurately measure the impact of fine-tuning and optimization efforts.

    Related Articles

    AutoGluon: A Complete Guide to AWS's AutoML Library for Real-World Machine Learning

    Tutorial Lengkap AutoGluon: Library AutoML dari AWS untuk Machine Learning Otomatis AutoGluon adalah library open-source...

    CatBoost Tutorial: Gradient Boosting for Tabular Data with Automatic Categorical Feature Handling

    Tutorial CatBoost: Gradient Boosting untuk Data Tabular dengan Penanganan Fitur Kategorikal Otomatis CatBoost (Categoric...

    Complete Guide to Hugging Face AutoTrain: Fine-Tune AI Models Without the Hassle

    Tutorial Lengkap Hugging Face AutoTrain: Fine-Tuning Model AI Tanpa Ribet Hugging Face AutoTrain adalah platform dan lib...

    MLflow Tutorial: Open-Source Platform for End-to-End MLOps

    Tutorial MLflow: Platform Open-Source untuk MLOps End-to-End MLflow adalah platform open-source yang dikembangkan oleh D...