Inspect AI: The LLM Evaluation Framework from the UK AI Safety Institute You Should Try
Hey friends, once you get serious about building applications that use large language models, sooner or later you run into one question that makes your head spin: "Which model is actually best for my use case?" Or even more specifically: "After I changed my prompt, did the results get better or worse?" For a long time many of us answered those questions using a method that is honestly a bit sloppy: try a few examples manually, look at the output, and say "seems okay". But "seems okay" is not an answer you can stand behind.
So in this tutorial I want to introduce you to a tool that changed the way I measure the quality of LLM work: it is called Inspect AI. It is an open-source framework built by the UK AI Safety Institute, which is an official body of the UK government whose job is literally to handle AI safety and evaluation. Because it was born from an institution that takes evaluation seriously, this framework is designed to make LLM evaluation structured, repeatable, and auditable. No more "seems okay", but clear numbers: model A scores 78 percent, model B scores 84 percent, on the same dataset, with the same scoring method.
I am going to take you from zero: starting from installation, understanding the core concepts (Task, Dataset, Solver, Scorer), writing your first task, using various solvers and scorers, running it through the inspect eval CLI, and viewing the results in the slick log viewer. All with Python examples you can run right away. Take it easy, we will go step by step. Ready? Let us get started.
Introduction: Why Inspect AI Matters
Before we dive into code, I want you to understand the problem Inspect AI is trying to solve. This is important so you understand why this tool is really worth learning.
Imagine you are building a customer service feature that answers customer questions using an LLM. You have written a good prompt, picked a model, and the results look okay when you try five or ten questions. But think about it: your application will face thousands of different questions. How do you know your model actually answers correctly in the majority of cases? And when a new model comes out, or you want to cut costs by using a smaller model, how do you compare them fairly?
The manual approach clearly does not scale. You cannot possibly test a thousand questions one by one every time there is a small change. What you need is a system that can: store a collection of questions along with their correct answers, run the model against all those questions automatically, score the model's answers consistently, and give you a clear report. This is exactly what Inspect AI provides.
Inspect AI has four core concepts you need to understand first. Once you get these four, you understand 80 percent of the framework:
First, the Dataset. This is the collection of questions you want to test. Each item in the dataset is called a Sample, and each Sample has at least two parts:input (the question or prompt given to the model) and target (the correct answer, aka ground truth). A dataset can come from a Python list directly, from a CSV/JSON file, or from Hugging Face.
Second, the Solver. This determines how the model works on each Sample. The most basic solver is generate(), which means "give the input to the model, ask it to generate an answer". But a Solver can be more complex: you can add a prompt template, a system message, chain-of-thought, multi-step reasoning, even use tools. A Solver is like a processing pipeline for each question.
Third, the Scorer. This grades the model's answer. After the model produces an answer, the Scorer compares that answer with the target and assigns a value. A Scorer can be simple like match() (does the string match or not), or sophisticated like modelgradedqa() which uses another LLM to grade answer quality.
Fourth, the Task. This unites all three. A Task is a combination of Dataset + Solver + Scorer. You declare a Task using the @task decorator, then you run it with the inspect eval command. The result is metrics (for example accuracy) plus detailed logs for each Sample.
Look at how these four concepts connect in one flow: Inspect takes each Sample from the Dataset, hands it to the Solver to be processed by the model, then the result is graded by the Scorer, and all of it is wrapped in one Task. The concept is that simple, but the power is enormous because every part can be swapped to fit your needs.
What I really like about Inspect AI compared to writing your own evaluation script: it already provides a nice visual log viewer, support for many model providers (OpenAI, Anthropic, Google, local models via Ollama, and more), efficient parallel handling so evaluations run fast, and a clean code structure so your evaluations can be shared and reproduced by others. Okay, enough theory. Let us install.
Installation
Installing Inspect AI is really easy because it is a normal Python package. The requirement is that you have Python 3.10 or newer. I strongly recommend you create a virtual environment first to keep things clean and avoid clashing with other projects.
# Create a virtual environment
python -m venv venv
Activate (Linux/Mac)
source venv/bin/activate
Activate (Windows)
venv\Scripts\activate
Install Inspect AI
pip install inspect-ai
That is enough for the core framework. But because Inspect AI needs to talk to models, you also need to install the SDK from the provider you want to use. For example if you want to use OpenAI or Anthropic models:
# If you want to use OpenAI
pip install openai
If you want to use Anthropic (Claude)
pip install anthropic
If you want to use Google Gemini
pip install google-generativeai
After that, you need to set the API key as an environment variable. Inspect AI automatically reads the key from the environment according to the provider you use. Here is how:
# For OpenAI
export OPENAIAPIKEY="sk-xxxxxxxxxxxx"
For Anthropic
export ANTHROPICAPIKEY="sk-ant-xxxxxxxxxxxx"
For Google
export GOOGLEAPIKEY="xxxxxxxxxxxx"
If you are too lazy to type export every time you open a terminal, you can put the keys in a .env file in your project folder. Inspect AI automatically reads that .env file, so it is very handy:
# File: .env
OPENAIAPIKEY=sk-xxxxxxxxxxxx
ANTHROPICAPIKEY=sk-ant-xxxxxxxxxxxx
To make sure your installation succeeded, try running this command in the terminal:
inspect --version
If a version number appears, then Inspect AI is installed correctly and its CLI is ready to use. This inspect command will be your friend for running evaluations, viewing logs, and various other things. Now we are ready to write our first task.
Basic Usage: Writing Your First Task
Okay, now the fun part. We are going to build a simple evaluation to test how good a model is at answering general knowledge questions. I will explain each line so you really understand it, not just copy-paste.
Create a new file called evalknowledge.py, then write the following code:
from inspectai import Task, task
from inspectai.dataset import Sample
from inspectai.solver import generate
from inspectai.scorer import match
@task
def generalknowledge():
# 1. Dataset: a collection of questions with their correct answers
dataset = [
Sample(
input="What is the capital of Indonesia?",
target="Jakarta",
),
Sample(
input="What is 12 multiplied by 8?",
target="96",
),
Sample(
input="Who was the first president of Indonesia?",
target="Soekarno",
),
]
# 2. Combine into a Task: Dataset + Solver + Scorer
return Task(
dataset=dataset,
solver=generate(),
scorer=match(location="any"),
)
Let us break down this code slowly. In the import lines, we take Task and task from inspectai. The lowercase task is the decorator, and the uppercase Task is the class. Then we take Sample to build dataset items, generate as the solver, and match as the scorer.
The @task decorator is really important. It marks the generalknowledge() function as a task that the inspect eval CLI can recognize. So later you just call this function name from the terminal.
Inside the function, I build a dataset as a list of three Samples. Each Sample has an input (the question) and a target (the correct answer). This is the most important part: this target is what will be used to grade the model's answer later.
Then we return a Task that unites three things: the dataset we just built, solver=generate() which means "tell the model to generate an answer from the input", and scorer=match(location="any") which means "score it correct if the target appears anywhere in the model's answer". I use location="any" because models usually answer in full sentences, not just one word. So if the model answers "The capital of Indonesia is Jakarta", the scorer still counts it correct because the word "Jakarta" is inside it.
Now, how do you run it? This is where the inspect eval CLI comes in. Open your terminal, then type:
inspect eval evalknowledge.py --model openai/gpt-4o-mini
Notice the model name format: openai/gpt-4o-mini. The format is always provider/model-name. If you want to use Claude, just switch it:
inspect eval evalknowledge.py --model anthropic/claude-3-5-sonnet-20241022
If you want to use a local model via Ollama:
inspect eval evalknowledge.py --model ollama/llama3.1
When you run it, Inspect AI takes each Sample, gives the question to the model, waits for its answer, then grades it using the scorer. At the end, it displays a summary in the terminal, like what percentage of answers were correct. On top of that, it automatically saves the full logs in a logs/ folder that you can later open in the log viewer.
Even better, you can control how many samples run at once to speed things up, and limit the number of samples for quick testing:
# Only run the first 2 samples for a quick check
inspect eval evalknowledge.py --model openai/gpt-4o-mini --limit 2
Run with 5 parallel connections to speed things up
inspect eval evalknowledge.py --model openai/gpt-4o-mini --max-connections 5
There you go, you just built and ran your first LLM evaluation. Easy, right? Now let us level up.
Advanced Usage: More Powerful Solvers and Scorers
The simple task above is great for getting acquainted, but the real world is usually more complicated. Models rarely answer in exactly the same format as the target, and questions often need smarter grading. In this section I will show you how to use more advanced solvers and scorers.
Solvers with Prompt Templates and System Messages
The plain generate() solver directly feeds the raw input to the model. But often you want to add extra instructions or a certain prompt format. For that, Inspect AI has the systemmessage() and prompttemplate() solvers that you can combine into a chain.
from inspectai import Task, task
from inspectai.dataset import Sample
from inspectai.solver import generate, systemmessage, prompttemplate
from inspectai.scorer import match
@task
def knowledgewithinstructions():
dataset = [
Sample(input="What is the capital of Japan?", target="Tokyo"),
Sample(input="What is the capital of France?", target="Paris"),
]
return Task(
dataset=dataset,
solver=[
systemmessage(
"You are a geography assistant. Answer with ONE word only, "
"which is the name of the city. Do not add any explanation."
),
prompttemplate("Question: {prompt}\nAnswer:"),
generate(),
],
scorer=match(location="any"),
)
Notice here that solver is no longer a single function but a list. This is what we call a solver chain. Inspect AI runs these solvers in order. First systemmessage() adds a system instruction that tells the model to answer with one word. Then prompttemplate() wraps our input using a template, where {prompt} is automatically replaced with the input from the Sample. Finally generate() actually calls the model. With this chain, you have full control over how the prompt is composed before it reaches the model.
Model-Graded Scorer for Open-Ended Answers
Now this is the part I think is the most powerful. The match() scorer only works for clear, definite answers. But what if the answer is an essay or a long explanation that cannot be matched by exact string? For example you ask "Explain why the sky is blue". There is no single fixed correct answer.
For cases like this, Inspect AI has modelgradedqa(). The core idea is clever: we use another LLM as a judge to grade whether the model's answer is correct and appropriate. This is called LLM-as-a-judge.
from inspectai import Task, task
from inspectai.dataset import Sample
from inspectai.solver import generate
from inspectai.scorer import modelgradedqa
@task
def explanationeval():
dataset = [
Sample(
input="Briefly explain why the sky is blue.",
target=(
"Because of Rayleigh scattering: air molecules scatter "
"blue light, which has a shorter wavelength, more strongly "
"than red light."
),
),
Sample(
input="Explain what photosynthesis is in one sentence.",
target=(
"The process by which plants convert sunlight, water, and "
"carbon dioxide into glucose and oxygen."
),
),
]
return Task(
dataset=dataset,
solver=generate(),
scorer=modelgradedqa(
instructions=(
"Grade whether the participant's answer is factually "
"correct and conveys the same core idea as the reference "
"answer. Ignore differences in writing style."
),
model="openai/gpt-4o",
),
)
Here modelgradedqa() has two important arguments. instructions is the guidance for the judge on how to grade. I tell the judge to focus on factual correctness and the core of the answer, not the writing style. Then model is the model that acts as the judge. I deliberately use a stronger model (gpt-4o) as the judge, while the model being tested can be a smaller one. This is common practice: the judge should be a smart model so its grading is accurate.
When you run this task, the flow becomes like this: the tested model answers the question, then that answer plus the target is sent to the judge model, and the judge decides whether the answer is correct or wrong. Cool, right? You can grade open-ended answers automatically and consistently.
Building Your Own Custom Scorer
Sometimes the built-in scorers are not enough and you need your own grading logic. Inspect AI lets you build custom scorers using the @scorer decorator. For example you want to grade based on whether the model's answer contains the correct number within a certain tolerance:
from inspectai.scorer import scorer, accuracy, stderr, Score, Target
from inspect
ai.solver import TaskState
import re
@scorer(metrics=[accuracy(), stderr()])
def numberwithintolerance(tolerance: float = 0.01):
async def score(state: TaskState, target: Target) -> Score:
# Get the model's answer and the target
answer = state.output.completion
targetvalue = float(target.text)
# Find the first number in the model's answer
numbers = re.findall(r"-?\d+\.?\d*", answer)
if not numbers:
return Score(value="I", explanation="No number in the answer")
modelvalue = float(numbers[0])
correct = abs(modelvalue - targetvalue) <= tolerance
return Score(
value="C" if correct else "I",
answer=str(modelvalue),
explanation=f"Model: {modelvalue}, Target: {targetvalue}",
)
return score
In this custom scorer, I use the @scorer decorator with metrics=[accuracy(), stderr()] which means the results are summarized into accuracy and standard error. The score function receives state (containing the model output) and target (the correct answer). I extract the number from the model's answer using regex, compare it with the target within a tolerance, then return a Score with value "C" (Correct) or "I" (Incorrect). The "C" and "I" values are Inspect AI's convention for correct and incorrect. With a custom scorer, you can grade anything according to your project's specific needs.
Viewing Results in the Inspect Log Viewer
After you run several evaluations, you will definitely want to see the details, not just the final numbers in the terminal. This is where the Inspect AI log viewer shines. Every time you run inspect eval, the results are stored in the logs/ folder. To open it, just type:
inspect view
This command starts a local web server (usually at http://localhost:7575) and opens a visual view in your browser. There you can see each Sample one by one: what the question was, what the model answered, what the target was, and why the scorer gave it a correct or wrong value. For modelgradedqa cases, you can even see the judge's reasoning for why it graded that way. This is super useful for debugging. If your model gets a lot wrong, you can immediately see the error patterns here.
You can also point the viewer to a specific log folder or a specific log file:
# Open logs from a specific folder
inspect view --log-dir ./logs
Sometimes you want to run it on a different port
inspect view --port 8080
This log viewer is, in my opinion, one of Inspect AI's distinguishing features. Many evaluation frameworks only give you numbers, but Inspect gives you full transparency down to the level of each sample. So you not only know "the model scored 80 percent", but you also know exactly why the 20 percent it got wrong went wrong.
Best Practices: Tips from Experience
After using Inspect AI for a while, there are a few things I learned that I want to share with you so you do not make the same mistakes.
First, start with a small dataset during development. When you are developing a task, do not immediately run it against a thousand samples. Use--limit 5 or --limit 10 to keep it fast and save on API costs. Once the task is correct and the scorer works well, then run it against the full dataset. I often forgot this at the start and wasted API money because of a bug in the scorer that forced all samples to be rerun.
Second, separate the dataset from the task code. For large datasets, do not hardcode them inside the task function. Store them in a separate CSV or JSON file, then load them using Inspect AI helpers. This makes your dataset easier to maintain and share:
from inspectai.dataset import csvdataset, jsondataset
Load from CSV with 'input' and 'target' columns
dataset = csvdataset("questions.csv")
Or from JSON
dataset = jsondataset("questions.json")
modelgradedqa() for questions with definite answers like "2 + 2 = 4", because it is expensive (it calls an additional LLM) and overkill. Just use match() or includes(). Conversely, do not use match() for essay answers, because string matching will grade them wrong. Match the scorer to the question's characteristics.
Fourth, watch out for costs when using model-graded scorers. Remember, each sample graded by modelgradedqa() means two LLM calls: one for the tested model, one for the judge. If your dataset is large, costs can pile up fast. Consider using a cheaper judge model if the questions are not too hard to grade.
Fifth, use epochs for models whose answers are inconsistent. LLMs are non-deterministic, so their answers can differ each time. If you want more stable grading, you can run each sample several times using the epochs parameter:
# Run each sample 3 times, take the average
inspect eval eval_knowledge.py --model openai/gpt-4o-mini --epochs 3
Conclusion
Okay friends, we have come a really long way in this tutorial. We started from zero, understood the four core concepts of Inspect AI (Dataset, Solver, Scorer, Task), installed the framework, wrote our first task, and used advanced solvers and scorers like prompt templates and model-graded QA. We also learned to build our own custom scorer and view detailed results in the log viewer.
What I hope you take home from this tutorial is not just the syntax, but the mindset. LLM evaluation is not something you should do with "seems okay". If you are serious about building products that use LLMs, you need a structured, repeatable, and auditable way to measure quality. Inspect AI gives you all of that in a clean, open-source package, built by an institution that is genuinely expert in AI evaluation.
The next step I recommend: try building a small dataset from real cases in your own project, maybe ten or twenty questions that represent what your users actually ask. Then build a task to evaluate the model you are currently using. From there you get a baseline number. After that, every time you change a prompt or change a model, run the evaluation again and compare. You will be surprised how many decisions you have been making based on "feeling" can actually be proven with numbers.
Inspect AI still has many features we have not covered: support for tools and agents, multi-turn conversation evaluation, code sandboxes, and integration with various popular benchmark datasets. But the foundation you have learned here is enough for you to get started. Take it slow, explore the rest as you go.
Happy experimenting, friends. I hope this tutorial makes you more confident about measuring the quality of your LLM work with data, not just feelings. If you build something cool with Inspect AI, I would love it if you shared it. See you in the next tutorial. Keep up the great learning.