Complete Braintrust Tutorial: Evaluate, Test, and Improve Your LLM Applications

Hey everyone, in this tutorial I want to introduce you to a platform that I think will seriously change how you build LLM applications, called Braintrust. If you have ever built an LLM feature and then had no idea whether your new prompt change actually made things better or somehow made them worse, then Braintrust is the answer. I have been stuck in that exact situation more than once. Change the prompt, change the model, change the temperature, and then just guess whether the result improved. It feels like driving with your eyes closed.

Braintrust is basically a platform for evaluating, testing, and improving LLM applications. It helps you measure the quality of model outputs objectively, not just by feel. In this article I will cover things from scratch, starting with the core concepts like experiments, datasets, scorers or evals, and logging, then moving on to installation, how to write a complete Eval() function, how to compare two prompt versions, how to log production traces, all the way to the prompt playground feature which is really fun. I will give you plenty of Python examples using the braintrust and autoevals libraries. Let's get started.

Why We Need Evaluation for LLM Applications

Before we dive into Braintrust, let me first explain why evaluation is so important in the LLM world. Unlike regular deterministic code, LLM output is probabilistic. The same input can produce different outputs. So you cannot write ordinary unit tests that say "the output must exactly equal this". The output can be correct in meaning but different in wording.

The problem is, when you develop an LLM application, you will constantly tinker with it. Change the prompt to make it clearer, switch the model from cheap to expensive or the other way around, add few-shot examples, tune the temperature, and so on. Every change can make some cases better but some others worse. Without a proper measuring tool, you will never understand the trade-off.

The naive approach is usually to try a few examples manually, look at the outputs, and then say "okay this looks good enough". But this is really dangerous, everyone. You only try five to ten examples, while in production there are thousands of input variations. A change that looks good on your examples might actually break cases you did not test. This is what people call "vibes-based development", building purely on gut feeling.

Braintrust gives us a way to measure this systematically. The idea is simple. We collect a set of inputs along with their ideal answers, we run our application against those inputs, then we score each output using a scoring function. We save the result as one experiment. Once we change something, we run it again and compare the new experiment against the old one. Now we have concrete numbers to see whether our change genuinely improves quality or not.

Core Concepts of Braintrust

Before we code, it is really important to understand the four main concepts in Braintrust. If you understand these four, the rest will be easy.

The first is the dataset. A dataset is a collection of examples you use for evaluation. Each example usually has an input (what you feed to the application) and an expected (the ideal answer or ground truth). This dataset becomes your benchmark. The more representative your dataset is of production conditions, the more accurate the evaluation.

The second is the task. A task is your application function that will be evaluated. It receives an input and returns an output. Its content can be anything, from just calling a single LLM to a complex RAG pipeline. Braintrust does not care what the task contains, as long as it can receive input and produce output.

The third is the scorer, often also called an eval. This is a function that assigns a value to your task's output, usually a number from 0 to 1. Scorers come in many kinds. Some are deterministic like checking whether the output exactly matches the expected, some are heuristic like measuring text similarity, and some are advanced using an LLM to judge the output (this is called LLM-as-a-judge). The autoevals library that comes with Braintrust provides many ready-to-use scorers.

The fourth is the experiment. An experiment is one complete execution of an evaluation. When you run a task against the entire dataset and score each of its outputs, the results are collected into one experiment. Each experiment has aggregate scores, and you can compare one experiment against another to spot regressions or improvements.

Besides those four, there is also the concept of logging for production. If evaluation is for before release, logging is for monitoring your application that is already running in production. You record every request and response along with their metadata and traces, so they can be analyzed and even turned into new datasets for evaluation. So there is a nice loop here, from production back into evaluation.

Installation

Alright, now let's get to the practice. Installation is really easy, just use pip. You need two main packages, namely braintrust itself and autoevals for the ready-to-use scorers.

pip install braintrust autoevals

If you are going to call the LLM directly inside your task, you also need the provider library. I usually install OpenAI as well since many examples use it.

pip install braintrust autoevals openai

As usual, I strongly recommend using a virtual environment so your project dependencies do not get mixed up. Activate your venv before installing.

python -m venv venv
source venv/bin/activate  # on Windows: venv\Scripts\activate
pip install braintrust autoevals openai

Once installed, you need an API key from Braintrust. Sign up on the Braintrust dashboard first, then create an API key on the settings page. Store that key as an environment variable so it stays safe and does not get hardcoded in your code.

export BRAINTRUSTAPIKEY="sk-your-braintrust-key"
export OPENAIAPIKEY="sk-your-openai-key"

To test whether the installation went well, you can try importing the libraries in Python.

import braintrust
import autoevals

print("Braintrust is ready to use")

If there are no errors, then you are ready to move on to the fun part.

Basic Usage: Writing Your First Eval

Now we get to the heart of Braintrust, namely the Eval() function. This function is the core of all evaluation. It needs three main things according to the concepts we discussed earlier, namely the dataset (data), the task, and the scorers (scores).

Let's build a simple example first. Suppose I have an application whose job is to answer general knowledge questions. I want to measure how often my application gives correct answers.

from braintrust import Eval
from autoevals import Levenshtein

def mytask(input):

    # this is a simple task, just a dictionary of answers.
    # in the real world this would call an LLM.
    answers = {
        "Capital of Indonesia?": "Jakarta",
        "What is 2 plus 2?": "4",
        "Color of the sky during the day?": "Blue",
    }
    return answers.get(input, "I don't know")

Eval(
    "first-qa-app",  # project name
    data=lambda: [
        {"input": "Capital of Indonesia?", "expected": "Jakarta"},
        {"input": "What is 2 plus 2?", "expected": "4"},
        {"input": "Color of the sky during the day?", "expected": "Blue"},
    ],
    task=mytask,
    scores=[Levenshtein],
)

Let's break down the code above one by one, everyone. The first argument "first-qa-app" is the project name in Braintrust. All your experiments will be collected under this project.

The data argument is a function that returns a list of examples. Each example has an input and an expected. I deliberately use a lambda so the data is fetched when needed. You could also pass a list directly, but using a function is more flexible if the data needs to be loaded from a file or an API.

The task argument is our application function. Braintrust will call this function for each input in the dataset, then store the output.

The scores argument is a list of scorers. Here I use Levenshtein from autoevals, which measures the text similarity between the output and the expected based on edit distance. The score ranges from 0 (totally different) to 1 (exactly the same).

To run it, save that code in a file, for example evalqa.py, then run it using the braintrust eval command.

braintrust eval evalqa.py

Braintrust will run the task against the entire dataset, score each output, then give you a link to the dashboard. On that dashboard you can see the average score, the details of each example, what output came out, and its score. This is a very different experience compared to just printing to the terminal.

A Task That Actually Calls an LLM

The previous example still had a fake task. Now let's build a task that genuinely calls an LLM. Braintrust has a wrapper for the OpenAI client so that every call is automatically logged.

from braintrust import Eval, wrapopenai
from autoevals import Factuality
from openai import OpenAI

client = wrapopenai(OpenAI())

def qatask(input):

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Answer concisely and accurately."},
            {"role": "user", "content": input},
        ],
    )
    return response.choices[0].message.content

Eval(
    "qa-llm-app",
    data=lambda: [
        {"input": "What is the capital of Japan?", "expected": "Tokyo"},
        {"input": "Who wrote the novel Pride and Prejudice?", "expected": "Jane Austen"},
        {"input": "How many planets are in the solar system?", "expected": "Eight"},
    ],
    task=qatask,
    scores=[Factuality],
)

Notice that here I use wrapopenai to wrap the OpenAI client. This is important because it makes every LLM call automatically recorded along with its input, output, tokens used, and latency. So on the dashboard you can see the details of each call.

For the scorer I switched to Factuality. This is an LLM-based scorer that checks whether the output is factually consistent with the expected. Unlike Levenshtein, Factuality does not care whether the wording is exactly the same, as long as the meaning is correct. So if the output is "Tokyo, the capital of Japan" and the expected is "Tokyo", Factuality still gives a high score even though the text is not identical. This is much more suitable for judging LLM output.

Advanced Usage

After understanding the basics, now we get to the more advanced features. In this section I will cover custom scorers, building your own LLM-based scorer, comparing versions, all the way to production logging.

Building Your Own Custom Scorer

The built-in autoevals scorers are indeed plentiful, but sometimes you need custom scoring logic specific to your case. Luckily, building your own scorer is really easy. A scorer is just a function that receives output and expected, then returns a score.

def reasonablelength(output, expected):
    # a scorer that gives a high value if the output length
    # is similar to the expected length.
    if not output:
        return 0
    ratio = min(len(output), len(expected)) / max(len(output), len(expected))
    return ratio

Eval(
    "length-app",
    data=lambda: [
        {"input": "Briefly explain what AI is", "expected": "AI is artificial intelligence in machines"},
    ],
    task=lambda input: "AI is artificial intelligence embedded into computer machines",
    scores=[reasonablelength],

)

Scorers like this are really useful for checking specific things, for example whether the output is too long, whether it contains forbidden words, whether the format is valid JSON, and so on. You can combine several scorers at once in the scores list, so each output is judged from several angles.

A scorer can also return a more detailed object, not just a number. You can give it a name and metadata so the result is more informative on the dashboard.

from autoevals import Score

def checkjsonformat(output, expected):

    import json
    try:
        json.loads(output)
        return Score(name="jsonformat", score=1, metadata={"valid": True})
    except Exception as e:
        return Score(name="jsonformat", score=0, metadata={"error": str(e)})

Custom LLM-as-a-Judge

Sometimes your scoring criteria are too subjective for a deterministic scorer. For example you want to judge whether an answer is "friendly" or "professional". Here we can use an LLM as a judge. Autoevals provides LLMClassifier to build LLM-based scorers easily.

from autoevals import LLMClassifier

politeness = LLMClassifier(
    name="Politeness",
    prompttemplate=(
        "Judge whether the following answer is polite and friendly.\n"
        "Question: {{input}}\n"
        "Answer: {{output}}\n"
        "Is this answer polite? Choose (a) Very polite, "
        "(b) Fairly polite, (c) Not polite."
    ),
    choicescores={"a": 1.0, "b": 0.5, "c": 0.0},

    usecot=True,
)

Notice the usecot=True argument. This makes the LLM judge think first (chain of thought) before giving its decision, so its judgment is more reliable. The choicescores field is a mapping from the LLM's choice to a numeric score. So if the LLM picks "a", the score is 1.0, and so on. This scorer can be directly plugged into the scores list in Eval() like any other scorer.

Comparing Versions

This is, in my opinion, the most powerful feature of Braintrust. Every time you run Eval(), the result is saved as a new experiment with a timestamp. Braintrust automatically compares the new experiment against the previous experiment in the same project. So you can immediately see whether the score went up, down, or stayed the same.

For example, suppose you have two prompt versions you want to compare. You can use metadata to mark which version is currently running.

from braintrust import Eval, wrapopenai
from autoevals import Factuality
from openai import OpenAI

client = wrapopenai(OpenAI())

PROMPTV1 = "Answer the question."

PROMPTV2 = "Answer the question concisely, accurately, and with facts only."

def maketask(systemprompt):
    def task(input):
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": systemprompt},

                {"role": "user", "content": input},
            ],
        )
        return response.choices[0].message.content
    return task

DATASET = [
    {"input": "What is the capital of Australia?", "expected": "Canberra"},
    {"input": "Who was the first president of the United States?", "expected": "George Washington"},
]

Eval(
    "prompt-comparison",
    data=lambda: DATASET,
    task=maketask(PROMPTV2),

    scores=[Factuality],
    metadata={"promptversion": "v2"},
)

You run it once with PROMPTV1 and metadata v1, then once more with PROMPTV2 and metadata v2. On the dashboard, Braintrust will display the two experiments side by side complete with their score diff. You can see which examples improved and which got worse. This is what lets you make decisions based on data, not feel.

What is even cooler, Braintrust shows regressions per example. So if the average score went up but there are actually a few cases that got worse, you can immediately see which cases those are and decide whether the trade-off is worth it.

Using Datasets Stored in Braintrust

So far our datasets have been hardcoded in the code. For serious projects, it is better to store datasets in Braintrust so they can be reused and managed by the team. Braintrust provides functions to create and access datasets.

from braintrust import initdataset

dataset = initdataset(project="prompt-comparison", name="geography-questions")

add examples to the dataset
dataset.insert(input="What is the capital of France?", expected="Paris")
dataset.insert(input="What is the tallest mountain in the world?", expected="Everest")

use the dataset in an eval
from braintrust import Eval
from autoevals import Factuality

Eval(
    "prompt-comparison",
    data=initdataset(project="prompt-comparison", name="geography-questions"),

    task=maketask(PROMPTV2),

    scores=[Factuality],
)

With this approach, your dataset becomes an asset that you can keep growing. Every time there is an interesting new case from production, you just add it to the dataset. Over time your dataset becomes more complete and the evaluation more accurate.

Logging Production Traces

Now we get to the production side. Besides being for evaluation before release, Braintrust can also trace your application that is already running in production. This is really important for monitoring quality in real time and gathering data for future evaluation.

The easiest way to start logging is to use initlogger and then leverage wrapopenai.

from braintrust import initlogger, wrapopenai
from openai import OpenAI

logger = initlogger(project="production-app")
client = wrapopenai(OpenAI())


def handlerequest(userquestion):

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": userquestion}],
    )
    return response.choices[0].message.content

every call here is automatically logged to Braintrust
answer = handlerequest("What is machine learning?")

print(answer)

Because the client is wrapped with wrapopenai, every call is automatically recorded along with its input, output, tokens, and latency. You do not need to add manual logging code. On the dashboard you can see all production traces, filter by time, and analyze patterns.

For more complex traces, for example a RAG pipeline that has several steps, you can use the @traced decorator to mark which functions you want to trace.

from braintrust import traced, initlogger, wrapopenai
from openai import OpenAI

logger = initlogger(project="rag-app")

client = wrapopenai(OpenAI())

@traced
def retrieve(query):
    # imagine this searches documents from a vector database
    return ["Document about " + query]

@traced
def generate(query, context):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Answer based on the context: " + str(context)},
            {"role": "user", "content": query},
        ],
    )
    return response.choices[0].message.content

@traced
def ragpipeline(query):

    context = retrieve(query)
    return generate(query, context)

result = ragpipeline("What is transfer learning?")
print(result)

With the @traced decorator, Braintrust creates a nested trace. So you can see the ragpipeline pipeline calls retrieve then generate, complete with the input and output of each step. This is really useful for debugging if there is a weird output, because you can trace at which step the problem appeared.

What completes the loop, these production logs can be turned into datasets for evaluation. So if there is an interesting or problematic production case, you just add it to your evaluation dataset. This keeps your evaluation increasingly relevant to real conditions.

Prompt Playground

The last feature I want to cover is the prompt playground. This is a feature on the Braintrust dashboard that gives you a place to tinker with prompts interactively without writing code. In the playground, you can write a prompt, choose a model, tune parameters like temperature, then run it against your dataset and see the results directly.

What is cool, this playground connects with the datasets and scorers you have already built. So you can try a new prompt against the same dataset, see its score, and compare it against the old prompt, all in one place without needing to deploy anything. This makes prompt iteration much faster.

Once you find a good prompt in the playground, you can save that prompt as an asset in Braintrust and call it from code. Braintrust provides a function to retrieve a saved prompt.

from braintrust import loadprompt

prompt = loadprompt(project="qa-llm-app", slug="accurate-answer")


build parameters to call the LLM with
params = prompt.build(input="What is the capital of Canada?")

from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(**params)
print(response.choices[0].message.content)

With this approach, your prompts become centralized in Braintrust. Your team can tinker with prompts in the playground, and the production code just calls the latest prompt version without needing to change the code. This separates prompt logic from application logic, which is really great for maintainability.

Best Practices

After using Braintrust across several projects, there are a few things that I think are really important for you to pay attention to so you get the most out of it.

First, start with a small but high-quality dataset. There is no need to immediately build thousands of examples. It is better to start with ten to twenty examples that are truly representative of your important cases. A small but good dataset is more useful than a large but sloppy one. You can keep adding examples over time, especially from problematic production cases.

Second, choose scorers that fit your case. Do not just use Levenshtein for everything. If you are judging output whose answer can differ in words but is the same in meaning, use Factuality or LLM-as-a-judge. If you are checking a format like JSON, use a deterministic scorer. Combine several scorers so the output is judged from multiple sides. But be careful, too many scorers also makes interpretation hard. Focus on the metrics that truly matter.

Third, always run evals before deploying changes. Make braintrust eval part of your workflow, and it can even be put into CI/CD. So every time there is a prompt or model change, the eval runs automatically and you immediately know if there is a regression. This saves you from deploying a change that looks good but actually breaks things in other cases.

Fourth, be careful with LLM-as-a-judge. LLM-based scorers are powerful but not perfect. The judge itself can misjudge. So do not swallow its scores whole. Occasionally check manually whether the judge's assessment makes sense. Use usecot=True so its judgment is more reliable, and write a clear and specific scoring prompt.

Fifth, make good use of metadata. Tag every experiment with the prompt version, model, and parameters used. This makes it easy for you to trace which experiment used which configuration when comparing. Without neat metadata, you will confuse yourself once the experiments pile up.

Sixth, close the loop from production to evaluation. Do not just log production and leave it alone. Regularly check production logs, look for interesting or problematic cases, then add them to your evaluation dataset. This way your evaluation keeps evolving to follow real conditions, and your application keeps getting better over time.

Seventh, keep your API keys secure. Never hardcode BRAINTRUSTAPIKEY or other provider keys in code. Use environment variables or a secret manager. This is basic but often forgotten, especially when you are in a hurry.

Conclusion

Alright everyone, we have covered Braintrust from scratch all the way to its advanced features. We started with why evaluation is important for LLM applications, then the core concepts like datasets, tasks, scorers, experiments, and logging. We also practiced writing our first Eval(), building a task that genuinely calls an LLM, building custom scorers including LLM-as-a-judge, comparing prompt versions, logging production traces using wrap_openai and the @traced decorator, all the way to the prompt playground feature.

In my opinion, the most important thing you can take away from Braintrust is a shift in mindset. From building LLM applications on feel, to building based on measurable data. Once you have a good dataset and scorers, every change you make can have its impact measured immediately. There is no more guessing whether the new prompt is better. You have concrete numbers to decide.

For those of you who are serious about building high-quality LLM applications that can be relied on in production, I strongly recommend starting to integrate evaluation from the beginning. It does not need to be perfect right away, just start with a few examples and one scorer, then develop it gradually. Braintrust gives you all the tools you need for that.

I hope this tutorial is useful for you. Happy tinkering and see you in the next tutorial.