Promptfoo Tutorial: Complete Guide to LLM Evaluation and Testing

Introduction

One of the biggest challenges in building applications powered by Large Language Models (LLMs) is ensuring that model outputs are consistent, accurate, and aligned with business requirements. Without a proper evaluation framework, developers often rely on manual testing that is time-consuming and unscalable. This is where Promptfoo comes in.

Promptfoo is an open-source tool for systematically evaluating and comparing LLM outputs. With Promptfoo, you can test various prompts, compare performance across models, detect regressions, and ensure output quality before deploying to production. The tool supports a wide range of providers including OpenAI, Anthropic Claude, Google Gemini, Ollama, and many more.

This tutorial will guide you from installation and basic configuration through advanced usage of Promptfoo to build a robust LLM evaluation pipeline.

Why Promptfoo?

Before diving into technical details, here is why Promptfoo deserves a place in your LLM development workflow:

• Side-by-Side Model Comparison: Compare outputs from different models and prompts in a single table
• Automated Testing: Define test cases once, run them repeatedly and automatically
• Regression Detection: Detect output quality degradation when changing prompts or models
• Cost Optimization: Compare expensive vs. affordable models to find the sweet spot between quality and cost
• CI/CD Integration: Integrate LLM evaluation into your CI/CD pipeline
• Red Teaming: Test prompt security and resilience against adversarial attacks

Installation

Prerequisites

Make sure you have Node.js version 18 or later installed:

node --version
v18.0.0 or later

Install Promptfoo

You can install Promptfoo globally using npm:

npm install -g promptfoo

Or use npx without global installation:

npx promptfoo@latest --help

For project-level installation:

mkdir llm-eval-project
cd llm-eval-project
npm init -y
npm install promptfoo

Verify Installation

promptfoo --version

Setup API Keys

Promptfoo requires API keys from the LLM providers you plan to use. Set environment variables accordingly:

# OpenAI
export OPENAIAPIKEY="sk-your-key-here"

Anthropic
export ANTHROPICAPIKEY="sk-ant-your-key-here"

Google Gemini
export GOOGLEAPIKEY="your-key-here"

You can also store keys in a .env file:

OPENAIAPIKEY=sk-your-key-here
ANTHROPICAPIKEY=sk-ant-your-key-here

Basic Usage

Initialize a Project

Start by initializing a Promptfoo project:

promptfoo init

This command creates a promptfooconfig.yaml file, which is the main configuration file.

Basic Configuration Structure

Create a promptfooconfig.yaml file with the following configuration:

description: "Customer Service Chatbot Evaluation"

prompts:
"You are a friendly customer service agent. Answer the following question: {{question}}"
"As a professional customer service assistant, provide a clear and concise answer for: {{question}}"

providers:
openai:gpt-4o-mini
openai:gpt-4o
anthropic:messages:claude-sonnet-4-20250514

tests:
vars:
      question: "How do I return a purchased item?"

    assert:
type: contains
        value: "return"

type: llm-rubric
        value: "The answer should clearly mention the steps for returning an item"

vars:
      question: "How long does shipping take for international orders?"

    assert:
type: contains
        value: "days"

type: llm-rubric
        value: "The answer should mention estimated shipping time"

vars:
      question: "Do you offer discounts for bulk purchases?"

    assert: