Weave Tutorial: LLM Application Observability and Evaluation with Weights & Biases

Building applications powered by Large Language Models (LLMs) is becoming increasingly accessible, but monitoring, debugging, and evaluating the quality of their outputs remains a significant challenge. How do you know if a new prompt performs better? How do you trace every LLM call in production? This is where Weave by Weights & Biases comes in.

Weave is an open-source framework for tracing, evaluating, and monitoring LLM applications. Unlike traditional W&B which focuses on experiment tracking for model training, Weave is specifically designed for the LLM application lifecycle from development to production.

In this tutorial, we will learn how to use Weave to trace LLM calls, create automated evaluations, manage datasets, and monitor LLM applications in production.

Installation and Setup

Installing Weave

pip install weave openai

Weave supports Python 3.9+ and integrates with various LLM providers including OpenAI, Anthropic, Google, Mistral, and more.

Configuring API Keys

You will need a Weights & Biases account. Sign up for free at wandb.ai and get your API key.

export WANDBAPIKEY="your-api-key-here"
export OPENAIAPIKEY="your-openai-key-here"

Initializing a Project

import weave

weave.init("my-llm-project")

After initialization, all traced operations will appear in the Weave dashboard on the W&B platform.

Basic Usage: Tracing LLM Calls

Auto-Tracing with Built-in Integrations

Weave automatically traces calls to popular LLM providers without any additional code.

import weave
from openai import OpenAI

weave.init("tracing-demo")
client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain what machine learning is in 2 sentences."}
    ]
)

print(response.choices[0].message.content)

By simply adding weave.init(), every OpenAI call is automatically traced. You can view inputs, outputs, latency, token usage, and cost in the Weave dashboard.

Custom Tracing with the @weave.op Decorator

For your custom functions, use the @weave.op decorator:

import weave
import json
from openai import OpenAI

weave.init("custom-tracing")
client = OpenAI()

@weave.op
def extractentities(text: str) -> dict:

    """Extract entities from text using an LLM."""
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": "Extract entities (people, places, organizations) from the text. Return in JSON format."
            },
            {"role": "user", "content": text}
        ],
        responseformat={"type": "jsonobject"}

    )
    return json.loads(response.choices[0].message.content)

@weave.op
def processdocument(document: str) -> dict:
    """Process a document: extract entities and summarize."""
    entities = extractentities(document)

    return {
        "entities": entities,
        "entitycount": sum(len(v) for v in entities.values() if isinstance(v, list))
    }

result = processdocument(

    "Sundar Pichai met with Tim Cook in San Francisco to discuss "
    "the partnership between Google and Apple on AI initiatives."
)
print(json.dumps(result, indent=2))

The @weave.op decorator creates hierarchical traces, allowing you to see how processdocument calls extractentities, complete with inputs and outputs at every level.

Tracing with Additional Metadata

You can add custom attributes to traces for easier filtering and analysis:

import weave

weave.init("metadata-demo")

@weave.op
def classifysentiment(text: str, language: str = "en") -> dict:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {