Tutorial 8: LangSmith - LLM Observability and Debugging

Table of Contents

Introduction

Prerequisites

Setting Up LangSmith

Tracing LLM Calls

Evaluation Datasets

Custom Evaluators

Prompt Versioning

Production Monitoring

Debugging Chains and Agents

Cost Tracking

Integration with LangChain

A/B Testing Prompts

Best Practices

Conclusion

---

Introduction

Building LLM-powered applications is only half the battle. Understanding how your models behave in production, why they fail, how much they cost, and whether your prompts are actually improving over time is the other critical half. LangSmith is the observability and evaluation platform built by LangChain specifically to address these challenges.

LangSmith provides end-to-end tracing of every LLM call, chain execution, and agent step. It enables you to build evaluation datasets, run automated evaluations, version your prompts, monitor production performance, and debug complex multi-step workflows. Whether you are prototyping a simple chatbot or running a production RAG pipeline, LangSmith gives you the visibility you need to ship reliable AI applications.

This tutorial walks you through every major feature of LangSmith with practical, production-ready code examples.

---

Prerequisites

• Python 3.9 or higher
• An active LangSmith account (sign up at smith.langchain.com)
• An OpenAI API key (or any supported LLM provider)
• Basic familiarity with LangChain concepts

Install the required packages:

pip install langsmith langchain langchain-openai openai

---

Setting Up LangSmith

Step 1: Configure Environment Variables

LangSmith requires a few environment variables to connect your application to the platform.

import os

os.environ["LANGCHAINTRACINGV2"] = "true"
os.environ["LANGCHAINAPIKEY"] = "ls_yourapikeyhere"
os.environ["LANGCHAINPROJECT"] = "my-first-project"

os.environ["LANGCHAINENDPOINT"] = "https://api.smith.langchain.com"

Step 2: Verify the Connection

from langsmith import Client

client = Client()

List existing projects to verify connectivity
projects = list(client.listprojects())

print(f"Connected to LangSmith. Found {len(projects)} project(s).")
for project in projects:
    print(f"  - {project.name} (created: {project.createdat})")

Step 3: Create a Dedicated Project

# Create a new project for organized tracking
projectname = "tutorial-langsmith-demo"


try:
    project = client.createproject(projectname)

    print(f"Project '{project.name}' created successfully.")
except Exception as e:
    print(f"Project may already exist: {e}")

Set the active project
os.environ["LANGCHAINPROJECT"] = projectname

---

Tracing LLM Calls

Basic Tracing with LangChain

When tracing is enabled, every LangChain call is automatically captured.

from langchainopenai import ChatOpenAI
from langchaincore.messages import HumanMessage, SystemMessage


llm = ChatOpenAI(model="gpt-4o", temperature=0.7)

messages = [
    SystemMessage(content="You are a helpful coding assistant."),
    HumanMessage(content="Explain Python decorators in 3 sentences.")
]

response = llm.invoke(messages)
print(response.content)
The trace is automatically sent to LangSmith

Manual Tracing with the @traceable Decorator

For custom functions that are not LangChain components, use the @traceable decorator.

from langsmith import traceable

@traceable(name="processuserquery")