Complete AgentOps Tutorial: Monitoring and Observability for AI Agents
Running AI agents in production without monitoring is like flying a plane without instruments. You have no idea when your agent fails, how much it costs, or where bottlenecks occur. AgentOps is an observability platform designed specifically for AI agents, giving you full visibility into every step your agent takes.
In this tutorial, we will cover how to use AgentOps from installation, basic integration with OpenAI and LangChain, to advanced features like custom event tracking, multi-agent monitoring, and cost analysis. All code examples can be run directly in your project.
What Is AgentOps?
AgentOps is an open-source observability platform built specifically for monitoring AI agents in production. Unlike traditional logging that only captures text, AgentOps records the entire agent lifecycle in a structured way, including LLM calls, tool usage, error traces, token costs, and latency.
Key features of AgentOps:
- Session Tracking: Records every agent session from start to finish
- LLM Call Monitoring: Tracks every LLM call including prompts, responses, token usage, and cost
- Tool Tracking: Monitors tool usage by agents
- Error Detection: Detects and reports errors in real-time
- Cost Analysis: Accurately calculates LLM usage costs
- Session Replay: Replays the entire agent workflow for debugging
- Multi-Agent Support: Supports monitoring for multi-agent systems
- Dashboard: Visualizes agent metrics through a web dashboard
Installation
Prerequisites
- Python 3.9 or later
- AgentOps account (free for developer tier)
- API key from an LLM provider (OpenAI, Anthropic, etc.)
Environment Setup
pip install agentops
For integration with specific frameworks, install additional dependencies:
# For OpenAI integration
pip install agentops openai
For LangChain integration
pip install agentops langchain langchain-openai
For CrewAI integration
pip install agentops crewai
For AutoGen integration
pip install agentops autogen
Getting Your API Key
export AGENTOPSAPIKEY="your-api-key-here"
export OPENAIAPIKEY="your-openai-key-here"
Or create a .env file:
AGENTOPSAPIKEY=your-api-key-here
OPENAIAPIKEY=your-openai-key-here
Basic Usage: Monitoring LLM Calls
Initializing AgentOps
The simplest way to use AgentOps is by adding just two lines of code. AgentOps automatically patches detected LLM libraries.
import agentops
import openai
Initialize AgentOps - automatically patches OpenAI
agentops.init()
client = openai.OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain what machine learning is in 3 sentences."}
]
)
print(response.choices[0].message.content)
End session and send data to dashboard
agentops.endsession("Success")
After running the code above, open the AgentOps dashboard to see:
- LLM call details (model, prompt, response)
- Token usage (input tokens, output tokens)
- Cost estimate
- Latency
Using API Key Explicitly
import agentops
agentops.init(apikey="your-api-key-here")
Session Tags for Organization
Tags help you categorize and filter sessions in the dashboard:
agentops.init(
tags=["production", "customer-support", "v2.1"]
)
Tracking Multiple LLM Calls
AgentOps automatically records every LLM call within a session. This is especially useful when your agent performs multiple reasoning steps.
import agentops
import openai
agentops.init(tags=["multi-step-reasoning"])
client = openai.OpenAI()
Step 1: Analyze the question
analysis = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "Analyze the following question and identify the main topic."},
{"role": "user", "content": "How do I optimize slow PostgreSQL database queries?"}
]
)
topic = analysis.choices[0].message.content
print(f"Topic identified: {topic}")
Step 2: Generate detailed answer
answer = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "Provide a detailed technical answer based on the topic analysis."},
{"role": "user", "content": f"Based on topic: {topic}\n\nProvide a complete optimization guide."}
]
)
print(f"Answer: {answer.choices[0].message.content}")
Step 3: Create summary
summary = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "Create a summary in 3 main points."},
{"role": "user", "content": answer.choices[0].message.content}
]
)
print(f"Summary: {summary.choices[0].message.content}")
agentops.endsession("Success")
In the dashboard, you will see all three LLM calls sequentially in one session, complete with timeline and total cost.
Decorator for Agent Classes
AgentOps provides the @trackagent decorator to track agents as separate entities within a session:
import agentops
from agentops import trackagent
agentops.init(tags=["agent-class-demo"])
@trackagent(name="research-agent")
class ResearchAgent:
def init(self):
self.client = openai.OpenAI()
self.findings = []
def research(self, topic: str) -> str:
response = self.client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "You are a thorough AI researcher."},
{"role": "user", "content": f"Conduct in-depth research on: {topic}"}
]
)
result = response.choices[0].message.content
self.findings.append(result)
return result
def summarize(self) -> str:
allfindings = "\n\n".join(self.findings)
response = self.client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "Summarize all the following research findings."},
{"role": "user", "content": allfindings}
]
)
return response.choices[0].message.content
agent = ResearchAgent()
agent.research("Transformer architecture for NLP")
agent.research("Efficient LLM fine-tuning techniques")
summary = agent.summarize()
print(summary)
agentops.endsession("Success")
Custom Event Tracking
Beyond automatically recorded LLM calls, you can record custom events to track business logic:
import agentops
from agentops import ActionEvent, ErrorEvent
agentops.init(tags=["custom-events"])
Record custom action
agentops.record(ActionEvent(
actiontype="databasequery",
params={"query": "SELECT FROM users WHERE active = true", "database": "production"},
returns={"rowcount": 1523}
))
Record error if one occurs
try:
result = processdata(rawdata)
except Exception as e:
agentops.record(ErrorEvent(
errortype=type(e).name,
details=str(e)
))
agentops.endsession("Fail")
raise
agentops.endsession("Success")
Tool Event Tracking
When agents use tools, record the results:
from agentops import ToolEvent
def searchdatabase(query: str) -> dict:
toolevent = ToolEvent(name="databasesearch")
toolevent.params = {"query": query}
try:
results = db.search(query)
toolevent.returns = {"count": len(results), "status": "success"}
toolevent.endtimestamp = agentops.gettimestamp()
agentops.record(toolevent)
return results
except Exception as e:
toolevent.returns = {"error": str(e), "status": "failed"}
agentops.record(toolevent)
raise
Integration with LangChain
AgentOps has built-in LangChain integration that enables automatic tracking of entire chains and agent runs:
import agentops
from langchainopenai import ChatOpenAI
from langchain.agents import AgentExecutor, createopenaitoolsagent
from langchain.tools import tool
from langchaincore.prompts import ChatPromptTemplate, MessagesPlaceholder
agentops.init(tags=["langchain-agent"])
@tool
def calculate(expression: str) -> str:
"""Calculate a mathematical expression."""
try:
result = eval(expression)
return f"Result: {result}"
except Exception as e:
return f"Error: {str(e)}"
@tool
def getweather(city: str) -> str:
"""Get weather information for a specific city."""
weatherdata = {
"New York": "Sunny, 28°C, Humidity 60%",
"London": "Cloudy, 18°C, Humidity 75%",
"Tokyo": "Clear, 30°C, Humidity 55%"
}
return weatherdata.get(city, f"Weather data for {city} is not available.")
llm = ChatOpenAI(model="gpt-4o", temperature=0)
prompt = ChatPromptTemplate.frommessages([
("system", "You are a smart assistant that can calculate and check weather."),
MessagesPlaceholder(variablename="chathistory", optional=True),
("human", "{input}"),
MessagesPlaceholder(variablename="agentscratchpad"),
])
tools = [calculate, getweather]
agent = createopenaitoolsagent(llm, tools, prompt)
executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
result = executor.invoke({
"input": "What's the average temperature if New York is 28 degrees and London is 18 degrees? Also check Tokyo's weather."
})
print(result["output"])
agentops.endsession("Success")
In the AgentOps dashboard, you will see:
- Every LLM call in the chain
- Tool calls made by the agent (calculate, getweather)
- Input and output of each step
- Total token usage and cost
Integration with CrewAI
AgentOps also integrates with CrewAI for multi-agent team monitoring:
import agentops
from crewai import Agent, Task, Crew
agentops.init(tags=["crewai-demo"])
researcher = Agent(
role="Technology Researcher",
goal="Gather the latest information about AI technology",
backstory="You are a senior researcher with 10 years of experience in AI.",
verbose=True
)
writer = Agent(
role="Content Writer",
goal="Write informative and easy-to-understand articles",
backstory="You are an experienced tech writer who excels at explaining complex concepts.",
verbose=True
)
researchtask = Task(
description="Research the latest developments in Large Language Models in 2026.",
expectedoutput="A research report with at least 5 important developments.",
agent=researcher
)
writingtask = Task(
description="Write a blog article based on the research about LLM developments.",
expectedoutput="A 500-word informative and engaging blog article.",
agent=writer
)
crew = Crew(
agents=[researcher, writer],
tasks=[researchtask, writingtask],
verbose=True
)
result = crew.kickoff()
print(result)
agentops.endsession("Success")
Advanced: Session Management
Multiple Sessions
You can run several sessions simultaneously, useful for comparing different agent strategies:
import agentops
Session for strategy A
sessiona = agentops.startsession(tags=["strategy-a", "experiment"])
Session for strategy B
sessionb = agentops.startsession(tags=["strategy-b", "experiment"])
Use specific sessions
clienta = openai.OpenAI()
clientb = openai.OpenAI()
responsea = clienta.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Explain quantum computing."}]
)
responseb = clientb.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Explain quantum computing."}]
)
sessiona.endsession("Success")
sessionb.endsession("Success")
Session with Context Manager
import agentops
with agentops.Session(tags=["context-manager"]) as session:
client = openai.OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)
Session automatically ends when exiting context
Advanced: Cost Tracking and Analytics
Monitoring Cost Per Session
AgentOps automatically calculates costs based on model and token usage:
import agentops
agentops.init(tags=["cost-tracking"])
client = openai.OpenAI()
Multiple calls with different models
for i in range(5):
client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": f"Complex query #{i+1}"}]
)
for i in range(10):
client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": f"Simple query #{i+1}"}]
)
agentops.endsession("Success")
Dashboard will show cost breakdown per model
Production Monitoring
For production, add robust error handling:
import agentops
import logging
logger = logging.getLogger(name)
def runagenttask(userinput: str) -> str:
session = agentops.startsession(
tags=["production", "customer-facing"]
)
try:
client = openai.OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "You are a customer support agent."},
{"role": "user", "content": userinput}
],
timeout=30
)
result = response.choices[0].message.content
session.endsession("Success")
return result
except openai.APITimeoutError:
logger.error("LLM call timeout")
agentops.record(ErrorEvent(
errortype="APITimeoutError",
details="LLM call exceeded 30s timeout"
))
session.endsession("Fail")
return "Sorry, the system is busy. Please try again."
except Exception as e:
logger.error(f"Agent error: {e}")
agentops.record(ErrorEvent(
errortype=type(e).name,
details=str(e)
))
session.endsession("Fail")
raise
Advanced: Callbacks and Webhooks
Webhook Configuration
AgentOps supports webhooks for real-time notifications:
agentops.init(
tags=["webhook-enabled"],
)
Configure webhooks through the AgentOps dashboard under Settings > Webhooks. You can send notifications to Slack, Discord, or custom endpoints when:
- A session fails
- Cost exceeds a threshold
- Latency exceeds limits
Best Practices
1. Always End Sessions
Make sure every session is ended with the appropriate status. Sessions that are not ended will appear as "Indeterminate" in the dashboard.
# Use try-finally to ensure session always ends
agentops.init()
try:
result = runagent()
agentops.endsession("Success")
except Exception:
agentops.endsession("Fail")
2. Use Descriptive Tags
agentops.init(tags=[
"production", # Environment
"v2.3", # Agent version
"customer-support", # Use case
"gpt-4o" # Primary model
])
3. Separate Environments
import os
env = os.getenv("ENVIRONMENT", "development")
agentops.init(tags=[env, "my-agent"])
4. Monitor Token Usage
Monitor token usage to optimize costs:
# Use smaller models for simple tasks
simpleresponse = client.chat.completions.create(
model="gpt-4o-mini", # Cheaper for classification
messages=[{"role": "user", "content": "Classify: is this a technical question? 'How to install Python'"}]
)
Use larger models for complex tasks
complexresponse = client.chat.completions.create(
model="gpt-4o", # More accurate for analysis
messages=[{"role": "user", "content": "Analyze the following microservices architecture and provide recommendations..."}]
)
5. Implement Tracked Retry Logic
import agentops
from agentops import ActionEvent
import time
def callwithretry(messages, maxretries=3):
for attempt in range(maxretries):
try:
response = client.chat.completions.create(
model="gpt-4o",
messages=messages
)
return response
except openai.RateLimitError:
agentops.record(ActionEvent(
actiontype="retry",
params={"attempt": attempt + 1, "reason": "ratelimit"}
))
time.sleep(2 attempt)
agentops.record(ErrorEvent(
errortype="MaxRetriesExceeded",
details=f"Failed after {maxretries} attempts"
))
raise Exception("Max retries exceeded")
6. Batch Processing with Session Per Batch
def processbatch(items: list, batchid: str):
session = agentops.startsession(
tags=["batch-processing", f"batch-{batchid}"]
)
successcount = 0
errorcount = 0
for item in items:
try:
processitem(item)
successcount += 1
except Exception as e:
errorcount += 1
agentops.record(ErrorEvent(
errortype=type(e).name,
details=f"Item {item['id']}: {str(e)}"
))
agentops.record(ActionEvent(
actiontype="batchcomplete",
params={"total": len(items)},
returns={"success": successcount, "errors": errorcount}
))
status = "Success" if errorcount == 0 else "Fail"
session.endsession(status)
Comparison with Other Monitoring Tools
| Feature | AgentOps | LangSmith | Helicone | Portkey |
|---------|----------|-----------|----------|---------|
| Agent-specific tracking | Yes | Yes | No | Partial |
| Auto-instrumentation | Yes | Yes | Yes | Yes |
| Multi-agent support | Yes | Partial | No | No |
| Cost tracking | Yes | Yes | Yes | Yes |
| Session replay | Yes | Yes | No | No |
| Open-source | Yes | No | Partial | No |
| Framework integrations | Many | LangChain | Agnostic | Agnostic |
| Self-hosted option | Yes | Enterprise | Yes | No |
Troubleshooting
Session Not Appearing in Dashboard
# Make sure init() is called before LLM calls
agentops.init() # MUST be at the beginning
Check connection
import agentops
result = agentops.init()
print(f"Init status: {result}")
Inaccurate Token Count
Some models may not return token counts. Make sure you are using supported models and the latest SDK version:
pip install --upgrade agentops openai
High Logging Latency
AgentOps sends data asynchronously, so it should not add significant latency. If it does, check your internet connection and consider using batch mode:
agentops.init(
maxwaittime=1000, # Maximum wait time in ms
maxqueuesize=100 # Queue size before flush
)
Complete Example: RAG Agent with Monitoring
Here is a complete example of a RAG agent monitored with AgentOps:
import agentops
from agentops import trackagent, ActionEvent, ToolEvent
import openai
import numpy as np
agentops.init(tags=["rag-agent", "production"])
@trackagent(name="rag-retrieval-agent")
class RAGAgent:
def init(self):
self.client = openai.OpenAI()
self.knowledgebase = self.loadknowledgebase()
def loadknowledgebase(self) -> list:
agentops.record(ActionEvent(
actiontype="loadknowledgebase",
returns={"status": "loaded", "documents": 150}
))
return [
{"text": "Python is a popular programming language.", "embedding": None},
{"text": "Machine learning requires quality data.", "embedding": None},
{"text": "Docker simplifies application deployment.", "embedding": None},
]
def retrieve(self, query: str, topk: int = 3) -> list:
toolevent = ToolEvent(name="vectorsearch")
toolevent.params = {"query": query, "topk": topk}
queryembedding = self.client.embeddings.create(
model="text-embedding-3-small",
input=query
)
results = self.knowledgebase[:topk]
toolevent.returns = {"resultscount": len(results)}
agentops.record(toolevent)
return results
def generate(self, query: str, context: list) -> str:
contexttext = "\n".join([doc["text"] for doc in context])
response = self.client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": f"Answer based on context:\n{contexttext}"},
{"role": "user", "content": query}
]
)
return response.choices[0].message.content
def answer(self, query: str) -> str:
context = self.retrieve(query)
return self.generate(query, context)
agent = RAGAgent()
answer = agent.answer("What are the benefits of using Docker?")
print(answer)
agentops.end_session("Success")
Conclusion
AgentOps provides the visibility you need when running AI agents in production. With auto-instrumentation, you only need to add a few lines of code to get comprehensive monitoring.
Key takeaways from this tutorial:
pip install agentops and agentops.init() to start trackingWith AgentOps, you can build AI agents that are not only functional but also observable and maintainable in production. Start with a simple integration, then add custom events as your monitoring needs grow.