Complete AgentOps Tutorial: Monitoring and Observability for AI Agents

# Tutorial Lengkap AgentOps: Monitoring dan Observability untuk AI Agent Membangun AI agent yang berjalan di production tanpa monitoring ibarat menerbangkan pesawat tanpa instrumen. Anda tidak tahu k...

By Ruby Abdullah · · tutorial
AgentOpsAI AgentsObservabilityLLM MonitoringPython

Complete AgentOps Tutorial: Monitoring and Observability for AI Agents

Running AI agents in production without monitoring is like flying a plane without instruments. You have no idea when your agent fails, how much it costs, or where bottlenecks occur. AgentOps is an observability platform designed specifically for AI agents, giving you full visibility into every step your agent takes.

In this tutorial, we will cover how to use AgentOps from installation, basic integration with OpenAI and LangChain, to advanced features like custom event tracking, multi-agent monitoring, and cost analysis. All code examples can be run directly in your project.

What Is AgentOps?

AgentOps is an open-source observability platform built specifically for monitoring AI agents in production. Unlike traditional logging that only captures text, AgentOps records the entire agent lifecycle in a structured way, including LLM calls, tool usage, error traces, token costs, and latency.

Key features of AgentOps:

  • Session Tracking: Records every agent session from start to finish
  • LLM Call Monitoring: Tracks every LLM call including prompts, responses, token usage, and cost
  • Tool Tracking: Monitors tool usage by agents
  • Error Detection: Detects and reports errors in real-time
  • Cost Analysis: Accurately calculates LLM usage costs
  • Session Replay: Replays the entire agent workflow for debugging
  • Multi-Agent Support: Supports monitoring for multi-agent systems
  • Dashboard: Visualizes agent metrics through a web dashboard

Installation

Prerequisites

  • Python 3.9 or later
  • AgentOps account (free for developer tier)
  • API key from an LLM provider (OpenAI, Anthropic, etc.)

Environment Setup

pip install agentops

For integration with specific frameworks, install additional dependencies:

# For OpenAI integration

pip install agentops openai

For LangChain integration

pip install agentops langchain langchain-openai

For CrewAI integration

pip install agentops crewai

For AutoGen integration

pip install agentops autogen

Getting Your API Key

  • Go to app.agentops.ai and create an account
  • Navigate to Settings > API Keys
  • Copy the available API key
  • Save it as an environment variable:
  • export AGENTOPSAPIKEY="your-api-key-here"
    

    export OPENAIAPIKEY="your-openai-key-here"

    Or create a .env file:

    AGENTOPSAPIKEY=your-api-key-here
    

    OPENAIAPIKEY=your-openai-key-here

    Basic Usage: Monitoring LLM Calls

    Initializing AgentOps

    The simplest way to use AgentOps is by adding just two lines of code. AgentOps automatically patches detected LLM libraries.

    import agentops
    

    import openai

    Initialize AgentOps - automatically patches OpenAI

    agentops.init()

    client = openai.OpenAI()

    response = client.chat.completions.create(

    model="gpt-4o",

    messages=[

    {"role": "system", "content": "You are a helpful assistant."},

    {"role": "user", "content": "Explain what machine learning is in 3 sentences."}

    ]

    )

    print(response.choices[0].message.content)

    End session and send data to dashboard

    agentops.endsession("Success")

    After running the code above, open the AgentOps dashboard to see:

    • LLM call details (model, prompt, response)
    • Token usage (input tokens, output tokens)
    • Cost estimate
    • Latency

    Using API Key Explicitly

    import agentops
    
    

    agentops.init(apikey="your-api-key-here")

    Session Tags for Organization

    Tags help you categorize and filter sessions in the dashboard:

    agentops.init(
    

    tags=["production", "customer-support", "v2.1"]

    )

    Tracking Multiple LLM Calls

    AgentOps automatically records every LLM call within a session. This is especially useful when your agent performs multiple reasoning steps.

    import agentops
    

    import openai

    agentops.init(tags=["multi-step-reasoning"])

    client = openai.OpenAI()

    Step 1: Analyze the question

    analysis = client.chat.completions.create(

    model="gpt-4o",

    messages=[

    {"role": "system", "content": "Analyze the following question and identify the main topic."},

    {"role": "user", "content": "How do I optimize slow PostgreSQL database queries?"}

    ]

    )

    topic = analysis.choices[0].message.content

    print(f"Topic identified: {topic}")

    Step 2: Generate detailed answer

    answer = client.chat.completions.create(

    model="gpt-4o",

    messages=[

    {"role": "system", "content": "Provide a detailed technical answer based on the topic analysis."},

    {"role": "user", "content": f"Based on topic: {topic}\n\nProvide a complete optimization guide."}

    ]

    )

    print(f"Answer: {answer.choices[0].message.content}")

    Step 3: Create summary

    summary = client.chat.completions.create(

    model="gpt-4o-mini",

    messages=[

    {"role": "system", "content": "Create a summary in 3 main points."},

    {"role": "user", "content": answer.choices[0].message.content}

    ]

    )

    print(f"Summary: {summary.choices[0].message.content}")

    agentops.endsession("Success")

    In the dashboard, you will see all three LLM calls sequentially in one session, complete with timeline and total cost.

    Decorator for Agent Classes

    AgentOps provides the @trackagent decorator to track agents as separate entities within a session:

    import agentops
    

    from agentops import trackagent

    agentops.init(tags=["agent-class-demo"])

    @trackagent(name="research-agent")

    class ResearchAgent:

    def init(self):

    self.client = openai.OpenAI()

    self.findings = []

    def research(self, topic: str) -> str:

    response = self.client.chat.completions.create(

    model="gpt-4o",

    messages=[

    {"role": "system", "content": "You are a thorough AI researcher."},

    {"role": "user", "content": f"Conduct in-depth research on: {topic}"}

    ]

    )

    result = response.choices[0].message.content

    self.findings.append(result)

    return result

    def summarize(self) -> str:

    allfindings = "\n\n".join(self.findings)

    response = self.client.chat.completions.create(

    model="gpt-4o-mini",

    messages=[

    {"role": "system", "content": "Summarize all the following research findings."},

    {"role": "user", "content": allfindings}

    ]

    )

    return response.choices[0].message.content

    agent = ResearchAgent()

    agent.research("Transformer architecture for NLP")

    agent.research("Efficient LLM fine-tuning techniques")

    summary = agent.summarize()

    print(summary)

    agentops.endsession("Success")

    Custom Event Tracking

    Beyond automatically recorded LLM calls, you can record custom events to track business logic:

    import agentops
    

    from agentops import ActionEvent, ErrorEvent

    agentops.init(tags=["custom-events"])

    Record custom action

    agentops.record(ActionEvent(

    actiontype="databasequery",

    params={"query": "SELECT FROM users WHERE active = true", "database": "production"},

    returns={"rowcount": 1523}

    ))

    Record error if one occurs

    try:

    result = processdata(rawdata)

    except Exception as e:

    agentops.record(ErrorEvent(

    errortype=type(e).name,

    details=str(e)

    ))

    agentops.endsession("Fail")

    raise

    agentops.endsession("Success")

    Tool Event Tracking

    When agents use tools, record the results:

    from agentops import ToolEvent
    
    

    def searchdatabase(query: str) -> dict:

    toolevent = ToolEvent(name="databasesearch")

    toolevent.params = {"query": query}

    try:

    results = db.search(query)

    toolevent.returns = {"count": len(results), "status": "success"}

    toolevent.endtimestamp = agentops.gettimestamp()

    agentops.record(toolevent)

    return results

    except Exception as e:

    toolevent.returns = {"error": str(e), "status": "failed"}

    agentops.record(toolevent)

    raise

    Integration with LangChain

    AgentOps has built-in LangChain integration that enables automatic tracking of entire chains and agent runs:

    import agentops
    

    from langchainopenai import ChatOpenAI

    from langchain.agents import AgentExecutor, createopenaitoolsagent

    from langchain.tools import tool

    from langchaincore.prompts import ChatPromptTemplate, MessagesPlaceholder

    agentops.init(tags=["langchain-agent"])

    @tool

    def calculate(expression: str) -> str:

    """Calculate a mathematical expression."""

    try:

    result = eval(expression)

    return f"Result: {result}"

    except Exception as e:

    return f"Error: {str(e)}"

    @tool

    def getweather(city: str) -> str:

    """Get weather information for a specific city."""

    weatherdata = {

    "New York": "Sunny, 28°C, Humidity 60%",

    "London": "Cloudy, 18°C, Humidity 75%",

    "Tokyo": "Clear, 30°C, Humidity 55%"

    }

    return weatherdata.get(city, f"Weather data for {city} is not available.")

    llm = ChatOpenAI(model="gpt-4o", temperature=0)

    prompt = ChatPromptTemplate.frommessages([

    ("system", "You are a smart assistant that can calculate and check weather."),

    MessagesPlaceholder(variablename="chathistory", optional=True),

    ("human", "{input}"),

    MessagesPlaceholder(variablename="agentscratchpad"),

    ])

    tools = [calculate, getweather]

    agent = createopenaitoolsagent(llm, tools, prompt)

    executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

    result = executor.invoke({

    "input": "What's the average temperature if New York is 28 degrees and London is 18 degrees? Also check Tokyo's weather."

    })

    print(result["output"])

    agentops.endsession("Success")

    In the AgentOps dashboard, you will see:

    • Every LLM call in the chain
    • Tool calls made by the agent (calculate, getweather)
    • Input and output of each step
    • Total token usage and cost

    Integration with CrewAI

    AgentOps also integrates with CrewAI for multi-agent team monitoring:

    import agentops
    

    from crewai import Agent, Task, Crew

    agentops.init(tags=["crewai-demo"])

    researcher = Agent(

    role="Technology Researcher",

    goal="Gather the latest information about AI technology",

    backstory="You are a senior researcher with 10 years of experience in AI.",

    verbose=True

    )

    writer = Agent(

    role="Content Writer",

    goal="Write informative and easy-to-understand articles",

    backstory="You are an experienced tech writer who excels at explaining complex concepts.",

    verbose=True

    )

    researchtask = Task(

    description="Research the latest developments in Large Language Models in 2026.",

    expectedoutput="A research report with at least 5 important developments.",

    agent=researcher

    )

    writingtask = Task(

    description="Write a blog article based on the research about LLM developments.",

    expectedoutput="A 500-word informative and engaging blog article.",

    agent=writer

    )

    crew = Crew(

    agents=[researcher, writer],

    tasks=[researchtask, writingtask],

    verbose=True

    )

    result = crew.kickoff()

    print(result)

    agentops.endsession("Success")

    Advanced: Session Management

    Multiple Sessions

    You can run several sessions simultaneously, useful for comparing different agent strategies:

    import agentops
    
    

    Session for strategy A

    sessiona = agentops.startsession(tags=["strategy-a", "experiment"])

    Session for strategy B

    sessionb = agentops.startsession(tags=["strategy-b", "experiment"])

    Use specific sessions

    clienta = openai.OpenAI()

    clientb = openai.OpenAI()

    responsea = clienta.chat.completions.create(

    model="gpt-4o",

    messages=[{"role": "user", "content": "Explain quantum computing."}]

    )

    responseb = clientb.chat.completions.create(

    model="gpt-4o-mini",

    messages=[{"role": "user", "content": "Explain quantum computing."}]

    )

    sessiona.endsession("Success")

    sessionb.endsession("Success")

    Session with Context Manager

    import agentops
    
    

    with agentops.Session(tags=["context-manager"]) as session:

    client = openai.OpenAI()

    response = client.chat.completions.create(

    model="gpt-4o",

    messages=[{"role": "user", "content": "Hello!"}]

    )

    print(response.choices[0].message.content)

    Session automatically ends when exiting context

    Advanced: Cost Tracking and Analytics

    Monitoring Cost Per Session

    AgentOps automatically calculates costs based on model and token usage:

    import agentops
    
    

    agentops.init(tags=["cost-tracking"])

    client = openai.OpenAI()

    Multiple calls with different models

    for i in range(5):

    client.chat.completions.create(

    model="gpt-4o",

    messages=[{"role": "user", "content": f"Complex query #{i+1}"}]

    )

    for i in range(10):

    client.chat.completions.create(

    model="gpt-4o-mini",

    messages=[{"role": "user", "content": f"Simple query #{i+1}"}]

    )

    agentops.endsession("Success")

    Dashboard will show cost breakdown per model

    Production Monitoring

    For production, add robust error handling:

    import agentops
    

    import logging

    logger = logging.getLogger(name)

    def runagenttask(userinput: str) -> str:

    session = agentops.startsession(

    tags=["production", "customer-facing"]

    )

    try:

    client = openai.OpenAI()

    response = client.chat.completions.create(

    model="gpt-4o",

    messages=[

    {"role": "system", "content": "You are a customer support agent."},

    {"role": "user", "content": userinput}

    ],

    timeout=30

    )

    result = response.choices[0].message.content

    session.endsession("Success")

    return result

    except openai.APITimeoutError:

    logger.error("LLM call timeout")

    agentops.record(ErrorEvent(

    errortype="APITimeoutError",

    details="LLM call exceeded 30s timeout"

    ))

    session.endsession("Fail")

    return "Sorry, the system is busy. Please try again."

    except Exception as e:

    logger.error(f"Agent error: {e}")

    agentops.record(ErrorEvent(

    errortype=type(e).name,

    details=str(e)

    ))

    session.endsession("Fail")

    raise

    Advanced: Callbacks and Webhooks

    Webhook Configuration

    AgentOps supports webhooks for real-time notifications:

    agentops.init(
    

    tags=["webhook-enabled"],

    )

    Configure webhooks through the AgentOps dashboard under Settings > Webhooks. You can send notifications to Slack, Discord, or custom endpoints when:

    • A session fails
    • Cost exceeds a threshold
    • Latency exceeds limits

    Best Practices

    1. Always End Sessions

    Make sure every session is ended with the appropriate status. Sessions that are not ended will appear as "Indeterminate" in the dashboard.

    # Use try-finally to ensure session always ends
    

    agentops.init()

    try:

    result = runagent()

    agentops.endsession("Success")

    except Exception:

    agentops.endsession("Fail")

    2. Use Descriptive Tags

    agentops.init(tags=[
    

    "production", # Environment

    "v2.3", # Agent version

    "customer-support", # Use case

    "gpt-4o" # Primary model

    ])

    3. Separate Environments

    import os
    
    

    env = os.getenv("ENVIRONMENT", "development")

    agentops.init(tags=[env, "my-agent"])

    4. Monitor Token Usage

    Monitor token usage to optimize costs:

    # Use smaller models for simple tasks
    

    simpleresponse = client.chat.completions.create(

    model="gpt-4o-mini", # Cheaper for classification

    messages=[{"role": "user", "content": "Classify: is this a technical question? 'How to install Python'"}]

    )

    Use larger models for complex tasks

    complexresponse = client.chat.completions.create(

    model="gpt-4o", # More accurate for analysis

    messages=[{"role": "user", "content": "Analyze the following microservices architecture and provide recommendations..."}]

    )

    5. Implement Tracked Retry Logic

    import agentops
    

    from agentops import ActionEvent

    import time

    def callwithretry(messages, maxretries=3):

    for attempt in range(maxretries):

    try:

    response = client.chat.completions.create(

    model="gpt-4o",

    messages=messages

    )

    return response

    except openai.RateLimitError:

    agentops.record(ActionEvent(

    actiontype="retry",

    params={"attempt": attempt + 1, "reason": "ratelimit"}

    ))

    time.sleep(2 attempt)

    agentops.record(ErrorEvent(

    errortype="MaxRetriesExceeded",

    details=f"Failed after {maxretries} attempts"

    ))

    raise Exception("Max retries exceeded")

    6. Batch Processing with Session Per Batch

    def processbatch(items: list, batchid: str):
    

    session = agentops.startsession(

    tags=["batch-processing", f"batch-{batchid}"]

    )

    successcount = 0

    errorcount = 0

    for item in items:

    try:

    processitem(item)

    successcount += 1

    except Exception as e:

    errorcount += 1

    agentops.record(ErrorEvent(

    errortype=type(e).name,

    details=f"Item {item['id']}: {str(e)}"

    ))

    agentops.record(ActionEvent(

    actiontype="batchcomplete",

    params={"total": len(items)},

    returns={"success": successcount, "errors": errorcount}

    ))

    status = "Success" if errorcount == 0 else "Fail"

    session.endsession(status)

    Comparison with Other Monitoring Tools

    | Feature | AgentOps | LangSmith | Helicone | Portkey |

    |---------|----------|-----------|----------|---------|

    | Agent-specific tracking | Yes | Yes | No | Partial |

    | Auto-instrumentation | Yes | Yes | Yes | Yes |

    | Multi-agent support | Yes | Partial | No | No |

    | Cost tracking | Yes | Yes | Yes | Yes |

    | Session replay | Yes | Yes | No | No |

    | Open-source | Yes | No | Partial | No |

    | Framework integrations | Many | LangChain | Agnostic | Agnostic |

    | Self-hosted option | Yes | Enterprise | Yes | No |

    Troubleshooting

    Session Not Appearing in Dashboard

    # Make sure init() is called before LLM calls
    

    agentops.init() # MUST be at the beginning

    Check connection

    import agentops

    result = agentops.init()

    print(f"Init status: {result}")

    Inaccurate Token Count

    Some models may not return token counts. Make sure you are using supported models and the latest SDK version:

    pip install --upgrade agentops openai
    

    High Logging Latency

    AgentOps sends data asynchronously, so it should not add significant latency. If it does, check your internet connection and consider using batch mode:

    agentops.init(
    

    maxwaittime=1000, # Maximum wait time in ms

    maxqueuesize=100 # Queue size before flush

    )

    Complete Example: RAG Agent with Monitoring

    Here is a complete example of a RAG agent monitored with AgentOps:

    import agentops
    

    from agentops import trackagent, ActionEvent, ToolEvent

    import openai

    import numpy as np

    agentops.init(tags=["rag-agent", "production"])

    @trackagent(name="rag-retrieval-agent")

    class RAGAgent:

    def init(self):

    self.client = openai.OpenAI()

    self.knowledgebase = self.loadknowledgebase()

    def loadknowledgebase(self) -> list:

    agentops.record(ActionEvent(

    actiontype="loadknowledgebase",

    returns={"status": "loaded", "documents": 150}

    ))

    return [

    {"text": "Python is a popular programming language.", "embedding": None},

    {"text": "Machine learning requires quality data.", "embedding": None},

    {"text": "Docker simplifies application deployment.", "embedding": None},

    ]

    def retrieve(self, query: str, topk: int = 3) -> list:

    toolevent = ToolEvent(name="vectorsearch")

    toolevent.params = {"query": query, "topk": topk}

    queryembedding = self.client.embeddings.create(

    model="text-embedding-3-small",

    input=query

    )

    results = self.knowledgebase[:topk]

    toolevent.returns = {"resultscount": len(results)}

    agentops.record(toolevent)

    return results

    def generate(self, query: str, context: list) -> str:

    contexttext = "\n".join([doc["text"] for doc in context])

    response = self.client.chat.completions.create(

    model="gpt-4o",

    messages=[

    {"role": "system", "content": f"Answer based on context:\n{contexttext}"},

    {"role": "user", "content": query}

    ]

    )

    return response.choices[0].message.content

    def answer(self, query: str) -> str:

    context = self.retrieve(query)

    return self.generate(query, context)

    agent = RAGAgent()

    answer = agent.answer("What are the benefits of using Docker?")

    print(answer)

    agentops.end_session("Success")

    Conclusion

    AgentOps provides the visibility you need when running AI agents in production. With auto-instrumentation, you only need to add a few lines of code to get comprehensive monitoring.

    Key takeaways from this tutorial:

  • Minimal setup: Just pip install agentops and agentops.init() to start tracking
  • Auto-instrumentation: LLM calls are automatically recorded without significant code changes
  • Custom events: Use ActionEvent, ToolEvent, and ErrorEvent to track business logic
  • Framework support: Built-in integration with LangChain, CrewAI, AutoGen, and more
  • Cost management: Dashboard shows cost breakdown per model and per session
  • Production-ready*: Error handling, retry tracking, and batch processing support
  • With AgentOps, you can build AI agents that are not only functional but also observable and maintainable in production. Start with a simple integration, then add custom events as your monitoring needs grow.

    Related Articles

    Logfire Tutorial: Observability Platform for Python and AI Applications

    Tutorial Logfire: Platform Observability untuk Aplikasi Python dan AI Logfire adalah platform observability buatan tim P...

    Complete Griptape Tutorial: Python Framework for Building AI Agents and Pipelines

    Tutorial Lengkap Griptape: Framework Python untuk Membangun AI Agent dan Pipeline Griptape adalah framework open-source ...

    Semantic Router Tutorial: Smart Routing for LLM Applications

    Tutorial Semantic Router: Routing Cerdas untuk Aplikasi LLM Semantic Router adalah library Python yang memungkinkan Anda...

    Complete Llama Stack Tutorial: Meta's Official Framework for Building AI Applications with Llama Models

    Tutorial Lengkap Llama Stack: Framework Resmi Meta untuk Membangun Aplikasi AI dengan Model Llama Llama Stack adalah fra...