LiveKit Agents Tutorial: Building Real-Time AI Voice Agents with Python

# Tutorial LiveKit Agents: Membangun AI Voice Agent Real-Time dengan Python ## Pengantar LiveKit Agents adalah framework open-source untuk membangun AI agent yang dapat berinteraksi dengan pengguna...

By Ruby Abdullah · · tutorial
LiveKitVoice AIReal-Time AISpeech-to-TextPython

LiveKit Agents Tutorial: Building Real-Time AI Voice Agents with Python

Introduction

LiveKit Agents is an open-source framework for building AI agents that interact with users in real-time through voice, video, and text. It provides high-level abstractions for connecting Large Language Models (LLMs), Speech-to-Text (STT), and Text-to-Speech (TTS) into a single responsive conversational pipeline.

Imagine building a virtual assistant that can answer phone calls, respond to customer inquiries, or guide users through complex processes, all in real-time with minimal latency. LiveKit Agents makes this possible with an architecture optimized for low-latency, streaming interactions.

In this tutorial, we will cover how to install LiveKit Agents, build your first voice agent, integrate various AI providers, and implement advanced techniques for creating responsive and intelligent agents.

Prerequisites

Before getting started, make sure you have:

  • Python 3.9 or later
  • A LiveKit Cloud account (free for development) or a self-hosted LiveKit server
  • An API key from at least one LLM provider (OpenAI, Anthropic, or Google)
  • Basic understanding of async programming in Python

Installation

Installing the Base Package

pip install livekit-agents

Installing Plugins

LiveKit Agents uses a plugin system for integration with various AI providers. Install plugins based on your needs:

# STT (Speech-to-Text) plugins

pip install livekit-plugins-deepgram

pip install livekit-plugins-google

LLM plugins

pip install livekit-plugins-openai

pip install livekit-plugins-anthropic

TTS (Text-to-Speech) plugins

pip install livekit-plugins-cartesia

pip install livekit-plugins-elevenlabs

VAD (Voice Activity Detection) plugin

pip install livekit-plugins-silero

Or install all plugins at once

pip install "livekit-agents[all]"

Setting Up Environment Variables

Create a .env file in your project root:

# LiveKit credentials

LIVEKITURL=wss://your-project.livekit.cloud

LIVEKITAPIKEY=yourapikey

LIVEKITAPISECRET=yourapisecret

AI Provider keys

OPENAIAPIKEY=sk-your-openai-key

DEEPGRAMAPIKEY=your-deepgram-key

CARTESIAAPIKEY=your-cartesia-key

You can get LiveKit credentials for free at LiveKit Cloud.

Core Concepts

LiveKit Agents Architecture

LiveKit Agents works with several key components:

  • Agent: The main logic handling user interactions
  • Worker: The process that runs agents and connects to the LiveKit server
  • Room: A virtual space where agents and users meet
  • Pipeline: A chain of components (STT → LLM → TTS) that process audio in real-time
  • Plugin: Integration modules for various AI providers
  • Voice Pipeline

    The voice conversation pipeline consists of:

    Audio Input → VAD → STT → LLM → TTS → Audio Output
    

    • VAD (Voice Activity Detection): Detects when the user starts and stops speaking
    • STT (Speech-to-Text): Converts user speech into text
    • LLM (Large Language Model): Processes text and generates responses
    • TTS (Text-to-Speech): Converts text responses into speech

    Building Your First Voice Agent

    A Simple Agent

    Here is the simplest voice agent example:

    import asyncio
    

    from livekit.agents import AutoSubscribe, JobContext, WorkerOptions, cli

    from livekit.agents.voice import AgentSession

    from livekit.plugins import deepgram, openai, cartesia, silero

    async def entrypoint(ctx: JobContext):

    await ctx.connect(autosubscribe=AutoSubscribe.AUDIOONLY)

    session = AgentSession(

    stt=deepgram.STT(),

    llm=openai.LLM(model="gpt-4o-mini"),

    tts=cartesia.TTS(),

    vad=silero.VAD.load(),

    )

    session.start(ctx.room)

    await session.say("Hello! I'm your virtual assistant. How can I help you today?")

    if name == "main":

    cli.runapp(WorkerOptions(entrypointfnc=entrypoint))

    Running the Agent

    python myagent.py dev
    

    The dev command runs the agent in development mode, which automatically creates a room and displays a URL for testing your agent in the browser.

    Testing in Browser

    After the agent is running, open the URL displayed in your terminal or use the LiveKit Playground:

    https://agents-playground.livekit.io
    

    Enter your LiveKit credentials and start talking to your agent.

    Configuring the System Prompt

    Setting Agent Instructions

    from livekit.agents.llm import ChatContext, ChatMessage
    
    

    async def entrypoint(ctx: JobContext):

    await ctx.connect(autosubscribe=AutoSubscribe.AUDIOONLY)

    initialcontext = ChatContext(

    messages=[

    ChatMessage(

    role="system",

    content="""You are a customer service assistant for 'TechStore Online'.

    Your responsibilities include:

    • Product information and pricing
    • Order shipment status
    • Return and refund processing
    • Handling complaints and suggestions

    Be polite, professional, and concise.

    Keep responses to 2-3 sentences maximum.

    If you don't know the answer, direct customers to a human agent.""",

    )

    ]

    )

    session = AgentSession(

    stt=deepgram.STT(),

    llm=openai.LLM(model="gpt-4o-mini"),

    tts=cartesia.TTS(),

    vad=silero.VAD.load(),

    chatctx=initialcontext,

    )

    session.start(ctx.room)

    await session.say("Welcome to TechStore! How can I help you today?")

    Function Calling (Tool Use)

    Registering Tools for Your Agent

    One of the most powerful features of LiveKit Agents is the ability to connect agents with external functions:

    from livekit.agents import llm
    
    

    class OrderLookup(llm.FunctionContext):

    @llm.aicallable(description="Look up order status by order number")

    async def checkorderstatus(self, orderid: str) -> str:

    # Simulated database lookup

    orders = {

    "ORD001": {"status": "In transit", "eta": "2 days"},

    "ORD002": {"status": "Delivered", "eta": "-"},

    "ORD003": {"status": "Processing", "eta": "3-5 business days"},

    }

    order = orders.get(orderid.upper())

    if order:

    return f"Order {orderid}: Status {order['status']}, ETA: {order['eta']}"

    return f"Order {orderid} was not found in our system."

    @llm.aicallable(description="Get product information by name")

    async def getproductinfo(self, productname: str) -> str:

    products = {

    "gaming laptop": {"price": "$1,200", "stock": 5},

    "wireless mouse": {"price": "$25", "stock": 50},

    "mechanical keyboard": {"price": "$80", "stock": 20},

    }

    product = products.get(productname.lower())

    if product:

    return f"{productname}: Price {product['price']}, Available stock: {product['stock']} units"

    return f"Product '{productname}' was not found in our catalog."

    async def entrypoint(ctx: JobContext):

    await ctx.connect(autosubscribe=AutoSubscribe.AUDIOONLY)

    fncctx = OrderLookup()

    session = AgentSession(

    stt=deepgram.STT(),

    llm=openai.LLM(model="gpt-4o"),

    tts=cartesia.TTS(),

    vad=silero.VAD.load(),

    fncctx=fncctx,

    )

    session.start(ctx.room)

    await session.say("Hello! I can help you check order status or product information. Go ahead and ask!")

    Using Alternative Providers

    Anthropic Claude as LLM

    from livekit.plugins import anthropic
    
    

    session = AgentSession(

    stt=deepgram.STT(),

    llm=anthropic.LLM(model="claude-sonnet-4-6"),

    tts=cartesia.TTS(),

    vad=silero.VAD.load(),

    )

    Google Gemini as LLM

    from livekit.plugins import google
    
    

    session = AgentSession(

    stt=google.STT(languages=["en-US"]),

    llm=google.LLM(model="gemini-2.0-flash"),

    tts=google.TTS(),

    vad=silero.VAD.load(),

    )

    ElevenLabs for High-Quality TTS

    from livekit.plugins import elevenlabs
    
    

    session = AgentSession(

    stt=deepgram.STT(),

    llm=openai.LLM(model="gpt-4o-mini"),

    tts=elevenlabs.TTS(

    voiceid="your-voice-id",

    model="eleventurbov25",

    ),

    vad=silero.VAD.load(),

    )

    Advanced Techniques

    Handling Interruptions (Turn-Taking)

    LiveKit Agents automatically handles interruptions, but you can customize the behavior:

    session = AgentSession(
    

    stt=deepgram.STT(),

    llm=openai.LLM(model="gpt-4o-mini"),

    tts=cartesia.TTS(),

    vad=silero.VAD.load(),

    minendpointingdelay=0.5,

    allowinterruptions=True,

    )

    The minendpointingdelay parameter controls how long the agent waits after the user stops speaking before starting to respond. Lower values make the agent more responsive but may cut off users who are pausing to think.

    Event Handlers

    You can listen to various events during the conversation:

    from livekit.agents.voice import AgentSession
    
    

    async def entrypoint(ctx: JobContext):

    await ctx.connect(autosubscribe=AutoSubscribe.AUDIOONLY)

    session = AgentSession(

    stt=deepgram.STT(),

    llm=openai.LLM(model="gpt-4o-mini"),

    tts=cartesia.TTS(),

    vad=silero.VAD.load(),

    )

    @session.on("userspeechcommitted")

    def onuserspeech(msg):

    print(f"User said: {msg.content}")

    @session.on("agentspeechcommitted")

    def onagentspeech(msg):

    print(f"Agent responded: {msg.content}")

    @session.on("functioncallsfinished")

    def onfunctiondone(calledfunctions):

    for fn in calledfunctions:

    print(f"Function {fn.callinfo.functioninfo.name} completed: {fn.result}")

    session.start(ctx.room)

    await session.say("Hello! I'm ready to help you.")

    Multi-Modal Agent (Voice + Video)

    LiveKit Agents also supports video input for agents that can see:

    from livekit.agents import AutoSubscribe
    
    

    async def entrypoint(ctx: JobContext):

    await ctx.connect(autosubscribe=AutoSubscribe.SUBSCRIBEALL)

    session = AgentSession(

    stt=deepgram.STT(),

    llm=openai.LLM(model="gpt-4o"),

    tts=cartesia.TTS(),

    vad=silero.VAD.load(),

    )

    session.start(

    ctx.room,

    agentvideo=True,

    )

    await session.say("I can see and hear you. Show me something and I'll describe it!")

    Session with Memory (Conversation Context)

    To maintain longer conversation context:

    from livekit.agents.llm import ChatContext, ChatMessage
    
    

    async def entrypoint(ctx: JobContext):

    await ctx.connect(autosubscribe=AutoSubscribe.AUDIOONLY)

    chatctx = ChatContext(

    messages=[

    ChatMessage(

    role="system",

    content="You are a medical assistant helping patients schedule appointments.",

    )

    ]

    )

    session = AgentSession(

    stt=deepgram.STT(),

    llm=openai.LLM(model="gpt-4o"),

    tts=cartesia.TTS(),

    vad=silero.VAD.load(),

    chatctx=chatctx,

    maxtoolsteps=5,

    )

    @session.on("userspeechcommitted")

    def onuserspeech(msg):

    with open("conversationlog.txt", "a") as f:

    f.write(f"User: {msg.content}\n")

    @session.on("agentspeechcommitted")

    def onagentspeech(msg):

    with open("conversationlog.txt", "a") as f:

    f.write(f"Agent: {msg.content}\n")

    session.start(ctx.room)

    Telephony Integration (SIP)

    LiveKit Agents supports phone integration via SIP to receive actual phone calls:

    from livekit.agents import AutoSubscribe, JobContext, WorkerOptions, cli, JobRequest
    
    

    async def requesthandler(req: JobRequest):

    await req.accept()

    async def entrypoint(ctx: JobContext):

    await ctx.connect(autosubscribe=AutoSubscribe.AUDIOONLY)

    isphonecall = any(

    p.kind == "SIP" for p in ctx.room.remoteparticipants.values()

    )

    systemprompt = """You are a virtual receptionist for HealthFirst Clinic.

    Help patients with:

  • Scheduling doctor appointments
  • Checking available doctor schedules
  • Providing general clinic information
  • """

    if isphonecall:

    systemprompt += "\nThe user is calling via phone. Use more formal language."

    session = AgentSession(

    stt=deepgram.STT(),

    llm=openai.LLM(model="gpt-4o-mini"),

    tts=cartesia.TTS(),

    vad=silero.VAD.load(),

    chatctx=ChatContext(

    messages=[ChatMessage(role="system", content=systemprompt)]

    ),

    )

    session.start(ctx.room)

    greeting = "Welcome to HealthFirst Clinic. How can I assist you today?"

    await session.say(greeting)

    if name == "main":

    cli.runapp(

    WorkerOptions(

    entrypointfnc=entrypoint,

    requestfnc=requesthandler,

    )

    )

    Deploying to Production

    Using Docker

    Create a Dockerfile for your agent:

    FROM python:3.11-slim
    
    

    WORKDIR /app

    COPY requirements.txt .

    RUN pip install --no-cache-dir -r requirements.txt

    COPY . .

    CMD ["python", "myagent.py", "start"]

    # requirements.txt
    

    livekit-agents>=1.0.0

    livekit-plugins-openai>=1.0.0

    livekit-plugins-deepgram>=1.0.0

    livekit-plugins-cartesia>=1.0.0

    livekit-plugins-silero>=1.0.0

    Deploying to LiveKit Cloud

    LiveKit Cloud provides managed infrastructure for running agents:

    # Login to LiveKit Cloud CLI
    

    lk cloud login

    Deploy agent

    lk cloud agent deploy --entry myagent.py

    Monitoring and Logging

    import logging
    
    

    logging.basicConfig(level=logging.INFO)

    logger = logging.getLogger("my-agent")

    async def entrypoint(ctx: JobContext):

    logger.info(f"Agent joined room: {ctx.room.name}")

    await ctx.connect(autosubscribe=AutoSubscribe.AUDIOONLY)

    session = AgentSession(

    stt=deepgram.STT(),

    llm=openai.LLM(model="gpt-4o-mini"),

    tts=cartesia.TTS(),

    vad=silero.VAD.load(),

    )

    @session.on("userspeechcommitted")

    def onuserspeech(msg):

    logger.info(f"User input: {msg.content}")

    @session.on("agentspeechcommitted")

    def onagentspeech(msg):

    logger.info(f"Agent output: {msg.content}")

    @session.on("error")

    def onerror(error):

    logger.error(f"Error occurred: {error}")

    session.start(ctx.room)

    logger.info("Agent session started")

    Best Practices

    1. Optimize for Low Latency

    Choose the fastest providers for each pipeline component:

    session = AgentSession(
    

    stt=deepgram.STT(

    model="nova-2",

    language="en",

    interimresults=True,

    ),

    llm=openai.LLM(

    model="gpt-4o-mini",

    temperature=0.7,

    ),

    tts=cartesia.TTS(

    modelid="sonic-english",

    speed=1.0,

    ),

    vad=silero.VAD.load(

    minspeechduration=0.1,

    minsilenceduration=0.3,

    ),

    minendpointingdelay=0.3,

    )

    2. Handle Errors Gracefully

    from livekit.agents.voice import AgentSession
    
    

    async def entrypoint(ctx: JobContext):

    await ctx.connect(autosubscribe=AutoSubscribe.AUDIOONLY)

    session = AgentSession(

    stt=deepgram.STT(),

    llm=openai.LLM(model="gpt-4o-mini"),

    tts=cartesia.TTS(),

    vad=silero.VAD.load(),

    )

    @session.on("error")

    def onerror(error):

    logger.error(f"Session error: {error}")

    session.start(ctx.room)

    try:

    await session.say("Hello! I'm ready to help you.")

    except Exception as e:

    logger.error(f"Greeting failed: {e}")

    await session.say("Sorry, there was a technical issue. Please try again shortly.")

    3. Use Streaming for Fast Responses

    LiveKit Agents uses streaming by default for both LLM and TTS. Tokens from the LLM are immediately sent to TTS as they become available, so the user starts hearing the response before the LLM finishes generating all the text.

    4. Keep Responses Short

    For voice agents, overly long responses create a poor user experience. Use system prompts that limit response length:

    systemprompt = """Keep your answers brief and clear.
    

    Maximum 2-3 sentences per response.

    If a topic is complex, ask the user if they want a more detailed explanation."""

    5. Implement Graceful Shutdown

    async def entrypoint(ctx: JobContext):
    

    await ctx.connect(autosubscribe=AutoSubscribe.AUDIOONLY)

    session = AgentSession(

    stt=deepgram.STT(),

    llm=openai.LLM(model="gpt-4o-mini"),

    tts=cartesia.TTS(),

    vad=silero.VAD.load(),

    )

    async def shutdown():

    await session.say("Thank you for contacting us. Goodbye!")

    await session.close()

    ctx.addshutdowncallback(shutdown)

    session.start(ctx.room)

    await session.say("Hello! How can I help you?")

    Full Example: Restaurant Order Agent

    Here is a complete agent example for taking orders at a restaurant:

    import asyncio
    

    from datetime import datetime

    from livekit.agents import AutoSubscribe, JobContext, WorkerOptions, cli, llm

    from livekit.agents.voice import AgentSession

    from livekit.agents.llm import ChatContext, ChatMessage

    from livekit.plugins import deepgram, openai, cartesia, silero

    class RestaurantTools(llm.FunctionContext):

    def init(self):

    super().init()

    self.orders = []

    @llm.aicallable(description="Add an item to the customer's order")

    async def addtoorder(self, item: str, quantity: int = 1, notes: str = "") -> str:

    self.orders.append({

    "item": item,

    "quantity": quantity,

    "notes": notes,

    "time": datetime.now().isoformat(),

    })

    return f"Added {quantity}x {item} to the order. {f'Note: {notes}' if notes else ''}"

    @llm.aicallable(description="View all items in the current order")

    async def vieworder(self) -> str:

    if not self.orders:

    return "The order is empty."

    orderlist = "\n".join(

    f"- {o['quantity']}x {o['item']}{f' ({o['notes']})' if o['notes'] else ''}"

    for o in self.orders

    )

    return f"Current order:\n{orderlist}"

    @llm.aicallable(description="Calculate the estimated total price for the order")

    async def calculatetotal(self) -> str:

    menuprices = {

    "burger": 12.99,

    "fries": 4.99,

    "grilled chicken": 15.99,

    "iced tea": 2.99,

    "orange juice": 4.49,

    }

    total = 0

    details = []

    for o in self.orders:

    itemlower = o["item"].lower()

    price = menuprices.get(itemlower, 10.00)

    subtotal = price * o["quantity"]

    total += subtotal

    details.append(f"{o['quantity']}x {o['item']}: ${subtotal:.2f}")

    return f"Order breakdown:\n" + "\n".join(details) + f"\n\nTotal: ${total:.2f}"

    async def entrypoint(ctx: JobContext):

    await ctx.connect(autosubscribe=AutoSubscribe.AUDIOONLY)

    tools = RestaurantTools()

    chatctx = ChatContext(

    messages=[

    ChatMessage(

    role="system",

    content="""You are an order-taking assistant for Sunshine Grill.

    Our menu:

    • Classic Burger: $12.99
    • French Fries: $4.99
    • Grilled Chicken: $15.99
    • Iced Tea: $2.99
    • Fresh Orange Juice: $4.49

    Your responsibilities:

  • Greet customers warmly
  • Help them choose from the menu
  • Record orders using the available tools
  • Confirm the order and total price before finalizing
  • Be friendly, natural, and helpful.""",

    )

    ]

    )

    session = AgentSession(

    stt=deepgram.STT(),

    llm=openai.LLM(model="gpt-4o"),

    tts=cartesia.TTS(),

    vad=silero.VAD.load(),

    chatctx=chatctx,

    fncctx=tools,

    )

    session.start(ctx.room)

    await session.say(

    "Welcome to Sunshine Grill! I'm here to help you place your order. "

    "What would you like today?"

    )

    if name == "main":

    cli.runapp(WorkerOptions(entrypoint_fnc=entrypoint))

    Conclusion

    LiveKit Agents simplifies the process of building real-time AI agents capable of interacting through voice, video, and text. With its modular pipeline architecture and plugin support for various AI providers, you can build sophisticated voice agents without having to manage WebRTC infrastructure or audio processing manually.

    Key takeaways from this tutorial:

    • Installation and setup of LiveKit Agents with the required plugins
    • Building a voice agent with a simple STT → LLM → TTS pipeline
    • Function calling to connect agents with business logic
    • Multi-provider integration (OpenAI, Anthropic, Google, Deepgram, etc.)
    • Advanced techniques including telephony integration, multi-modal agents, and event handling
    • Best practices for latency optimization, error handling, and deployment

    This framework is well-suited for building customer service automation, virtual receptionists, voice-enabled IoT devices, interactive tutoring systems, and many other real-time AI use cases.

    For more information, visit the official LiveKit Agents documentation and explore the example projects available in their GitHub repository.

    Related Articles

    OpenAI Whisper Tutorial: Speech-to-Text and Audio Transcription

    OpenAI Whisper - Tutorial Lengkap Speech-to-Text Daftar Isi Pendahuluan Prasyarat Memahami Ukuran Model Whisper [Tran...

    Flower (flwr): A Complete Guide to Federated Learning in Python

    Tutorial Lengkap Flower (flwr): Framework Federated Learning dengan Python Federated learning adalah paradigma machine l...

    Complete Aisuite Tutorial: Unified Multi-LLM Interface by Andrew Ng

    Tutorial Lengkap Aisuite: Unified Multi-LLM Interface dari Andrew Ng Apa itu Aisuite? Aisuite adalah library Python open...

    Complete Neo4j Tutorial: Graph Database for Knowledge Graphs and AI

    Tutorial Lengkap Neo4j: Graph Database untuk Knowledge Graphs dan AI Graph database telah menjadi komponen penting dalam...