LiveKit Agents Tutorial: Building Real-Time AI Voice Agents with Python
Introduction
LiveKit Agents is an open-source framework for building AI agents that interact with users in real-time through voice, video, and text. It provides high-level abstractions for connecting Large Language Models (LLMs), Speech-to-Text (STT), and Text-to-Speech (TTS) into a single responsive conversational pipeline.
Imagine building a virtual assistant that can answer phone calls, respond to customer inquiries, or guide users through complex processes, all in real-time with minimal latency. LiveKit Agents makes this possible with an architecture optimized for low-latency, streaming interactions.
In this tutorial, we will cover how to install LiveKit Agents, build your first voice agent, integrate various AI providers, and implement advanced techniques for creating responsive and intelligent agents.
Prerequisites
Before getting started, make sure you have:
- Python 3.9 or later
- A LiveKit Cloud account (free for development) or a self-hosted LiveKit server
- An API key from at least one LLM provider (OpenAI, Anthropic, or Google)
- Basic understanding of async programming in Python
Installation
Installing the Base Package
pip install livekit-agents
Installing Plugins
LiveKit Agents uses a plugin system for integration with various AI providers. Install plugins based on your needs:
# STT (Speech-to-Text) plugins
pip install livekit-plugins-deepgram
pip install livekit-plugins-google
LLM plugins
pip install livekit-plugins-openai
pip install livekit-plugins-anthropic
TTS (Text-to-Speech) plugins
pip install livekit-plugins-cartesia
pip install livekit-plugins-elevenlabs
VAD (Voice Activity Detection) plugin
pip install livekit-plugins-silero
Or install all plugins at once
pip install "livekit-agents[all]"
Setting Up Environment Variables
Create a .env file in your project root:
# LiveKit credentials
LIVEKITURL=wss://your-project.livekit.cloud
LIVEKITAPIKEY=yourapikey
LIVEKITAPISECRET=yourapisecret
AI Provider keys
OPENAIAPIKEY=sk-your-openai-key
DEEPGRAMAPIKEY=your-deepgram-key
CARTESIAAPIKEY=your-cartesia-key
You can get LiveKit credentials for free at LiveKit Cloud.
Core Concepts
LiveKit Agents Architecture
LiveKit Agents works with several key components:
Voice Pipeline
The voice conversation pipeline consists of:
Audio Input → VAD → STT → LLM → TTS → Audio Output
- VAD (Voice Activity Detection): Detects when the user starts and stops speaking
- STT (Speech-to-Text): Converts user speech into text
- LLM (Large Language Model): Processes text and generates responses
- TTS (Text-to-Speech): Converts text responses into speech
Building Your First Voice Agent
A Simple Agent
Here is the simplest voice agent example:
import asyncio
from livekit.agents import AutoSubscribe, JobContext, WorkerOptions, cli
from livekit.agents.voice import AgentSession
from livekit.plugins import deepgram, openai, cartesia, silero
async def entrypoint(ctx: JobContext):
await ctx.connect(autosubscribe=AutoSubscribe.AUDIOONLY)
session = AgentSession(
stt=deepgram.STT(),
llm=openai.LLM(model="gpt-4o-mini"),
tts=cartesia.TTS(),
vad=silero.VAD.load(),
)
session.start(ctx.room)
await session.say("Hello! I'm your virtual assistant. How can I help you today?")
if name == "main":
cli.runapp(WorkerOptions(entrypointfnc=entrypoint))
Running the Agent
python myagent.py dev
The dev command runs the agent in development mode, which automatically creates a room and displays a URL for testing your agent in the browser.
Testing in Browser
After the agent is running, open the URL displayed in your terminal or use the LiveKit Playground:
https://agents-playground.livekit.io
Enter your LiveKit credentials and start talking to your agent.
Configuring the System Prompt
Setting Agent Instructions
from livekit.agents.llm import ChatContext, ChatMessage
async def entrypoint(ctx: JobContext):
await ctx.connect(autosubscribe=AutoSubscribe.AUDIOONLY)
initialcontext = ChatContext(
messages=[
ChatMessage(
role="system",
content="""You are a customer service assistant for 'TechStore Online'.
Your responsibilities include:
- Product information and pricing
- Order shipment status
- Return and refund processing
- Handling complaints and suggestions
Be polite, professional, and concise.
Keep responses to 2-3 sentences maximum.
If you don't know the answer, direct customers to a human agent.""",
)
]
)
session = AgentSession(
stt=deepgram.STT(),
llm=openai.LLM(model="gpt-4o-mini"),
tts=cartesia.TTS(),
vad=silero.VAD.load(),
chatctx=initialcontext,
)
session.start(ctx.room)
await session.say("Welcome to TechStore! How can I help you today?")
Function Calling (Tool Use)
Registering Tools for Your Agent
One of the most powerful features of LiveKit Agents is the ability to connect agents with external functions:
from livekit.agents import llm
class OrderLookup(llm.FunctionContext):
@llm.aicallable(description="Look up order status by order number")
async def checkorderstatus(self, orderid: str) -> str:
# Simulated database lookup
orders = {
"ORD001": {"status": "In transit", "eta": "2 days"},
"ORD002": {"status": "Delivered", "eta": "-"},
"ORD003": {"status": "Processing", "eta": "3-5 business days"},
}
order = orders.get(orderid.upper())
if order:
return f"Order {orderid}: Status {order['status']}, ETA: {order['eta']}"
return f"Order {orderid} was not found in our system."
@llm.aicallable(description="Get product information by name")
async def getproductinfo(self, productname: str) -> str:
products = {
"gaming laptop": {"price": "$1,200", "stock": 5},
"wireless mouse": {"price": "$25", "stock": 50},
"mechanical keyboard": {"price": "$80", "stock": 20},
}
product = products.get(productname.lower())
if product:
return f"{productname}: Price {product['price']}, Available stock: {product['stock']} units"
return f"Product '{productname}' was not found in our catalog."
async def entrypoint(ctx: JobContext):
await ctx.connect(autosubscribe=AutoSubscribe.AUDIOONLY)
fncctx = OrderLookup()
session = AgentSession(
stt=deepgram.STT(),
llm=openai.LLM(model="gpt-4o"),
tts=cartesia.TTS(),
vad=silero.VAD.load(),
fncctx=fncctx,
)
session.start(ctx.room)
await session.say("Hello! I can help you check order status or product information. Go ahead and ask!")
Using Alternative Providers
Anthropic Claude as LLM
from livekit.plugins import anthropic
session = AgentSession(
stt=deepgram.STT(),
llm=anthropic.LLM(model="claude-sonnet-4-6"),
tts=cartesia.TTS(),
vad=silero.VAD.load(),
)
Google Gemini as LLM
from livekit.plugins import google
session = AgentSession(
stt=google.STT(languages=["en-US"]),
llm=google.LLM(model="gemini-2.0-flash"),
tts=google.TTS(),
vad=silero.VAD.load(),
)
ElevenLabs for High-Quality TTS
from livekit.plugins import elevenlabs
session = AgentSession(
stt=deepgram.STT(),
llm=openai.LLM(model="gpt-4o-mini"),
tts=elevenlabs.TTS(
voiceid="your-voice-id",
model="eleventurbov25",
),
vad=silero.VAD.load(),
)
Advanced Techniques
Handling Interruptions (Turn-Taking)
LiveKit Agents automatically handles interruptions, but you can customize the behavior:
session = AgentSession(
stt=deepgram.STT(),
llm=openai.LLM(model="gpt-4o-mini"),
tts=cartesia.TTS(),
vad=silero.VAD.load(),
minendpointingdelay=0.5,
allowinterruptions=True,
)
The minendpointingdelay parameter controls how long the agent waits after the user stops speaking before starting to respond. Lower values make the agent more responsive but may cut off users who are pausing to think.
Event Handlers
You can listen to various events during the conversation:
from livekit.agents.voice import AgentSession
async def entrypoint(ctx: JobContext):
await ctx.connect(autosubscribe=AutoSubscribe.AUDIOONLY)
session = AgentSession(
stt=deepgram.STT(),
llm=openai.LLM(model="gpt-4o-mini"),
tts=cartesia.TTS(),
vad=silero.VAD.load(),
)
@session.on("userspeechcommitted")
def onuserspeech(msg):
print(f"User said: {msg.content}")
@session.on("agentspeechcommitted")
def onagentspeech(msg):
print(f"Agent responded: {msg.content}")
@session.on("functioncallsfinished")
def onfunctiondone(calledfunctions):
for fn in calledfunctions:
print(f"Function {fn.callinfo.functioninfo.name} completed: {fn.result}")
session.start(ctx.room)
await session.say("Hello! I'm ready to help you.")
Multi-Modal Agent (Voice + Video)
LiveKit Agents also supports video input for agents that can see:
from livekit.agents import AutoSubscribe
async def entrypoint(ctx: JobContext):
await ctx.connect(autosubscribe=AutoSubscribe.SUBSCRIBEALL)
session = AgentSession(
stt=deepgram.STT(),
llm=openai.LLM(model="gpt-4o"),
tts=cartesia.TTS(),
vad=silero.VAD.load(),
)
session.start(
ctx.room,
agentvideo=True,
)
await session.say("I can see and hear you. Show me something and I'll describe it!")
Session with Memory (Conversation Context)
To maintain longer conversation context:
from livekit.agents.llm import ChatContext, ChatMessage
async def entrypoint(ctx: JobContext):
await ctx.connect(autosubscribe=AutoSubscribe.AUDIOONLY)
chatctx = ChatContext(
messages=[
ChatMessage(
role="system",
content="You are a medical assistant helping patients schedule appointments.",
)
]
)
session = AgentSession(
stt=deepgram.STT(),
llm=openai.LLM(model="gpt-4o"),
tts=cartesia.TTS(),
vad=silero.VAD.load(),
chatctx=chatctx,
maxtoolsteps=5,
)
@session.on("userspeechcommitted")
def onuserspeech(msg):
with open("conversationlog.txt", "a") as f:
f.write(f"User: {msg.content}\n")
@session.on("agentspeechcommitted")
def onagentspeech(msg):
with open("conversationlog.txt", "a") as f:
f.write(f"Agent: {msg.content}\n")
session.start(ctx.room)
Telephony Integration (SIP)
LiveKit Agents supports phone integration via SIP to receive actual phone calls:
from livekit.agents import AutoSubscribe, JobContext, WorkerOptions, cli, JobRequest
async def requesthandler(req: JobRequest):
await req.accept()
async def entrypoint(ctx: JobContext):
await ctx.connect(autosubscribe=AutoSubscribe.AUDIOONLY)
isphonecall = any(
p.kind == "SIP" for p in ctx.room.remoteparticipants.values()
)
systemprompt = """You are a virtual receptionist for HealthFirst Clinic.
Help patients with:
Scheduling doctor appointments
Checking available doctor schedules
Providing general clinic information
"""
if isphonecall:
systemprompt += "\nThe user is calling via phone. Use more formal language."
session = AgentSession(
stt=deepgram.STT(),
llm=openai.LLM(model="gpt-4o-mini"),
tts=cartesia.TTS(),
vad=silero.VAD.load(),
chatctx=ChatContext(
messages=[ChatMessage(role="system", content=systemprompt)]
),
)
session.start(ctx.room)
greeting = "Welcome to HealthFirst Clinic. How can I assist you today?"
await session.say(greeting)
if name == "main":
cli.runapp(
WorkerOptions(
entrypointfnc=entrypoint,
requestfnc=requesthandler,
)
)
Deploying to Production
Using Docker
Create a Dockerfile for your agent:
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "myagent.py", "start"]
# requirements.txt
livekit-agents>=1.0.0
livekit-plugins-openai>=1.0.0
livekit-plugins-deepgram>=1.0.0
livekit-plugins-cartesia>=1.0.0
livekit-plugins-silero>=1.0.0
Deploying to LiveKit Cloud
LiveKit Cloud provides managed infrastructure for running agents:
# Login to LiveKit Cloud CLI
lk cloud login
Deploy agent
lk cloud agent deploy --entry myagent.py
Monitoring and Logging
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("my-agent")
async def entrypoint(ctx: JobContext):
logger.info(f"Agent joined room: {ctx.room.name}")
await ctx.connect(autosubscribe=AutoSubscribe.AUDIOONLY)
session = AgentSession(
stt=deepgram.STT(),
llm=openai.LLM(model="gpt-4o-mini"),
tts=cartesia.TTS(),
vad=silero.VAD.load(),
)
@session.on("userspeechcommitted")
def onuserspeech(msg):
logger.info(f"User input: {msg.content}")
@session.on("agentspeechcommitted")
def onagentspeech(msg):
logger.info(f"Agent output: {msg.content}")
@session.on("error")
def onerror(error):
logger.error(f"Error occurred: {error}")
session.start(ctx.room)
logger.info("Agent session started")
Best Practices
1. Optimize for Low Latency
Choose the fastest providers for each pipeline component:
session = AgentSession(
stt=deepgram.STT(
model="nova-2",
language="en",
interimresults=True,
),
llm=openai.LLM(
model="gpt-4o-mini",
temperature=0.7,
),
tts=cartesia.TTS(
modelid="sonic-english",
speed=1.0,
),
vad=silero.VAD.load(
minspeechduration=0.1,
minsilenceduration=0.3,
),
minendpointingdelay=0.3,
)
2. Handle Errors Gracefully
from livekit.agents.voice import AgentSession
async def entrypoint(ctx: JobContext):
await ctx.connect(autosubscribe=AutoSubscribe.AUDIOONLY)
session = AgentSession(
stt=deepgram.STT(),
llm=openai.LLM(model="gpt-4o-mini"),
tts=cartesia.TTS(),
vad=silero.VAD.load(),
)
@session.on("error")
def onerror(error):
logger.error(f"Session error: {error}")
session.start(ctx.room)
try:
await session.say("Hello! I'm ready to help you.")
except Exception as e:
logger.error(f"Greeting failed: {e}")
await session.say("Sorry, there was a technical issue. Please try again shortly.")
3. Use Streaming for Fast Responses
LiveKit Agents uses streaming by default for both LLM and TTS. Tokens from the LLM are immediately sent to TTS as they become available, so the user starts hearing the response before the LLM finishes generating all the text.
4. Keep Responses Short
For voice agents, overly long responses create a poor user experience. Use system prompts that limit response length:
systemprompt = """Keep your answers brief and clear.
Maximum 2-3 sentences per response.
If a topic is complex, ask the user if they want a more detailed explanation."""
5. Implement Graceful Shutdown
async def entrypoint(ctx: JobContext):
await ctx.connect(autosubscribe=AutoSubscribe.AUDIOONLY)
session = AgentSession(
stt=deepgram.STT(),
llm=openai.LLM(model="gpt-4o-mini"),
tts=cartesia.TTS(),
vad=silero.VAD.load(),
)
async def shutdown():
await session.say("Thank you for contacting us. Goodbye!")
await session.close()
ctx.addshutdowncallback(shutdown)
session.start(ctx.room)
await session.say("Hello! How can I help you?")
Full Example: Restaurant Order Agent
Here is a complete agent example for taking orders at a restaurant:
import asyncio
from datetime import datetime
from livekit.agents import AutoSubscribe, JobContext, WorkerOptions, cli, llm
from livekit.agents.voice import AgentSession
from livekit.agents.llm import ChatContext, ChatMessage
from livekit.plugins import deepgram, openai, cartesia, silero
class RestaurantTools(llm.FunctionContext):
def init(self):
super().init()
self.orders = []
@llm.aicallable(description="Add an item to the customer's order")
async def addtoorder(self, item: str, quantity: int = 1, notes: str = "") -> str:
self.orders.append({
"item": item,
"quantity": quantity,
"notes": notes,
"time": datetime.now().isoformat(),
})
return f"Added {quantity}x {item} to the order. {f'Note: {notes}' if notes else ''}"
@llm.aicallable(description="View all items in the current order")
async def vieworder(self) -> str:
if not self.orders:
return "The order is empty."
orderlist = "\n".join(
f"- {o['quantity']}x {o['item']}{f' ({o['notes']})' if o['notes'] else ''}"
for o in self.orders
)
return f"Current order:\n{orderlist}"
@llm.aicallable(description="Calculate the estimated total price for the order")
async def calculatetotal(self) -> str:
menuprices = {
"burger": 12.99,
"fries": 4.99,
"grilled chicken": 15.99,
"iced tea": 2.99,
"orange juice": 4.49,
}
total = 0
details = []
for o in self.orders:
itemlower = o["item"].lower()
price = menuprices.get(itemlower, 10.00)
subtotal = price * o["quantity"]
total += subtotal
details.append(f"{o['quantity']}x {o['item']}: ${subtotal:.2f}")
return f"Order breakdown:\n" + "\n".join(details) + f"\n\nTotal: ${total:.2f}"
async def entrypoint(ctx: JobContext):
await ctx.connect(autosubscribe=AutoSubscribe.AUDIOONLY)
tools = RestaurantTools()
chatctx = ChatContext(
messages=[
ChatMessage(
role="system",
content="""You are an order-taking assistant for Sunshine Grill.
Our menu:
- Classic Burger: $12.99
- French Fries: $4.99
- Grilled Chicken: $15.99
- Iced Tea: $2.99
- Fresh Orange Juice: $4.49
Your responsibilities:
Greet customers warmly
Help them choose from the menu
Record orders using the available tools
Confirm the order and total price before finalizing
Be friendly, natural, and helpful.""",
)
]
)
session = AgentSession(
stt=deepgram.STT(),
llm=openai.LLM(model="gpt-4o"),
tts=cartesia.TTS(),
vad=silero.VAD.load(),
chatctx=chatctx,
fncctx=tools,
)
session.start(ctx.room)
await session.say(
"Welcome to Sunshine Grill! I'm here to help you place your order. "
"What would you like today?"
)
if name == "main":
cli.runapp(WorkerOptions(entrypoint_fnc=entrypoint))
Conclusion
LiveKit Agents simplifies the process of building real-time AI agents capable of interacting through voice, video, and text. With its modular pipeline architecture and plugin support for various AI providers, you can build sophisticated voice agents without having to manage WebRTC infrastructure or audio processing manually.
Key takeaways from this tutorial:
- Installation and setup of LiveKit Agents with the required plugins
- Building a voice agent with a simple STT → LLM → TTS pipeline
- Function calling to connect agents with business logic
- Multi-provider integration (OpenAI, Anthropic, Google, Deepgram, etc.)
- Advanced techniques including telephony integration, multi-modal agents, and event handling
- Best practices for latency optimization, error handling, and deployment
This framework is well-suited for building customer service automation, virtual receptionists, voice-enabled IoT devices, interactive tutoring systems, and many other real-time AI use cases.
For more information, visit the official LiveKit Agents documentation and explore the example projects available in their GitHub repository.