Helicone: How I Monitor and Control Every LLM Call in Production
Folks, once you start getting serious about building LLM-powered applications, whether it's a chatbot, an agent, or an AI feature inside a product, sooner or later you will hit the same wall: you have no idea what is actually happening behind each call to the model. How many tokens did that cost? What is my total bill this month? Why is this one request so painfully slow? Which user is hammering the API the most? Which prompt version is quietly blowing up my costs? All of these questions are hard to answer if you just fire off openai.chat.completions.create() and forget about it.
In this tutorial I want to introduce you to Helicone. Helicone is an open-source observability platform built specifically for LLMs. Its core job is simple but crucial: it logs, monitors, and gives you full control over every LLM call your application makes. What I love about it is that using it is not complicated at all. You just change the baseurl to the Helicone proxy and add one authentication header, and suddenly all your requests show up neatly in a dashboard. No need to rebuild your architecture, no heavy SDK to install, no changes to your business logic.
Throughout this tutorial I will walk you through everything from scratch: from account setup and the most basic integration, to request logging, custom properties, per-user tracking, caching to save money, rate limiting to keep users honest, all the way to sessions and tracing for agents with many steps. Every example is written in Python using the OpenAI SDK, because that is the most common combination out there. Let's get started.
Introduction
Before we dive into the code, I want you to understand why observability matters and how Helicone solves it.
While your LLM app is still a prototype, everything looks fine. You type a prompt, you get an answer, you are happy. But once it hits production with hundreds or thousands of users, the situation changes completely. API costs can spiral out of control because of an overly long prompt. Requests fail silently and users complain. Latency spikes during peak hours and you have no idea why. Without observability, you are flying blind.
Helicone solves this with two integration approaches:
- Proxy (Gateway): You route your LLM requests through the Helicone proxy. This is the fastest way, you just change the
baseurl. Because requests go through Helicone, it can add features like caching and rate limiting at the network level. - Async Logging: If you would rather not route requests through a proxy (for latency or security policy reasons), you can send logs asynchronously to Helicone after the request completes. The LLM call itself goes directly to OpenAI, and the log is sent separately.
For most cases, the proxy approach is the most practical, and that is what I will focus on here. The reason is that with the proxy you get all the features at once without writing much extra code.
What makes Helicone attractive compared to just storing your own logs in a database:
- Cost and latency dashboards that come ready to use. You can see total spend, breakdown per model, per user, per feature.
- Custom properties to tag every request so you can filter and analyze it later.
- Built-in caching that can dramatically cut costs for repeated requests.
- Rate limiting per user or per property, without having to build your own system.
- Sessions and tracing for agents with many steps, so you can see the full execution flow from start to finish.
And because it is open-source, you can self-host it if you really want full control over your data. But to get started, the free cloud version is more than enough.
Installation
The good news is that to use Helicone with the proxy approach, you actually don't need to install the Helicone package at all. You just need the OpenAI SDK you are already using. But for completeness, let me explain everything you need.
Step 1: Create an Account and Get an API Key
First, sign up at helicone.ai with your email or a GitHub account. Once you're in, open the Settings menu then the API Keys section. Generate a new key, which usually starts with sk-helicone-. Keep this key safe and never leak it into a public repo.
Step 2: Install Dependencies
The only requirement is the OpenAI SDK:
pip install openai