Portkey: One AI Gateway to Manage Every LLM Across Many Providers

Friends, if you have ever built an application that calls an LLM, whether it is OpenAI, Anthropic, Google, or an open source model on Together, you have probably felt the same thing: the code gets messier over time. You have keys scattered everywhere, retries you wrote by hand every time the API times out, fallbacks you built with long try-except blocks, and once you hit production you have no idea why the bill exploded because nobody is tracking how many tokens get burned each day. I have lived through that phase myself and honestly it wears you out.

In this tutorial I want to introduce you to a tool that in my opinion you absolutely need to know if you are serious about building LLM applications. It is called Portkey. Portkey is an AI gateway and observability layer for LLM apps. The core idea is beautifully simple: you put one gateway in front of many providers, then all of your LLM requests flow through that gateway. From there you get a pile of features for free that you used to have to build yourself, things like automatic retries, fallbacks, load balancing, caching, logging, tracing, and even guardrails.

In this article I will walk you through it slowly from scratch. We start with the concept, then installation, then basic usage with the SDK and the OpenAI-compatible way, and then we move on to advanced features like virtual keys, retries, fallbacks, load balancing, semantic caching, observability, and guardrails. Everything comes with runnable Python examples. Let us get started.

Introduction: Why You Need an AI Gateway

Before we jump into code, I want you to understand the problem Portkey is trying to solve. Imagine you have a chatbot application. In the beginning you only use OpenAI. The code is clean, you just call client.chat.completions.create. But once your application gets serious, new needs start showing up one by one.

The first need is usually reliability. LLM APIs do not always run smoothly. Sometimes they time out, sometimes you hit a rate limit, sometimes you get a 500 error from the provider. When a request fails you do not want to give up immediately. You want to retry a few times first. And if OpenAI is badly down, you want to automatically switch over to Anthropic so your application keeps running. That is what we call fallback.

The second need is about cost and visibility. You need to know how many requests come in, how many tokens get used, how much it costs, which model gets called the most, and which requests are slow or failing. Without observability you are completely blind. When the bill arrives you can only be shocked.

The third need is about efficiency. Many requests are actually similar or even exactly the same. If you have to hit a real LLM every single time, that is wasted money and wasted latency. This is where caching plays a role, especially semantic caching which can recognize questions that mean the same thing even when the words differ.

The fourth need is about security and output quality. You do not want your model leaking sensitive data or producing output that does not match the required format. Guardrails help you validate input and output automatically.

Portkey answers all of those needs in a single place. So instead of writing all that logic by hand in each application, you simply route your requests through the Portkey gateway and configure everything through settings. The gateway itself is open source and you can self-host it, but Portkey also offers a hosted version that is free to start with. Even better, Portkey has an OpenAI-compatible interface, so if your code already uses the OpenAI SDK, migrating to Portkey is just changing the base URL and headers. There is almost no big change involved.

Installation

Alright, let us start practicing. First, install the Portkey SDK for Python. Open your terminal and run this command.

pip install portkey-ai

If you want to use the OpenAI-compatible way, you can also install the OpenAI SDK because Portkey can sit on top of it.

pip install openai portkey-ai

After that you need an API key from Portkey. You can sign up for free on the Portkey dashboard, then generate an API key in the settings menu. This API key is used to authenticate to the Portkey gateway. I recommend storing the key in an environment variable so it does not end up written in your code and does not get committed to git.

export PORTKEYAPIKEY="pk-xxxxxxxxxxxxxxxxx"
export OPENAIAPIKEY="sk-xxxxxxxxxxxxxxxxx"