Portkey: One AI Gateway to Manage Every LLM Across Many Providers

Friends, if you have ever built an application that calls an LLM, whether it is OpenAI, Anthropic, Google, or an open source model on Together, you have probably felt the same thing: the code gets messier over time. You have keys scattered everywhere, retries you wrote by hand every time the API times out, fallbacks you built with long try-except blocks, and once you hit production you have no idea why the bill exploded because nobody is tracking how many tokens get burned each day. I have lived through that phase myself and honestly it wears you out.

In this tutorial I want to introduce you to a tool that in my opinion you absolutely need to know if you are serious about building LLM applications. It is called Portkey. Portkey is an AI gateway and observability layer for LLM apps. The core idea is beautifully simple: you put one gateway in front of many providers, then all of your LLM requests flow through that gateway. From there you get a pile of features for free that you used to have to build yourself, things like automatic retries, fallbacks, load balancing, caching, logging, tracing, and even guardrails.

In this article I will walk you through it slowly from scratch. We start with the concept, then installation, then basic usage with the SDK and the OpenAI-compatible way, and then we move on to advanced features like virtual keys, retries, fallbacks, load balancing, semantic caching, observability, and guardrails. Everything comes with runnable Python examples. Let us get started.

Introduction: Why You Need an AI Gateway

Before we jump into code, I want you to understand the problem Portkey is trying to solve. Imagine you have a chatbot application. In the beginning you only use OpenAI. The code is clean, you just call client.chat.completions.create. But once your application gets serious, new needs start showing up one by one.

The first need is usually reliability. LLM APIs do not always run smoothly. Sometimes they time out, sometimes you hit a rate limit, sometimes you get a 500 error from the provider. When a request fails you do not want to give up immediately. You want to retry a few times first. And if OpenAI is badly down, you want to automatically switch over to Anthropic so your application keeps running. That is what we call fallback.

The second need is about cost and visibility. You need to know how many requests come in, how many tokens get used, how much it costs, which model gets called the most, and which requests are slow or failing. Without observability you are completely blind. When the bill arrives you can only be shocked.

The third need is about efficiency. Many requests are actually similar or even exactly the same. If you have to hit a real LLM every single time, that is wasted money and wasted latency. This is where caching plays a role, especially semantic caching which can recognize questions that mean the same thing even when the words differ.

The fourth need is about security and output quality. You do not want your model leaking sensitive data or producing output that does not match the required format. Guardrails help you validate input and output automatically.

Portkey answers all of those needs in a single place. So instead of writing all that logic by hand in each application, you simply route your requests through the Portkey gateway and configure everything through settings. The gateway itself is open source and you can self-host it, but Portkey also offers a hosted version that is free to start with. Even better, Portkey has an OpenAI-compatible interface, so if your code already uses the OpenAI SDK, migrating to Portkey is just changing the base URL and headers. There is almost no big change involved.

Installation

Alright, let us start practicing. First, install the Portkey SDK for Python. Open your terminal and run this command.

pip install portkey-ai

If you want to use the OpenAI-compatible way, you can also install the OpenAI SDK because Portkey can sit on top of it.

pip install openai portkey-ai

After that you need an API key from Portkey. You can sign up for free on the Portkey dashboard, then generate an API key in the settings menu. This API key is used to authenticate to the Portkey gateway. I recommend storing the key in an environment variable so it does not end up written in your code and does not get committed to git.

export PORTKEYAPIKEY="pk-xxxxxxxxxxxxxxxxx"
export OPENAIAPIKEY="sk-xxxxxxxxxxxxxxxxx"

For LLM providers like OpenAI or Anthropic, you have two options for supplying the key. The first option is to send the provider key directly in the header of each request. The second and cleaner option is to create a virtual key on the Portkey dashboard. I will cover virtual keys in more detail in the advanced section. For now let us start with the most basic approach.

Just to make sure your installation works, try this small snippet to check the version.

import portkeyai

print("Portkey version:", portkeyai.version)

If the version prints without errors, then you are ready to continue. Really easy, right.

Basic Usage

Now let us make our first LLM call through Portkey. There are two styles you can use. The first style uses the Portkey SDK directly, the second uses the OpenAI SDK pointed at the Portkey gateway. I will show you both so you are free to choose.

Using the Portkey SDK Directly

This is the most native way. You create a Portkey object, give it your Portkey API key, then specify the provider and the provider key. The call structure is very similar to OpenAI so you will feel right at home.

import os
from portkeyai import Portkey


client = Portkey(
    apikey=os.environ["PORTKEYAPIKEY"],
    provider="openai",
    Authorization=f"Bearer {os.environ['OPENAIAPIKEY']}",
)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are a friendly and concise assistant."},
        {"role": "user", "content": "Explain what an AI gateway is in two sentences."},
    ],
)

print(response.choices[0].message.content)

Notice that the request body is exactly the same as calling OpenAI normally. The only difference is the client object. Every request that goes through here is automatically logged on the Portkey dashboard, so without adding any code you already get free logging.

Using the OpenAI SDK with the Portkey Gateway

If you already have a codebase full of OpenAI SDK usage and you do not want to change much, you can keep using the OpenAI SDK but point its base URL to the Portkey gateway. Portkey provides a helper to build the correct base URL and headers.

import os
from openai import OpenAI
from portkeyai import PORTKEYGATEWAYURL, createHeaders


client = OpenAI(
    apikey=os.environ["OPENAIAPIKEY"],
    baseurl=PORTKEYGATEWAYURL,

    defaultheaders=createHeaders(
        apikey=os.environ["PORTKEYAPIKEY"],

        provider="openai",
    ),
)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "user", "content": "List three benefits of using an AI gateway."},
    ],
)

print(response.choices[0].message.content)

See, the rest of the code is identical to plain OpenAI. You only add baseurl and defaultheaders. This is exactly why migrating to Portkey is almost painless. For teams with a large existing codebase, this is a lifesaver.

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Write a short poem about AI."}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)
print()

import os
from portkeyai import Portkey

client = Portkey(
    apikey=os.environ["PORTKEYAPIKEY"],

    virtualkey="openai-prod-xxxx",
)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hi, are you running through a virtual key?"}],
)

print(response.choices[0].message.content)

Easy, right. You no longer need to mention Authorization or the provider key in your code. Everything is handled on the Portkey side through the virtual key.

Config: The Brain Behind Every Advanced Feature

Before we get into retries, fallbacks, and load balancing, you need to meet the concept of Config. A Config is a JSON configuration object that controls how the gateway treats your request. Inside a config you can define retry strategy, fallback, load balancing, caching, and guardrails. You can build a config directly in code as a dict, or you can store it on the dashboard and just reference its config ID.

I will be using config a lot in the following examples, so get comfortable with its shape. The gist is that you pass config through the config parameter when creating the client.

Automatic Retries

Retry is the simplest feature but one of the most impactful. If your request fails due to a transient error, like a rate limit or server error, the gateway will automatically try again. You just configure how many times and for which status codes.

import os
from portkeyai import Portkey


config = {
    "retry": {
        "attempts": 3,
        "onstatuscodes": [429, 500, 502, 503, 504],

    },
    "virtualkey": "openai-prod-xxxx",
}

client = Portkey(
    apikey=os.environ["PORTKEYAPIKEY"],

    config=config,
)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Test automatic retry."}],
)

print(response.choices[0].message.content)

import os
from portkeyai import Portkey

config = {
    "strategy": {"mode": "fallback"},
    "targets": [
        {
            "virtualkey": "openai-prod-xxxx",

            "overrideparams": {"model": "gpt-4o-mini"},
        },
        {
            "virtualkey": "anthropic-prod-xxxx",

            "overrideparams": {"model": "claude-3-5-sonnet-latest"},
        },
    ],
}

client = Portkey(
    apikey=os.environ["PORTKEYAPIKEY"],

    config=config,
)

response = client.chat.completions.create(
    messages=[{"role": "user", "content": "If OpenAI is down, please fall back to Anthropic."}],
)

print(response.choices[0].message.content)

Notice that here I do not write model in the call, because each target has its own overrideparams. So if OpenAI fails, the gateway immediately tries Anthropic using the Claude model. For production applications that must not go down, this pattern is extremely important. Your application becomes resilient even when one provider has problems.

Load Balancing

Sometimes you have several keys or several equivalent providers, and you want to split the request load among them. The reasons could be to raise throughput, to work around per-key rate limits, or to spread out cost. Load balancing in Portkey lets you assign a weight to each target.

import os
from portkeyai import Portkey


config = {
    "strategy": {"mode": "loadbalance"},
    "targets": [
        {
            "virtualkey": "openai-key-a",
            "weight": 0.7,
            "overrideparams": {"model": "gpt-4o-mini"},

        },
        {
            "virtualkey": "openai-key-b",
            "weight": 0.3,
            "overrideparams": {"model": "gpt-4o-mini"},

        },
    ],
}

client = Portkey(
    apikey=os.environ["PORTKEYAPIKEY"],
    config=config,
)

for i in range(5):
    response = client.chat.completions.create(
        messages=[{"role": "user", "content": f"Request number {i}"}],
    )
    print(response.choices[0].message.content)

With weights of 0.7 and 0.3, roughly 70 percent of requests go to key A and 30 percent to key B. You can also combine load balancing with fallback, so each load-balanced target can have its own fallback. Very flexible for a serious production setup.

Semantic Caching

Caching is a quick way to save both money and time. Portkey has two caching modes. The first is simple cache, which stores an answer when the request is exactly the same. The second is semantic cache, which is smarter because it recognizes questions that mean the same thing even when the words differ. For example "what is the capital of Indonesia" and "name the capital city of the country Indonesia" are different text but mean the same thing. Semantic cache can serve a cached answer for both.

import os
from portkeyai import Portkey


config = {
    "cache": {
        "mode": "semantic",
        "maxage": 3600,
    },
    "virtualkey": "openai-prod-xxxx",

}

client = Portkey(
    apikey=os.environ["PORTKEYAPIKEY"],
    config=config,
)

The first call hits the real LLM
r1 = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "What is the capital of Indonesia?"}],
)
print("First:", r1.choices[0].message.content)

The second call is semantically similar, so it should be served from cache
r2 = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Please name the capital city of the country Indonesia."}],
)
print("Second:", r2.choices[0].message.content)

The maxage parameter determines how long the cache is considered valid, in seconds. In this example it is one hour. For FAQ or customer support applications with lots of repeated questions, semantic caching can slash your cost dramatically. I have seen cases where more than half of the requests could be served from cache. That is a great win for your wallet.

import os
from portkeyai import Portkey

client = Portkey(
    apikey=os.environ["PORTKEYAPIKEY"],

    virtualkey="openai-prod-xxxx",
    traceid="conversation-abc-123",

    metadata={
        "userid": "user42",

        "feature": "support-chatbot",
        "environment": "production",
    },
)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "How do I reset my password?"}],
)

print(response.choices[0].message.content)

import os
from portkeyai import Portkey

config = {
    "virtualkey": "openai-prod-xxxx",

    "inputguardrails": ["guardrail-pii-check-xxxx"],
    "outputguardrails": ["guardrail-valid-json-xxxx"],

}

client = Portkey(
    apikey=os.environ["PORTKEYAPIKEY"],
    config=config,
)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "Reply only in valid JSON with an answer key."},
        {"role": "user", "content": "What is the capital of Japan?"},
    ],
)

print(response.choices[0].message.content)

In this example I attach an input guardrail to check for PII before the request is sent, and an output guardrail to make sure the reply is valid JSON. If the output is not JSON, you can configure Portkey to retry or reject. These guardrails are extremely important if your application operates in a sensitive domain like healthcare or finance, where a bad output can cause real trouble.

Best Practices

Now that you know all the features, I want to share some advice so your Portkey usage stays clean and safe.

First, always use virtual keys for production and never hardcode provider keys in your code. Store the real keys in the Portkey vault, and store your Portkey API key in an environment variable or a secret manager. This reduces the risk of a key leaking and makes key rotation easy.

Second, separate configs per environment. Create a development config that maybe skips expensive fallbacks, and a production config complete with retries, fallbacks, and guardrails. By storing configs on the dashboard and referencing them by ID, you can change gateway behavior without redeploying your application.

Third, start simple and add features as needed. Do not turn on every feature on day one. Start with logging first so you understand your traffic patterns. After that add retries, then fallbacks, then caching once you notice lots of repeated requests. Turn on features based on the data you see on the dashboard, not just because the feature exists.

Fourth, be careful with semantic caching in cases that need fresh answers every time. Semantic cache is great for questions whose answers are stable, but do not use it for requests whose results must always be new or time-dependent, like real-time prices or frequently changing data. Set a max_age that makes sense for your context.

Fifth, use metadata and trace IDs from the start. It looks trivial, but once your application grows and you need to debug or produce cost reports per feature, this data will be a lifesaver. Make it a habit to attach a user ID, feature name, and environment to every request.

Sixth, combine fallback with guardrails for output reliability. For example if an output guardrail fails JSON validation, you can configure a fallback to a smarter model. So your reliability is not only about a provider being alive or dead, but also about consistent output quality.

Seventh, monitor cost regularly through the dashboard. One of Portkey's biggest advantages is cost visibility. Set aside time each week to see which model is the most expensive and whether you can downgrade to a cheaper model without sacrificing quality. Many teams are surprised to find they can save a lot just by moving some traffic to a smaller model.

Conclusion

Alright friends, we have gone a long way from just the concept to complete code examples. I hope you now have a clear picture of why an AI gateway like Portkey is not just a cool add-on, but genuinely helps you build LLM applications that are reliable, cost-efficient, and easy to monitor.

Here is the gist. Portkey puts one gateway in front of all your LLM providers. From that point you get automatic retries so you survive transient errors, fallbacks so your application stays alive even when one provider dies, load balancing so the load is spread evenly, semantic caching so you save cost and time, observability so you are not blind about traffic and cost, plus guardrails so your output is safe and follows the rules. And the best part, because the interface is OpenAI-compatible, the migration is almost painless.

My advice, do not just read. Try it right away. Start by signing up for a Portkey account, install pip install portkey-ai, then run the basic usage examples above. Feel for yourself how nice it is to have all your requests logged on the dashboard without writing any logging code. After that gradually turn on other features as your application needs them. If you are building a serious LLM application that is going to production, having a gateway like this will be the difference between a fragile application and one that is resilient.

That is all from me for now. Happy tinkering with Portkey, and may your LLM applications get even stronger. If you have questions, I am always happy to help. See you in the next tutorial, friends.