NVIDIA NIM: A Complete Guide to NVIDIA Inference Microservices

Introduction

NVIDIA NIM (NVIDIA Inference Microservices) is a set of optimized cloud-native microservices designed to accelerate the deployment of AI models in production environments. NIM provides pre-built, GPU-optimized inference containers that allow developers and enterprises to deploy large language models (LLMs), embedding models, vision models, and other AI workloads with minimal configuration and maximum performance.

In the rapidly evolving landscape of AI deployment, one of the most significant challenges organizations face is bridging the gap between model training and production-ready inference. While frameworks like PyTorch and TensorFlow excel at training, serving models at scale with low latency and high throughput requires specialized infrastructure. This is precisely the problem NVIDIA NIM solves.

Why NVIDIA NIM Matters

Traditional approaches to model deployment often involve complex pipelines: converting models to optimized formats, configuring inference servers, tuning GPU memory allocation, and implementing request batching. NIM abstracts away this complexity by packaging everything into ready-to-deploy containers that leverage NVIDIA TensorRT-LLM and Triton Inference Server under the hood.

Key advantages of NVIDIA NIM include:

• Optimized Inference Performance: NIM containers are built with TensorRT-LLM optimization, delivering up to 3-5x faster inference compared to vanilla PyTorch serving for many popular models.
• OpenAI-Compatible API: NIM exposes an API that is fully compatible with the OpenAI API specification, making migration from OpenAI to self-hosted models straightforward.
• Broad Model Support: NIM supports a wide range of models including Llama 3.1, Mistral, Mixtral, Gemma, Phi-3, embedding models like NV-Embed, and vision-language models.
• Enterprise-Grade Reliability: Built-in health checks, monitoring endpoints, and horizontal scaling capabilities make NIM suitable for production workloads.
• Flexible Deployment Options: Use NIM through the cloud API at build.nvidia.com for quick prototyping, or deploy self-hosted containers in your own infrastructure for data privacy and cost control.

NIM vs. Alternatives

Compared to alternatives like vLLM, Ollama, or direct TGI (Text Generation Inference) deployments, NIM offers deeper GPU optimization through its tight integration with NVIDIA hardware. While vLLM provides excellent open-source inference serving, NIM adds enterprise support, pre-optimized model profiles for specific GPU configurations (A100, H100, L40S), and a managed cloud API for rapid prototyping. For organizations already invested in the NVIDIA ecosystem, NIM provides the most streamlined path to production AI inference.

Prerequisites and Setup

Before diving into NIM, you need to set up a few things. This section covers everything from account creation to SDK installation.

Step 1: Create an NVIDIA NGC Account

NVIDIA NGC (NVIDIA GPU Cloud) is the hub for GPU-optimized software, including NIM containers and API access.

Step 2: Generate an API Key

Once logged in to build.nvidia.com:

Set your API key as an environment variable:

export NVIDIAAPIKEY="nvapi-your-api-key-here"

For persistent configuration, add it to your shell profile:

echo 'export NVIDIAAPIKEY="nvapi-your-api-key-here"' >> ~/.bashrc
source ~/.bashrc

Step 3: Install Python Dependencies

Install the required Python packages for working with NIM:

pip install openai langchain-nvidia-ai-endpoints requests

The openai package is the primary SDK since NIM uses an OpenAI-compatible API. The langchain-nvidia-ai-endpoints package provides LangChain integration for building more complex AI applications.

Step 4: Verify Your Setup

Run this quick verification script to confirm everything is working:

import os

apikey = os.environ.get("NVIDIAAPIKEY")

if apikey and apikey.startswith("nvapi-"):

    print("NVIDIA API key is configured correctly.")
else:
    print("ERROR: NVIDIAAPIKEY not set or invalid.")

    print("Please set it with: export NVIDIAAPIKEY='nvapi-...'")

curl -s https://integrate.api.nvidia.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $NVIDIAAPIKEY" \

  -d '{
    "model": "meta/llama-3.1-70b-instruct",
    "messages": [
      {"role": "system", "content": "You are a helpful AI assistant specializing in data science."},
      {"role": "user", "content": "Explain the difference between L1 and L2 regularization in machine learning, with a practical example."}
    ],
    "temperature": 0.7,
    "maxtokens": 1024,
    "topp": 0.9

  }' | python3 -m json.tool

curl -s https://integrate.api.nvidia.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $NVIDIAAPIKEY" \

  -d '{
    "model": "mistralai/mistral-large-2-instruct",
    "messages": [
      {"role": "user", "content": "Write a Python function to calculate the Fibonacci sequence using dynamic programming."}
    ],
    "temperature": 0.3,
    "maxtokens": 512
  }'

Embedding Generation

NIM also supports embedding models, which are essential for semantic search, RAG (Retrieval-Augmented Generation), and similarity matching. Here is an example using the NV-Embed model:

curl -s https://integrate.api.nvidia.com/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $NVIDIAAPIKEY" \
  -d '{
    "model": "nvidia/nv-embedqa-e5-v5",
    "input": ["What is machine learning?", "How does neural network training work?"],
    "inputtype": "query",

    "encodingformat": "float"
  }' | python3 -c "
import json, sys
data = json.load(sys.stdin)
for item in data['data']:
    print(f'Embedding {item[\"index\"]}: dimension={len(item[\"embedding\"])}, first 5 values={item[\"embedding\"][:5]}')
"

Vision Model Examples (Multimodal)

NIM supports vision-language models that can process both images and text. Here is an example using a multimodal model to analyze an image:

curl -s https://integrate.api.nvidia.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $NVIDIAAPIKEY" \
  -d '{
    "model": "microsoft/phi-3.5-vision-instruct",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Describe what you see in this image and identify any objects of interest."
          },
          {
            "type": "imageurl",

            "imageurl": {
              "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/4/47/PNGtransparencydemonstration1.png/300px-PNGtransparencydemonstration1.png"
            }
          }
        ]
      }
    ],
    "maxtokens": 512,

    "temperature": 0.5
  }'

import base64
import requests
import os

def analyzelocalimage(imagepath: str, prompt: str) -> str:
    with open(imagepath, "rb") as f:

        imageb64 = base64.b64encode(f.read()).decode("utf-8")

    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {os.environ['NVIDIAAPIKEY']}"
    }

    payload = {
        "model": "microsoft/phi-3.5-vision-instruct",
        "messages": [
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": prompt},
                    {
                        "type": "imageurl",

                        "imageurl": {
                            "url": f"data:image/png;base64,{imageb64}"

                        }
                    }
                ]
            }
        ],
        "maxtokens": 1024,
        "temperature": 0.5
    }

    response = requests.post(
        "https://integrate.api.nvidia.com/v1/chat/completions",
        headers=headers,
        json=payload
    )
    response.raiseforstatus()
    return response.json()["choices"][0]["message"]["content"]

Usage
result = analyzelocalimage("screenshot.png", "What does this screenshot show?")
print(result)

Understanding the Request and Response Format

NIM follows the OpenAI API specification. Here is a breakdown of the request and response structure:

| Parameter | Type | Description |

|-----------|------|-------------|

| model | string | Model identifier (e.g., meta/llama-3.1-70b-instruct) |

| messages | array | Array of message objects with role and content |

| temperature | float | Sampling temperature (0.0 to 1.0). Lower = more deterministic |

| maxtokens | integer | Maximum number of tokens to generate |

| topp | float | Nucleus sampling parameter (0.0 to 1.0) |

| stream | boolean | Whether to stream the response token by token |

| stop | array | Stop sequences to end generation |

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1695000000,
  "model": "meta/llama-3.1-70b-instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The model's response text..."
      },
      "finishreason": "stop"

    }
  ],
  "usage": {
    "prompttokens": 42,
    "completiontokens": 256,

    "totaltokens": 298
  }
}

Advanced Usage: Self-Hosted Deployment

While the cloud API is excellent for prototyping, production workloads often require self-hosted deployment for data privacy, cost control, or compliance requirements. NIM containers make self-hosting straightforward.

Deploying NIM Containers with Docker

First, authenticate with NVIDIA's container registry:

echo "$NGCAPIKEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin

Pull and run a NIM container for Llama 3.1 8B:

docker run -d \
  --name nim-llama-3.1-8b \
  --gpus all \
  --shm-size=16g \
  -e NGCAPIKEY="$NGCAPIKEY" \
  -p 8000:8000 \
  nvcr.io/nim/meta/llama-3.1-8b-instruct:latest

The container will download the optimized model on first run and start serving on port 8000. You can monitor the startup process:

docker logs -f nim-llama-3.1-8b

Wait for the message indicating the server is ready, then test it:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta/llama-3.1-8b-instruct",
    "messages": [{"role": "user", "content": "Hello, how are you?"}],
    "maxtokens": 100

  }'

# Llama 3.1 8B - Works on a single GPU with 24GB+ VRAM (e.g., RTX 4090, A10G)
docker run -d --gpus '"device=0"' --shm-size=16g \
  -e NGCAPIKEY="$NGCAPIKEY" \

  -p 8000:8000 \
  nvcr.io/nim/meta/llama-3.1-8b-instruct:latest

Llama 3.1 70B - Requires multiple GPUs (e.g., 2x A100 80GB or 4x A100 40GB)
docker run -d --gpus all --shm-size=16g \
  -e NGCAPIKEY="$NGCAPIKEY" \

  -e NIMTENSORPARALLELSIZE=2 \
  -p 8000:8000 \
  nvcr.io/nim/meta/llama-3.1-70b-instruct:latest

Mistral 7B with specific GPU selection
docker run -d --gpus '"device=1"' --shm-size=16g \
  -e NGCAPIKEY="$NGCAPIKEY" \
  -p 8001:8000 \
  nvcr.io/nim/mistralai/mistral-7b-instruct-v03:latest

To persist downloaded models and avoid re-downloading on container restart, mount a local cache directory:

mkdir -p /opt/nim/cache

docker run -d --gpus all --shm-size=16g \
  -e NGCAPIKEY="$NGCAPIKEY" \
  -v /opt/nim/cache:/opt/nim/.cache \
  -p 8000:8000 \
  nvcr.io/nim/meta/llama-3.1-8b-instruct:latest

Scaling with Docker Compose

For multi-model deployments or load balancing, use Docker Compose:

# docker-compose.yml
version: '3.8'

services:
  nim-llama-8b:
    image: nvcr.io/nim/meta/llama-3.1-8b-instruct:latest
    deploy:
      resources:
        reservations:
          devices:
driver: nvidia
              deviceids: ['0']

              capabilities: [gpu]
    environment:
NGCAPIKEY=${NGCAPIKEY}

    ports:

"8000:8000"
    shmsize: '16g'

    volumes:
nim-cache:/opt/nim/.cache
    healthcheck:

      test: ["CMD", "curl", "-f", "http://localhost:8000/v1/health/ready"]
      interval: 30s
      timeout: 10s
      retries: 5
      startperiod: 300s


  nim-embedding:
    image: nvcr.io/nim/nvidia/nv-embedqa-e5-v5:latest
    deploy:
      resources:
        reservations:
          devices:
driver: nvidia
              deviceids: ['1']

              capabilities: [gpu]
    environment:
NGCAPIKEY=${NGCAPIKEY}
    ports:

"8001:8000"
    shmsize: '8g'

    volumes:
nim-cache:/opt/nim/.cache

  nginx:
    image: nginx:alpine
    ports:
"80:80"
    volumes:

./nginx.conf:/etc/nginx/nginx.conf:ro
    dependson:

      nim-llama-8b:
        condition: servicehealthy


volumes:
  nim-cache:

# Check if the model is loaded and ready to serve
curl http://localhost:8000/v1/health/ready

Check if the container is alive
curl http://localhost:8000/v1/health/live

Get model information
curl http://localhost:8000/v1/models

Prometheus-compatible metrics endpoint
curl http://localhost:8000/metrics

#!/bin/bash
nim-health-check.sh

NIMENDPOINT="http://localhost:8000"
ALERTWEBHOOK="https://hooks.slack.com/your-webhook-url"


health=$(curl -s -o /dev/null -w "%{httpcode}" "$NIMENDPOINT/v1/health/ready")


if [ "$health" != "200" ]; then
    echo "[$(date)] NIM health check FAILED (HTTP $health)"
    curl -s -X POST "$ALERTWEBHOOK" \
      -H "Content-Type: application/json" \
      -d "{\"text\": \"NIM health check failed at $(date). HTTP status: $health\"}"
else
    echo "[$(date)] NIM health check OK"
fi

Python Integration

Python is the primary language for AI development, and NIM provides first-class Python support through multiple libraries.

Using the OpenAI Python SDK

Since NIM is OpenAI API-compatible, you can use the official openai Python library with minimal changes:

from openai import OpenAI
import os

For cloud API (build.nvidia.com)
client = OpenAI(
    baseurl="https://integrate.api.nvidia.com/v1",

    apikey=os.environ["NVIDIAAPIKEY"]
)

For self-hosted NIM
client = OpenAI(
baseurl="http://localhost:8000/v1",

apikey="not-used"  # Self-hosted NIM doesn't require an API key by default
)

response = client.chat.completions.create(
    model="meta/llama-3.1-70b-instruct",
    messages=[
        {"role": "system", "content": "You are a senior Python developer. Provide concise, well-commented code."},
        {"role": "user", "content": "Write a Python class for a thread-safe connection pool with configurable max connections."}
    ],
    temperature=0.3,
    maxtokens=1024

)

print(response.choices[0].message.content)
print(f"\nTokens used: {response.usage.totaltokens}")

Streaming Responses

For real-time applications like chatbots, streaming is essential. NIM supports SSE (Server-Sent Events) streaming:

from openai import OpenAI
import os

client = OpenAI(
    baseurl="https://integrate.api.nvidia.com/v1",

    apikey=os.environ["NVIDIAAPIKEY"]
)

stream = client.chat.completions.create(
    model="meta/llama-3.1-70b-instruct",
    messages=[
        {"role": "user", "content": "Write a detailed explanation of how transformers work in deep learning."}
    ],
    temperature=0.7,
    maxtokens=2048,

    stream=True
)

fullresponse = ""
for chunk in stream:
    if chunk.choices[0].delta.content is not None:
        token = chunk.choices[0].delta.content
        print(token, end="", flush=True)
        fullresponse += token


print(f"\n\nTotal characters received: {len(fullresponse)}")

Using LangChain NVIDIA AI Endpoints

For more complex applications involving chains, agents, or RAG pipelines, the langchain-nvidia-ai-endpoints package provides native LangChain integration:

from langchainnvidiaaiendpoints import ChatNVIDIA, NVIDIAEmbeddings
from langchaincore.messages import HumanMessage, SystemMessage
import os

Initialize the chat model
llm = ChatNVIDIA(
    model="meta/llama-3.1-70b-instruct",
    apikey=os.environ["NVIDIAAPIKEY"],

    temperature=0.5,
    maxtokens=1024
)

Simple invocation
messages = [
    SystemMessage(content="You are a data engineering expert."),
    HumanMessage(content="What are the key differences between Apache Kafka and Apache Pulsar?")
]

response = llm.invoke(messages)
print(response.content)

Initialize embeddings
embeddings = NVIDIAEmbeddings(
    model="nvidia/nv-embedqa-e5-v5",
    apikey=os.environ["NVIDIAAPIKEY"]

)

Generate embeddings for documents
docs = [
    "Machine learning is a subset of artificial intelligence.",
    "Deep learning uses neural networks with many layers.",
    "Natural language processing deals with text and speech."
]

vectors = embeddings.embeddocuments(docs)
print(f"Generated {len(vectors)} embeddings, each with {len(vectors[0])} dimensions")

Generate query embedding for search
queryvector = embeddings.embedquery("What is deep learning?")
print(f"Query embedding dimension: {len(queryvector)}")

from openai import OpenAI
import json
import os

client = OpenAI(
    baseurl="https://integrate.api.nvidia.com/v1",
    apikey=os.environ["NVIDIAAPIKEY"]

)

tools = [
    {
        "type": "function",
        "function": {
            "name": "getweather",
            "description": "Get the current weather for a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "City and country, e.g. 'Jakarta, Indonesia'"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "searchdatabase",

            "description": "Search a product database by query",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {"type": "string", "description": "Search query"},
                    "limit": {"type": "integer", "description": "Max results to return"}
                },
                "required": ["query"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="meta/llama-3.1-70b-instruct",
    messages=[
        {"role": "user", "content": "What is the weather like in Jakarta right now?"}
    ],
    tools=tools,
    toolchoice="auto"
)

message = response.choices[0].message

if message.toolcalls:

    for toolcall in message.toolcalls:

        print(f"Function: {toolcall.function.name}")
        print(f"Arguments: {toolcall.function.arguments}")


        # In production, you would call the actual function here
        # and send the result back to the model
        args = json.loads(toolcall.function.arguments)
        print(f"Parsed args: {args}")

Batch Processing

For processing large volumes of requests efficiently, implement batch processing with concurrency control:

import asyncio
from openai import AsyncOpenAI
import os
import time

client = AsyncOpenAI(
    baseurl="https://integrate.api.nvidia.com/v1",

    apikey=os.environ["NVIDIAAPIKEY"]
)

async def processsingle(prompt: str, semaphore: asyncio.Semaphore) -> dict:

    """Process a single prompt with concurrency control."""
    async with semaphore:
        try:
            response = await client.chat.completions.create(
                model="meta/llama-3.1-70b-instruct",
                messages=[{"role": "user", "content": prompt}],
                temperature=0.3,
                maxtokens=256
            )
            return {
                "prompt": prompt,
                "response": response.choices[0].message.content,
                "tokens": response.usage.totaltokens,

                "status": "success"
            }
        except Exception as e:
            return {
                "prompt": prompt,
                "response": None,
                "tokens": 0,
                "status": f"error: {str(e)}"
            }

async def batchprocess(prompts: list[str], maxconcurrent: int = 5) -> list[dict]:

    """Process multiple prompts concurrently with a concurrency limit."""
    semaphore = asyncio.Semaphore(maxconcurrent)
    tasks = [processsingle(prompt, semaphore) for prompt in prompts]

    results = await asyncio.gather(tasks)

    return results

Example usage
prompts = [
    "Summarize the benefits of containerization in one paragraph.",
    "What is the CAP theorem in distributed systems?",
    "Explain the difference between SQL and NoSQL databases.",
    "What are microservices and when should you use them?",
    "Describe the observer design pattern with a use case.",
    "What is eventual consistency in distributed databases?",
    "Explain CQRS pattern and its advantages.",
    "What is the difference between gRPC and REST?",
]

async def main():
    start = time.time()
    results = await batchprocess(prompts, maxconcurrent=4)

    elapsed = time.time() - start

    for r in results:
        status = r["status"]
        tokens = r["tokens"]
        preview = r["response"][:80] + "..." if r["response"] else "N/A"
        print(f"[{status}] ({tokens} tokens) {preview}")

    successful = sum(1 for r in results if r["status"] == "success")
    totaltokens = sum(r["tokens"] for r in results)
    print(f"\nProcessed {successful}/{len(prompts)} prompts in {elapsed:.2f}s")
    print(f"Total tokens: {totaltokens}")


asyncio.run(main())

# Optimize for latency: smaller maxtokens, lower temperature
fastresponse = client.chat.completions.create(

    model="meta/llama-3.1-8b-instruct",
    messages=[{"role": "user", "content": "Classify this text as positive or negative: 'Great product!'"}],
    temperature=0.0,       # Deterministic output, faster
    maxtokens=10,         # Limit output length
    topp=1.0

)

Optimize for throughput: use streaming and process in batches
Set NIMMAXBATCHSIZE environment variable when running the container
docker run ... -e NIMMAXBATCHSIZE=64 ...

import hashlib
import json
from functools import lrucache


class NIMCache:
    """Simple in-memory cache for NIM responses."""

    def init(self, client, maxsize=1000):
        self.client = client
        self.cache = {}
        self.maxsize = maxsize
        self.hits = 0
        self.misses = 0

    def makekey(self, model: str, messages: list, temperature: float) -> str:
        content = json.dumps({"model": model, "messages": messages, "temperature": temperature}, sortkeys=True)

        return hashlib.sha256(content.encode()).hexdigest()

    def chatcompletion(self, model: str, messages: list, temperature: float = 0.0, kwargs):

        # Only cache deterministic requests (temperature=0)
        if temperature > 0:
            self.misses += 1
            return self.client.chat.completions.create(
                model=model, messages=messages, temperature=temperature, kwargs
            )

        key = self.makekey(model, messages, temperature)

        if key in self.cache:
            self.hits += 1
            return self.cache[key]

        self.misses += 1
        response = self.client.chat.completions.create(
            model=model, messages=messages, temperature=temperature, kwargs

        )

        if len(self.cache) >= self.maxsize:

            oldestkey = next(iter(self.cache))
            del self.cache[oldestkey]


        self.cache[key] = response
        return response

    def stats(self) -> dict:
        total = self.hits + self.misses
        hitrate = (self.hits / total  100) if total > 0 else 0
        return {"hits": self.hits, "misses": self.misses, "hitrate": f"{hitrate:.1f}%"}

Usage
cachedclient = NIMCache(client)

response = cachedclient.chatcompletion(

    model="meta/llama-3.1-70b-instruct",
    messages=[{"role": "user", "content": "What is 2+2?"}],
    temperature=0.0,
    maxtokens=50
)
print(cachedclient.stats())

import time
from openai import OpenAI, APIError, RateLimitError, APIConnectionError

def robustcompletion(
    client: OpenAI,
    model: str,
    messages: list,
    maxretries: int = 3,

    basedelay: float = 1.0,
    kwargs

) -> dict:
    """Make an API call with exponential backoff retry logic."""
    lastexception = None


    for attempt in range(maxretries):
        try:
            response = client.chat.completions.create(
                model=model,
                messages=messages,
                kwargs
            )
            return response

        except RateLimitError as e:
            lastexception = e

            delay = basedelay  (2  attempt)

            print(f"Rate limited. Retrying in {delay}s (attempt {attempt + 1}/{maxretries})")

            time.sleep(delay)

        except APIConnectionError as e:
            lastexception = e
            delay = basedelay  (2 * attempt)

            print(f"Connection error. Retrying in {delay}s (attempt {attempt + 1}/{maxretries})")
            time.sleep(delay)

        except APIError as e:
            if e.statuscode and e.statuscode >= 500:
                lastexception = e

                delay = basedelay  (2  attempt)

                print(f"Server error ({e.statuscode}). Retrying in {delay}s")

                time.sleep(delay)
            else:
                raise  # Client errors (4xx) should not be retried

    raise lastexception

Usage
response = robustcompletion(

    client=client,
    model="meta/llama-3.1-70b-instruct",
    messages=[{"role": "user", "content": "Hello!"}],
    maxretries=3,
    temperature=0.7,
    maxtokens=256

)

class TokenBudget:
    """Track and limit token usage."""

    def init(self, dailylimit: int = 1000000):
        self.dailylimit = dailylimit
        self.tokensused = 0


    def track(self, response) -> bool:
        tokens = response.usage.totaltokens
        self.tokensused += tokens

        remaining = self.dailylimit - self.tokensused

        print(f"Tokens used: {tokens} | Total today: {self.tokensused} | Remaining: {remaining}")
        return self.tokensused < self.dailylimit

    def canproceed(self, estimatedtokens: int = 500) -> bool:
        return (self.tokensused + estimatedtokens) < self.dailylimit


budget = TokenBudget(dailylimit=500000)


Before each request
if budget.canproceed():
    response = client.chat.completions.create(
        model="meta/llama-3.1-70b-instruct",
        messages=[{"role": "user", "content": "Your prompt here"}],
        max_tokens=256
    )
    budget.track(response)
else:
    print("Daily token budget exceeded. Request blocked.")