Groq API Tutorial: Super Fast LLM Inference for AI Applications

Introduction

Groq has become one of the most popular AI inference platforms thanks to the remarkable speed offered by their proprietary Language Processing Unit (LPU) chip. If you have ever been frustrated by high latency when calling large language model APIs, Groq provides a solution with inference speeds that far surpass traditional GPUs.

In this tutorial, we will learn how to use the Groq API comprehensively, from initial setup, basic chat completion usage, to advanced features like streaming, tool use (function calling), vision models, and integration with popular frameworks like LangChain and LlamaIndex. All code examples in this tutorial can be run directly in your local environment.

Groq provides access to various popular open-source models including Llama, Mixtral, and Gemma with output speeds reaching hundreds of tokens per second. Interestingly, the Groq API uses a format compatible with the OpenAI API, making migration from OpenAI to Groq straightforward.

Installation and Setup

Getting an API Key

The first step is to register and obtain an API key from Groq:

Visit console.groq.com and create an account

Navigate to the API Keys section

Click "Create API Key" and save the generated key

Installing the Library

Groq provides an official Python SDK that can be installed via pip:

pip install groq

For more complete projects, install additional dependencies:

pip install groq python-dotenv httpx Pillow

Environment Configuration

Create a .env file to store your API key:

GROQAPIKEY=gskyourapikeyhere

Verify the installation with a simple script:

import os
from dotenv import loaddotenv

from groq import Groq

loaddotenv()

client = Groq(apikey=os.environ.get("GROQAPIKEY"))


Test connection
models = client.models.list()
for model in models.data:
    print(f"Model: {model.id}")

If successful, you will see a list of available models on Groq.

Basic Usage

Simple Chat Completion

The most basic usage of the Groq API is chat completion:

from groq import Groq

client = Groq()

chatcompletion = client.chat.completions.create(
    messages=[
        {
            "role": "system",
            "content": "You are a helpful and friendly AI assistant."
        },
        {
            "role": "user",
            "content": "Explain what machine learning is in 3 sentences."
        }
    ],
    model="llama-3.3-70b-versatile",
    temperature=0.7,
    maxtokens=1024,

)

print(chatcompletion.choices[0].message.content)

Available Models

Groq provides several popular models. Here are usage recommendations:

# Model for general tasks and reasoning
MODELGENERAL = "llama-3.3-70b-versatile"


Fast model for simple tasks
MODELFAST = "llama-3.1-8b-instant"

Mixtral model for multilingual tasks
MODELMULTILINGUAL = "mixtral-8x7b-32768"


Model with large context window
MODELLONGCONTEXT = "llama-3.3-70b-versatile"  # 128K context


Vision model for images
MODELVISION = "llama-3.2-90b-vision-preview"

Configuring Parameters

You can control model output with various parameters:

response = client.chat.completions.create(
    messages=[
        {"role": "user", "content": "Write a short poem about coding."}
    ],
    model="llama-3.3-70b-versatile",
    temperature=0.9,       # Creativity (0.0 - 2.0)
    maxtokens=512,        # Maximum output tokens

    topp=0.9,             # Nucleus sampling
    frequencypenalty=0.5, # Word repetition penalty

    presencepenalty=0.3,  # Penalty for already mentioned topics
    stop=["---"],          # Stop sequence
)

Multi-turn Conversation

For multi-turn conversations, send the entire conversation history:

conversationhistory = [
    {"role": "system", "content": "You are a patient Python tutor."}
]

def chat(usermessage):