Fireworks AI: A Super Fast Inference Platform for Open LLMs and Multimodal Models
Hey everyone, in this tutorial I want to introduce you to a platform that has been gaining a lot of traction in the LLM world lately, called Fireworks AI. If you have been calling closed-source models all this time and started thinking "why is the latency so slow" or "why does the bill keep growing", well Fireworks AI might be the answer I really want you to try.
Personally I love platforms that focus on speed, because for production applications, latency is not just about looking nice, it genuinely affects the user experience and your costs. Fireworks AI sells itself as a low-latency, high-throughput inference platform that can run many popular open-source models like Llama, Qwen, DeepSeek, Mixtral, and even multimodal models for vision. In this tutorial we will cover everything from the very beginning, starting with getting an API key, installation, chat completions, streaming, structured output aka JSON mode, function calling, vision models, embeddings, and the basics of fine-tuning. I will give you Python examples you can run right away.
Introduction
Before we jump into the code, I want you to understand first what Fireworks AI actually is and why it is interesting. So here is the deal, Fireworks AI is an inference platform. That means they provide the infrastructure to run AI models, especially open-source LLMs, with performance that has been heavily optimized. You do not have to worry about GPUs, deployment, scaling, or inference engine optimization. You just call the API, and the model responds quickly.
A few things drew me to Fireworks. First, the speed. They use their own inference engine called FireAttention, optimized for high throughput and low latency. For applications that need real-time responses like chatbots, coding assistants, or agents, this matters a lot. Second, model flexibility. You can pick from dozens of already-hosted open-source models, or even deploy your own model through their dedicated deployment feature. Third, compatibility. The Fireworks AI API is compatible with the OpenAI format, so if you already have code that uses the OpenAI library, migration is super easy, you just swap the base URL and API key.
For those who cannot picture the use cases yet, Fireworks AI is a great fit for building smart chatbots, RAG (Retrieval Augmented Generation) systems, coding assistants, applications that need image analysis, or pipelines that need structured output for further processing. Because the models are open-source and pricing is per-token with competitive rates, this is also a cost-friendly choice compared to some closed-source providers.
One thing I like to emphasize to everyone, Fireworks AI has two main deployment modes. The first is serverless, where you pay per token and the model is shared among many users. This is great for prototyping and traffic that is not too heavy. The second is on-demand or dedicated deployment, where you get a dedicated GPU for your model, latency is more consistent, and it is ideal for production with high traffic. We will touch on both later.
Installation
Alright, now let us get into the technical part. Before installation, the first thing you need to do is create an account and get an API key.
Get an API Key
The steps are easy. Just open fireworks.ai, then sign up for an account. Once you are in the dashboard, find the API Keys menu. There you can generate a new API key. Keep it safe, do not commit it to Git or share it with anyone else. This API key is what we will use to authenticate all requests.
My advice, store this API key in an environment variable to keep it safe. Never hard-code it directly in your code, especially if your repo is public. Here is how in the terminal:
export FIREWORKSAPIKEY="fwxxxxxxxxxxxxxxxxxxxx"
Or you can use a .env file with the python-dotenv library. I will show you how to use it.
Install the Library
Fireworks AI provides an official SDK for Python. Installation is just one line:
pip install fireworks-ai
If you want to use a .env file to keep things tidy, add this too:
pip install python-dotenv
Besides using the official Fireworks SDK, you can also use the OpenAI library because the Fireworks API is compatible. So if you want, install this as well:
pip install openai
Personally I like showing both approaches, because each has its own advantages. The Fireworks SDK is more native and directly supports Fireworks-specific features. Meanwhile using the OpenAI SDK is convenient if you already have an existing codebase that uses OpenAI, migration is almost effortless.
Verify the Installation
After installation, let us make sure everything works with a small test:
import os
from fireworks.client import Fireworks
client = Fireworks(apikey=os.environ.get("FIREWORKSAPIKEY"))
response = client.chat.completions.create(
model="accounts/fireworks/models/llama-v3p1-8b-instruct",
messages=[{"role": "user", "content": "Hello, what model are you?"}],
)
print(response.choices[0].message.content)
If you get a response from the model, that means your installation and API key are correct. Notice the model name, the Fireworks format always uses the full path accounts/fireworks/models/. This is different from OpenAI which just uses short names like gpt-4. So do not be confused if it looks long, that is indeed the format.
Basic Usage
Now let us get into basic usage. In this section I will explain chat completions, how to use it via OpenAI-compatible, and streaming. This is the foundation you need to master before moving on to more advanced features.
Chat Completions
Chat completions are the most common way to interact with an LLM. You send a list of messages, and the model replies. The message structure is the same as what you usually see, with roles system, user, and assistant.
import os
from fireworks.client import Fireworks
client = Fireworks(apikey=os.environ.get("FIREWORKSAPIKEY"))
response = client.chat.completions.create(
model="accounts/fireworks/models/llama-v3p1-70b-instruct",
messages=[
{"role": "system", "content": "You are a friendly assistant who answers concisely."},
{"role": "user", "content": "Explain what inference latency is in one short paragraph."},
],
temperature=0.6,
maxtokens=512,
)
print(response.choices[0].message.content)
A few important parameters you should know. temperature controls how creative or random the model's answer is, low values like 0.2 make answers more deterministic, high values like 0.9 make them more varied. maxtokens limits the output length. There is also topp for nucleus sampling. I recommend using a low temperature for applications that need consistency.
Using It via OpenAI-Compatible
Now this is what I said earlier is really nice. If you are already familiar with the OpenAI SDK, you just swap baseurl and apikey. Everything else stays almost unchanged.
import os
from openai import OpenAI
client = OpenAI(
baseurl="https://api.fireworks.ai/inference/v1",
apikey=os.environ.get("FIREWORKSAPIKEY"),
)
response = client.chat.completions.create(
model="accounts/fireworks/models/llama-v3p1-70b-instruct",
messages=[
{"role": "user", "content": "List 3 benefits of using open-source LLMs."},
],
)
print(response.choices[0].message.content)
See, the only difference is the baseurl. This is what I really love about Fireworks, because migrating from OpenAI to Fireworks becomes super easy. If you want to try saving costs or want to use open-source models but are too lazy to refactor your code, this approach is the most practical.
Streaming
For real-time applications like chatbots, you definitely want responses to appear word by word, not wait for the whole answer to finish before it all shows up at once. Well, streaming is the solution. With streaming, tokens are sent one by one as they are generated, so the user immediately sees text appear and the perceived latency is much lower.
import os
from fireworks.client import Fireworks
client = Fireworks(apikey=os.environ.get("FIREWORKSAPIKEY"))
stream = client.chat.completions.create(
model="accounts/fireworks/models/llama-v3p1-70b-instruct",
messages=[
{"role": "user", "content": "Write a short story about a robot learning to cook."},
],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)
print()
Notice stream=True and how we iterate each chunk. Each chunk contains a piece of text in delta.content. We print it directly without a newline so it looks like it flows. For those building web applications, these chunks are usually forwarded to the frontend via Server-Sent Events or WebSocket. This is what makes the chatbot experience feel alive and responsive.
If you use OpenAI-compatible, the streaming approach is exactly the same, just add stream=True and iterate the chunks.
Advanced Usage
Okay everyone, now let us get into the more exciting part. Here we will cover structured output aka JSON mode, function calling, vision models, embeddings, and the basics of fine-tuning. This section is what makes Fireworks AI truly powerful for production applications.
Structured Output and JSON Mode
We often need output from the LLM in a structured format so it is easy for a program to process. For example we want to extract data from text and the result must be valid JSON. Fireworks AI supports JSON mode and even structured output with a schema, so you can force the model to produce output in the shape you want.
The most basic approach, you can use responseformat with type jsonobject:
import os
import json
from fireworks.client import Fireworks
client = Fireworks(apikey=os.environ.get("FIREWORKSAPIKEY"))
response = client.chat.completions.create(
model="accounts/fireworks/models/llama-v3p1-70b-instruct",
messages=[
{"role": "system", "content": "Extract the information and reply in JSON."},
{"role": "user", "content": "Budi is 28 years old, works as a data scientist in Jakarta."},
],
responseformat={"type": "jsonobject"},
)
data = json.loads(response.choices[0].message.content)
print(data)
But if you want to be even stricter, Fireworks supports structured output using a JSON schema. This is cool because the model is guaranteed to follow the structure you define. I like using Pydantic to keep it clean:
import os
import json
from pydantic import BaseModel
from fireworks.client import Fireworks
client = Fireworks(apikey=os.environ.get("FIREWORKSAPIKEY"))
class Person(BaseModel):
name: str
age: int
job: str
city: str
response = client.chat.completions.create(
model="accounts/fireworks/models/llama-v3p1-70b-instruct",
messages=[
{"role": "user", "content": "Budi, 28 years old, data scientist in Jakarta."},
],
responseformat={
"type": "jsonobject",
"schema": Person.modeljsonschema(),
},
)
person = Person.modelvalidatejson(response.choices[0].message.content)
print(person)
With this approach, you do not have to worry about the output being broken or not matching. The model will follow the Person schema. This is really important for production applications where the LLM output goes straight into a database or is processed by another program.
Function Calling
Function calling, sometimes called tool calling, is a feature that lets an LLM "call" functions you provide. So instead of the model just answering text, it can decide to call a specific tool with the appropriate arguments. This is the foundation for building AI agents that can interact with external systems, like checking the weather, querying a database, or calling another API.
import os
import json
from fireworks.client import Fireworks
client = Fireworks(apikey=os.environ.get("FIREWORKSAPIKEY"))
tools = [
{
"type": "function",
"function": {
"name": "getweather",
"description": "Get current weather info for a city",
"parameters": {
"type": "object",
"properties": {
"city": {"type": "string", "description": "City name"},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
},
"required": ["city"],
},
},
}
]
response = client.chat.completions.create(
model="accounts/fireworks/models/llama-v3p1-70b-instruct",
messages=[{"role": "user", "content": "How is the weather in Bandung today?"}],
tools=tools,
toolchoice="auto",
)
message = response.choices[0].message
if message.toolcalls:
call = message.toolcalls[0]
print("Model wants to call:", call.function.name)
print("Arguments:", json.loads(call.function.arguments))
Here is the full flow. The model looks at the user's question, decides it needs to call getweather, and provides the city and unit arguments. Your job in the code is to catch that call, actually run the getweather function, then send the result back to the model with role tool. The model will turn the result into a natural answer. This pattern is the same as OpenAI, so if you have ever built an agent, the concept is identical.
Here is a complete loop example that sends the function result back to the model:
def getweather(city, unit="celsius"):
return {"city": city, "temp": 27, "unit": unit, "condition": "partly cloudy"}
messages = [{"role": "user", "content": "How is the weather in Bandung today?"}]
response = client.chat.completions.create(
model="accounts/fireworks/models/llama-v3p1-70b-instruct",
messages=messages,
tools=tools,
toolchoice="auto",
)
msg = response.choices[0].message
if msg.toolcalls:
call = msg.toolcalls[0]
args = json.loads(call.function.arguments)
result = getweather(args)
messages.append(msg)
messages.append({
"role": "tool",
"toolcallid": call.id,
"content": json.dumps(result),
})
final = client.chat.completions.create(
model="accounts/fireworks/models/llama-v3p1-70b-instruct",
messages=messages,
)
print(final.choices[0].message.content)
Vision Models
Fireworks AI also supports multimodal models that can "see" images. This is really useful for use cases like image analysis, OCR, visual description, or content moderation. To use it, you send the image inside a message, either via a URL or base64.
import os
from fireworks.client import Fireworks
client = Fireworks(apikey=os.environ.get("FIREWORKSAPIKEY"))
response = client.chat.completions.create(
model="accounts/fireworks/models/llama-v3p2-11b-vision-instruct",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "What is in this image? Describe it in detail."},
{
"type": "imageurl",
"imageurl": {"url": "https://images.unsplash.com/photo-1518791841217-8f162f1e1131"},
},
],
}
],
)
print(response.choices[0].message.content)
If your image is local, you can encode it to base64 first:
import base64
def encodeimage(path):
with open(path, "rb") as f:
return base64.b64encode(f.read()).decode("utf-8")
b64 = encodeimage("cat.jpg")
response = client.chat.completions.create(
model="accounts/fireworks/models/llama-v3p2-11b-vision-instruct",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image."},
{
"type": "imageurl",
"imageurl": {"url": f"data:image/jpeg;base64,{b64}"},
},
],
}
],
)
print(response.choices[0].message.content)
The message format is the same as OpenAI vision, so again if you have ever played with GPT-4 Vision, this will feel familiar.
Embeddings
Embeddings are numerical representations of text in the form of vectors. This is the foundation for semantic search, RAG systems, clustering, and recommendation. Fireworks AI provides embedding models you can call through the embeddings endpoint.
import os
from fireworks.client import Fireworks
client = Fireworks(apikey=os.environ.get("FIREWORKSAPIKEY"))
response = client.embeddings.create(
model="nomic-ai/nomic-embed-text-v1.5",
input=["Fireworks AI is fast", "An inference platform for open LLMs"],
)
for item in response.data:
print(len(item.embedding), item.embedding[:5])
The resulting vectors can be stored in a vector database like Pinecone, Qdrant, or pgvector. Then when a query comes in, you embed the query too, find the most similar vectors using cosine similarity, and retrieve the relevant documents. This is the core of a RAG system. For those who want to build a chatbot connected to your own knowledge base, the combination of embeddings plus chat completions from Fireworks is a complete package.
Fine-Tuning Basics
Sometimes a general model just is not enough, you need a model that understands your specific style, domain, or format. This is where fine-tuning comes in. Fireworks AI supports fine-tuning using the efficient LoRA technique, so you do not have to retrain the entire model.
Here is the general flow. First, you prepare a dataset in JSONL format, where each line contains a conversation example. The format looks roughly like this:
import json
examples = [
{"messages": [
{"role": "user", "content": "What is latency?"},
{"role": "assistant", "content": "Latency is the delay between a request and a response."},
]},
{"messages": [
{"role": "user", "content": "What is throughput?"},
{"role": "assistant", "content": "Throughput is the number of requests that can be processed per unit of time."},
]},
]
with open("dataset.jsonl", "w", encoding="utf-8") as f:
for ex in examples:
f.write(json.dumps(ex, ensureascii=False) + "\n")
Once the dataset is ready, you use the Fireworks CLI to run the fine-tuning process. Install the CLI via pip install firectl, then log in. The command flow looks roughly like:
# Log in to Fireworks
firectl signin
Upload the dataset
firectl create dataset my-dataset dataset.jsonl
Start a fine-tuning job with LoRA
firectl create fine-tuning-job \
--base-model accounts/fireworks/models/llama-v3p1-8b-instruct \
--dataset my-dataset \
--output-model my-tuned-model
Once the job finishes, you can deploy the fine-tuned model and call it just like a regular model, only the model name changes to your own model. The nice thing about using LoRA is you can have many model variants without expensive deployment costs, because a LoRA adapter is lightweight. For those who have a specific use case and need a model that truly fits your domain, fine-tuning is really worth trying.
Best Practices
After covering all the features, I want to share some tips from experience so you feel more comfortable using Fireworks AI in production.
First, on API key security. I will repeat this because it is really important. Never hard-code your API key in your code. Always use environment variables or a secret manager. If your API key leaks, people can use your quota and you will foot the bill.
Second, choose a model that fits your needs. Fireworks has many model sizes. The 8B model is fast and cheap, great for simple tasks like classification or extraction. The 70B model is smarter but slower and more expensive, suited for complex reasoning. Do not just use the biggest model when the task is simple, that wastes cost and latency. I usually start with a small model, and only move up if the quality is lacking.
Third, take advantage of streaming for interactive applications. Perceived latency matters for user experience. Even though the total generation time is the same, a response that appears gradually feels much faster than waiting for everything to finish.
Fourth, set maxtokens wisely. Do not set it too high if you do not need to, because you pay per token and long output increases latency. Adjust it to your needs.
Fifth, handle errors and rate limits. Sometimes requests can fail due to the network or hitting a rate limit. I recommend wrapping API calls with retries and exponential backoff. Here is a simple example:
import time
import os
from fireworks.client import Fireworks
client = Fireworks(apikey=os.environ.get("FIREWORKSAPIKEY"))
def chatwithretry(messages, model, maxretries=3):
for attempt in range(max_retries):
try:
return client.chat.completions.create(model=model, messages=messages)
except Exception as e:
wait = 2 attempt
print(f"Error: {e}. Retrying in {wait} seconds...")
time.sleep(wait)
raise RuntimeError("Failed after several attempts")
Sixth, consider the deployment option based on scale. If your traffic is still small or you are prototyping, just use serverless, pay per token, practical. But if you have moved into production with high traffic and need consistent latency, it is better to upgrade to on-demand deployment to get a dedicated GPU. The latency is more stable and you do not get queued behind other users.
Seventh, use a low temperature for tasks that need consistency. If you build a system that needs deterministic output like data extraction or structured output, set temperature to 0 or close to 0. If you need creativity like writing stories, then raise it.
Eighth, monitor usage and cost. The Fireworks dashboard provides token usage info. Check it regularly so you do not get surprised at the end of the month. If needed, set a budget alert.
Conclusion
Okay everyone, that was a complete tutorial about Fireworks AI. We covered everything from scratch, starting with getting an API key, installation via pip install fireworks-ai as well as using OpenAI-compatible, then chat completions, streaming, structured output with JSON mode, function calling for building agents, vision models for image analysis, embeddings for RAG, and the basics of fine-tuning with LoRA.
What I want you to take away from this tutorial is, Fireworks AI is a really solid choice if you need fast, cheap, and flexible inference for open-source models. The compatibility with the OpenAI format makes migration easy, and their focus on low latency makes this platform a great fit for production applications that need lightning-fast responses. Add to that the deployment options from serverless to dedicated, you can start small and scale up as your application grows.
My advice, just start with the simple stuff first. Try chat completions, then explore streaming, then move on to advanced features like function calling or vision as your application needs. Do not forget to always pay attention to best practices around security, model selection, and cost management. Happy tinkering everyone, I hope this tutorial is useful and makes you more confident building your own AI applications. See you in the next tutorial!