Together AI: A Complete Guide to Inference and Fine-Tuning Open Source Models with One API
Hey everyone! This time I want to talk about a platform that I have been using a lot lately for my AI projects, and it is called Together AI. For those of you who so far only know OpenAI or Anthropic as language model API providers, Together AI offers something different: access to popular open source models like Llama, Qwen, DeepSeek, Mixtral, and dozens of other models, through a single API that is compatible with the OpenAI format. So if you are already used to coding with the OpenAI library, there is almost no learning curve at all.
What makes me stick with Together AI is not just about inference, meaning running a model to generate text. The platform also provides fine-tuning that is super easy, embeddings for semantic search, image generation models, plus function calling and structured output. So it is genuinely a one-stop shop for almost all your AI needs. In this tutorial I will explain everything from scratch: how to get an API key, install the library, all the way to Python code examples you can run right away on your laptop or server.
I wrote this tutorial assuming you already understand the basics of Python and know a little bit about LLMs. But do not worry, I will explain things slowly. Let us get started!
Introduction
Before we jump into code, I want to give you an overview of why Together AI is interesting and when you should use this platform.
What is Together AI
Together AI is a cloud platform that focuses on running open source AI models at scale. Imagine you want to use Llama 3.3 70B or DeepSeek R1 for your application. If you run it yourself, you need expensive GPUs, a complicated setup, and electricity costs that are not cheap. Together AI provides all of that as a service, so you just call the API and pay per token you use. No need to think about infrastructure at all.
There are a few reasons why I really like this platform:
First, the model collection is very broad. From chat models like Llama, Qwen, Mistral, and DeepSeek, to models specialized for coding, reasoning, embeddings, and even image generation. You are free to pick whichever model fits your use case best without being locked into a single vendor.
Second, the pricing is transparent and competitive. Because we use open source models, the cost is much cheaper than proprietary models. For startups or indie developers like us, this helps a lot in saving budget.
Third, and this is important, the API is OpenAI-compatible. So if you have old code that uses the openai library, you just swap the base URL and API key, and it runs immediately. Migration becomes super easy.
When should you use Together AI
In my opinion Together AI is a great fit when you want more control over the model you use, when you want to fine-tune your own model with your data, or when you want to save cost while still getting good quality. If you need data privacy and do not want your data used for training, Together AI also has a clear policy on this. For me personally, this platform is my go-to choice when I am building fast prototypes or production applications that need open source models.
Installation
Okay now we get into the technical part. First we need to get an API key and install the library.
Get an API Key
The first step, open the together.ai website and register an account. The process is quick, just use email or sign in with Google. After you enter the dashboard, look for the Settings or API Keys menu. There you can generate a new API key. Keep this key safe, do not let it get committed to a public repository or leaked to other people. Usually you also get free credits when you first sign up, so you can experiment right away without spending money.
The safest way to store an API key is using an environment variable. So do not hardcode it in your code. In the terminal, you can set it like this:
export TOGETHERAPIKEY="your-api-key-here"
If you are on Windows using PowerShell:
$env:TOGETHERAPIKEY="your-api-key-here"
Or even neater, create a .env file in your project folder and store the key there. I will show you how to load it later.
Install the Python Library
Now we install the official Together AI library. Open your terminal and run:
pip install together
If you want to load the API key from a .env file, also install python-dotenv:
pip install together python-dotenv
I recommend using a virtual environment so your dependencies stay tidy and do not clash with other projects:
python -m venv venv
source venv/bin/activate # Linux/Mac
or on Windows: venv\Scripts\activate
pip install together python-dotenv
Verify the Installation
After installing, let us check whether everything works by making a simple script. Create a file testtogether.py:
import os
from together import Together
Initialize the client. If TOGETHERAPIKEY is already set
in the environment, we do not need to pass it manually
client = Together()
If you want to pass it manually:
client = Together(apikey="your-api-key-here")
Try listing some available models
models = client.models.list()
print(f"Total models available: {len(models)}")
for model in models[:5]:
print(f"- {model.id}")
If you use a .env file, the code is slightly different:
import os
from dotenv import loaddotenv
from together import Together
loaddotenv() # read the .env file
client = Together(apikey=os.environ.get("TOGETHERAPIKEY"))
print("Client initialized successfully!")
Run it with python testtogether.py. If a list of models appears, your setup is correct and we are ready to move on to the more exciting parts.
Basic Usage
Now we get into basic usage. I will show you the most common use case which is chat completions, then streaming, and how to use the OpenAI library directly for those who are already comfortable with it.
Your First Chat Completion
This is the most common use case: we send a message to the model and the model replies with an answer. The format is exactly like OpenAI, so there is the concept of messages with roles system, user, and assistant.
from together import Together
client = Together()
response = client.chat.completions.create(
model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
messages=[
{"role": "system", "content": "You are a friendly assistant who answers clearly."},
{"role": "user", "content": "Explain what machine learning is with a simple analogy."}
],
)
print(response.choices[0].message.content)
Super easy right? We just specify which model to use, give a list of messages, and the model replies. The system field is for setting the behavior and persona of the model, user is our question, and the model's answer is in response.choices[0].message.content.
I use the model meta-llama/Llama-3.3-70B-Instruct-Turbo in this example because in my opinion it is the sweet spot between quality and speed. But you are free to switch to another model based on your needs.
Controlling Generation Parameters
Models have several parameters we can adjust to control the output. The ones I use most often are temperature, maxtokens, and topp.
from together import Together
client = Together()
response = client.chat.completions.create(
model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
messages=[
{"role": "user", "content": "Write one paragraph of a story about a robot learning to paint."}
],
temperature=0.9, # higher means more creative/random (0 to 2)
maxtokens=300, # limit on answer length in tokens
topp=0.9, # nucleus sampling, an alternative way to control randomness
)
print(response.choices[0].message.content)
print(f"\nTokens used: {response.usage.totaltokens}")
A low temperature (say 0.1) makes answers more deterministic and focused, good for tasks that need definite answers like data extraction. A high temperature (say 0.9) makes answers more creative and varied, good for brainstorming or writing stories. maxtokens is for limiting output length so it does not get too long and to control cost.
Streaming Responses
If you build a chat application like ChatGPT, you definitely want the answer to appear token by token in real time, rather than waiting for everything to finish before it shows up. This is called streaming. Together AI supports this easily.
from together import Together
client = Together()
stream = client.chat.completions.create(
model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
messages=[
{"role": "user", "content": "Tell a brief history of the Python programming language."}
],
stream=True, # this is the key
)
for chunk in stream:
content = chunk.choices[0].delta.content
if content:
print(content, end="", flush=True)
print() # newline at the end
With stream=True, the response becomes a generator that yields chunks of text. We loop over each chunk and grab delta.content. Notice I use end="" and flush=True in print so the text appears smoothly without newlines and renders immediately. The effect is exactly like typing in ChatGPT, it looks more responsive to the user.
Using the OpenAI Library Directly
Now this is my favorite part. Because the Together API is OpenAI-compatible, you can use the openai library that you may already have installed, just swap the base URL and API key. So if you have a codebase that already uses OpenAI, migrating to Together AI is just changing two lines.
from openai import OpenAI
client = OpenAI(
apikey="your-together-api-key",
baseurl="https://api.together.xyz/v1",
)
response = client.chat.completions.create(
model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
messages=[
{"role": "user", "content": "What is the difference between a list and a tuple in Python?"}
],
)
print(response.choices[0].message.content)
Cool right? The rest of your application code does not need to change at all. This makes Together AI a super easy drop-in replacement for OpenAI, especially if you want to save cost or use open source models.
Advanced Usage
Okay now we get into the more advanced part. Here I will cover JSON mode and structured output, function calling, embeddings, image generation, all the way to fine-tuning your own model. This is the part that makes Together AI genuinely powerful.
JSON Mode and Structured Output
Often we need the model's output in structured JSON format so it is easy to process by a program. For example we want to extract information from text into a tidy object. Together AI supports JSON mode and can even be forced to follow a specific schema.
import json
from together import Together
client = Together()
response = client.chat.completions.create(
model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
messages=[
{"role": "system", "content": "You extract information and reply ONLY in JSON format."},
{"role": "user", "content": "Budi is 28 years old, lives in Jakarta, works as a data scientist."}
],
responseformat={"type": "jsonobject"},
)
data = json.loads(response.choices[0].message.content)
print(data)
For tighter control, we can use a JSON schema. I often use Pydantic to define the schema so it is tidy and type-safe:
import json
from pydantic import BaseModel
from together import Together
client = Together()
class Person(BaseModel):
name: str
age: int
city: str
job: str
response = client.chat.completions.create(
model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
messages=[
{"role": "system", "content": "Extract person information from text into JSON."},
{"role": "user", "content": "Sari, 32 years old, an architect from Bandung."}
],
responseformat={
"type": "jsonobject",
"schema": Person.modeljsonschema(),
},
)
person = Person.modelvalidatejson(response.choices[0].message.content)
print(f"Name: {person.name}, Age: {person.age}, City: {person.city}")
With this approach, the model's output is guaranteed to follow the structure we want, so there are no more stories of parsing failing because of inconsistent format. This is really important for production applications.
Function Calling
Function calling or tool calling is the model's ability to "call" functions that we define. So the model not only replies with text, but can decide when it needs to call an external function, for example to check the weather, query a database, or call another API. This is the foundation for building AI agents.
import json
from together import Together
client = Together()
Definition of a tool the model can call
tools = [
{
"type": "function",
"function": {
"name": "getweather",
"description": "Get the current weather for a city",
"parameters": {
"type": "object",
"properties": {
"city": {"type": "string", "description": "City name"},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
},
"required": ["city"]
}
}
}
]
response = client.chat.completions.create(
model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
messages=[
{"role": "user", "content": "What is the weather in Surabaya today?"}
],
tools=tools,
toolchoice="auto",
)
message = response.choices[0].message
if message.toolcalls:
for call in message.toolcalls:
print(f"Model wants to call function: {call.function.name}")
args = json.loads(call.function.arguments)
print(f"With arguments: {args}")
Here the model does not answer the weather directly, but says "hey I need to call the getweather function with city Surabaya". Our job in code is to run the actual function, get the result, then send it back to the model so it can compose the final answer. Here is a complete loop example:
def getweather(city, unit="celsius"):
# This is an example, in the real world you call a real weather API
return {"city": city, "temp": 30, "unit": unit, "condition": "sunny"}
messages = [{"role": "user", "content": "What is the weather in Surabaya?"}]
response = client.chat.completions.create(
model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
messages=messages,
tools=tools,
)
msg = response.choices[0].message
if msg.toolcalls:
messages.append(msg) # save the tool call request from the model
for call in msg.toolcalls:
args = json.loads(call.function.arguments)
result = getweather(*args)
# send the function result back to the model
messages.append({
"role": "tool",
"toolcallid": call.id,
"content": json.dumps(result),
})
# call the model again so it composes the final answer
final = client.chat.completions.create(
model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
messages=messages,
)
print(final.choices[0].message.content)
This pattern is the basis for building agents that can interact with the outside world. You can add many tools at once, and the model will smartly pick which one needs to be called.
Embeddings for Semantic Search
Embeddings are numeric representations of text in the form of vectors. They are useful for measuring how similar two pieces of text are in meaning, not just word by word. This is the foundation for semantic search, RAG (Retrieval Augmented Generation), and recommendation systems. Together AI provides embedding models too.
from together import Together
client = Together()
response = client.embeddings.create(
model="togethercomputer/m2-bert-80M-8k-retrieval",
input="Machine learning is a branch of artificial intelligence.",
)
embedding = response.data[0].embedding
print(f"Vector dimension: {len(embedding)}")
print(f"First 5 values: {embedding[:5]}")
A real example using embeddings to find the most relevant document from a collection of texts:
import numpy as np
from together import Together
client = Together()
def getembedding(text):
resp = client.embeddings.create(
model="togethercomputer/m2-bert-80M-8k-retrieval",
input=text,
)
return np.array(resp.data[0].embedding)
def cosinesimilarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) np.linalg.norm(b))
documents = [
"Python is a popular programming language.",
"Cats are adorable pets.",
"Deep learning uses artificial neural networks.",
]
query = "What is a neural network?"
queryemb = getembedding(query)
Compute similarity of query with each document
scores = [(doc, cosinesimilarity(queryemb, getembedding(doc))) for doc in documents]
scores.sort(key=lambda x: x[1], reverse=True)
print("Most relevant documents:")
for doc, s in scores:
print(f" {s:.3f} - {doc}")
In this example, the document about deep learning will come out on top because its meaning is closest to the query about neural networks, even though the words are different. This is the power of semantic search.
Image Generation
Together AI can also generate images using models like FLUX. So you can create images from text descriptions directly through the API.
from together import Together
client = Together()
response = client.images.generate(
model="black-forest-labs/FLUX.1-schnell",
prompt="An orange cat wearing a chef hat cooking, cartoon style, bright colors",
width=1024,
height=1024,
steps=4,
)
URL of the generated image
imageurl = response.data[0].url
print(f"Image created successfully: {imageurl}")
The FLUX.1-schnell model is super fast and good enough for most needs. If you need higher quality, there is also another variant like FLUX.1-dev. The result is a URL that you can download or display in your application.
Fine-Tuning Your Own Model
Now this is the feature that makes Together AI different from most platforms. We can fine-tune open source models with our own data. Fine-tuning is the process of retraining a model so it becomes better at a specific task or follows a certain style. For example you want to build a model that answers in your brand's voice, or a model that is an expert in the medical domain. I will explain the flow step by step.
The flow has three main stages: upload the dataset, create a fine-tuning job, and use the resulting model.
First, we prepare the dataset. The format is usually JSONL (JSON Lines), where each line is one example conversation. We create a file trainingdata.jsonl:
import json
Example training data in conversational format
data = [
{"messages": [
{"role": "user", "content": "What does API stand for?"},
{"role": "assistant", "content": "API stands for Application Programming Interface, a way for applications to communicate with each other."}
]},
{"messages": [
{"role": "user", "content": "What is Python?"},
{"role": "assistant", "content": "Python is a high-level programming language that is easy to read and versatile."}
]},
# ... ideally at least dozens to hundreds of examples
]
with open("trainingdata.jsonl", "w", encoding="utf-8") as f:
for item in data:
f.write(json.dumps(item, ensureascii=False) + "\n")
print("Dataset ready!")
Second, we upload the dataset to Together AI and create a fine-tuning job:
from together import Together
client = Together()
Upload the dataset file
uploaded = client.files.upload(
file="trainingdata.jsonl",
purpose="fine-tune",
)
print(f"File uploaded with ID: {uploaded.id}")
Create a fine-tuning job
job = client.finetuning.create(
trainingfile=uploaded.id,
model="meta-llama/Meta-Llama-3.1-8B-Instruct-Reference",
nepochs=3,
learningrate=1e-5,
suffix="my-custom-assistant",
)
print(f"Fine-tuning job started with ID: {job.id}")
nepochs is how many times the model sees the entire dataset. learningrate controls how big the adjustment is each step. suffix is for giving a name so it is easy to recognize. After the job runs, we can check its status:
from together import Together
client = Together()
Check job status
jobid = "ft-xxxxxxxx" # replace with your job ID
status = client.finetuning.retrieve(jobid)
print(f"Status: {status.status}")
If you want to see all jobs
jobs = client.finetuning.list()
for j in jobs.data:
print(f"{j.id}: {j.status}")
The fine-tuning process can take from a few minutes to a few hours depending on the size of the dataset and model. You can monitor it through the dashboard or through the code above. Once the status is completed, we will get the name of the fine-tuned model.
Third, we use the fine-tuned model just like a regular model, just swap the model name:
from together import Together
client = Together()
The model name usually looks like: username/Meta-Llama-3.1-8B-Instruct-my-custom-assistant-xxxx
response = client.chat.completions.create(
model="username/Meta-Llama-3.1-8B-Instruct-my-custom-assistant-xxxx",
messages=[
{"role": "user", "content": "What is Python?"}
],
)
print(response.choices[0].message.content)
That is the whole flow. The fine-tuned model will now answer with the style and knowledge we trained it on. For those of you who want to build AI products with special characteristics, this feature is a genuine game changer.
Best Practices
After using Together AI for quite a while across various projects, I have a few tips that may be useful for you.
Secure Your API Key
This is the most important and people forget it very often. Never hardcode your API key inside your code, especially if you are going to push it to GitHub. Always use environment variables or a .env file that you add to .gitignore. If your API key leaks, other people can use it and you are the one paying the bill. I have seen a friend whose API bill ballooned because the key got committed to a public repo. Be careful.
Choose the Right Model
Do not just always use the biggest model. A 70B model is indeed smarter, but it is slower and more expensive. For simple tasks like classification or data extraction, a small model like Llama 8B is often more than enough and much cheaper. I usually start from a small model first, and if the result is not good enough then I move up to a bigger model. The principle is, use the smallest model possible that still gives satisfactory results.
Handle Errors and Retries
APIs can fail for various reasons, whether it is rate limits, timeouts, or the server being busy. So always wrap your API calls with error handling. I recommend implementing retries with exponential backoff:
import time
from together import Together
client = Together()
def chatwithretry(messages, model="meta-llama/Llama-3.3-70B-Instruct-Turbo", maxretry=3):
for attempt in range(maxretry):
try:
response = client.chat.completions.create(
model=model,
messages=messages,
)
return response.choices[0].message.content
except Exception as e:
wait = 2 ** attempt # 1, 2, 4 seconds
print(f"Error: {e}. Retrying in {wait} seconds...")
time.sleep(wait)
raise Exception("Failed after several attempts")
result = chatwithretry([{"role": "user", "content": "Hello!"}])
print(result)
Monitor Token Usage
Each response has usage info that tells you how many tokens were used. Get into the habit of monitoring this so you do not get surprised when you see the bill. Tokens are the unit of cost in language APIs. If your application sends long prompts repeatedly, the cost can rise quickly. Consider trimming unnecessary context or using caching where possible.
response = client.chat.completions.create(
model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
messages=[{"role": "user", "content": "Hello!"}],
)
print(f"Prompt tokens: {response.usage.prompttokens}")
print(f"Completion tokens: {response.usage.completiontokens}")
print(f"Total tokens: {response.usage.totaltokens}")
Effective Fine-Tuning
If you want to fine-tune, data quality is far more important than quantity. It is better to have 100 clean and consistent examples than 1000 messy ones. Make sure your data format is consistent, and the examples genuinely represent the task you want. Start with a small nepochs (2 to 3) so the model does not overfit, meaning memorizing the training data too much until it cannot generalize to new data.
Leverage Streaming for UX
If you build an application that interacts directly with users, always use streaming. The difference between a user waiting 5 seconds staring at a blank screen versus seeing text appear gradually is huge for the user experience. Technically the speed is the same, but perceptually streaming feels much faster and more responsive.
Conclusion
Okay everyone, that is the complete guide to Together AI from me. We have covered everything from how to get an API key, install the library, all the way to various features starting from chat completions, streaming, JSON mode, function calling, embeddings, image generation, and the most exciting part which is fine-tuning your own model.
In my opinion Together AI is a very solid choice for anyone who wants to work with open source models without the hassle of managing infrastructure. Its OpenAI-compatible API makes migration super easy, the model collection is broad, and the pricing is friendly. For me personally, the combination of cheap inference and easy fine-tuning capability is the main selling point that keeps me coming back to this platform.
My advice, try starting from something simple first. Make a simple chat completion, play with its parameters, then slowly move up to more advanced features as your project needs. Do not forget to always keep your API key secure and monitor your token usage so your wallet stays safe.
I hope this tutorial is useful. If you have questions or want to share a project you built with Together AI, do not hesitate to contact me. Happy coding and see you in the next tutorial. Keep learning AI, everyone!