Complete Replicate Tutorial: Run and Deploy ML Models via API

Replicate is a cloud platform that allows you to run machine learning models through an API without managing GPU infrastructure yourself. With Replicate, you can access thousands of open-source models ranging from image generation, LLMs, speech-to-text, to video generation with just a few lines of code.

The platform is extremely popular among developers for its simplicity: no server setup required, no CUDA configuration needed, and models are ready to use through REST API or Python SDK. Replicate also provides tools to deploy your custom models using an open-source tool called Cog.

In this tutorial, we will learn how to use Replicate from initial setup, running predictions, streaming responses, using webhooks, fine-tuning models, to packaging custom models with Cog.

Installation and Setup

Creating an Account and API Token

The first step is to create an account on Replicate. After signing up, you can obtain your API token from the Account Settings page.

# Set API token as environment variable
export REPLICATEAPITOKEN="r8yourapitokenhere"

Installing the Python SDK

# Install using pip
pip install replicate

Or using uv (recommended)
uv pip install replicate

Verifying Installation

import replicate
import os

Make sure token is set
assert os.environ.get("REPLICATEAPITOKEN"), "Token not set!"

Test connection by running a simple model
output = replicate.run(
    "meta/meta-llama-3.1-8b-instruct",
    input={"prompt": "Hello, world!"}
)
print("".join(output))

Running Predictions (Basic Usage)

Text Generation with LLMs

The simplest way to run a model on Replicate is using the replicate.run() function.

import replicate

Run Llama 3.1 for text generation
output = replicate.run(
    "meta/meta-llama-3.1-70b-instruct",
    input={
        "prompt": "Explain the concept of machine learning in 3 paragraphs",
        "maxtokens": 512,

        "temperature": 0.7,
        "topp": 0.9,
        "systemprompt": "You are a helpful AI assistant that explains technology concepts clearly."

    }
)

Output is an iterator, join into a string
result = "".join(output)
print(result)

import replicate

Generate images using SDXL
output = replicate.run(
    "stability-ai/sdxl:39ed52f2a78e934b3ba6e2a89f5b1c712de7dfea535525255b1aa35c5565e08b",
    input={
        "prompt": "A futuristic city skyline at sunset, cyberpunk style, highly detailed",
        "negativeprompt": "blurry, low quality, distorted",
        "width": 1024,
        "height": 1024,
        "numoutputs": 1,

        "scheduler": "KEULER",
        "numinferencesteps": 30,
        "guidancescale": 7.5

    }
)

Output is a list of image URLs
for i, url in enumerate(output):
    print(f"Image {i+1}: {url}")

import replicate

Use LLaVA for image understanding
output = replicate.run(
    "yorickvp/llava-v1.6-34b:41ecfbfb261e6c1adf3ad896c9066ca98346996d7c4045c5bc944a79d430f174",
    input={
        "image": "https://example.com/photo.jpg",
        "prompt": "Describe this image in detail"
    }
)

result = "".join(output)
print(result)

import replicate

Transcribe audio using Whisper
output = replicate.run(
    "openai/whisper:4d50797a6f35677e3f7e36c5b0d0c3c50e8a0b0e4e0e0e0e0e0e0e0e0e0e0e",
    input={
        "audio": open("recording.mp3", "rb"),
        "model": "large-v3",
        "language": "en",
        "translate": False
    }
)

print(output["transcription"])