LitServe Tutorial: Fast and Easy AI Model Serving Framework

Introduction

LitServe is an open-source framework from Lightning AI designed for serving AI/ML models with high performance and ease of use. Built on top of FastAPI, LitServe provides enterprise-grade features such as batching, streaming, GPU autoscaling, and multi-model endpoints without complex configuration.

Unlike other serving frameworks that require extensive boilerplate code and infrastructure setup, LitServe lets you turn any AI model into a production-ready API with just a few lines of code. The framework supports all model types including LLMs, computer vision, NLP, audio, and classic scikit-learn models.

In this tutorial, we will cover everything from installation, creating a simple API, to advanced features like batching, streaming, and authentication.

Why LitServe?

Before diving into implementation, here is why LitServe stands out:

• High Performance: Up to 2x faster than plain FastAPI thanks to internal optimizations
• Minimal Boilerplate: Just define setup(), decoderequest(), predict(), and encoderesponse()
• Automatic Batching: Combines multiple requests for higher GPU throughput
• Streaming Response: Native support for streaming output (ideal for LLMs)
• Multi-Model: Single server can serve multiple models simultaneously
• GPU Management: Automatic GPU allocation and multi-worker handling
• OpenAPI Docs: Auto-generated API documentation (Swagger UI)

Installation

Prerequisites

Make sure you have Python 3.8 or later. Using a virtual environment is recommended.

python -m venv litserve-env
source litserve-env/bin/activate  # Linux/Mac
or
litserve-env\Scripts\activate     # Windows

Install LitServe

pip install litserve

For additional features, you can install optional dependencies:

# For all features
pip install litserve[all]

For PyTorch models
pip install torch torchvision

For Hugging Face models
pip install transformers

For scikit-learn models
pip install scikit-learn

Verify installation:

import litserve as ls
print(ls.version)

Basic Usage: Creating Your First API

Core Concept: LitAPI

The core of LitServe is the LitAPI class. You need to implement at minimum 4 methods:

setup(device) - Initialize model and resources (called once at server startup)

decoderequest(request) - Transform HTTP request into model input

predict(x) - Run model inference

encoderesponse(output) - Transform model output into HTTP response

Example 1: Simple API with Scikit-Learn

Let's build an API for an iris classification model using scikit-learn.

# server.py
import litserve as ls
from sklearn.datasets import loadiris

from sklearn.ensemble import RandomForestClassifier
import numpy as np

class IrisAPI(ls.LitAPI):
    def setup(self, device):
        """Load and train model at server startup."""
        iris = loadiris()
        self.model = RandomForestClassifier(nestimators=100, randomstate=42)
        self.model.fit(iris.data, iris.target)
        self.classnames = iris.targetnames.tolist()

    def decoderequest(self, request):

        """Parse input from HTTP request."""
        return np.array(request["features"]).reshape(1, -1)

    def predict(self, x):
        """Run prediction."""
        prediction = self.model.predict(x)
        probability = self.model.predictproba(x)
        return {
            "classidx": int(prediction[0]),

            "probabilities": probability[0].tolist()
        }

    def encoderesponse(self, output):
        """Format response for client."""
        return {
            "predictedclass": self.classnames[output["classidx"]],

            "probabilities": {
                name: round(prob, 4)