TorchServe: Deploying PyTorch Models to Production
Author: Ruby Abdullah Category: Tutorial Tags: TorchServe, PyTorch, Model Serving, MLOps, Deep LearningIntroduction
Deploying a machine learning model to production is often the most underestimated phase of an ML project. Training a model that achieves strong accuracy metrics is one challenge; serving that model reliably at scale under real-world traffic is another problem entirely. TorchServe is PyTorch's official model serving framework, designed to close the gap between research and production deployment.
TorchServe is a flexible, high-performance serving framework that handles the operational concerns of running PyTorch models in production: HTTP endpoints, model management, batching, logging, metrics, and multi-model deployments. It was developed collaboratively by AWS and Facebook AI, and it is now maintained as part of the PyTorch ecosystem.
This tutorial walks through everything an ML engineer needs to know to go from a trained PyTorch model to a fully operational serving endpoint. You will learn how to package models, configure TorchServe, write custom handlers, enable batch inference, monitor your service, and deploy on GPU infrastructure.
Prerequisites
Before starting, ensure you have:
- Python 3.8 or higher
- Java 11 or higher (required by TorchServe)
- PyTorch 2.0 or higher
- A trained model (this tutorial uses ResNet-50 from torchvision)
- Basic familiarity with the PyTorch model lifecycle
Why TorchServe
Before TorchServe, ML engineers serving PyTorch models typically built custom Flask or FastAPI services. These ad-hoc approaches work for simple cases but quickly become difficult to maintain. They lack standardized model versioning, dynamic loading and unloading, built-in batching, and production-grade metrics.
TorchServe provides:
- A RESTful management API to register, update, and remove models at runtime without restarting the server
- An inference API that clients use to send prediction requests
- A metrics API compatible with Prometheus and other monitoring systems
- A model archive format (
.mar) that bundles model weights, code, and dependencies into a single deployable artifact - Support for multi-GPU inference and automatic batching
Installation and Setup
Installing Java
TorchServe requires Java 11. On Ubuntu:
sudo apt-get update
sudo apt-get install -y openjdk-11-jdk
java -version
On macOS with Homebrew:
brew install openjdk@11
export PATH="/opt/homebrew/opt/openjdk@11/bin:$PATH"
Installing TorchServe
Install TorchServe and the model archiver tool using pip:
pip install torchserve torch-model-archiver torch-workflow-archiver
For GPU support, ensure you have the CUDA-enabled version of PyTorch installed first:
# For CUDA 12.1
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install torchserve torch-model-archiver torch-workflow-archiver
Verify the installation:
torchserve --version
torch-model-archiver --version
Directory Structure
TorchServe expects a specific working directory layout. Create the following structure:
mkdir -p torchserve-demo/{model-store,logs,config}
cd torchserve-demo
Your working directory should look like this:
torchserve-demo/
model-store/ # .mar files go here
logs/ # TorchServe log output
config/ # config.properties file
Packaging a Model with torch-model-archiver
The torch-model-archiver command-line tool creates a .mar (Model ARchive) file. A .mar file is a self-contained artifact that includes:
- The serialized model weights (
.ptor.pth) - The model definition (optional, for eager mode models)
- A handler script that defines how to process requests
- Extra files such as label maps or configuration files
Exporting a ResNet-50 Model
First, download and export a pretrained ResNet-50:
# exportresnet.py
import torch
import torchvision.models as models
model = models.resnet50(pretrained=True)
model.eval()
TorchScript export (recommended for production)
example
input = torch.rand(1, 3, 224, 224)
tracedmodel = torch.jit.trace(model, exampleinput)
tracedmodel.save("resnet50traced.pt")
print("Model exported to resnet50traced.pt")
python exportresnet.py
Download the ImageNet class index file that maps output indices to human-readable labels:
wget https://s3.amazonaws.com/deep-learning-models/image-models/imagenetclassindex.json
Creating the Model Archive
torch-model-archiver \
--model-name resnet50 \
--version 1.0 \
--serialized-file resnet50traced.pt \
--handler imageclassifier \
--extra-files imagenetclassindex.json \
--export-path model-store \
--force
Key flags explained:
--model-name: The name used to identify the model in API requests--version: Model version string; allows multiple versions of the same model to coexist--serialized-file: Path to the model weights file--handler: Either a built-in handler name or a path to a custom handler script--extra-files: Comma-separated list of additional files bundled into the archive--export-path: Directory where the.marfile is written
TorchServe ships with four built-in handlers: imageclassifier, imagesegmenter, objectdetector, and textclassifier. For anything else, you need a custom handler.
Starting TorchServe
Configuration File
Create config/config.properties:
inferenceaddress=http://0.0.0.0:8080
management
address=http://0.0.0.0:8081
metricsaddress=http://0.0.0.0:8082
modelstore=model-store
loadmodels=resnet50.mar
numberofnettythreads=32
jobqueuesize=1000
defaultworkerspermodel=1
defaultresponsetimeout=120
unregistermodeltimeout=180
Starting the Server
torchserve \
--start \
--model-store model-store \
--models resnet50=resnet50.mar \
--ts-config config/config.properties \
--log-config config/log4j2.xml \
--foreground
Alternatively, without a config file:
torchserve \
--start \
--model-store model-store \
--models resnet50=resnet50.mar \
--ncs
The --ncs flag disables snapshot saving. Remove it in production to enable crash recovery.
Stopping the Server
torchserve --stop
Health Check
curl http://localhost:8080/ping
Expected response:
{"status": "Healthy"}
Inference API
Sending an Inference Request
Download a test image and send it to the prediction endpoint:
curl -X POST \
http://localhost:8080/predictions/resnet50 \
-T kitten.jpg
Or using Python:
# inferenceclient.py
import requests
url = "http://localhost:8080/predictions/resnet50"
with open("kitten.jpg", "rb") as f:
response = requests.post(url, data=f, headers={"Content-Type": "application/octet-stream"})
print(response.json())
The response includes the top-5 predicted class labels with their probabilities:
[
{"tabby": 0.4079},
{"tigercat": 0.3461},
{"Egyptiancat": 0.1810},
{"lynx": 0.0229},
{"tiger": 0.0045}
]
Versioned Inference
If you have registered multiple versions of a model, you can target a specific version:
curl http://localhost:8080/predictions/resnet50/1.0 -T kitten.jpg
Management API
The management API runs on port 8081 by default and allows you to register, update, scale, and unregister models without restarting TorchServe.
Registering a Model
curl -X POST \
"http://localhost:8081/models?url=resnet50.mar&modelname=resnet50&initialworkers=2&synchronous=true"
Listing Registered Models
curl http://localhost:8081/models
{
"models": [
{
"modelName": "resnet50",
"modelUrl": "resnet50.mar"
}
]
}
Describing a Model
curl http://localhost:8081/models/resnet50
This returns detailed information including the number of workers, batch size, and version.
Scaling Workers
Dynamically increase or decrease the number of worker processes for a model:
# Scale up to 4 workers
curl -X PUT \
"http://localhost:8081/models/resnet50?minworker=2&maxworker=4"
TorchServe manages a pool of workers per model. Each worker is an independent Python process that loads the model into memory. Scaling workers up increases throughput at the cost of memory.
Unregistering a Model
curl -X DELETE http://localhost:8081/models/resnet50/1.0
Setting the Default Version
curl -X PUT http://localhost:8081/models/resnet50/1.0/set-default
Custom Handlers
Built-in handlers cover standard vision and text classification tasks. For custom architectures, preprocessing pipelines, or multi-output models, you need to implement a custom handler.
Handler Interface
A custom handler is a Python class that inherits from BaseHandler and implements four methods:
# customhandler.py
import json
import logging
import os
import torch
from torchvision import transforms
from PIL import Image
import io
from ts.torch
handler.basehandler import BaseHandler
logger = logging.getLogger(name)
class CustomResNetHandler(BaseHandler):
def initialize(self, context):
"""
Load the model and any supporting resources.
Called once when the worker starts.
"""
self.manifest = context.manifest
properties = context.system
properties
modeldir = properties.get("modeldir")
self.device = torch.device(
"cuda:" + str(properties.get("gpuid")) if torch.cuda.isavailable() else "cpu"
)
# Load the serialized model
serializedfile = self.manifest["model"]["serializedFile"]
modelptpath = os.path.join(modeldir, serializedfile)
self.model = torch.jit.load(modelptpath, maplocation=self.device)
self.model.eval()
# Load class index
mappingfilepath = os.path.join(modeldir, "imagenetclassindex.json")
with open(mappingfilepath) as f:
self.mapping = json.load(f)
# Define preprocessing pipeline
self.transform = transforms.Compose([
transforms.Resize(256),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize(
mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225]
),
])
self.initialized = True
logger.info("CustomResNetHandler initialized on device: %s", self.device)
def preprocess(self, data):
"""
Transform raw input bytes into model-ready tensors.
data is a list of dicts, one per request in the batch.
"""
images = []
for row in data:
# Input can be raw bytes or base64-encoded JSON
imagebytes = row.get("data") or row.get("body")
if isinstance(imagebytes, (bytes, bytearray)):
image = Image.open(io.BytesIO(imagebytes)).convert("RGB")
else:
raise ValueError("Unsupported input format")
tensor = self.transform(image)
images.append(tensor)
# Stack into a single batch tensor
return torch.stack(images).to(self.device)
def inference(self, inputs):
"""
Run the forward pass. Receives the output of preprocess.
"""
with torch.nograd():
outputs = self.model(inputs)
return outputs
def postprocess(self, outputs):
"""
Convert model outputs to JSON-serializable predictions.
Returns a list with one prediction per input in the batch.
"""
predictions = []
probabilities = torch.nn.functional.softmax(outputs, dim=1)
top5probs, top5indices = torch.topk(probabilities, 5)
for probs, indices in zip(top5probs, top5indices):
result = {}
for prob, idx in zip(probs.tolist(), indices.tolist()):
label = self.mapping[str(idx)][1]
result[label] = round(prob, 4)
predictions.append(result)
return predictions
Packaging with the Custom Handler
torch-model-archiver \
--model-name resnet50custom \
--version 2.0 \
--serialized-file resnet50traced.pt \
--handler customhandler.py \
--extra-files imagenetclassindex.json \
--export-path model-store \
--force
Batch Inference
Batching is one of the most important optimizations for GPU throughput. TorchServe supports both request-level batching (waiting to accumulate a batch before running inference) and dynamic batching.
Configuring Batch Inference
Batch settings can be specified at registration time:
curl -X POST \
"http://localhost:8081/models?url=resnet50custom.mar&modelname=resnet50custom&batchsize=8&maxbatchdelay=100&initialworkers=2&synchronous=true"
batchsize: Maximum number of requests to process in a single forward passmaxbatchdelay: Maximum time in milliseconds to wait for a batch to fill before processing with whatever has arrived
Or in config.properties:
# Global defaults
batchsize=8
maxbatchdelay=100
Handler Considerations for Batching
The preprocess method already receives a list of requests. The torch.stack call in the custom handler above correctly assembles them into a batch tensor. Ensure your handler always returns a list from postprocess with one element per input, as TorchServe uses this to route each response back to the correct client.
Testing Batch Throughput
# batchtest.py
import requests
import threading
import time
url = "http://localhost:8080/predictions/resnet50custom"
imagepath = "kitten.jpg"
numrequests = 50
def sendrequest():
with open(imagepath, "rb") as f:
requests.post(url, data=f)
start = time.time()
threads = [threading.Thread(target=sendrequest) for in range(numrequests)]
for t in threads:
t.start()
for t in threads:
t.join()
elapsed = time.time() - start
print(f"Processed {numrequests} requests in {elapsed:.2f}s")
print(f"Throughput: {numrequests / elapsed:.1f} req/s")
Logging and Metrics
Log Configuration
TorchServe uses Log4j2 for logging. The default log directory is logs/. Key log files:
tslog.log: Main TorchServe server logmodellog.log: Output from handler code (uselogger.info()in your handler)accesslog.log: Inference and management API access logs
To add logging in your handler:
import logging
logger = logging.getLogger(name)
def inference(self, inputs):
logger.info("Running inference on batch of size %d", inputs.shape[0])
with torch.nograd():
outputs = self.model(inputs)
return outputs
Metrics API
The metrics endpoint exposes Prometheus-compatible metrics:
curl http://localhost:8082/metrics
Key built-in metrics:
tsinferencerequeststotal: Total number of inference requeststsinferencelatencymicroseconds: Inference latency histogramtsqueuelatencymicroseconds: Time requests spend waiting in the job queueWorkerThreadTime: CPU time spent in worker threadsCPUUtilization,MemoryUsed,MemoryAvailable,GPUUtilization,GPUMemoryUsed
Custom Metrics
You can emit custom metrics from your handler using the metrics context:
from ts.metrics.metricsstore import MetricsStore
def inference(self, inputs):
# Emit a counter metric
metrics = self.context.metrics
metrics.addcounter("CustomInferenceCount", 1, idx=0)
with torch.nograd():
outputs = self.model(inputs)
return outputs
Prometheus and Grafana Integration
Add a scrape target to your Prometheus configuration:
# prometheus.yml
scrapeconfigs:
- jobname: "torchserve"
staticconfigs:
- targets: ["localhost:8082"]
GPU Support
Single GPU Deployment
TorchServe automatically assigns workers to available GPUs. When you start TorchServe with CUDA-enabled PyTorch, it detects GPUs and passes the gpuid property to each worker's initialize method.
In your handler:
def initialize(self, context):
properties = context.systemproperties
gpuid = properties.get("gpuid")
self.device = torch.device(f"cuda:{gpuid}" if torch.cuda.isavailable() and gpuid is not None else "cpu")
self.model = torch.jit.load(modelptpath, maplocation=self.device)
Multi-GPU Configuration
To distribute workers across multiple GPUs, set numberofgpu in config.properties:
numberofgpu=4
default
workerspermodel=4
TorchServe assigns one GPU per worker, round-robin across available devices.
Mixed Precision Inference
For faster GPU inference with FP16:
def initialize(self, context):
# ... load model ...
if self.device.type == "cuda":
self.model = self.model.half() # Convert to FP16
self.usefp16 = True
else:
self.usefp16 = False
def preprocess(self, data):
# ... build tensor ...
if self.usefp16:
return batchtensor.half().to(self.device)
return batchtensor.to(self.device)
Best Practices for Production Deployment
Model Versioning Strategy
Always include the version flag when creating model archives. Use semantic versioning and keep previous versions registered to enable instant rollback:
# Register new version
curl -X POST "http://localhost:8081/models?url=resnet50.mar&modelname=resnet50&modelversion=2.0&initialworkers=2"
Test new version
curl http://localhost:8080/predictions/resnet50/2.0 -T testimage.jpg
Promote to default if testing passes
curl -X PUT http://localhost:8081/models/resnet50/2.0/set-default
Remove old version after validation period
curl -X DELETE http://localhost:8081/models/resnet50/1.0
Worker Sizing
The right number of workers depends on your hardware and workload:
- CPU-only: Set workers equal to the number of CPU cores, then tune based on observed queue latency
- GPU: One worker per GPU is typically sufficient; multiple workers on the same GPU compete for memory and may reduce throughput
- Mixed workloads: Profile under realistic traffic before committing to a worker count in production
Monitor tsqueuelatencymicroseconds. If it is consistently high, add more workers or increase batch size.
Health Checks and Load Balancers
TorchServe's /ping endpoint is suitable for liveness probes. For readiness probes, verify that at least one model is registered and healthy:
curl http://localhost:8080/ping # Liveness
curl http://localhost:8081/models # Readiness (check response is not empty)
A simple readiness check script:
# readinesscheck.py
import requests
import sys
try:
response = requests.get("http://localhost:8081/models", timeout=5)
models = response.json().get("models", [])
if len(models) > 0:
print("Ready")
sys.exit(0)
else:
print("No models loaded")
sys.exit(1)
except Exception as e:
print(f"Health check failed: {e}")
sys.exit(1)
Containerization with Docker
TorchServe's official Docker images are on Docker Hub:
# Dockerfile
FROM pytorch/torchserve:latest-gpu
COPY model-store/ /home/model-server/model-store/
COPY config/config.properties /home/model-server/config.properties
USER model-server
CMD ["torchserve", \
"--start", \
"--model-store", "/home/model-server/model-store", \
"--models", "resnet50=resnet50.mar", \
"--ts-config", "/home/model-server/config.properties", \
"--foreground"]
docker build -t my-torchserve .
docker run --gpus all -p 8080:8080 -p 8081:8081 -p 8082:8082 my-torchserve
Securing the Management API
The management API should never be publicly exposed. In production, bind it to localhost or an internal network interface and use a reverse proxy for the inference API:
# config.properties
inferenceaddress=http://0.0.0.0:8080
managementaddress=http://127.0.0.1:8081
metricsaddress=http://127.0.0.1:8082
Place an NGINX reverse proxy in front of the inference API to handle TLS termination and rate limiting.
Snapshot and Recovery
TorchServe maintains a snapshot of the registered models on disk. When TorchServe restarts, it restores the last snapshot automatically. Ensure the snapshot directory (logs/) is on persistent storage in containerized environments.
Enable snapshot saving by omitting the --ncs flag and ensuring the modelstore directory is writable.
Tuning the Job Queue
jobqueuesize=1000
The job queue size limits how many pending requests TorchServe will hold in memory before rejecting new ones with HTTP 503. Set this based on your acceptable latency budget and expected traffic spikes. A queue that is too large causes requests to pile up and experience high latency; a queue that is too small causes unnecessary 503 errors under burst traffic.
Conclusion
TorchServe addresses a real and common gap in ML workflows: the lack of a standardized, production-ready serving layer for PyTorch models. By adopting TorchServe, ML engineering teams gain a consistent model deployment pattern, built-in operational tooling, and a path to scale from a single-GPU development machine to a multi-node production cluster.
The key takeaways from this tutorial:
- Use
torch-model-archiverto package models into self-contained.mararchives that include weights, code, and dependencies - TorchServe exposes three APIs: inference (port 8080), management (port 8081), and metrics (port 8082)
- The management API enables zero-downtime model updates and rollbacks by supporting multiple model versions simultaneously
- Custom handlers give you full control over preprocessing, inference, and postprocessing pipelines
- Batch inference is essential for GPU efficiency; configure
batchsizeandmaxbatchdelaybased on your latency and throughput requirements - The metrics API integrates with Prometheus and Grafana for production observability
- Always bind the management API to an internal interface and place a TLS-terminating reverse proxy in front of the inference API
TorchServe is actively developed and the ecosystem around it continues to mature. Features such as workflow chaining (for multi-model pipelines), gRPC support, and integration with Kubernetes autoscalers make it a viable foundation for enterprise-grade ML serving at significant scale.