TorchServe: Deploying PyTorch Models to Production

# TorchServe: Panduan Lengkap Deployment Model PyTorch ke Production **Penulis**: Ruby Abdullah **Kategori**: Tutorial **Tags**: PyTorch, TorchServe, MLOps, Model Serving, Production ML ## Pendahulu...

By Ruby Abdullah · · tutorial
TorchServePyTorchModel ServingMLOpsDeep Learning

TorchServe: Deploying PyTorch Models to Production

Author: Ruby Abdullah Category: Tutorial Tags: TorchServe, PyTorch, Model Serving, MLOps, Deep Learning

Introduction

Deploying a machine learning model to production is often the most underestimated phase of an ML project. Training a model that achieves strong accuracy metrics is one challenge; serving that model reliably at scale under real-world traffic is another problem entirely. TorchServe is PyTorch's official model serving framework, designed to close the gap between research and production deployment.

TorchServe is a flexible, high-performance serving framework that handles the operational concerns of running PyTorch models in production: HTTP endpoints, model management, batching, logging, metrics, and multi-model deployments. It was developed collaboratively by AWS and Facebook AI, and it is now maintained as part of the PyTorch ecosystem.

This tutorial walks through everything an ML engineer needs to know to go from a trained PyTorch model to a fully operational serving endpoint. You will learn how to package models, configure TorchServe, write custom handlers, enable batch inference, monitor your service, and deploy on GPU infrastructure.

Prerequisites

Before starting, ensure you have:

  • Python 3.8 or higher
  • Java 11 or higher (required by TorchServe)
  • PyTorch 2.0 or higher
  • A trained model (this tutorial uses ResNet-50 from torchvision)
  • Basic familiarity with the PyTorch model lifecycle

Why TorchServe

Before TorchServe, ML engineers serving PyTorch models typically built custom Flask or FastAPI services. These ad-hoc approaches work for simple cases but quickly become difficult to maintain. They lack standardized model versioning, dynamic loading and unloading, built-in batching, and production-grade metrics.

TorchServe provides:

  • A RESTful management API to register, update, and remove models at runtime without restarting the server
  • An inference API that clients use to send prediction requests
  • A metrics API compatible with Prometheus and other monitoring systems
  • A model archive format (.mar) that bundles model weights, code, and dependencies into a single deployable artifact
  • Support for multi-GPU inference and automatic batching

Installation and Setup

Installing Java

TorchServe requires Java 11. On Ubuntu:

sudo apt-get update

sudo apt-get install -y openjdk-11-jdk

java -version

On macOS with Homebrew:

brew install openjdk@11

export PATH="/opt/homebrew/opt/openjdk@11/bin:$PATH"

Installing TorchServe

Install TorchServe and the model archiver tool using pip:

pip install torchserve torch-model-archiver torch-workflow-archiver

For GPU support, ensure you have the CUDA-enabled version of PyTorch installed first:

# For CUDA 12.1

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

pip install torchserve torch-model-archiver torch-workflow-archiver

Verify the installation:

torchserve --version

torch-model-archiver --version

Directory Structure

TorchServe expects a specific working directory layout. Create the following structure:

mkdir -p torchserve-demo/{model-store,logs,config}

cd torchserve-demo

Your working directory should look like this:

torchserve-demo/

model-store/ # .mar files go here

logs/ # TorchServe log output

config/ # config.properties file

Packaging a Model with torch-model-archiver

The torch-model-archiver command-line tool creates a .mar (Model ARchive) file. A .mar file is a self-contained artifact that includes:

  • The serialized model weights (.pt or .pth)
  • The model definition (optional, for eager mode models)
  • A handler script that defines how to process requests
  • Extra files such as label maps or configuration files

Exporting a ResNet-50 Model

First, download and export a pretrained ResNet-50:

# exportresnet.py

import torch

import torchvision.models as models

model = models.resnet50(pretrained=True)

model.eval()

TorchScript export (recommended for production)

exampleinput = torch.rand(1, 3, 224, 224)

tracedmodel = torch.jit.trace(model, exampleinput)

tracedmodel.save("resnet50traced.pt")

print("Model exported to resnet50traced.pt")

python exportresnet.py

Download the ImageNet class index file that maps output indices to human-readable labels:

wget https://s3.amazonaws.com/deep-learning-models/image-models/imagenetclassindex.json

Creating the Model Archive

torch-model-archiver \

--model-name resnet50 \

--version 1.0 \

--serialized-file resnet50traced.pt \

--handler imageclassifier \

--extra-files imagenetclassindex.json \

--export-path model-store \

--force

Key flags explained:

  • --model-name: The name used to identify the model in API requests
  • --version: Model version string; allows multiple versions of the same model to coexist
  • --serialized-file: Path to the model weights file
  • --handler: Either a built-in handler name or a path to a custom handler script
  • --extra-files: Comma-separated list of additional files bundled into the archive
  • --export-path: Directory where the .mar file is written

TorchServe ships with four built-in handlers: imageclassifier, imagesegmenter, objectdetector, and textclassifier. For anything else, you need a custom handler.

Starting TorchServe

Configuration File

Create config/config.properties:

inferenceaddress=http://0.0.0.0:8080

managementaddress=http://0.0.0.0:8081

metricsaddress=http://0.0.0.0:8082

modelstore=model-store

loadmodels=resnet50.mar

numberofnettythreads=32

jobqueuesize=1000

defaultworkerspermodel=1

defaultresponsetimeout=120

unregistermodeltimeout=180

Starting the Server

torchserve \

--start \

--model-store model-store \

--models resnet50=resnet50.mar \

--ts-config config/config.properties \

--log-config config/log4j2.xml \

--foreground

Alternatively, without a config file:

torchserve \

--start \

--model-store model-store \

--models resnet50=resnet50.mar \

--ncs

The --ncs flag disables snapshot saving. Remove it in production to enable crash recovery.

Stopping the Server

torchserve --stop

Health Check

curl http://localhost:8080/ping

Expected response:

{"status": "Healthy"}

Inference API

Sending an Inference Request

Download a test image and send it to the prediction endpoint:

curl -X POST \

http://localhost:8080/predictions/resnet50 \

-T kitten.jpg

Or using Python:

# inferenceclient.py

import requests

url = "http://localhost:8080/predictions/resnet50"

with open("kitten.jpg", "rb") as f:

response = requests.post(url, data=f, headers={"Content-Type": "application/octet-stream"})

print(response.json())

The response includes the top-5 predicted class labels with their probabilities:

[

{"tabby": 0.4079},

{"tigercat": 0.3461},

{"Egyptiancat": 0.1810},

{"lynx": 0.0229},

{"tiger": 0.0045}

]

Versioned Inference

If you have registered multiple versions of a model, you can target a specific version:

curl http://localhost:8080/predictions/resnet50/1.0 -T kitten.jpg

Management API

The management API runs on port 8081 by default and allows you to register, update, scale, and unregister models without restarting TorchServe.

Registering a Model

curl -X POST \

"http://localhost:8081/models?url=resnet50.mar&modelname=resnet50&initialworkers=2&synchronous=true"

Listing Registered Models

curl http://localhost:8081/models

{

"models": [

{

"modelName": "resnet50",

"modelUrl": "resnet50.mar"

}

]

}

Describing a Model

curl http://localhost:8081/models/resnet50

This returns detailed information including the number of workers, batch size, and version.

Scaling Workers

Dynamically increase or decrease the number of worker processes for a model:

# Scale up to 4 workers

curl -X PUT \

"http://localhost:8081/models/resnet50?minworker=2&maxworker=4"

TorchServe manages a pool of workers per model. Each worker is an independent Python process that loads the model into memory. Scaling workers up increases throughput at the cost of memory.

Unregistering a Model

curl -X DELETE http://localhost:8081/models/resnet50/1.0

Setting the Default Version

curl -X PUT http://localhost:8081/models/resnet50/1.0/set-default

Custom Handlers

Built-in handlers cover standard vision and text classification tasks. For custom architectures, preprocessing pipelines, or multi-output models, you need to implement a custom handler.

Handler Interface

A custom handler is a Python class that inherits from BaseHandler and implements four methods:

# customhandler.py

import json

import logging

import os

import torch

from torchvision import transforms

from PIL import Image

import io

from ts.torchhandler.basehandler import BaseHandler

logger = logging.getLogger(name)

class CustomResNetHandler(BaseHandler):

def initialize(self, context):

"""

Load the model and any supporting resources.

Called once when the worker starts.

"""

self.manifest = context.manifest

properties = context.systemproperties

modeldir = properties.get("modeldir")

self.device = torch.device(

"cuda:" + str(properties.get("gpuid")) if torch.cuda.isavailable() else "cpu"

)

# Load the serialized model

serializedfile = self.manifest["model"]["serializedFile"]

modelptpath = os.path.join(modeldir, serializedfile)

self.model = torch.jit.load(modelptpath, maplocation=self.device)

self.model.eval()

# Load class index

mappingfilepath = os.path.join(modeldir, "imagenetclassindex.json")

with open(mappingfilepath) as f:

self.mapping = json.load(f)

# Define preprocessing pipeline

self.transform = transforms.Compose([

transforms.Resize(256),

transforms.CenterCrop(224),

transforms.ToTensor(),

transforms.Normalize(

mean=[0.485, 0.456, 0.406],

std=[0.229, 0.224, 0.225]

),

])

self.initialized = True

logger.info("CustomResNetHandler initialized on device: %s", self.device)

def preprocess(self, data):

"""

Transform raw input bytes into model-ready tensors.

data is a list of dicts, one per request in the batch.

"""

images = []

for row in data:

# Input can be raw bytes or base64-encoded JSON

imagebytes = row.get("data") or row.get("body")

if isinstance(imagebytes, (bytes, bytearray)):

image = Image.open(io.BytesIO(imagebytes)).convert("RGB")

else:

raise ValueError("Unsupported input format")

tensor = self.transform(image)

images.append(tensor)

# Stack into a single batch tensor

return torch.stack(images).to(self.device)

def inference(self, inputs):

"""

Run the forward pass. Receives the output of preprocess.

"""

with torch.nograd():

outputs = self.model(inputs)

return outputs

def postprocess(self, outputs):

"""

Convert model outputs to JSON-serializable predictions.

Returns a list with one prediction per input in the batch.

"""

predictions = []

probabilities = torch.nn.functional.softmax(outputs, dim=1)

top5probs, top5indices = torch.topk(probabilities, 5)

for probs, indices in zip(top5probs, top5indices):

result = {}

for prob, idx in zip(probs.tolist(), indices.tolist()):

label = self.mapping[str(idx)][1]

result[label] = round(prob, 4)

predictions.append(result)

return predictions

Packaging with the Custom Handler

torch-model-archiver \

--model-name resnet50custom \

--version 2.0 \

--serialized-file resnet50traced.pt \

--handler customhandler.py \

--extra-files imagenetclassindex.json \

--export-path model-store \

--force

Batch Inference

Batching is one of the most important optimizations for GPU throughput. TorchServe supports both request-level batching (waiting to accumulate a batch before running inference) and dynamic batching.

Configuring Batch Inference

Batch settings can be specified at registration time:

curl -X POST \

"http://localhost:8081/models?url=resnet50custom.mar&modelname=resnet50custom&batchsize=8&maxbatchdelay=100&initialworkers=2&synchronous=true"

  • batchsize: Maximum number of requests to process in a single forward pass
  • maxbatchdelay: Maximum time in milliseconds to wait for a batch to fill before processing with whatever has arrived

Or in config.properties:

# Global defaults

batchsize=8

maxbatchdelay=100

Handler Considerations for Batching

The preprocess method already receives a list of requests. The torch.stack call in the custom handler above correctly assembles them into a batch tensor. Ensure your handler always returns a list from postprocess with one element per input, as TorchServe uses this to route each response back to the correct client.

Testing Batch Throughput

# batchtest.py

import requests

import threading

import time

url = "http://localhost:8080/predictions/resnet50custom"

imagepath = "kitten.jpg"

numrequests = 50

def sendrequest():

with open(imagepath, "rb") as f:

requests.post(url, data=f)

start = time.time()

threads = [threading.Thread(target=sendrequest) for in range(numrequests)]

for t in threads:

t.start()

for t in threads:

t.join()

elapsed = time.time() - start

print(f"Processed {numrequests} requests in {elapsed:.2f}s")

print(f"Throughput: {numrequests / elapsed:.1f} req/s")

Logging and Metrics

Log Configuration

TorchServe uses Log4j2 for logging. The default log directory is logs/. Key log files:

  • tslog.log: Main TorchServe server log
  • modellog.log: Output from handler code (use logger.info() in your handler)
  • accesslog.log: Inference and management API access logs

To add logging in your handler:

import logging

logger = logging.getLogger(name)

def inference(self, inputs):

logger.info("Running inference on batch of size %d", inputs.shape[0])

with torch.nograd():

outputs = self.model(inputs)

return outputs

Metrics API

The metrics endpoint exposes Prometheus-compatible metrics:

curl http://localhost:8082/metrics

Key built-in metrics:

  • tsinferencerequeststotal: Total number of inference requests
  • tsinferencelatencymicroseconds: Inference latency histogram
  • tsqueuelatencymicroseconds: Time requests spend waiting in the job queue
  • WorkerThreadTime: CPU time spent in worker threads
  • CPUUtilization, MemoryUsed, MemoryAvailable, GPUUtilization, GPUMemoryUsed

Custom Metrics

You can emit custom metrics from your handler using the metrics context:

from ts.metrics.metricsstore import MetricsStore

def inference(self, inputs):

# Emit a counter metric

metrics = self.context.metrics

metrics.addcounter("CustomInferenceCount", 1, idx=0)

with torch.nograd():

outputs = self.model(inputs)

return outputs

Prometheus and Grafana Integration

Add a scrape target to your Prometheus configuration:

# prometheus.yml

scrapeconfigs:

  • jobname: "torchserve"
staticconfigs:

  • targets: ["localhost:8082"]

GPU Support

Single GPU Deployment

TorchServe automatically assigns workers to available GPUs. When you start TorchServe with CUDA-enabled PyTorch, it detects GPUs and passes the gpuid property to each worker's initialize method.

In your handler:

def initialize(self, context):

properties = context.systemproperties

gpuid = properties.get("gpuid")

self.device = torch.device(f"cuda:{gpuid}" if torch.cuda.isavailable() and gpuid is not None else "cpu")

self.model = torch.jit.load(modelptpath, maplocation=self.device)

Multi-GPU Configuration

To distribute workers across multiple GPUs, set numberofgpu in config.properties:

numberofgpu=4

defaultworkerspermodel=4

TorchServe assigns one GPU per worker, round-robin across available devices.

Mixed Precision Inference

For faster GPU inference with FP16:

def initialize(self, context):

# ... load model ...

if self.device.type == "cuda":

self.model = self.model.half() # Convert to FP16

self.usefp16 = True

else:

self.usefp16 = False

def preprocess(self, data):

# ... build tensor ...

if self.usefp16:

return batchtensor.half().to(self.device)

return batchtensor.to(self.device)

Best Practices for Production Deployment

Model Versioning Strategy

Always include the version flag when creating model archives. Use semantic versioning and keep previous versions registered to enable instant rollback:

# Register new version

curl -X POST "http://localhost:8081/models?url=resnet50.mar&modelname=resnet50&modelversion=2.0&initialworkers=2"

Test new version

curl http://localhost:8080/predictions/resnet50/2.0 -T testimage.jpg

Promote to default if testing passes

curl -X PUT http://localhost:8081/models/resnet50/2.0/set-default

Remove old version after validation period

curl -X DELETE http://localhost:8081/models/resnet50/1.0

Worker Sizing

The right number of workers depends on your hardware and workload:

  • CPU-only: Set workers equal to the number of CPU cores, then tune based on observed queue latency
  • GPU: One worker per GPU is typically sufficient; multiple workers on the same GPU compete for memory and may reduce throughput
  • Mixed workloads: Profile under realistic traffic before committing to a worker count in production

Monitor tsqueuelatencymicroseconds. If it is consistently high, add more workers or increase batch size.

Health Checks and Load Balancers

TorchServe's /ping endpoint is suitable for liveness probes. For readiness probes, verify that at least one model is registered and healthy:

curl http://localhost:8080/ping         # Liveness

curl http://localhost:8081/models # Readiness (check response is not empty)

A simple readiness check script:

# readinesscheck.py

import requests

import sys

try:

response = requests.get("http://localhost:8081/models", timeout=5)

models = response.json().get("models", [])

if len(models) > 0:

print("Ready")

sys.exit(0)

else:

print("No models loaded")

sys.exit(1)

except Exception as e:

print(f"Health check failed: {e}")

sys.exit(1)

Containerization with Docker

TorchServe's official Docker images are on Docker Hub:

# Dockerfile

FROM pytorch/torchserve:latest-gpu

COPY model-store/ /home/model-server/model-store/

COPY config/config.properties /home/model-server/config.properties

USER model-server

CMD ["torchserve", \

"--start", \

"--model-store", "/home/model-server/model-store", \

"--models", "resnet50=resnet50.mar", \

"--ts-config", "/home/model-server/config.properties", \

"--foreground"]

docker build -t my-torchserve .

docker run --gpus all -p 8080:8080 -p 8081:8081 -p 8082:8082 my-torchserve

Securing the Management API

The management API should never be publicly exposed. In production, bind it to localhost or an internal network interface and use a reverse proxy for the inference API:

# config.properties

inferenceaddress=http://0.0.0.0:8080

managementaddress=http://127.0.0.1:8081

metricsaddress=http://127.0.0.1:8082

Place an NGINX reverse proxy in front of the inference API to handle TLS termination and rate limiting.

Snapshot and Recovery

TorchServe maintains a snapshot of the registered models on disk. When TorchServe restarts, it restores the last snapshot automatically. Ensure the snapshot directory (logs/) is on persistent storage in containerized environments.

Enable snapshot saving by omitting the --ncs flag and ensuring the modelstore directory is writable.

Tuning the Job Queue

jobqueuesize=1000

The job queue size limits how many pending requests TorchServe will hold in memory before rejecting new ones with HTTP 503. Set this based on your acceptable latency budget and expected traffic spikes. A queue that is too large causes requests to pile up and experience high latency; a queue that is too small causes unnecessary 503 errors under burst traffic.

Conclusion

TorchServe addresses a real and common gap in ML workflows: the lack of a standardized, production-ready serving layer for PyTorch models. By adopting TorchServe, ML engineering teams gain a consistent model deployment pattern, built-in operational tooling, and a path to scale from a single-GPU development machine to a multi-node production cluster.

The key takeaways from this tutorial:

  • Use torch-model-archiver to package models into self-contained .mar archives that include weights, code, and dependencies
  • TorchServe exposes three APIs: inference (port 8080), management (port 8081), and metrics (port 8082)
  • The management API enables zero-downtime model updates and rollbacks by supporting multiple model versions simultaneously
  • Custom handlers give you full control over preprocessing, inference, and postprocessing pipelines
  • Batch inference is essential for GPU efficiency; configure batchsize and maxbatchdelay based on your latency and throughput requirements
  • The metrics API integrates with Prometheus and Grafana for production observability
  • Always bind the management API to an internal interface and place a TLS-terminating reverse proxy in front of the inference API

TorchServe is actively developed and the ecosystem around it continues to mature. Features such as workflow chaining (for multi-model pipelines), gRPC support, and integration with Kubernetes autoscalers make it a viable foundation for enterprise-grade ML serving at significant scale.

Related Articles

Complete Keras 3 Tutorial: Multi-Backend Deep Learning Framework for TensorFlow, JAX, and PyTorch

Tutorial Lengkap Keras 3: Framework Deep Learning Multi-Backend untuk TensorFlow, JAX, dan PyTorch Keras 3 adalah evolus...

LitServe Tutorial: Fast and Easy AI Model Serving Framework

Tutorial LitServe: Framework Serving Model AI yang Cepat dan Mudah Pendahuluan LitServe adalah framework open-source dar...

PyTorch Geometric Tutorial: Graph Neural Networks in PyTorch

Memulai dengan PyTorch Geometric: Graph Neural Network di PyTorch Sebagian besar machine learning bekerja pada grid (gam...

Ray Train & Ray Tune Tutorial: Distributed Training and Hyperparameter Tuning

Ray Train & Ray Tune: Pelatihan Terdistribusi dan Penyetelan Hiperparameter Sebagian besar proyek machine learning dimul...