Tutorial 19: Triton Inference Server - High-Performance Model Serving

Table of Contents

Introduction

NVIDIA Triton Inference Server is an open-source inference serving software that enables teams to deploy AI models from any framework on any GPU or CPU-based infrastructure. It supports multiple model formats simultaneously — TensorFlow, PyTorch, ONNX Runtime, TensorRT, and custom Python backends — all from a single server instance.

Triton addresses the hard problems of production model serving: dynamic batching to maximize GPU utilization, model ensembles for multi-stage pipelines, concurrent model execution, model versioning, health monitoring, and metrics export. This tutorial covers everything from initial setup to production-grade Kubernetes deployment.

Prerequisites

• Docker installed with NVIDIA Container Toolkit
• NVIDIA GPU with CUDA 12.0+ (for GPU serving)
• Python 3.9+ with tritonclient library
• Basic understanding of model serving concepts
• kubectl and helm (for Kubernetes section)

# Install Triton client libraries
pip install tritonclient[all] grpcio numpy

import tritonclient.grpc as grpcclient
import tritonclient.http as httpclient
import numpy as np

Setting Up Triton Inference Server

Pulling and Running Triton

# Pull the Triton Inference Server container
docker pull nvcr.io/nvidia/tritonserver:24.01-py3

Create model repository directory
mkdir -p /opt/triton/modelrepository


Run Triton with GPU support
docker run --gpus all \
    --rm -p 8000:8000 -p 8001:8001 -p 8002:8002 \
    -v /opt/triton/modelrepository:/models \
    nvcr.io/nvidia/tritonserver:24.01-py3 \
    tritonserver --model-repository=/models \
                 --log-verbose=1

Ports:
8000 - HTTP/REST endpoint
8001 - gRPC endpoint
8002 - Metrics endpoint (Prometheus format)

Verifying the Server

import tritonclient.http as httpclient

Check server health
client = httpclient.InferenceServerClient(url="localhost:8000")

print(f"Server live: {client.isserverlive()}")
print(f"Server ready: {client.isserverready()}")

List loaded models
repositoryindex = client.getmodelrepositoryindex()
for model in repositoryindex:

    print(f"Model: {model['name']}, Version: {model.get('version', 'N/A')}, "
          f"State: {model.get('state', 'N/A')}")

modelrepository/
├── imageclassifier/

│   ├── config.pbtxt
│   ├── 1/
│   │   └── model.onnx
│   └── 2/
│       └── model.onnx
├── textencoder/
│   ├── config.pbtxt
│   └── 1/
│       └── model.pt
├── featureextractor/

│   ├── config.pbtxt
│   └── 1/
│       └── model.savedmodel/
│           ├── savedmodel.pb
│           └── variables/
└── preprocessor/
    ├── config.pbtxt
    └── 1/
        └── model.py

Model Configuration (config.pbtxt)

# config.pbtxt for an ONNX image classifier
name: "imageclassifier"

platform: "onnxruntimeonnx"
maxbatchsize: 32

input [
  {
    name: "input"
    datatype: TYPEFP32
    dims: [ 3, 224, 224 ]
  }
]

output [
  {
    name: "output"
    datatype: TYPEFP32
    dims: [ 1000 ]
    labelfilename: "labels.txt"

  }
]

Version policy: serve only the latest version
versionpolicy: { latest { numversions: 1 }}


Instance configuration
instancegroup [
  {
    count: 2
    kind: KINDGPU

    gpus: [ 0 ]
  }
]

# Export PyTorch model for Triton
import torch
import torchvision.models as models

model = models.resnet50(pretrained=True)
model.eval()

Create TorchScript model
scriptedmodel = torch.jit.script(model)
scriptedmodel.save("modelrepository/resnet50pytorch/1/model.pt")

# config.pbtxt for PyTorch model
name: "resnet50pytorch"
platform: "pytorchlibtorch"

maxbatchsize: 64


input [
  {
    name: "INPUT0"

    datatype: TYPEFP32

    dims: [ 3, 224, 224 ]
  }
]

output [
  {
    name: "OUTPUT0"

    datatype: TYPEFP32

    dims: [ 1000 ]
  }
]

# config.pbtxt for TensorFlow SavedModel
name: "textclassifiertf"

platform: "tensorflowsavedmodel"
maxbatchsize: 128

input [
  {
    name: "inputids"

    datatype: TYPEINT32

    dims: [ 512 ]
  },
  {
    name: "attentionmask"
    datatype: TYPEINT32
    dims: [ 512 ]
  }
]

output [
  {
    name: "logits"
    datatype: TYPEFP32
    dims: [ 3 ]
  }
]

ONNX Runtime Backend

# Convert a model to ONNX for Triton
import torch
import onnx

model = models.resnet50(pretrained=True)
model.eval()

dummyinput = torch.randn(1, 3, 224, 224)

torch.onnx.export(
    model,
    dummyinput,
    "modelrepository/resnet50onnx/1/model.onnx",
    inputnames=['input'],

    outputnames=['output'],
    dynamicaxes={

        'input': {0: 'batchsize'},
        'output': {0: 'batchsize'}

    },
    opsetversion=17
)

Verify the ONNX model
onnxmodel = onnx.load("modelrepository/resnet50onnx/1/model.onnx")

onnx.checker.checkmodel(onnxmodel)

print("ONNX model is valid")

# modelrepository/preprocessor/1/model.py
import tritonpythonbackendutils as pbutils
import numpy as np
from PIL import Image
import io
import json

class TritonPythonModel:
    """Custom Python backend for image preprocessing."""

    def initialize(self, args):
        """Called once when the model is loaded."""
        self.modelconfig = json.loads(args['modelconfig'])
        self.outputdtype = pbutils.tritonstringtonumpy(

            pbutils.getoutputconfigbyname(
                self.modelconfig, "preprocessedimage"
            )['datatype']

        )

        # Standard ImageNet normalization
        self.mean = np.array([0.485, 0.456, 0.406], dtype=np.float32)
        self.std = np.array([0.229, 0.224, 0.225], dtype=np.float32)

    def execute(self, requests):
        """Process each inference request."""
        responses = []

        for request in requests:
            # Get raw image bytes
            rawimage = pbutils.getinputtensorbyname(

                request, "rawimage"
            ).asnumpy()


            # Decode and preprocess
            image = Image.open(io.BytesIO(rawimage.tobytes()))
            image = image.resize((224, 224))
            image = np.array(image, dtype=np.float32) / 255.0

            # Normalize
            image = (image - self.mean) / self.std

            # Transpose to CHW format
            image = np.transpose(image, (2, 0, 1))
            image = np.expanddims(image, axis=0)


            # Create output tensor
            outputtensor = pbutils.Tensor(

                "preprocessedimage",
                image.astype(self.outputdtype)

            )

            responses.append(
                pbutils.InferenceResponse(
                    outputtensors=[outputtensor]
                )
            )

        return responses

    def finalize(self):
        """Called once when the model is unloaded."""
        print("Preprocessor model finalized")

# config.pbtxt for Python backend
name: "preprocessor"
backend: "python"
maxbatchsize: 0

input [
  {
    name: "rawimage"

    datatype: TYPEUINT8

    dims: [ -1 ]
  }
]

output [
  {
    name: "preprocessedimage"
    datatype: TYPEFP32
    dims: [ 1, 3, 224, 224 ]
  }
]

instancegroup [

  {
    count: 2
    kind: KINDCPU
  }
]

Dynamic Batching

Dynamic batching allows Triton to combine individual inference requests into a batch, maximizing GPU throughput without requiring the client to batch requests manually.

# config.pbtxt with dynamic batching
name: "imageclassifier"

platform: "onnxruntimeonnx"
maxbatchsize: 64

input [
  {
    name: "input"
    datatype: TYPEFP32
    dims: [ 3, 224, 224 ]
  }
]

output [
  {
    name: "output"
    datatype: TYPEFP32
    dims: [ 1000 ]
  }
]

dynamicbatching {

  preferredbatchsize: [ 8, 16, 32 ]

  maxqueuedelaymicroseconds: 100

  # Priority levels for different request types
  prioritylevels: 3

  defaultprioritylevel: 2


  # Queue policy
  defaultqueuepolicy {

    timeoutaction: REJECT
    defaulttimeoutmicroseconds: 5000000
    allowtimeoutoverride: true
    maxqueuesize: 100
  }
}

Client-Side Batching

import tritonclient.grpc as grpcclient
import numpy as np
from concurrent.futures import ThreadPoolExecutor
import time

def sendinferencerequest(client, modelname, imagedata):
    """Send a single inference request."""
    inputs = [
        grpcclient.InferInput("input", imagedata.shape, "FP32")

    ]
    inputs[0].setdatafromnumpy(imagedata)


    outputs = [
        grpcclient.InferRequestedOutput("output")
    ]

    result = client.infer(
        modelname=modelname,

        inputs=inputs,
        outputs=outputs
    )

    return result.asnumpy("output")


Send concurrent requests (Triton batches them automatically)
client = grpcclient.InferenceServerClient(url="localhost:8001")
modelname = "imageclassifier"

Simulate 100 concurrent requests
images = [
    np.random.rand(1, 3, 224, 224).astype(np.float32)
    for  in range(100)

]

starttime = time.time()

with ThreadPoolExecutor(maxworkers=16) as executor:

    futures = [
        executor.submit(sendinferencerequest, client, modelname, img)
        for img in images
    ]
    results = [f.result() for f in futures]

elapsed = time.time() - starttime

print(f"Processed {len(images)} images in {elapsed:.2f}s")
print(f"Throughput: {len(images) / elapsed:.1f} images/sec")

# config.pbtxt for ensemble model
name: "imagepipeline"
platform: "ensemble"
maxbatchsize: 32

input [
  {
    name: "rawimage"

    datatype: TYPEUINT8

    dims: [ -1 ]
  }
]

output [
  {
    name: "classification"
    datatype: TYPEFP32

    dims: [ 1000 ]
  }
]

ensemblescheduling {
  step [
    {
      modelname: "preprocessor"

      modelversion: -1
      inputmap {

        key: "rawimage"
        value: "rawimage"

      }
      outputmap {
        key: "preprocessedimage"

        value: "preprocessedoutput"
      }
    },
    {
      modelname: "imageclassifier"
      modelversion: -1

      inputmap {
        key: "input"
        value: "preprocessedoutput"

      }
      outputmap {
        key: "output"
        value: "classification"
      }
    }
  ]
}

Multi-Model Ensemble Example

# A more complex ensemble: preprocess -> extract features -> classify + detect
name: "multitaskpipeline"
platform: "ensemble"
maxbatchsize: 16

input [
  {
    name: "rawimage"

    datatype: TYPEUINT8

    dims: [ -1 ]
  }
]

output [
  {
    name: "classlabels"
    datatype: TYPEFP32
    dims: [ 1000 ]
  },
  {
    name: "boundingboxes"

    datatype: TYPEFP32

    dims: [ 100, 6 ]
  }
]

ensemblescheduling {
  step [
    {
      modelname: "preprocessor"

      modelversion: -1
      inputmap { key: "rawimage" value: "rawimage" }

      outputmap { key: "preprocessedimage" value: "processed" }

    },
    {
      modelname: "featureextractor"

      modelversion: -1
      inputmap { key: "input" value: "processed" }

      outputmap { key: "features" value: "sharedfeatures" }

    },
    {
      modelname: "classifierhead"

      modelversion: -1
      inputmap { key: "features" value: "sharedfeatures" }
      outputmap { key: "logits" value: "classlabels" }
    },
    {
      modelname: "detectionhead"
      modelversion: -1

      inputmap { key: "features" value: "sharedfeatures" }

      outputmap { key: "boxes" value: "boundingboxes" }

    }
  ]
}

# Run multiple instances across GPUs
name: "largemodel"
platform: "pytorchlibtorch"

maxbatchsize: 8


instancegroup [
  {
    # 2 instances on GPU 0
    count: 2
    kind: KINDGPU

    gpus: [ 0 ]
  },
  {
    # 1 instance on GPU 1
    count: 1
    kind: KINDGPU
    gpus: [ 1 ]
  },
  {
    # 1 CPU fallback instance
    count: 1
    kind: KINDCPU

  }
]

Rate limiting to prevent GPU memory exhaustion
ratelimiter {
  resources [
    {
      name: "gpumemory"

      global: true
      count: 4
    }
  ]
}

# Warm up model to avoid cold-start latency
name: "imageclassifier"
platform: "onnxruntimeonnx"

maxbatchsize: 32


modelwarmup [
  {
    name: "warmupbatch1"
    batchsize: 1

    inputs {
      key: "input"
      value: {
        datatype: TYPEFP32

        dims: [ 3, 224, 224 ]
        randomdata: true
      }
    }
  },
  {
    name: "warmupbatch32"
    batchsize: 32

    inputs {
      key: "input"
      value: {
        datatype: TYPEFP32

        dims: [ 3, 224, 224 ]
        randomdata: true
      }
    }
  }
]

Performance Analyzer

Triton's perfanalyzer tool measures the throughput and latency characteristics of your models under various load conditions.

# Install perfanalyzer
It comes bundled in the Triton SDK container
docker pull nvcr.io/nvidia/tritonserver:24.01-py3-sdk

Basic throughput test
perfanalyzer -m imageclassifier \
    -u localhost:8001 \
    -i grpc \
    --concurrency-range 1:16:2 \
    --measurement-interval 10000

Detailed latency analysis
perfanalyzer -m imageclassifier \
    -u localhost:8001 \
    -i grpc \
    --concurrency-range 1:32:4 \
    --percentile=95 \
    --measurement-interval 10000 \
    -f results.csv

Test with specific input shapes
perfanalyzer -m imageclassifier \
    -u localhost:8001 \
    --shape input:3,224,224 \
    --concurrency-range 1:64:8 \
    -i grpc

Programmatic Performance Testing

import tritonclient.grpc as grpcclient
import numpy as np
import time
from concurrent.futures import ThreadPoolExecutor, ascompleted

from dataclasses import dataclass

@dataclass
class BenchmarkResult:
    totalrequests: int
    totaltimes: float
    throughputrps: float

    avglatencyms: float

    p50latencyms: float

    p95latencyms: float

    p99latencyms: float

    errorcount: int

def benchmarkmodel(url, modelname, inputshape,

                    concurrency=8, numrequests=1000):
    """Run a comprehensive benchmark against a Triton model."""
    client = grpcclient.InferenceServerClient(url=url)
    latencies = []
    errors = 0

    def singlerequest():

        nonlocal errors
        try:
            data = np.random.rand(inputshape).astype(np.float32)
            inputs = [grpcclient.InferInput("input", data.shape, "FP32")]
            inputs[0].setdatafromnumpy(data)

            outputs = [grpcclient.InferRequestedOutput("output")]

            start = time.perfcounter()
            client.infer(modelname=modelname,
                        inputs=inputs, outputs=outputs)
            elapsed = (time.perfcounter() - start)  1000
            return elapsed
        except Exception:
            errors += 1
            return None

    # Warmup
    for  in range(min(50, numrequests // 10)):

        singlerequest()

    # Benchmark
    starttime = time.time()

    with ThreadPoolExecutor(maxworkers=concurrency) as executor:
        futures = [executor.submit(singlerequest)

                   for  in range(numrequests)]

        for f in ascompleted(futures):
            result = f.result()
            if result is not None:
                latencies.append(result)

    totaltime = time.time() - starttime
    latencies = np.array(latencies)

    return BenchmarkResult(
        totalrequests=numrequests,
        totaltimes=totaltime,

        throughputrps=len(latencies) / totaltime,

        avglatencyms=np.mean(latencies),

        p50latencyms=np.percentile(latencies, 50),

        p95latencyms=np.percentile(latencies, 95),

        p99latencyms=np.percentile(latencies, 99),

        errorcount=errors
    )

Run benchmark
result = benchmarkmodel(

    url="localhost:8001",
    modelname="imageclassifier",

    inputshape=(1, 3, 224, 224),
    concurrency=16,
    numrequests=5000

)

print(f"Throughput: {result.throughputrps:.1f} req/s")
print(f"Avg latency: {result.avglatencyms:.2f} ms")
print(f"P95 latency: {result.p95latencyms:.2f} ms")
print(f"P99 latency: {result.p99latencyms:.2f} ms")
print(f"Errors: {result.errorcount}")

# triton-values.yaml - Helm chart values for Triton deployment
replicaCount: 3

image:
  repository: nvcr.io/nvidia/tritonserver
  tag: "24.01-py3"
  pullPolicy: IfNotPresent

resources:
  limits:
    nvidia.com/gpu: 1
    memory: "16Gi"
    cpu: "8"
  requests:
    nvidia.com/gpu: 1
    memory: "8Gi"
    cpu: "4"

service:
  type: ClusterIP
  httpPort: 8000
  grpcPort: 8001
  metricsPort: 8002

modelRepository:
  storageType: "pvc"  # pvc, s3, gcs, azure
  pvc:
    existingClaim: "triton-models-pvc"

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetGPUUtilization: 70
  targetQueueLatency: 50  # ms

# triton-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: triton-inference-server
  labels:
    app: triton
spec:
  replicas: 3
  selector:
    matchLabels:
      app: triton
  template:
    metadata:
      labels:
        app: triton
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8002"
    spec:
      containers:
name: triton
        image: nvcr.io/nvidia/tritonserver:24.01-py3

        command: ["tritonserver"]
        args:
"--model-repository=/models"
"--strict-model-config=false"
"--log-verbose=0"
"--exit-on-error=false"
"--rate-limit=executioncount"
"--rate-limit-resource=R1:4"
        ports:

containerPort: 8000
          name: http

containerPort: 8001
          name: grpc

containerPort: 8002
          name: metrics

        resources:
          limits:
            nvidia.com/gpu: 1
            memory: "16Gi"
          requests:
            nvidia.com/gpu: 1
            memory: "8Gi"
        volumeMounts:
name: model-repository
          mountPath: /models

        livenessProbe:
          httpGet:
            path: /v2/health/live
            port: 8000
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /v2/health/ready
            port: 8000
          initialDelaySeconds: 30
          periodSeconds: 10
      volumes:
name: model-repository
        persistentVolumeClaim:

          claimName: triton-models-pvc

apiVersion: v1
kind: Service
metadata:
  name: triton-service
spec:
  selector:
    app: triton
  ports:
name: http
    port: 8000

    targetPort: 8000
name: grpc
    port: 8001

    targetPort: 8001
name: metrics
    port: 8002

    targetPort: 8002

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: triton-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: triton-inference-server
  minReplicas: 2
  maxReplicas: 10
  metrics:
type: Pods
    pods:

      metric:
        name: nvinferencequeuedurationus
      target:
        type: AverageValue
        averageValue: "50000"  # 50ms queue latency target

Monitoring with Prometheus and Grafana

# Triton exposes Prometheus metrics at :8002/metrics
Key metrics to monitor:

nvinferencerequestsuccess    - successful inference count

nvinferencerequestfailure    - failed inference count
nvinferencecount              - total inference count
nvinferenceexeccount         - execution count (batches)

nvinferencerequestdurationus - request latency

nvinferencequeuedurationus  - queue wait time

nvinferencecomputeinputdurationus  - input processing time
nvinferencecomputeinferdurationus  - actual inference time

nvinferencecomputeoutputdurationus - output processing time
nvgpuutilization              - GPU utilization percentage
nvgpumemoryusedbytes        - GPU memory usage

import requests

def gettritonmetrics(metricsurl="http://localhost:8002/metrics"):

    """Fetch and parse Triton Prometheus metrics."""
    response = requests.get(metricsurl)
    metrics = {}
    for line in response.text.split('\n'):
        if line and not line.startswith('#'):
            parts = line.split()
            if len(parts) == 2:
                metrics[parts[0]] = float(parts[1])
    return metrics

metrics = gettritonmetrics()
for key, value in sorted(metrics.items()):
    if 'nvinference' in key or 'nvgpu' in key:
        print(f"{key}: {value}")

Best Practices

Conclusion

Triton Inference Server solves the operational complexity of serving ML models in production. By supporting multiple frameworks from a single server, providing dynamic batching out of the box, enabling model ensembles for complex pipelines, and integrating naturally with Kubernetes and Prometheus, Triton lets engineering teams focus on model quality rather than infrastructure plumbing. The performance analyzer ensures you can find and fix bottlenecks before they reach production, and the GPU scheduling features maximize hardware utilization across your model fleet.