Serverless Model Serving on Kubernetes with KServe

KServe is a Kubernetes-native platform for serving machine learning models, built around a single custom resource called the InferenceService. If you already run workloads on Kubernetes and want model serving that scales to zero, supports canary rollouts, and exposes a standard inference protocol, KServe fits into your existing cluster instead of asking you to adopt a separate stack. This tutorial walks through the architecture, installation, and the day-to-day patterns you will use in production.

What KServe Is

KServe (formerly KFServing) is a CNCF model inference platform that runs on Kubernetes. Its core idea is to hide the boilerplate of building a serving deployment behind a declarative CRD. Instead of writing a Deployment, Service, HorizontalPodAutoscaler, and an ingress route by hand, you describe one InferenceService and KServe reconciles everything else.

A few characteristics distinguish KServe from a plain serving container:

• It is protocol-aware. KServe standardizes prediction endpoints around the V1 data plane and the Open Inference Protocol (V2), so clients written against the spec work across different model frameworks.
• It is runtime-aware. Built-in ServingRuntime definitions cover common frameworks (scikit-learn, XGBoost, PyTorch, TensorFlow, Triton, MLServer, Hugging Face) so you only point at a model artifact.
• It is composable. A request can flow through a transformer (pre/post-processing) and then a predictor, optionally with an explainer attached.

Unlike a general-purpose serving framework that you self-host, KServe is a control plane: it leans on Kubernetes primitives and, in serverless mode, on Knative and Istio.

Relationship to Knative and Istio

KServe supports two deployment modes, and the difference comes down to which dependencies you install.

Serverless mode

In serverless mode, KServe uses Knative Serving to manage the lifecycle of predictor pods and Istio (or another Knative-supported networking layer) for ingress and traffic routing. Knative gives you two things that are hard to build yourself: request-driven autoscaling down to zero replicas, and revision-based traffic splitting that powers canary rollouts. This is the default and most feature-complete mode.

Standard (raw) mode

Standard mode (sometimes called "raw Kubernetes" deployment) skips Knative and Istio. KServe creates plain Kubernetes Deployment, Service, and HorizontalPodAutoscaler objects directly. You lose scale-to-zero and revision-based canary, but you also drop two large dependencies. Standard mode is a good fit when your platform team does not want Istio in the cluster, or when steady traffic means scale-to-zero offers little benefit.

You select the mode per service with an annotation:

metadata:
  annotations:
    serving.kserve.io/deploymentMode: "RawDeployment"  # or "Serverless"

Prerequisites

You need:

• A Kubernetes cluster, version 1.28 or newer. A local kind or minikube cluster works for learning; a managed cluster (GKE, EKS, AKS) works for production.
• kubectl configured to talk to that cluster.
• Cluster-admin permissions to install CRDs and webhooks.
• For serverless mode, enough headroom for Knative and Istio control-plane pods (roughly 2 vCPU and 4 GB of free capacity beyond your workloads).

Confirm your context before installing:

kubectl config current-context
kubectl get nodes
kubectl version --short

Installation

Quickstart script (development)

For a development cluster, the quickstart script installs KServe together with Knative, Istio, and cert-manager in one step. Always read a script before piping it to a shell.

# Inspect first
curl -sL "https://raw.githubusercontent.com/kserve/kserve/release-0.14/hack/quickinstall.sh" -o quickinstall.sh
less quickinstall.sh


Then run
bash quickinstall.sh

When it finishes, verify the control plane is healthy:

kubectl get pods -n kserve
kubectl get pods -n knative-serving
kubectl get pods -n istio-system

Manifests (production)

For production you typically install each dependency from its own pinned manifest so you control versions and can audit changes. The general order is: cert-manager, then Istio, then Knative Serving, then KServe.

# KServe core, after dependencies are installed
kubectl apply --server-side -f \
  "https://github.com/kserve/kserve/releases/download/v0.14.0/kserve.yaml"

Built-in serving runtimes (ClusterServingRuntime objects)
kubectl apply --server-side -f \
  "https://github.com/kserve/kserve/releases/download/v0.14.0/kserve-cluster-resources.yaml"

Confirm the CRDs registered:

kubectl get crd | grep kserve.io
kubectl get clusterservingruntimes

The InferenceService CRD

The InferenceService (often abbreviated isvc) is the resource you will spend most of your time with. At minimum it names a predictor and points at a model.

Deploying a model from object storage

The example below serves a scikit-learn model stored in object storage, using the built-in sklearn runtime. KServe downloads the artifact at storageUri into the model server's storage initializer, so you never bake the model into the image.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: sklearn-iris
  namespace: ml-serving
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      storageUri: "gs://kfserving-examples/models/sklearn/1.0/model"
      resources:
        requests:
          cpu: "100m"
          memory: "256Mi"
        limits:
          cpu: "1"
          memory: "1Gi"

Apply it and wait for the service to become ready:

kubectl create namespace ml-serving
kubectl apply -f sklearn-iris.yaml
kubectl get isvc sklearn-iris -n ml-serving -w

The READY column flips to True and the URL column shows the external endpoint once Knative has provisioned the revision.

Pointing at S3 or other backends

apiVersion: v1
kind: Secret
metadata:
  name: s3-creds
  namespace: ml-serving
  annotations:
    serving.kserve.io/s3-endpoint: "s3.amazonaws.com"
    serving.kserve.io/s3-region: "us-east-1"
    serving.kserve.io/s3-usehttps: "1"
type: Opaque
stringData:
  AWSACCESSKEYID: ""

  AWSSECRETACCESSKEY: ""

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-s3
  namespace: ml-serving
secrets:
name: s3-creds

Then reference the service account in the predictor:

spec:
  predictor:
    serviceAccountName: sa-s3
    model:
      modelFormat:
        name: sklearn
      storageUri: "s3://my-bucket/models/iris"

Prediction Endpoints: V1 and the Open Inference Protocol (V2)

KServe exposes two data-plane protocols, and the runtime determines which are available.

V1 protocol

The V1 protocol is simple and JSON-based. The prediction path is /v1/models/:predict. The request body wraps your input in an instances array.

First, find the ingress address. On a local cluster with Istio, you usually port-forward the ingress gateway:

kubectl port-forward -n istio-system svc/istio-ingressgateway 8080:80

SERVICEHOST=$(kubectl get isvc sklearn-iris -n ml-serving \

  -o jsonpath='{.status.url}' | cut -d "/" -f3)

curl -v -H "Host: ${SERVICEHOST}" \
  -H "Content-Type: application/json" \
  http://localhost:8080/v1/models/sklearn-iris:predict \
  -d '{"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}'

A typical response:

{"predictions": [1, 1]}

Open Inference Protocol (V2)

The V2 protocol (Open Inference Protocol, originally KFServing/Triton's protocol) is more structured and supports typed tensors over both HTTP and gRPC. The prediction path is /v2/models//infer. Runtimes such as MLServer and Triton speak V2.

curl -H "Host: ${SERVICEHOST}" \
  -H "Content-Type: application/json" \
  http://localhost:8080/v2/models/sklearn-iris/infer \
  -d '{
    "inputs": [
      {
        "name": "input-0",
        "shape": [2, 4],
        "datatype": "FP32",
        "data": [6.8, 2.8, 4.8, 1.4, 6.0, 3.4, 4.5, 1.6]
      }
    ]
  }'

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: sklearn-iris
  namespace: ml-serving
  annotations:
    autoscaling.knative.dev/target: "10"
spec:
  predictor:
    minReplicas: 0
    maxReplicas: 5
    model:
      modelFormat:
        name: sklearn
      storageUri: "gs://kfserving-examples/models/sklearn/1.0/model"

metadata:
  annotations:
    autoscaling.knative.dev/scaleToZeroPodRetentionPeriod: "60s"
    autoscaling.knative.dev/window: "60s"

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: sklearn-iris
  namespace: ml-serving
spec:
  predictor:
    canaryTrafficPercent: 10
    model:
      modelFormat:
        name: sklearn
      storageUri: "gs://kfserving-examples/models/sklearn/1.0/model-v2"

# Inspect revisions and their traffic split
kubectl get isvc sklearn-iris -n ml-serving -o yaml | grep -A20 "status:"
kubectl get revisions -n ml-serving

# model.py
import joblib
import numpy as np
from kserve import Model, ModelServer, modelserver


class CustomSklearnModel(Model):
    def init(self, name: str):
        super().init(name)
        self.name = name
        self.model = None
        self.ready = False

    def load(self) -> bool:
        # Runs once when the pod starts.
        self.model = joblib.load("/mnt/models/model.joblib")
        self.ready = True
        return self.ready

    def predict(self, payload: dict, headers: dict = None) -> dict:
        instances = payload["instances"]
        inputs = np.asarray(instances)
        predictions = self.model.predict(inputs)
        return {"predictions": predictions.tolist()}


if name == "main":
    model = CustomSklearnModel("custom-sklearn")
    model.load()
    ModelServer().start([model])

Packaging with a Dockerfile

FROM python:3.11-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY model.py .

KServe defaults to port 8080 for HTTP
EXPOSE 8080
ENTRYPOINT ["python", "model.py"]

A minimal requirements.txt:

kserve>=0.14.0
scikit-learn
joblib
numpy

Build and push the image, then reference it in an InferenceService using a raw container spec instead of the model block:

docker build -t registry.example.com/ml/custom-sklearn:1.0 .
docker push registry.example.com/ml/custom-sklearn:1.0

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: custom-sklearn
  namespace: ml-serving
spec:
  predictor:
    containers:
name: kserve-container
        image: registry.example.com/ml/custom-sklearn:1.0

        args:
--modelname=custom-sklearn

        resources:

          requests:
            cpu: "200m"
            memory: "512Mi"

# transformer.py
from typing import Dict
from kserve import Model, ModelServer


class ImageTransformer(Model):
    def init(self, name: str, predictorhost: str):
        super().init(name)
        self.predictorhost = predictorhost
        self.ready = True

    def preprocess(self, request: Dict, headers: Dict = None) -> Dict:
        # Decode/normalize before sending to the predictor.
        raw = request["instances"]
        normalized = [[float(v) / 255.0 for v in row] for row in raw]
        return {"instances": normalized}

    def postprocess(self, response: Dict, headers: Dict = None) -> Dict:
        # Map class indices to labels after the predictor responds.
        labels = ["cat", "dog"]
        return {"predictions": [labels[i] for i in response["predictions"]]}


if name == "main":
    ModelServer().start([ImageTransformer("custom-model", predictor_host="")])

Attach it in the same InferenceService as the predictor. KServe wires the request flow client -> transformer -> predictor -> transformer -> client automatically:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: custom-model
  namespace: ml-serving
spec:
  transformer:
    containers:
name: kserve-container
        image: registry.example.com/ml/image-transformer:1.0

  predictor:
    model:
      modelFormat:
        name: pytorch
      storageUri: "gs://my-bucket/models/resnet"

ModelMesh for Multi-Model Serving

The standard InferenceService gives each model its own pods, which is efficient when you have a handful of high-traffic models. When you need to serve hundreds or thousands of models, dedicating pods per model wastes memory. KServe integrates ModelMesh for this case: a shared pool of serving runtime pods loads and evicts models on demand, keeping hot models in memory and lazily loading cold ones. You enable ModelMesh at install time and then deploy models with InferenceService resources annotated for the ModelMesh runtime. It trades a small per-request loading risk for much higher model density per node.

GPU Serving Notes

To serve on GPUs, request the GPU resource and ensure the node pool has the NVIDIA device plugin installed.

spec:
  predictor:
    model:
      modelFormat:
        name: pytorch
      storageUri: "gs://my-bucket/models/llm"
      resources:
        limits:
          nvidia.com/gpu: "1"
        requests:
          cpu: "2"
          memory: "8Gi"

A few practical points:

• Schedule GPU workloads onto GPU nodes with nodeSelector or taints/tolerations so CPU pods do not consume GPU nodes.
• Scale-to-zero on GPUs reclaims expensive hardware between bursts, but cold starts are longer because large models take time to load. Measure before enabling it.
• For large language models, the Hugging Face serving runtime and Triton (with TensorRT-LLM or vLLM backends) are common choices, configured through the same model or containers spec.

Monitoring and Logging

KServe model servers expose Prometheus metrics on a metrics port, including request counts, latency histograms, and (for V2 runtimes) per-model statistics. Scrape them with a ServiceMonitor if you run the Prometheus Operator, and build dashboards on request rate, p95/p99 latency, and error ratio per InferenceService.

KServe also supports a payload logger that asynchronously forwards request and response bodies to a sink (for example a broker or collector) for audit, drift detection, or building training datasets:

spec:
  predictor:
    logger:
      mode: all          # request, response, or all
      url: http://logging-collector.monitoring.svc.cluster.local

For pod-level debugging, the usual Kubernetes tools apply:

kubectl logs -n ml-serving -l serving.kserve.io/inferenceservice=sklearn-iris -c kserve-container
kubectl describe isvc sklearn-iris -n ml-serving

Best Practices

• Pin versions. Pin KServe, Knative, and Istio to known-compatible releases, and pin runtime images. Treat the control plane like any other dependency.
• Set resource requests and limits. Without them, the autoscaler has nothing to schedule against and noisy neighbors can starve your predictors.
• Choose the deployment mode deliberately. Use serverless mode when scale-to-zero and revision-based canary earn their keep; use standard mode for steady traffic or clusters without Istio.
• Keep models out of images. Use storageUri so you can roll model versions without rebuilding containers, and so the same image serves many models.
• Roll out with canaries. Never send 100 percent of traffic to a fresh revision. Start small, watch latency and error rate, then promote.
• Mind cold starts. Reserve minReplicas: 0 for tolerant, low-traffic endpoints. Keep at least one warm replica for latency-sensitive paths.
• Separate transformers from predictors when pre/post-processing is heavy, so the CPU-bound transformer and the GPU-bound predictor scale independently.
• Monitor from day one. Wire up Prometheus and structured logging before you have an incident, not during one.

Conclusion

KServe turns model serving on Kubernetes into a declarative operation: you describe an InferenceService, and the platform handles the deployment, networking, scaling, and traffic management. The serverless mode, backed by Knative and Istio, brings scale-to-zero and revision-based canary rollouts; the standard mode keeps things lean for steady workloads. Built-in runtimes cover the common frameworks, the Python SDK handles the rest through the Model class, and transformers let you separate data shaping from inference. For teams already invested in Kubernetes, KServe offers a consistent, protocol-driven path from a trained artifact to a production endpoint.

Key Takeaways

• The InferenceService CRD is the single abstraction; everything else is reconciled for you.
• Serverless mode (Knative + Istio) enables scale-to-zero and canary; standard mode trades those for fewer dependencies.
• V1 is a simple JSON protocol; the Open Inference Protocol (V2) is typed and supports HTTP and gRPC.
• minReplicas: 0 plus a concurrency target controls autoscaling; canaryTrafficPercent controls progressive rollout.
• Custom predictors and transformers are Python classes packaged as images; ModelMesh handles high-density multi-model serving.