Complete Vertex AI Model Monitoring Tutorial: Continuous ML Monitoring

Vertex AI Model Monitoring automatically detects data drift, prediction drift, and feature attribution changes in deployed models. It helps maintain model performance and reliability in production.

Why Model Monitoring?

• Drift detection: Identify data and concept drift
• Automatic alerts: Get notified of issues
• Feature attribution: Track feature importance changes
• Continuous monitoring: 24/7 model oversight
• Integration: Works with Vertex AI endpoints

Prerequisites

pip install google-cloud-aiplatform
gcloud auth login

Setup Monitoring

1. Enable Monitoring on Endpoint

from google.cloud import aiplatform

aiplatform.init(project="your-project", location="us-central1")

Get endpoint
endpoint = aiplatform.Endpoint("projects/123/locations/us-central1/endpoints/456")

Create monitoring job
monitoringjob = aiplatform.ModelDeploymentMonitoringJob.create(

    displayname="model-monitoring-job",
    endpoint=endpoint,
    loggingsamplingstrategy={
        "randomsampleconfig": {"samplerate": 0.8}

    },
    scheduleconfig={"monitorinterval": {"seconds": 3600}},  # Hourly

    driftthresholds={
        "numericalfeatures": 0.3,

        "categoricalfeatures": 0.3
    }
)

print(f"Monitoring job created: {monitoringjob.resourcename}")

2. Configure Drift Detection

from google.cloud.aiplatformv1 import (
    ModelDeploymentMonitoringJob,
    ModelDeploymentMonitoringObjectiveConfig,
    ModelDeploymentMonitoringScheduleConfig,
    SamplingStrategy
)

Training dataset for baseline
trainingdataset = "bq://project.dataset.trainingdata"


Configure monitoring objectives
objectiveconfig = ModelDeploymentMonitoringObjectiveConfig(
    deployedmodelid=deployedmodelid,
    objectiveconfig={

        "trainingdataset": trainingdataset,

        "trainingpredictionskewdetectionconfig": {

            "skewthresholds": {
                "age": {"value": 0.3},
                "monthlycharges": {"value": 0.3}

            }
        },
        "predictiondriftdetectionconfig": {
            "driftthresholds": {

                "prediction": {"value": 0.2}
            }
        }
    }
)

# Configure email alerts
monitoringjob = aiplatform.ModelDeploymentMonitoringJob.create(
    displayname="monitoring-with-alerts",

    endpoint=endpoint,
    alertconfig={
        "emailalertconfig": {
            "useremails": ["team@company.com"]

        },
        "enablelogging": True
    },
    driftthresholds={

        "defaultdriftthreshold": 0.2

    }
)

# Detect differences between training and serving data
skewconfig = {
    "skewthresholds": {

        "age": {"value": 0.3},
        "tenuremonths": {"value": 0.3},
        "monthlycharges": {"value": 0.25}

    },
    "attributionscoreskewthresholds": {
        "age": {"value": 0.2}
    }
}

2. Prediction Drift

# Monitor prediction distribution changes
driftconfig = {

    "driftthresholds": {
        "predictionscore": {"value": 0.15}

    }
}

# Monitor feature importance changes
attributionconfig = {
    "attributionscoredriftthresholds": {

        "age": {"value": 0.2},
        "tenuremonths": {"value": 0.2}
    }
}