Tutorial Lengkap Vertex AI Model Monitoring: Monitoring ML Berkelanjutan

Vertex AI Model Monitoring secara otomatis mendeteksi data drift, prediction drift, dan perubahan feature attribution pada model yang dideploy. Membantu menjaga performa dan reliabilitas model di production.

Mengapa Model Monitoring?

• Deteksi drift: Identifikasi data dan concept drift
• Alert otomatis: Dapat notifikasi masalah
• Feature attribution: Lacak perubahan importance
• Monitoring berkelanjutan: Pengawasan model 24/7
• Integrasi: Bekerja dengan Vertex AI endpoints

Prerequisites

pip install google-cloud-aiplatform
gcloud auth login

Setup Monitoring

1. Aktifkan Monitoring pada Endpoint

from google.cloud import aiplatform

aiplatform.init(project="your-project", location="us-central1")

Dapatkan endpoint
endpoint = aiplatform.Endpoint("projects/123/locations/us-central1/endpoints/456")

Buat monitoring job
monitoringjob = aiplatform.ModelDeploymentMonitoringJob.create(

    displayname="model-monitoring-job",
    endpoint=endpoint,
    loggingsamplingstrategy={
        "randomsampleconfig": {"samplerate": 0.8}

    },
    scheduleconfig={"monitorinterval": {"seconds": 3600}},  # Per jam

    driftthresholds={
        "numericalfeatures": 0.3,

        "categoricalfeatures": 0.3
    }
)

print(f"Monitoring job dibuat: {monitoringjob.resourcename}")

2. Konfigurasi Deteksi Drift

from google.cloud.aiplatformv1 import (
    ModelDeploymentMonitoringJob,
    ModelDeploymentMonitoringObjectiveConfig,
    ModelDeploymentMonitoringScheduleConfig,
    SamplingStrategy
)

Training dataset untuk baseline
trainingdataset = "bq://project.dataset.trainingdata"


Konfigurasi monitoring objectives
objectiveconfig = ModelDeploymentMonitoringObjectiveConfig(
    deployedmodelid=deployedmodelid,
    objectiveconfig={

        "trainingdataset": trainingdataset,

        "trainingpredictionskewdetectionconfig": {

            "skewthresholds": {
                "age": {"value": 0.3},
                "monthlycharges": {"value": 0.3}

            }
        },
        "predictiondriftdetectionconfig": {
            "driftthresholds": {

                "prediction": {"value": 0.2}
            }
        }
    }
)

# Konfigurasi email alerts
monitoringjob = aiplatform.ModelDeploymentMonitoringJob.create(
    displayname="monitoring-with-alerts",

    endpoint=endpoint,
    alertconfig={
        "emailalertconfig": {
            "useremails": ["team@company.com"]

        },
        "enablelogging": True
    },
    driftthresholds={

        "defaultdriftthreshold": 0.2

    }
)

# Deteksi perbedaan antara training dan serving data
skewconfig = {
    "skewthresholds": {

        "age": {"value": 0.3},
        "tenuremonths": {"value": 0.3},
        "monthlycharges": {"value": 0.25}

    },
    "attributionscoreskewthresholds": {
        "age": {"value": 0.2}
    }
}

2. Prediction Drift

# Monitor perubahan distribusi prediksi
driftconfig = {

    "driftthresholds": {
        "predictionscore": {"value": 0.15}

    }
}

# Monitor perubahan feature importance
attributionconfig = {
    "attributionscoredriftthresholds": {

        "age": {"value": 0.2},
        "tenuremonths": {"value": 0.2}
    }
}