Azure DevOps for MLOps Tutorial: CI/CD for Machine Learning

# Tutorial Lengkap Azure DevOps untuk MLOps: CI/CD untuk Machine Learning Azure DevOps menyediakan kemampuan CI/CD komprehensif untuk proyek machine learning. Tutorial ini mencakup pembangunan pipeli...

By Ruby Abdullah · · tutorial
AzureDevOpsMLOpsCI/CDAutomationPipeline

Complete Azure DevOps for MLOps Tutorial: CI/CD for Machine Learning

Azure DevOps provides comprehensive CI/CD capabilities for machine learning projects. This tutorial covers building automated ML pipelines, model deployment, and continuous delivery using Azure DevOps.

Why Azure DevOps for MLOps?

Key Benefits:
  • End-to-end automation: From code to deployment
  • Version control: Git repos for code and data
  • Pipeline orchestration: Multi-stage ML workflows
  • Integration: Native Azure ML integration
  • Collaboration: Team-based development

Components:
  • Azure Repos: Git repositories
  • Azure Pipelines: CI/CD automation
  • Azure Artifacts: Package management
  • Azure Boards: Work tracking

Prerequisites

pip install azure-devops azure-ai-ml


Azure CLI


az login


az extension add --name azure-devops


az devops configure --defaults organization=https://dev.azure.com/myorg

Project Setup

1. Create DevOps Project

# Create project

az devops project create --name "MLOps-Project" --org https://dev.azure.com/myorg



Create repository


az repos create --name "ml-models" --project "MLOps-Project"

2. Repository Structure

ml-models/

├── src/


│   ├── train.py


│   ├── evaluate.py


│   └── score.py


├── tests/


│   └── testmodel.py


├── pipelines/


│   ├── train-pipeline.yml


│   ├── deploy-pipeline.yml


│   └── cd-pipeline.yml


├── infrastructure/


│   └── arm-templates/


├── environment.yml


├── requirements.txt


└── azure-pipelines.yml

3. Service Connection

# Create service connection to Azure

az devops service-endpoint azurerm create \


    --azure-rm-service-principal-id "your-sp-id" \


    --azure-rm-subscription-id "your-subscription-id" \


    --azure-rm-subscription-name "Your Subscription" \


    --azure-rm-tenant-id "your-tenant-id" \


    --name "azure-ml-connection"

CI Pipeline for ML

1. Basic CI Pipeline

# azure-pipelines.yml

trigger:


  branches:


    include:


  • main
    • 

  • develop
    • 

  paths:


    include:


  • src/
    • 

  • tests/
    • 



pool:


  vmImage: 'ubuntu-latest'



variables:


  pythonVersion: '3.9'



stages:


  • stage: Build
    • 

    displayName: 'Build and Test'


    jobs:


  • job: BuildJob
    • 

        steps:


  • task: UsePythonVersion@0
    • 

            inputs:


              versionSpec: '$(pythonVersion)'


            displayName: 'Use Python $(pythonVersion)'


  • script: |
    • 

              python -m pip install --upgrade pip


              pip install -r requirements.txt


              pip install pytest pytest-cov


            displayName: 'Install dependencies'


  • script: |
    • 

              python -m pytest tests/ --cov=src --cov-report=xml


            displayName: 'Run tests'


  • task: PublishTestResults@2
    • 

            inputs:


              testResultsFiles: '*/test-.xml'


              testRunTitle: 'Python Tests'


  • task: PublishCodeCoverageResults@1
    • 

            inputs:


              codeCoverageTool: Cobertura


              summaryFileLocation: '$(System.DefaultWorkingDirectory)/*/coverage.xml'

2. Linting and Code Quality

# Add to azure-pipelines.yml

  • script: |
    • 

              pip install flake8 black mypy


              flake8 src/ --max-line-length=100


              black --check src/


              mypy src/


            displayName: 'Code quality checks'

Training Pipeline

1. ML Training Pipeline

# pipelines/train-pipeline.yml

trigger:


  branches:


    include:


  • main
    • 

  paths:


    include:


  • src/
    • 

  • data/
    • 



variables:


  • group: ml-variables
    • 

  • name: resourceGroup
    • 

    value: 'ml-rg'


  • name: workspaceName
    • 

    value: 'ml-workspace'


  • name: computeName
    • 

    value: 'cpu-cluster'



stages:


  • stage: Train
    • 

    displayName: 'Train Model'


    jobs:


  • job: TrainJob
    • 

        pool:


          vmImage: 'ubuntu-latest'


        steps:


  • task: UsePythonVersion@0
    • 

            inputs:


              versionSpec: '3.9'


  • task: AzureCLI@2
    • 

            displayName: 'Install Azure ML CLI'


            inputs:


              azureSubscription: 'azure-ml-connection'


              scriptType: 'bash'


              scriptLocation: 'inlineScript'


              inlineScript: |


                az extension add -n ml


  • task: AzureCLI@2
    • 

            displayName: 'Submit Training Job'


            inputs:


              azureSubscription: 'azure-ml-connection'


              scriptType: 'bash'


              scriptLocation: 'inlineScript'


              inlineScript: |


                az ml job create \


                  --file jobs/train-job.yml \


                  --resource-group $(resourceGroup) \


                  --workspace-name $(workspaceName) \


                  --set compute=$(computeName)

2. Training Job Definition

# jobs/train-job.yml

$schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json


type: command


code: ./src


command: python train.py --data ${{inputs.data}} --output ${{outputs.model}}


inputs:


  data:


    type: urifolder


    path: azureml:training-data:1


outputs:


  model:


    type: urifolder


environment: azureml:sklearn-env:1


compute: azureml:cpu-cluster


experimentname: mlops-training


displayname: model-training-$(Build.BuildId)

3. Training Script

# src/train.py

import argparse


import os


import mlflow


import pandas as pd


from sklearn.ensemble import RandomForestClassifier


from sklearn.modelselection import traintestsplit


from sklearn.metrics import accuracyscore, f1score


import joblib



def main():


    parser = argparse.ArgumentParser()


    parser.addargument("--data", type=str, required=True)


    parser.addargument("--output", type=str, required=True)


    args = parser.parseargs()



    # Enable autologging


    mlflow.autolog()



    # Load data


    df = pd.readcsv(os.path.join(args.data, "train.csv"))


    X = df.drop("target", axis=1)


    y = df["target"]



    Xtrain, Xtest, ytrain, ytest = traintestsplit(


        X, y, testsize=0.2, randomstate=42


    )



    # Train model


    with mlflow.startrun():


        model = RandomForestClassifier(nestimators=100, randomstate=42)


        model.fit(Xtrain, ytrain)



        # Evaluate


        predictions = model.predict(Xtest)


        accuracy = accuracyscore(ytest, predictions)


        f1 = f1score(ytest, predictions, average="weighted")



        mlflow.logmetric("accuracy", accuracy)


        mlflow.logmetric("f1score", f1)



        # Save model


        os.makedirs(args.output, existok=True)


        joblib.dump(model, os.path.join(args.output, "model.joblib"))



        print(f"Accuracy: {accuracy:.4f}")


        print(f"F1 Score: {f1:.4f}")



if name == "main":


    main()

Model Registration Pipeline

1. Register Model Pipeline

# pipelines/register-pipeline.yml

stages:


  • stage: Register
    • 

    displayName: 'Register Model'


    dependsOn: Train


    jobs:


  • job: RegisterJob
    • 

        steps:


  • task: AzureCLI@2
    • 

            displayName: 'Register Model'


            inputs:


              azureSubscription: 'azure-ml-connection'


              scriptType: 'bash'


              scriptLocation: 'inlineScript'


              inlineScript: |


                # Get latest job


                JOBNAME=$(az ml job list \


                  --resource-group $(resourceGroup) \


                  --workspace-name $(workspaceName) \


                  --query "[0].name" -o tsv)



                # Register model


                az ml model create \


                  --name my-model \


                  --version $(Build.BuildId) \


                  --path azureml://jobs/$JOBNAME/outputs/model \


                  --resource-group $(resourceGroup) \


                  --workspace-name $(workspaceName)

2. Model with Metadata

# models/model.yml

$schema: https://azuremlschemas.azureedge.net/latest/model.schema.json


name: classification-model


path: azureml://jobs/{jobname}/outputs/model


description: Random Forest classifier trained on customer data


properties:


  accuracy: 0.95


  framework: sklearn


  task: classification


tags:


  team: data-science


  environment: production

Deployment Pipeline

1. Deploy to Online Endpoint

# pipelines/deploy-pipeline.yml

stages:


  • stage: DeployStaging
    • 

    displayName: 'Deploy to Staging'


    jobs:


  • deployment: DeployStaging
    • 

        environment: staging


        strategy:


          runOnce:


            deploy:


              steps:


  • task: AzureCLI@2
    • 

                  displayName: 'Create/Update Endpoint'


                  inputs:


                    azureSubscription: 'azure-ml-connection'


                    scriptType: 'bash'


                    scriptLocation: 'inlineScript'


                    inlineScript: |


                      # Create endpoint if not exists


                      az ml online-endpoint create \


                        --name staging-endpoint \


                        --resource-group $(resourceGroup) \


                        --workspace-name $(workspaceName) \


                        --file endpoints/staging-endpoint.yml \


                        || true



                      # Create/update deployment


                      az ml online-deployment create \


                        --name blue \


                        --endpoint-name staging-endpoint \


                        --resource-group $(resourceGroup) \


                        --workspace-name $(workspaceName) \


                        --file endpoints/staging-deployment.yml



                      # Set traffic


                      az ml online-endpoint update \


                        --name staging-endpoint \


                        --resource-group $(resourceGroup) \


                        --workspace-name $(workspaceName) \


                        --traffic "blue=100"


  • stage: TestStaging
    • 

    displayName: 'Test Staging'


    dependsOn: DeployStaging


    jobs:


  • job: SmokeTest
    • 

        steps:


  • task: AzureCLI@2
    • 

            displayName: 'Run Smoke Tests'


            inputs:


              azureSubscription: 'azure-ml-connection'


              scriptType: 'bash'


              scriptLocation: 'inlineScript'


              inlineScript: |


                # Get endpoint key


                KEY=$(az ml online-endpoint get-credentials \


                  --name staging-endpoint \


                  --resource-group $(resourceGroup) \


                  --workspace-name $(workspaceName) \


                  --query "primaryKey" -o tsv)



                # Get endpoint URL


                URL=$(az ml online-endpoint show \


                  --name staging-endpoint \


                  --resource-group $(resourceGroup) \


                  --workspace-name $(workspaceName) \


                  --query "scoringuri" -o tsv)



                # Test endpoint


                curl -X POST "$URL" \


                  -H "Authorization: Bearer $KEY" \


                  -H "Content-Type: application/json" \


                  -d '{"features": [[5.1, 3.5, 1.4, 0.2]]}'


  • stage: DeployProduction
    • 

    displayName: 'Deploy to Production'


    dependsOn: TestStaging


    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))


    jobs:


  • deployment: DeployProduction
    • 

        environment: production


        strategy:


          runOnce:


            deploy:


              steps:


  • task: AzureCLI@2
    • 

                  displayName: 'Deploy to Production'


                  inputs:


                    azureSubscription: 'azure-ml-connection'


                    scriptType: 'bash'


                    scriptLocation: 'inlineScript'


                    inlineScript: |


                      az ml online-deployment create \


                        --name blue-$(Build.BuildId) \


                        --endpoint-name production-endpoint \


                        --resource-group $(resourceGroup) \


                        --workspace-name $(workspaceName) \


                        --file endpoints/production-deployment.yml

2. Endpoint Configuration

# endpoints/production-endpoint.yml

$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineEndpoint.schema.json


name: production-endpoint


authmode: key



endpoints/production-deployment.yml


$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json


name: blue


endpointname: production-endpoint


model: azureml:classification-model:latest


instancetype: StandardDS3v2


instancecount: 2


codeconfiguration:


  code: ./scoring


  scoringscript: score.py


environment: azureml:sklearn-env:1

Blue-Green Deployment

1. Blue-Green Pipeline

# pipelines/blue-green-pipeline.yml

stages:


  • stage: DeployGreen
    • 

    displayName: 'Deploy Green'


    jobs:


  • job: DeployGreen
    • 

        steps:


  • task: AzureCLI@2
    • 

            displayName: 'Deploy Green Version'


            inputs:


              azureSubscription: 'azure-ml-connection'


              scriptType: 'bash'


              scriptLocation: 'inlineScript'


              inlineScript: |


                # Deploy new version as green


                az ml online-deployment create \


                  --name green \


                  --endpoint-name production-endpoint \


                  --resource-group $(resourceGroup) \


                  --workspace-name $(workspaceName) \


                  --file endpoints/green-deployment.yml


  • stage: TestGreen
    • 

    displayName: 'Test Green'


    dependsOn: DeployGreen


    jobs:


  • job: TestGreen
    • 

        steps:


  • task: AzureCLI@2
    • 

            displayName: 'Test Green Deployment'


            inputs:


              azureSubscription: 'azure-ml-connection'


              scriptType: 'bash'


              scriptLocation: 'inlineScript'


              inlineScript: |


                python tests/integrationtests.py --deployment green


  • stage: ShiftTraffic
    • 

    displayName: 'Shift Traffic'


    dependsOn: TestGreen


    jobs:


  • job: ShiftTraffic
    • 

        steps:


  • task: AzureCLI@2
    • 

            displayName: 'Gradual Traffic Shift'


            inputs:


              azureSubscription: 'azure-ml-connection'


              scriptType: 'bash'


              scriptLocation: 'inlineScript'


              inlineScript: |


                # 10% to green


                az ml online-endpoint update \


                  --name production-endpoint \


                  --traffic "blue=90 green=10"



                sleep 300  # Monitor for 5 minutes



                # 50% to green


                az ml online-endpoint update \


                  --name production-endpoint \


                  --traffic "blue=50 green=50"



                sleep 300



                # 100% to green


                az ml online-endpoint update \


                  --name production-endpoint \


                  --traffic "blue=0 green=100"


  • stage: Cleanup
    • 

    displayName: 'Cleanup Old Deployment'


    dependsOn: ShiftTraffic


    jobs:


  • job: Cleanup
    • 

        steps:


  • task: AzureCLI@2
    • 

            displayName: 'Delete Blue Deployment'


            inputs:


              azureSubscription: 'azure-ml-connection'


              scriptType: 'bash'


              scriptLocation: 'inlineScript'


              inlineScript: |


                az ml online-deployment delete \


                  --name blue \


                  --endpoint-name production-endpoint \


                  --yes

Monitoring Pipeline

1. Model Monitoring

# pipelines/monitor-pipeline.yml

schedules:


  • cron: "0 0   "  # Daily at midnight
    • 

    displayName: Daily monitoring


    branches:


      include:


  • main
    • 



stages:


  • stage: Monitor
    • 

    displayName: 'Model Monitoring'


    jobs:


  • job: MonitorJob
    • 

        steps:


  • task: AzureCLI@2
    • 

            displayName: 'Check Model Drift'


            inputs:


              azureSubscription: 'azure-ml-connection'


              scriptType: 'bash'


              scriptLocation: 'inlineScript'


              inlineScript: |


                python monitoring/checkdrift.py \


                  --endpoint production-endpoint \


                  --threshold 0.1


  • task: AzureCLI@2
    • 

            displayName: 'Check Performance'


            inputs:


              azureSubscription: 'azure-ml-connection'


              scriptType: 'bash'


              scriptLocation: 'inlineScript'


              inlineScript: |


                python monitoring/checkperformance.py \


                  --endpoint production-endpoint \


                  --accuracy-threshold 0.85

2. Drift Detection Script

# monitoring/checkdrift.py

import argparse


from azure.ai.ml import MLClient


from azure.identity import DefaultAzureCredential



def checkdrift(endpointname, threshold):


    mlclient = MLClient.fromconfig(credential=DefaultAzureCredential())



    # Get recent predictions


    # Compare with training distribution


    # Alert if drift detected



    driftscore = calculatedrift()



    if driftscore > threshold:


        print(f"##vso[task.logissue type=warning]Data drift detected: {driftscore}")


        print("##vso[task.setvariable variable=driftDetected]true")


    else:


        print(f"Drift score: {driftscore} - within acceptable range")


        print("##vso[task.setvariable variable=driftDetected]false")



if name == "main":


    parser = argparse.ArgumentParser()


    parser.addargument("--endpoint", required=True)


    parser.addargument("--threshold", type=float, default=0.1)


    args = parser.parseargs()


    checkdrift(args.endpoint, args.threshold)

Variable Groups

1. Create Variable Group

# Create variable group

az pipelines variable-group create \


  --name "ml-variables" \


  --variables \


    AZURESUBSCRIPTIONID="your-subscription-id" \


    AZURERESOURCEGROUP="ml-rg" \


    AZUREMLWORKSPACE="ml-workspace" \


  --authorize

2. Use in Pipeline

variables:

  • group: ml-variables
    • 

  • name: environment
    • 

    value: 'production'



steps:


  • script: |
    • 

      echo "Deploying to $(AZUREML_WORKSPACE)"

Best Practices

1. Pipeline Templates

# templates/train-template.yml

parameters:


  • name: computeTarget
    • 

    type: string


  • name: experimentName
    • 

    type: string



steps:


  • task: AzureCLI@2
    • 

    inputs:


      scriptType: 'bash'


      inlineScript: |


        az ml job create \


          --compute ${{ parameters.computeTarget }} \


          --experiment-name ${{ parameters.experimentName }}



Usage in main pipeline


stages:


  • stage: Train
    • 

    jobs:


  • template: templates/train-template.yml
    • 

        parameters:


          computeTarget: cpu-cluster


          experimentName: my-experiment

2. Environment Approvals

# Configure environment with approvals

environments:


  • name: production
    • 

    resourceName: production


    resourceType: VirtualMachine


    tags: production

Conclusion

Azure DevOps for MLOps provides:

  • Automation: End-to-end CI/CD
  • Version control: Code and model tracking
  • Pipeline orchestration: Multi-stage workflows
  • Deployment strategies: Blue-green, canary
  • Monitoring: Continuous monitoring

    • Key takeaways:

    • Structure repos for ML projects
    • Use multi-stage pipelines
    • Implement proper testing
    • Use blue-green deployments
    • Monitor models continuously

    Related Articles

    Azure ML Pipelines Tutorial: ML Pipeline Automation

    Tutorial Lengkap Azure ML Pipelines: CI/CD untuk Machine Learning Azure ML Pipelines memungkinkan Anda membangun workflo...

    Complete GitHub Actions Tutorial for ML CI/CD

    Tutorial Lengkap GitHub Actions untuk ML CI/CD GitHub Actions memungkinkan Anda mengotomasi workflow Machine Learning da...

    AWS SageMaker Pipelines Tutorial: ML Pipeline Automation

    Tutorial Lengkap AWS SageMaker Pipelines: Automasi ML Workflows SageMaker Pipelines adalah layanan CI/CD yang dibuat khu...

    ClearML Tutorial: Open-Source MLOps Platform for Experiment Tracking and Pipeline Automation

    Tutorial ClearML: Platform MLOps Open-Source untuk Experiment Tracking dan Pipeline Automation ClearML adalah platform M...