Tutorial Lengkap DVC: Git untuk Data dan Model Machine Learning

# Tutorial Lengkap DVC: Git untuk Data dan Model Machine Learning DVC (Data Version Control) adalah open-source tool yang memperluas Git untuk menangani data, model, dan eksperimen ML. Dengan DVC, An...

By Ruby Abdullah · · tutorial
DVCMLOpsData Version ControlMachine LearningGitPipeline

Tutorial Lengkap DVC: Git untuk Data dan Model Machine Learning

DVC (Data Version Control) adalah open-source tool yang memperluas Git untuk menangani data, model, dan eksperimen ML. Dengan DVC, Anda dapat melakukan version control pada dataset besar dan model tanpa menyimpannya langsung di Git repository.

Mengapa DVC?

Masalah umum dalam ML projects:

  • Large files: Git tidak efisien untuk file besar (datasets, models)
  • Reproducibility: Sulit mereproduksi eksperimen tanpa tracking data version
  • Collaboration: Berbagi dataset besar antar tim sulit
  • Storage: Dataset besar memakan space di Git repository

Solusi DVC:
  • Version control untuk data dan model
  • Remote storage (S3, GCS, Azure, SSH)
  • Pipeline reproducibility
  • Experiment tracking
  • Integrasi seamless dengan Git

Instalasi

# Install DVC

pip install dvc

Install dengan remote storage support

pip install "dvc[s3]" # AWS S3

pip install "dvc[gs]" # Google Cloud Storage

pip install "dvc[azure]" # Azure Blob Storage

pip install "dvc[ssh]" # SSH/SFTP

pip install "dvc[all]" # Semua remotes

Verify installation

dvc version

Quick Start

1. Initialize DVC

# Pastikan sudah di Git repository

git init

dvc init

DVC akan membuat:

- .dvc/ directory

- .dvcignore file

git status

.dvc/.gitignore

.dvc/config

.dvcignore

git add .dvc .dvcignore

git commit -m "Initialize DVC"

2. Track Data Files

# Tambahkan file data ke DVC

dvc add data/dataset.csv

DVC akan membuat:

- data/dataset.csv.dvc (metadata file)

- Update .gitignore

Add .dvc file ke Git

git add data/dataset.csv.dvc data/.gitignore

git commit -m "Add dataset"

Untuk folder

dvc add data/images/

git add data/images.dvc data/.gitignore

git commit -m "Add images folder"

3. Setup Remote Storage

# AWS S3

dvc remote add -d myremote s3://mybucket/dvc-storage

Google Cloud Storage

dvc remote add -d myremote gs://mybucket/dvc-storage

Azure Blob Storage

dvc remote add -d myremote azure://mycontainer/dvc-storage

Local/Network storage

dvc remote add -d myremote /path/to/storage

SSH

dvc remote add -d myremote ssh://user@server/path/to/storage

Commit remote config

git add .dvc/config

git commit -m "Configure remote storage"

4. Push dan Pull Data

# Push data ke remote

dvc push

Pull data dari remote

dvc pull

Push/pull specific file

dvc push data/dataset.csv.dvc

dvc pull data/dataset.csv.dvc

Working with Data Versions

1. Update Data

# Setelah data berubah

dvc add data/dataset.csv

git add data/dataset.csv.dvc

git commit -m "Update dataset v2"

dvc push

2. Switch Data Versions

# Checkout ke versi sebelumnya

git checkout HEAD~1 data/dataset.csv.dvc

dvc checkout

Kembali ke latest

git checkout HEAD data/dataset.csv.dvc

dvc checkout

3. View Data History

# Lihat history

git log --oneline data/dataset.csv.dvc

Diff antara versi

dvc diff HEAD~1

Metrics diff

dvc metrics diff HEAD~1

DVC Pipelines

1. Membuat Pipeline

Pipeline didefinisikan dalam dvc.yaml:

# dvc.yaml

stages:

prepare:

cmd: python src/prepare.py

deps:

  • src/prepare.py
  • data/raw/
outs:

  • data/processed/

train:

cmd: python src/train.py

deps:

  • src/train.py
  • data/processed/
params:

  • train.nestimators
  • train.maxdepth
outs:

  • models/model.pkl
metrics:

  • metrics/trainmetrics.json:
cache: false

evaluate:

cmd: python src/evaluate.py

deps:

  • src/evaluate.py
  • models/model.pkl
  • data/processed/test.csv
metrics:

  • metrics/evalmetrics.json:
cache: false

plots:

  • plots/confusionmatrix.png

2. Parameters File

# params.yaml

prepare:

testsize: 0.2

randomstate: 42

train:

nestimators: 100

maxdepth: 5

randomstate: 42

evaluate:

threshold: 0.5

3. Implementasi Scripts

# src/prepare.py

import pandas as pd

from sklearn.modelselection import traintestsplit

import yaml

import os

Load params

with open("params.yaml") as f:

params = yaml.safeload(f)["prepare"]

Load raw data

df = pd.readcsv("data/raw/dataset.csv")

Split data

train, test = traintestsplit(

df,

testsize=params["testsize"],

randomstate=params["randomstate"]

)

Save processed data

os.makedirs("data/processed", existok=True)

train.tocsv("data/processed/train.csv", index=False)

test.tocsv("data/processed/test.csv", index=False)

print(f"Train: {len(train)}, Test: {len(test)}")

# src/train.py

import pandas as pd

from sklearn.ensemble import RandomForestClassifier

import joblib

import yaml

import json

import os

Load params

with open("params.yaml") as f:

params = yaml.safeload(f)["train"]

Load data

train = pd.readcsv("data/processed/train.csv")

Xtrain = train.drop("target", axis=1)

ytrain = train["target"]

Train model

model = RandomForestClassifier(

nestimators=params["nestimators"],

maxdepth=params["maxdepth"],

randomstate=params["randomstate"]

)

model.fit(Xtrain, ytrain)

Save model

os.makedirs("models", existok=True)

joblib.dump(model, "models/model.pkl")

Save training metrics

trainscore = model.score(Xtrain, ytrain)

os.makedirs("metrics", existok=True)

with open("metrics/trainmetrics.json", "w") as f:

json.dump({"trainaccuracy": trainscore}, f, indent=2)

print(f"Train accuracy: {trainscore:.4f}")

# src/evaluate.py

import pandas as pd

from sklearn.metrics import accuracyscore, f1score, confusionmatrix

import joblib

import json

import matplotlib.pyplot as plt

import seaborn as sns

import os

Load model and data

model = joblib.load("models/model.pkl")

test = pd.readcsv("data/processed/test.csv")

Xtest = test.drop("target", axis=1)

ytest = test["target"]

Predict

ypred = model.predict(Xtest)

Calculate metrics

accuracy = accuracyscore(ytest, ypred)

f1 = f1score(ytest, ypred, average="weighted")

Save metrics

os.makedirs("metrics", existok=True)

with open("metrics/evalmetrics.json", "w") as f:

json.dump({

"accuracy": accuracy,

"f1score": f1

}, f, indent=2)

Save confusion matrix plot

os.makedirs("plots", existok=True)

cm = confusionmatrix(ytest, ypred)

plt.figure(figsize=(8, 6))

sns.heatmap(cm, annot=True, fmt="d", cmap="Blues")

plt.xlabel("Predicted")

plt.ylabel("Actual")

plt.title("Confusion Matrix")

plt.savefig("plots/confusionmatrix.png", dpi=100, bboxinches="tight")

print(f"Accuracy: {accuracy:.4f}, F1: {f1:.4f}")

4. Run Pipeline

# Run entire pipeline

dvc repro

Run specific stage

dvc repro train

Run dengan force (ignore cache)

dvc repro -f

Dry run (see what would run)

dvc repro --dry

Visualize pipeline

dvc dag

5. Pipeline Visualization

# Show DAG as text

dvc dag

Output:

+----------+

| prepare |

+----------+

+----------+

| train |

+----------+

+----------+

| evaluate |

+----------+

Experiment Management

1. Run Experiments

# Run experiment dengan parameter berbeda

dvc exp run -S train.nestimators=200

Multiple parameters

dvc exp run -S train.nestimators=200 -S train.maxdepth=10

Queue experiments

dvc exp run --queue -S train.nestimators=100

dvc exp run --queue -S train.nestimators=200

dvc exp run --queue -S train.nestimators=300

Run queued experiments

dvc exp run --run-all

Run in parallel

dvc exp run --run-all --parallel 4

2. Compare Experiments

# List experiments

dvc exp show

Output table dengan metrics

dvc exp show --md

Compare specific experiments

dvc exp diff exp-abc123 exp-def456

Show plots

dvc plots show

3. Apply Best Experiment

# Apply experiment ke workspace

dvc exp apply exp-abc123

Branch dari experiment

dvc exp branch exp-abc123 best-model

Remove experiments

dvc exp remove exp-abc123

dvc exp gc --workspace # Clean up

Remote Storage Configuration

1. AWS S3

# Add remote

dvc remote add -d s3remote s3://mybucket/dvc

Configure credentials (method 1: AWS profile)

dvc remote modify s3remote profile myprofile

Method 2: Environment variables

export AWSACCESSKEYID=your-key

export AWSSECRETACCESSKEY=your-secret

Method 3: Direct config (not recommended for secrets)

dvc remote modify s3remote accesskeyid your-key

dvc remote modify s3remote secretaccesskey your-secret

Use specific region

dvc remote modify s3remote region us-west-2

2. Google Cloud Storage

# Add remote

dvc remote add -d gcsremote gs://mybucket/dvc

Configure credentials

export GOOGLEAPPLICATIONCREDENTIALS=/path/to/credentials.json

Or use project

dvc remote modify gcsremote projectname myproject

3. Azure Blob Storage

# Add remote

dvc remote add -d azureremote azure://mycontainer/dvc

Configure

dvc remote modify azureremote accountname myaccount

dvc remote modify azureremote accountkey mykey

Or use connection string

dvc remote modify azureremote connectionstring "DefaultEndpointsProtocol=..."

4. SSH/SFTP

# Add remote

dvc remote add -d sshremote ssh://user@server.com/path/to/storage

Configure key

dvc remote modify sshremote keyfile ~/.ssh/idrsa

Or password (not recommended)

dvc remote modify sshremote password mypassword

Python API

1. Basic Operations

import dvc.api

Get URL of tracked file

url = dvc.api.geturl("data/dataset.csv", repo=".")

print(url)

Read file directly

with dvc.api.open(

"data/dataset.csv",

repo="https://github.com/user/repo",

rev="v1.0"

) as f:

data = f.read()

Get params

params = dvc.api.paramsshow()

print(params)

2. Programmatic Pipeline

import subprocess

def rundvcpipeline():

"""Run DVC pipeline programmatically"""

result = subprocess.run(

["dvc", "repro"],

captureoutput=True,

text=True

)

print(result.stdout)

if result.returncode != 0:

print(f"Error: {result.stderr}")

return False

return True

def runexperiment(params: dict):

"""Run experiment with custom params"""

paramargs = []

for key, value in params.items():

paramargs.extend(["-S", f"{key}={value}"])

result = subprocess.run(

["dvc", "exp", "run"] + paramargs,

captureoutput=True,

text=True

)

return result.returncode == 0

CI/CD Integration

1. GitHub Actions

# .github/workflows/ml-pipeline.yml

name: ML Pipeline

on:

push:

branches: [main]

pullrequest:

branches: [main]

jobs:

train:

runs-on: ubuntu-latest

steps:

  • uses: actions/checkout@v3
  • name: Setup Python
uses: actions/setup-python@v4

with:

python-version: "3.10"

  • name: Install dependencies
run: |

pip install -r requirements.txt

pip install dvc[s3]

  • name: Configure DVC
env:

AWSACCESSKEYID: ${{ secrets.AWSACCESSKEYID }}

AWSSECRETACCESSKEY: ${{ secrets.AWSSECRETACCESSKEY }}

run: |

dvc remote modify s3remote --local accesskeyid $AWSACCESSKEYID

dvc remote modify s3remote --local secretaccesskey $AWSSECRETACCESSKEY

  • name: Pull data
run: dvc pull

  • name: Run pipeline
run: dvc repro

  • name: Push results
run: dvc push

  • name: Show metrics
run: dvc metrics show

2. GitLab CI

# .gitlab-ci.yml

stages:

  • train
  • evaluate

variables:

PIPCACHEDIR: "$CIPROJECTDIR/.pip-cache"

train:

stage: train

image: python:3.10

cache:

paths:

  • .pip-cache/
before
script:

  • pip install -r requirements.txt
  • pip install dvc[s3]
script:

  • dvc pull
  • dvc repro train
  • dvc push
artifacts:

paths:

  • models/
  • metrics/

evaluate:

stage: evaluate

image: python:3.10

dependencies:

  • train
script:

  • pip install -r requirements.txt
  • pip install dvc[s3]
  • dvc repro evaluate
  • dvc metrics show

Best Practices

1. Project Structure

my-ml-project/

├── .dvc/

│ ├── config

│ └── .gitignore

├── .github/

│ └── workflows/

│ └── ml-pipeline.yml

├── data/

│ ├── raw/

│ │ └── dataset.csv.dvc

│ └── processed/

├── src/

│ ├── prepare.py

│ ├── train.py

│ └── evaluate.py

├── models/

├── metrics/

├── plots/

├── dvc.yaml

├── dvc.lock

├── params.yaml

├── requirements.txt

└── README.md

2. .gitignore dan .dvcignore

# .gitignore

/data/raw/.csv

/data/processed/

/models/.pkl

pycache/

.env

# .dvcignore
.tmp

.log

pycache

.ipynb_checkpoints

3. Tips

# Check status

dvc status

Clean unused cache

dvc gc -w # Workspace only

dvc gc -a # All branches

Import data dari repo lain

dvc import https://github.com/user/repo data/dataset.csv

Update imported data

dvc update dataset.csv.dvc

Fetch tanpa checkout

dvc fetch

Kesimpulan

DVC adalah tool essential untuk MLOps yang memungkinkan:

  • Version Control: Track data dan model seperti code
  • Reproducibility: Pipeline yang reproducible
  • Collaboration: Share data via remote storage
  • Experiment Tracking: Compare dan manage experiments
  • CI/CD Integration: Automate ML workflows
  • Key takeaways:

    • Gunakan DVC untuk semua file besar (>100MB)
    • Definisikan pipeline di dvc.yaml
    • Setup remote storage untuk collaboration
    • Integrate dengan CI/CD untuk automation
    • Gunakan experiments untuk hyperparameter tuning

    Artikel Terkait

    Tutorial MLflow: Platform Open-Source untuk MLOps End-to-End

    Tutorial MLflow: Platform Open-Source untuk MLOps End-to-End MLflow adalah platform open-source yang dikembangkan oleh D...

    ZenML: Bikin Pipeline ML dan LLM yang Portable dan Siap Produksi

    ZenML: Bikin Pipeline ML dan LLM yang Portable dan Siap Produksi Halo temen-temen, ketemu lagi sama aku, Ruby Abdullah. ...

    Tutorial Lengkap Replicate: Menjalankan dan Deploy Model ML via API

    Tutorial Lengkap Replicate: Menjalankan dan Deploy Model ML via API Replicate adalah platform cloud yang memungkinkan An...

    Tutorial Lengkap Comet ML: Platform MLOps untuk Experiment Tracking dan Model Management

    Tutorial Lengkap Comet ML: Platform MLOps untuk Experiment Tracking dan Model Management Dalam dunia machine learning mo...