Tutorial Lengkap DVC: Git untuk Data dan Model Machine Learning
DVC (Data Version Control) adalah open-source tool yang memperluas Git untuk menangani data, model, dan eksperimen ML. Dengan DVC, Anda dapat melakukan version control pada dataset besar dan model tanpa menyimpannya langsung di Git repository.
Mengapa DVC?
Masalah umum dalam ML projects:
- Large files: Git tidak efisien untuk file besar (datasets, models)
- Reproducibility: Sulit mereproduksi eksperimen tanpa tracking data version
- Collaboration: Berbagi dataset besar antar tim sulit
- Storage: Dataset besar memakan space di Git repository
- Version control untuk data dan model
- Remote storage (S3, GCS, Azure, SSH)
- Pipeline reproducibility
- Experiment tracking
- Integrasi seamless dengan Git
Instalasi
# Install DVC
pip install dvc
Install dengan remote storage support
pip install "dvc[s3]" # AWS S3
pip install "dvc[gs]" # Google Cloud Storage
pip install "dvc[azure]" # Azure Blob Storage
pip install "dvc[ssh]" # SSH/SFTP
pip install "dvc[all]" # Semua remotes
Verify installation
dvc version
Quick Start
1. Initialize DVC
# Pastikan sudah di Git repository
git init
dvc init
DVC akan membuat:
- .dvc/ directory
- .dvcignore file
git status
.dvc/.gitignore
.dvc/config
.dvcignore
git add .dvc .dvcignore
git commit -m "Initialize DVC"
2. Track Data Files
# Tambahkan file data ke DVC
dvc add data/dataset.csv
DVC akan membuat:
- data/dataset.csv.dvc (metadata file)
- Update .gitignore
Add .dvc file ke Git
git add data/dataset.csv.dvc data/.gitignore
git commit -m "Add dataset"
Untuk folder
dvc add data/images/
git add data/images.dvc data/.gitignore
git commit -m "Add images folder"
3. Setup Remote Storage
# AWS S3
dvc remote add -d myremote s3://mybucket/dvc-storage
Google Cloud Storage
dvc remote add -d myremote gs://mybucket/dvc-storage
Azure Blob Storage
dvc remote add -d myremote azure://mycontainer/dvc-storage
Local/Network storage
dvc remote add -d myremote /path/to/storage
SSH
dvc remote add -d myremote ssh://user@server/path/to/storage
Commit remote config
git add .dvc/config
git commit -m "Configure remote storage"
4. Push dan Pull Data
# Push data ke remote
dvc push
Pull data dari remote
dvc pull
Push/pull specific file
dvc push data/dataset.csv.dvc
dvc pull data/dataset.csv.dvc
Working with Data Versions
1. Update Data
# Setelah data berubah
dvc add data/dataset.csv
git add data/dataset.csv.dvc
git commit -m "Update dataset v2"
dvc push
2. Switch Data Versions
# Checkout ke versi sebelumnya
git checkout HEAD~1 data/dataset.csv.dvc
dvc checkout
Kembali ke latest
git checkout HEAD data/dataset.csv.dvc
dvc checkout
3. View Data History
# Lihat history
git log --oneline data/dataset.csv.dvc
Diff antara versi
dvc diff HEAD~1
Metrics diff
dvc metrics diff HEAD~1
DVC Pipelines
1. Membuat Pipeline
Pipeline didefinisikan dalam dvc.yaml:
# dvc.yaml
stages:
prepare:
cmd: python src/prepare.py
deps:
- src/prepare.py
- data/raw/
outs:
- data/processed/
train:
cmd: python src/train.py
deps:
- src/train.py
- data/processed/
params:
- train.nestimators
- train.maxdepth
outs:
- models/model.pkl
metrics:
- metrics/trainmetrics.json:
cache: false
evaluate:
cmd: python src/evaluate.py
deps:
- src/evaluate.py
- models/model.pkl
- data/processed/test.csv
metrics:
- metrics/evalmetrics.json:
cache: false
plots:
- plots/confusionmatrix.png
2. Parameters File
# params.yaml
prepare:
testsize: 0.2
randomstate: 42
train:
nestimators: 100
maxdepth: 5
randomstate: 42
evaluate:
threshold: 0.5
3. Implementasi Scripts
# src/prepare.py
import pandas as pd
from sklearn.modelselection import traintestsplit
import yaml
import os
Load params
with open("params.yaml") as f:
params = yaml.safeload(f)["prepare"]
Load raw data
df = pd.readcsv("data/raw/dataset.csv")
Split data
train, test = traintestsplit(
df,
testsize=params["testsize"],
randomstate=params["randomstate"]
)
Save processed data
os.makedirs("data/processed", existok=True)
train.tocsv("data/processed/train.csv", index=False)
test.tocsv("data/processed/test.csv", index=False)
print(f"Train: {len(train)}, Test: {len(test)}")
# src/train.py
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
import joblib
import yaml
import json
import os
Load params
with open("params.yaml") as f:
params = yaml.safeload(f)["train"]
Load data
train = pd.readcsv("data/processed/train.csv")
Xtrain = train.drop("target", axis=1)
ytrain = train["target"]
Train model
model = RandomForestClassifier(
nestimators=params["nestimators"],
maxdepth=params["maxdepth"],
randomstate=params["randomstate"]
)
model.fit(Xtrain, ytrain)
Save model
os.makedirs("models", existok=True)
joblib.dump(model, "models/model.pkl")
Save training metrics
trainscore = model.score(Xtrain, ytrain)
os.makedirs("metrics", existok=True)
with open("metrics/trainmetrics.json", "w") as f:
json.dump({"trainaccuracy": trainscore}, f, indent=2)
print(f"Train accuracy: {trainscore:.4f}")
# src/evaluate.py
import pandas as pd
from sklearn.metrics import accuracyscore, f1score, confusionmatrix
import joblib
import json
import matplotlib.pyplot as plt
import seaborn as sns
import os
Load model and data
model = joblib.load("models/model.pkl")
test = pd.readcsv("data/processed/test.csv")
Xtest = test.drop("target", axis=1)
ytest = test["target"]
Predict
ypred = model.predict(Xtest)
Calculate metrics
accuracy = accuracyscore(ytest, ypred)
f1 = f1score(ytest, ypred, average="weighted")
Save metrics
os.makedirs("metrics", existok=True)
with open("metrics/evalmetrics.json", "w") as f:
json.dump({
"accuracy": accuracy,
"f1score": f1
}, f, indent=2)
Save confusion matrix plot
os.makedirs("plots", existok=True)
cm = confusionmatrix(ytest, ypred)
plt.figure(figsize=(8, 6))
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues")
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.title("Confusion Matrix")
plt.savefig("plots/confusionmatrix.png", dpi=100, bboxinches="tight")
print(f"Accuracy: {accuracy:.4f}, F1: {f1:.4f}")
4. Run Pipeline
# Run entire pipeline
dvc repro
Run specific stage
dvc repro train
Run dengan force (ignore cache)
dvc repro -f
Dry run (see what would run)
dvc repro --dry
Visualize pipeline
dvc dag
5. Pipeline Visualization
# Show DAG as text
dvc dag
Output:
+----------+
| prepare |
+----------+
+----------+
| train |
+----------+
+----------+
| evaluate |
+----------+
Experiment Management
1. Run Experiments
# Run experiment dengan parameter berbeda
dvc exp run -S train.nestimators=200
Multiple parameters
dvc exp run -S train.nestimators=200 -S train.maxdepth=10
Queue experiments
dvc exp run --queue -S train.nestimators=100
dvc exp run --queue -S train.nestimators=200
dvc exp run --queue -S train.nestimators=300
Run queued experiments
dvc exp run --run-all
Run in parallel
dvc exp run --run-all --parallel 4
2. Compare Experiments
# List experiments
dvc exp show
Output table dengan metrics
dvc exp show --md
Compare specific experiments
dvc exp diff exp-abc123 exp-def456
Show plots
dvc plots show
3. Apply Best Experiment
# Apply experiment ke workspace
dvc exp apply exp-abc123
Branch dari experiment
dvc exp branch exp-abc123 best-model
Remove experiments
dvc exp remove exp-abc123
dvc exp gc --workspace # Clean up
Remote Storage Configuration
1. AWS S3
# Add remote
dvc remote add -d s3remote s3://mybucket/dvc
Configure credentials (method 1: AWS profile)
dvc remote modify s3remote profile myprofile
Method 2: Environment variables
export AWSACCESSKEYID=your-key
export AWSSECRETACCESSKEY=your-secret
Method 3: Direct config (not recommended for secrets)
dvc remote modify s3remote accesskeyid your-key
dvc remote modify s3remote secretaccesskey your-secret
Use specific region
dvc remote modify s3remote region us-west-2
2. Google Cloud Storage
# Add remote
dvc remote add -d gcsremote gs://mybucket/dvc
Configure credentials
export GOOGLEAPPLICATIONCREDENTIALS=/path/to/credentials.json
Or use project
dvc remote modify gcsremote projectname myproject
3. Azure Blob Storage
# Add remote
dvc remote add -d azureremote azure://mycontainer/dvc
Configure
dvc remote modify azureremote accountname myaccount
dvc remote modify azureremote accountkey mykey
Or use connection string
dvc remote modify azureremote connectionstring "DefaultEndpointsProtocol=..."
4. SSH/SFTP
# Add remote
dvc remote add -d sshremote ssh://user@server.com/path/to/storage
Configure key
dvc remote modify sshremote keyfile ~/.ssh/idrsa
Or password (not recommended)
dvc remote modify sshremote password mypassword
Python API
1. Basic Operations
import dvc.api
Get URL of tracked file
url = dvc.api.geturl("data/dataset.csv", repo=".")
print(url)
Read file directly
with dvc.api.open(
"data/dataset.csv",
repo="https://github.com/user/repo",
rev="v1.0"
) as f:
data = f.read()
Get params
params = dvc.api.paramsshow()
print(params)
2. Programmatic Pipeline
import subprocess
def rundvcpipeline():
"""Run DVC pipeline programmatically"""
result = subprocess.run(
["dvc", "repro"],
captureoutput=True,
text=True
)
print(result.stdout)
if result.returncode != 0:
print(f"Error: {result.stderr}")
return False
return True
def runexperiment(params: dict):
"""Run experiment with custom params"""
paramargs = []
for key, value in params.items():
paramargs.extend(["-S", f"{key}={value}"])
result = subprocess.run(
["dvc", "exp", "run"] + paramargs,
captureoutput=True,
text=True
)
return result.returncode == 0
CI/CD Integration
1. GitHub Actions
# .github/workflows/ml-pipeline.yml
name: ML Pipeline
on:
push:
branches: [main]
pullrequest:
branches: [main]
jobs:
train:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: "3.10"
- name: Install dependencies
run: |
pip install -r requirements.txt
pip install dvc[s3]
- name: Configure DVC
env:
AWSACCESSKEYID: ${{ secrets.AWSACCESSKEYID }}
AWSSECRETACCESSKEY: ${{ secrets.AWSSECRETACCESSKEY }}
run: |
dvc remote modify s3remote --local accesskeyid $AWSACCESSKEYID
dvc remote modify s3remote --local secretaccesskey $AWSSECRETACCESSKEY
- name: Pull data
run: dvc pull
- name: Run pipeline
run: dvc repro
- name: Push results
run: dvc push
- name: Show metrics
run: dvc metrics show
2. GitLab CI
# .gitlab-ci.yml
stages:
- train
- evaluate
variables:
PIPCACHEDIR: "$CIPROJECTDIR/.pip-cache"
train:
stage: train
image: python:3.10
cache:
paths:
- .pip-cache/
beforescript:
- pip install -r requirements.txt
- pip install dvc[s3]
script:
- dvc pull
- dvc repro train
- dvc push
artifacts:
paths:
- models/
- metrics/
evaluate:
stage: evaluate
image: python:3.10
dependencies:
- train
script:
- pip install -r requirements.txt
- pip install dvc[s3]
- dvc repro evaluate
- dvc metrics show
Best Practices
1. Project Structure
my-ml-project/
├── .dvc/
│ ├── config
│ └── .gitignore
├── .github/
│ └── workflows/
│ └── ml-pipeline.yml
├── data/
│ ├── raw/
│ │ └── dataset.csv.dvc
│ └── processed/
├── src/
│ ├── prepare.py
│ ├── train.py
│ └── evaluate.py
├── models/
├── metrics/
├── plots/
├── dvc.yaml
├── dvc.lock
├── params.yaml
├── requirements.txt
└── README.md
2. .gitignore dan .dvcignore
# .gitignore
/data/raw/.csv
/data/processed/
/models/.pkl
pycache/
.env
# .dvcignore
.tmp
.log
pycache
.ipynb_checkpoints
3. Tips
# Check status
dvc status
Clean unused cache
dvc gc -w # Workspace only
dvc gc -a # All branches
Import data dari repo lain
dvc import https://github.com/user/repo data/dataset.csv
Update imported data
dvc update dataset.csv.dvc
Fetch tanpa checkout
dvc fetch
Kesimpulan
DVC adalah tool essential untuk MLOps yang memungkinkan:
Key takeaways:
- Gunakan DVC untuk semua file besar (>100MB)
- Definisikan pipeline di
dvc.yaml - Setup remote storage untuk collaboration
- Integrate dengan CI/CD untuk automation
- Gunakan experiments untuk hyperparameter tuning