Complete DVC Tutorial: Git for Data and Machine Learning Models

# Tutorial Lengkap DVC: Git untuk Data dan Model Machine Learning DVC (Data Version Control) adalah open-source tool yang memperluas Git untuk menangani data, model, dan eksperimen ML. Dengan DVC, An...

By Ruby Abdullah · · tutorial
DVCMLOpsData Version ControlMachine LearningGitPipeline

Complete DVC Tutorial: Git for Data and Machine Learning Models

DVC (Data Version Control) is an open-source tool that extends Git to handle data, models, and ML experiments. With DVC, you can version control large datasets and models without storing them directly in your Git repository.

Why DVC?

Common problems in ML projects:

  • Large files: Git is not efficient for large files (datasets, models)
  • Reproducibility: Difficult to reproduce experiments without data version tracking
  • Collaboration: Sharing large datasets across teams is challenging
  • Storage: Large datasets consume space in Git repository

DVC Solutions:
  • Version control for data and models
  • Remote storage (S3, GCS, Azure, SSH)
  • Pipeline reproducibility
  • Experiment tracking
  • Seamless integration with Git

Installation

# Install DVC

pip install dvc

Install with remote storage support

pip install "dvc[s3]" # AWS S3

pip install "dvc[gs]" # Google Cloud Storage

pip install "dvc[azure]" # Azure Blob Storage

pip install "dvc[ssh]" # SSH/SFTP

pip install "dvc[all]" # All remotes

Verify installation

dvc version

Quick Start

1. Initialize DVC

# Make sure you're in a Git repository

git init

dvc init

DVC will create:

- .dvc/ directory

- .dvcignore file

git status

.dvc/.gitignore

.dvc/config

.dvcignore

git add .dvc .dvcignore

git commit -m "Initialize DVC"

2. Track Data Files

# Add data file to DVC

dvc add data/dataset.csv

DVC will create:

- data/dataset.csv.dvc (metadata file)

- Update .gitignore

Add .dvc file to Git

git add data/dataset.csv.dvc data/.gitignore

git commit -m "Add dataset"

For folders

dvc add data/images/

git add data/images.dvc data/.gitignore

git commit -m "Add images folder"

3. Setup Remote Storage

# AWS S3

dvc remote add -d myremote s3://mybucket/dvc-storage

Google Cloud Storage

dvc remote add -d myremote gs://mybucket/dvc-storage

Azure Blob Storage

dvc remote add -d myremote azure://mycontainer/dvc-storage

Local/Network storage

dvc remote add -d myremote /path/to/storage

SSH

dvc remote add -d myremote ssh://user@server/path/to/storage

Commit remote config

git add .dvc/config

git commit -m "Configure remote storage"

4. Push and Pull Data

# Push data to remote

dvc push

Pull data from remote

dvc pull

Push/pull specific file

dvc push data/dataset.csv.dvc

dvc pull data/dataset.csv.dvc

Working with Data Versions

1. Update Data

# After data changes

dvc add data/dataset.csv

git add data/dataset.csv.dvc

git commit -m "Update dataset v2"

dvc push

2. Switch Data Versions

# Checkout to previous version

git checkout HEAD~1 data/dataset.csv.dvc

dvc checkout

Return to latest

git checkout HEAD data/dataset.csv.dvc

dvc checkout

3. View Data History

# View history

git log --oneline data/dataset.csv.dvc

Diff between versions

dvc diff HEAD~1

Metrics diff

dvc metrics diff HEAD~1

DVC Pipelines

1. Creating a Pipeline

Pipeline is defined in dvc.yaml:

# dvc.yaml

stages:

prepare:

cmd: python src/prepare.py

deps:

  • src/prepare.py
  • data/raw/
outs:

  • data/processed/

train:

cmd: python src/train.py

deps:

  • src/train.py
  • data/processed/
params:

  • train.nestimators
  • train.maxdepth
outs:

  • models/model.pkl
metrics:

  • metrics/trainmetrics.json:
cache: false

evaluate:

cmd: python src/evaluate.py

deps:

  • src/evaluate.py
  • models/model.pkl
  • data/processed/test.csv
metrics:

  • metrics/evalmetrics.json:
cache: false

plots:

  • plots/confusionmatrix.png

2. Parameters File

# params.yaml

prepare:

testsize: 0.2

randomstate: 42

train:

nestimators: 100

maxdepth: 5

randomstate: 42

evaluate:

threshold: 0.5

3. Implementing Scripts

# src/prepare.py

import pandas as pd

from sklearn.modelselection import traintestsplit

import yaml

import os

Load params

with open("params.yaml") as f:

params = yaml.safeload(f)["prepare"]

Load raw data

df = pd.readcsv("data/raw/dataset.csv")

Split data

train, test = traintestsplit(

df,

testsize=params["testsize"],

randomstate=params["randomstate"]

)

Save processed data

os.makedirs("data/processed", existok=True)

train.tocsv("data/processed/train.csv", index=False)

test.tocsv("data/processed/test.csv", index=False)

print(f"Train: {len(train)}, Test: {len(test)}")

# src/train.py

import pandas as pd

from sklearn.ensemble import RandomForestClassifier

import joblib

import yaml

import json

import os

Load params

with open("params.yaml") as f:

params = yaml.safeload(f)["train"]

Load data

train = pd.readcsv("data/processed/train.csv")

Xtrain = train.drop("target", axis=1)

ytrain = train["target"]

Train model

model = RandomForestClassifier(

nestimators=params["nestimators"],

maxdepth=params["maxdepth"],

randomstate=params["randomstate"]

)

model.fit(Xtrain, ytrain)

Save model

os.makedirs("models", existok=True)

joblib.dump(model, "models/model.pkl")

Save training metrics

trainscore = model.score(Xtrain, ytrain)

os.makedirs("metrics", existok=True)

with open("metrics/trainmetrics.json", "w") as f:

json.dump({"trainaccuracy": trainscore}, f, indent=2)

print(f"Train accuracy: {trainscore:.4f}")

# src/evaluate.py

import pandas as pd

from sklearn.metrics import accuracyscore, f1score, confusionmatrix

import joblib

import json

import matplotlib.pyplot as plt

import seaborn as sns

import os

Load model and data

model = joblib.load("models/model.pkl")

test = pd.readcsv("data/processed/test.csv")

Xtest = test.drop("target", axis=1)

ytest = test["target"]

Predict

ypred = model.predict(Xtest)

Calculate metrics

accuracy = accuracyscore(ytest, ypred)

f1 = f1score(ytest, ypred, average="weighted")

Save metrics

os.makedirs("metrics", existok=True)

with open("metrics/evalmetrics.json", "w") as f:

json.dump({

"accuracy": accuracy,

"f1score": f1

}, f, indent=2)

Save confusion matrix plot

os.makedirs("plots", existok=True)

cm = confusionmatrix(ytest, ypred)

plt.figure(figsize=(8, 6))

sns.heatmap(cm, annot=True, fmt="d", cmap="Blues")

plt.xlabel("Predicted")

plt.ylabel("Actual")

plt.title("Confusion Matrix")

plt.savefig("plots/confusionmatrix.png", dpi=100, bboxinches="tight")

print(f"Accuracy: {accuracy:.4f}, F1: {f1:.4f}")

4. Run Pipeline

# Run entire pipeline

dvc repro

Run specific stage

dvc repro train

Run with force (ignore cache)

dvc repro -f

Dry run (see what would run)

dvc repro --dry

Visualize pipeline

dvc dag

5. Pipeline Visualization

# Show DAG as text

dvc dag

Output:

+----------+

| prepare |

+----------+

+----------+

| train |

+----------+

+----------+

| evaluate |

+----------+

Experiment Management

1. Run Experiments

# Run experiment with different parameters

dvc exp run -S train.nestimators=200

Multiple parameters

dvc exp run -S train.nestimators=200 -S train.maxdepth=10

Queue experiments

dvc exp run --queue -S train.nestimators=100

dvc exp run --queue -S train.nestimators=200

dvc exp run --queue -S train.nestimators=300

Run queued experiments

dvc exp run --run-all

Run in parallel

dvc exp run --run-all --parallel 4

2. Compare Experiments

# List experiments

dvc exp show

Output table with metrics

dvc exp show --md

Compare specific experiments

dvc exp diff exp-abc123 exp-def456

Show plots

dvc plots show

3. Apply Best Experiment

# Apply experiment to workspace

dvc exp apply exp-abc123

Branch from experiment

dvc exp branch exp-abc123 best-model

Remove experiments

dvc exp remove exp-abc123

dvc exp gc --workspace # Clean up

Remote Storage Configuration

1. AWS S3

# Add remote

dvc remote add -d s3remote s3://mybucket/dvc

Configure credentials (method 1: AWS profile)

dvc remote modify s3remote profile myprofile

Method 2: Environment variables

export AWSACCESSKEYID=your-key

export AWSSECRETACCESSKEY=your-secret

Method 3: Direct config (not recommended for secrets)

dvc remote modify s3remote accesskeyid your-key

dvc remote modify s3remote secretaccesskey your-secret

Use specific region

dvc remote modify s3remote region us-west-2

2. Google Cloud Storage

# Add remote

dvc remote add -d gcsremote gs://mybucket/dvc

Configure credentials

export GOOGLEAPPLICATIONCREDENTIALS=/path/to/credentials.json

Or use project

dvc remote modify gcsremote projectname myproject

3. Azure Blob Storage

# Add remote

dvc remote add -d azureremote azure://mycontainer/dvc

Configure

dvc remote modify azureremote accountname myaccount

dvc remote modify azureremote accountkey mykey

Or use connection string

dvc remote modify azureremote connectionstring "DefaultEndpointsProtocol=..."

4. SSH/SFTP

# Add remote

dvc remote add -d sshremote ssh://user@server.com/path/to/storage

Configure key

dvc remote modify sshremote keyfile ~/.ssh/idrsa

Or password (not recommended)

dvc remote modify sshremote password mypassword

Python API

1. Basic Operations

import dvc.api

Get URL of tracked file

url = dvc.api.geturl("data/dataset.csv", repo=".")

print(url)

Read file directly

with dvc.api.open(

"data/dataset.csv",

repo="https://github.com/user/repo",

rev="v1.0"

) as f:

data = f.read()

Get params

params = dvc.api.paramsshow()

print(params)

2. Programmatic Pipeline

import subprocess

def rundvcpipeline():

"""Run DVC pipeline programmatically"""

result = subprocess.run(

["dvc", "repro"],

captureoutput=True,

text=True

)

print(result.stdout)

if result.returncode != 0:

print(f"Error: {result.stderr}")

return False

return True

def runexperiment(params: dict):

"""Run experiment with custom params"""

paramargs = []

for key, value in params.items():

paramargs.extend(["-S", f"{key}={value}"])

result = subprocess.run(

["dvc", "exp", "run"] + paramargs,

captureoutput=True,

text=True

)

return result.returncode == 0

CI/CD Integration

1. GitHub Actions

# .github/workflows/ml-pipeline.yml

name: ML Pipeline

on:

push:

branches: [main]

pullrequest:

branches: [main]

jobs:

train:

runs-on: ubuntu-latest

steps:

  • uses: actions/checkout@v3
  • name: Setup Python
uses: actions/setup-python@v4

with:

python-version: "3.10"

  • name: Install dependencies
run: |

pip install -r requirements.txt

pip install dvc[s3]

  • name: Configure DVC
env:

AWSACCESSKEYID: ${{ secrets.AWSACCESSKEYID }}

AWSSECRETACCESSKEY: ${{ secrets.AWSSECRETACCESSKEY }}

run: |

dvc remote modify s3remote --local accesskeyid $AWSACCESSKEYID

dvc remote modify s3remote --local secretaccesskey $AWSSECRETACCESSKEY

  • name: Pull data
run: dvc pull

  • name: Run pipeline
run: dvc repro

  • name: Push results
run: dvc push

  • name: Show metrics
run: dvc metrics show

2. GitLab CI

# .gitlab-ci.yml

stages:

  • train
  • evaluate

variables:

PIPCACHEDIR: "$CIPROJECTDIR/.pip-cache"

train:

stage: train

image: python:3.10

cache:

paths:

  • .pip-cache/
before
script:

  • pip install -r requirements.txt
  • pip install dvc[s3]
script:

  • dvc pull
  • dvc repro train
  • dvc push
artifacts:

paths:

  • models/
  • metrics/

evaluate:

stage: evaluate

image: python:3.10

dependencies:

  • train
script:

  • pip install -r requirements.txt
  • pip install dvc[s3]
  • dvc repro evaluate
  • dvc metrics show

Best Practices

1. Project Structure

my-ml-project/

├── .dvc/

│ ├── config

│ └── .gitignore

├── .github/

│ └── workflows/

│ └── ml-pipeline.yml

├── data/

│ ├── raw/

│ │ └── dataset.csv.dvc

│ └── processed/

├── src/

│ ├── prepare.py

│ ├── train.py

│ └── evaluate.py

├── models/

├── metrics/

├── plots/

├── dvc.yaml

├── dvc.lock

├── params.yaml

├── requirements.txt

└── README.md

2. .gitignore and .dvcignore

# .gitignore

/data/raw/.csv

/data/processed/

/models/.pkl

pycache/

.env

# .dvcignore
.tmp

.log

pycache

.ipynb_checkpoints

3. Tips

# Check status

dvc status

Clean unused cache

dvc gc -w # Workspace only

dvc gc -a # All branches

Import data from another repo

dvc import https://github.com/user/repo data/dataset.csv

Update imported data

dvc update dataset.csv.dvc

Fetch without checkout

dvc fetch

Conclusion

DVC is an essential tool for MLOps that enables:

  • Version Control: Track data and models like code
  • Reproducibility: Reproducible pipelines
  • Collaboration: Share data via remote storage
  • Experiment Tracking: Compare and manage experiments
  • CI/CD Integration: Automate ML workflows
  • Key takeaways:

    • Use DVC for all large files (>100MB)
    • Define pipelines in dvc.yaml
    • Setup remote storage for collaboration
    • Integrate with CI/CD for automation
    • Use experiments for hyperparameter tuning

    Related Articles

    MLflow Tutorial: Open-Source Platform for End-to-End MLOps

    Tutorial MLflow: Platform Open-Source untuk MLOps End-to-End MLflow adalah platform open-source yang dikembangkan oleh D...

    ZenML: Build Portable, Production-Ready ML and LLM Pipelines

    ZenML: Bikin Pipeline ML dan LLM yang Portable dan Siap Produksi Halo temen-temen, ketemu lagi sama aku, Ruby Abdullah. ...

    Complete Replicate Tutorial: Run and Deploy ML Models via API

    Tutorial Lengkap Replicate: Menjalankan dan Deploy Model ML via API Replicate adalah platform cloud yang memungkinkan An...

    Complete Comet ML Tutorial: MLOps Platform for Experiment Tracking and Model Management

    Tutorial Lengkap Comet ML: Platform MLOps untuk Experiment Tracking dan Model Management Dalam dunia machine learning mo...