Tutorial Lengkap Comet ML: Platform MLOps untuk Experiment Tracking dan Model Management

Dalam dunia machine learning modern, mengelola eksperimen secara manual menjadi tantangan besar seiring bertambahnya kompleksitas proyek. Comet ML hadir sebagai platform MLOps yang memungkinkan data scientist dan ML engineer untuk melacak eksperimen, membandingkan model, dan mengelola lifecycle machine learning secara efisien. Tutorial ini akan membahas cara menggunakan Comet ML dari instalasi hingga fitur-fitur lanjutan untuk proyek ML production.

Apa Itu Comet ML?

Comet ML adalah platform experiment tracking dan model management yang menyediakan dashboard terpusat untuk mencatat parameter, metrik, artefak, dan kode dari setiap eksperimen ML. Berbeda dengan pencatatan manual menggunakan spreadsheet atau log file, Comet ML secara otomatis menangkap semua informasi penting dan menyajikannya dalam antarmuka visual yang intuitif.

Keunggulan utama Comet ML meliputi:

• Automatic logging untuk framework populer seperti PyTorch, TensorFlow, scikit-learn, dan XGBoost
• Experiment comparison dengan visualisasi metrik secara real-time
• Model registry untuk versioning dan deployment tracking
• Artifact management untuk dataset dan model versioning
• Team collaboration dengan fitur sharing dan reporting

Instalasi dan Setup

Instalasi Package

Instal Comet ML menggunakan pip:

pip install cometml

pip install cometml[pytorch]
pip install cometml[tensorflow]

pip install cometml[sklearn]

Konfigurasi API Key

Setelah membuat akun di comet.com, dapatkan API key Anda dari halaman Settings. Ada beberapa cara untuk mengkonfigurasi API key:

export COMETAPIKEY="your-api-key-here"

Buat file .comet.config di home directory:

[comet]
apikey=your-api-key-here

projectname=my-ml-project
workspace=my-workspace

import cometml

cometml.login(apikey="your-api-key-here")

import cometml

experiment = cometml.Experiment(

    projectname="test-project",
    autometriclogging=True,
    autoparamlogging=True,
)

experiment.logparameter("testparam", "hello")
experiment.logmetric("testmetric", 0.95)
experiment.end()

print("Comet ML berhasil dikonfigurasi!")

Basic Usage: Experiment Tracking

Logging Parameters dan Metrics

Berikut contoh dasar penggunaan Comet ML untuk melacak eksperimen klasifikasi sederhana:

import cometml
from sklearn.datasets import loadiris
from sklearn.modelselection import traintestsplit

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracyscore, f1score


experiment = cometml.Experiment(
    projectname="iris-classification",

    autometriclogging=False,

)

X, y = loadiris(returnXy=True)
Xtrain, Xtest, ytrain, ytest = traintestsplit(
    X, y, testsize=0.2, randomstate=42
)

params = {
    "nestimators": 100,

    "maxdepth": 5,
    "minsamplessplit": 2,
    "randomstate": 42,

}
experiment.logparameters(params)

model = RandomForestClassifier(params)

model.fit(Xtrain, ytrain)

ypred = model.predict(Xtest)

accuracy = accuracyscore(ytest, ypred)

f1 = f1score(ytest, ypred, average="weighted")

experiment.logmetric("accuracy", accuracy)

experiment.logmetric("f1score", f1)


experiment.addtag("baseline")
experiment.addtag("random-forest")


experiment.end()
print(f"Accuracy: {accuracy:.4f}, F1: {f1:.4f}")

import cometml
import numpy as np

experiment = cometml.Experiment(projectname="training-loop-demo")

numepochs = 50

for epoch in range(numepochs):
    trainloss = 1.0 / (epoch + 1) + np.random.normal(0, 0.02)

    valloss = 1.0 / (epoch + 1) + np.random.normal(0, 0.05) + 0.1
    trainacc = 1 - trainloss + np.random.normal(0, 0.01)
    valacc = 1 - valloss + np.random.normal(0, 0.02)

    experiment.logmetric("trainloss", trainloss, step=epoch)

    experiment.logmetric("valloss", valloss, step=epoch)
    experiment.logmetric("trainaccuracy", trainacc, step=epoch)

    experiment.logmetric("valaccuracy", valacc, step=epoch)

experiment.end()

import cometml
import matplotlib.pyplot as plt
from sklearn.metrics import confusionmatrix, ConfusionMatrixDisplay
from sklearn.datasets import loadiris

from sklearn.modelselection import traintestsplit
from sklearn.ensemble import RandomForestClassifier

experiment = cometml.Experiment(projectname="visualization-demo")

X, y = loadiris(returnXy=True)

Xtrain, Xtest, ytrain, ytest = traintestsplit(X, y, testsize=0.2)

model = RandomForestClassifier(nestimators=100)

model.fit(Xtrain, ytrain)

ypred = model.predict(Xtest)


cm = confusionmatrix(ytest, ypred)
disp = ConfusionMatrixDisplay(confusionmatrix=cm)

disp.plot(cmap="Blues")
plt.title("Confusion Matrix")

experiment.logfigure(figurename="confusionmatrix", figure=plt)
plt.close()

experiment.logconfusionmatrix(
    ytrue=ytest.tolist(),
    ypredicted=ypred.tolist(),
    labels=["setosa", "versicolor", "virginica"],
)

experiment.end()

import cometml
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset

experiment = cometml.Experiment(
    projectname="pytorch-demo",

    autometriclogging=True,

    autoparamlogging=True,

    loggraph=True,
)

class SimpleNet(nn.Module):
    def init(self, inputdim, hiddendim, outputdim):

        super().init()
        self.fc1 = nn.Linear(inputdim, hiddendim)

        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(0.3)
        self.fc2 = nn.Linear(hiddendim, outputdim)


    def forward(self, x):
        x = self.fc1(x)
        x = self.relu(x)
        x = self.dropout(x)
        x = self.fc2(x)
        return x

params = {
    "inputdim": 20,
    "hiddendim": 64,

    "outputdim": 3,
    "learningrate": 0.001,

    "batchsize": 32,
    "epochs": 20,
}
experiment.logparameters(params)


Xtrain = torch.randn(500, params["inputdim"])

ytrain = torch.randint(0, params["outputdim"], (500,))


dataset = TensorDataset(Xtrain, ytrain)

dataloader = DataLoader(dataset, batchsize=params["batchsize"], shuffle=True)


model = SimpleNet(params["inputdim"], params["hiddendim"], params["outputdim"])
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=params["learningrate"])


for epoch in range(params["epochs"]):
    model.train()
    totalloss = 0
    correct = 0
    total = 0

    for batchX, batchy in dataloader:
        optimizer.zerograd()

        outputs = model(batchX)
        loss = criterion(outputs, batchy)

        loss.backward()
        optimizer.step()

        totalloss += loss.item()
        , predicted = torch.max(outputs, 1)

        total += batchy.size(0)
        correct += (predicted == batchy).sum().item()


    avgloss = totalloss / len(dataloader)

    accuracy = correct / total

    experiment.logmetric("epochloss", avgloss, step=epoch)
    experiment.logmetric("epochaccuracy", accuracy, step=epoch)

experiment.logmodel("simplenet", "./model.pth")
experiment.end()

import cometml
from sklearn.datasets import fetchcaliforniahousing

from sklearn.modelselection import traintestsplit, crossvalscore
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import meansquarederror, r2score

import numpy as np

experiment = cometml.Experiment(
    projectname="sklearn-regression",

    autooutputlogging="native",

)

data = fetchcaliforniahousing()

Xtrain, Xtest, ytrain, ytest = traintestsplit(

    data.data, data.target, testsize=0.2, randomstate=42

)

params = {
    "nestimators": 200,
    "maxdepth": 4,

    "learningrate": 0.1,
    "subsample": 0.8,
    "minsamplesleaf": 10,
}
experiment.logparameters(params)


model = GradientBoostingRegressor(params, randomstate=42)
model.fit(Xtrain, ytrain)

ypred = model.predict(Xtest)
mse = meansquarederror(ytest, ypred)
rmse = np.sqrt(mse)
r2 = r2score(ytest, ypred)


experiment.logmetric("mse", mse)
experiment.logmetric("rmse", rmse)

experiment.logmetric("r2score", r2)


cvscores = crossvalscore(model, data.data, data.target, cv=5, scoring="r2")
experiment.logmetric("cvmeanr2", cvscores.mean())
experiment.logmetric("cvstdr2", cvscores.std())

featureimportance = dict(zip(data.featurenames, model.featureimportances))
experiment.logparameters({"featureimportance": featureimportance})


experiment.end()

import cometml
import xgboost as xgb
from sklearn.datasets import loadbreastcancer
from sklearn.modelselection import traintestsplit

from sklearn.metrics import accuracyscore, rocaucscore

experiment = cometml.Experiment(projectname="xgboost-demo")

data = loadbreastcancer()
Xtrain, Xtest, ytrain, ytest = traintestsplit(
    data.data, data.target, testsize=0.2, randomstate=42
)

dtrain = xgb.DMatrix(Xtrain, label=ytrain)
dtest = xgb.DMatrix(Xtest, label=ytest)

params = {
    "maxdepth": 4,

    "learningrate": 0.1,
    "objective": "binary:logistic",
    "evalmetric": "auc",

    "nestimators": 100,
}
experiment.logparameters(params)


model = xgb.train(
    params,
    dtrain,
    numboostround=params["nestimators"],
    evals=[(dtrain, "train"), (dtest, "eval")],
    verboseeval=10,

)

ypredproba = model.predict(dtest)

ypred = (ypredproba > 0.5).astype(int)

accuracy = accuracyscore(ytest, ypred)

auc = rocaucscore(ytest, ypredproba)

experiment.logmetric("accuracy", accuracy)

experiment.logmetric("auc", auc)
experiment.end()

Advanced Usage

Artifact Management

Comet ML Artifacts memungkinkan Anda melacak versi dataset dan model:

import cometml
from cometml import Artifact
import json

experiment = cometml.Experiment(projectname="artifact-demo")

datasetartifact = Artifact(

    name="training-dataset",
    artifacttype="dataset",
    version="1.0.0",
    metadata={
        "numsamples": 10000,

        "numfeatures": 20,
        "split": "train",
        "preprocessing": "standardscaler",

    },
)

datasetartifact.add("./data/train.csv")
datasetartifact.add("./data/metadata.json")


experiment.logartifact(datasetartifact)


modelartifact = Artifact(
    name="trained-model",
    artifacttype="model",

    version="1.0.0",
    metadata={
        "framework": "pytorch",
        "accuracy": 0.95,
        "architecture": "resnet50",
    },
)
modelartifact.add("./models/model.pth")
modelartifact.add("./models/config.json")


experiment.logartifact(modelartifact)

experiment.end()

import cometml
from cometml import API


api = API()

experiment = cometml.Experiment(projectname="model-registry-demo")


experiment.logparameter("modeltype", "gradientboosting")
experiment.logmetric("accuracy", 0.94)

experiment.logmetric("f1score", 0.93)

experiment.logmodel("mymodel", "./model/")


experiment.registermodel("mymodel", registryname="production-classifier")

experiment.end()

registeredmodel = api.getmodel(
    workspace="my-workspace",
    modelname="production-classifier"

)
print(f"Model versions: {registeredmodel.findversions()}")

import cometml
from cometml import Optimizer

from sklearn.datasets import loaddigits
from sklearn.modelselection import crossvalscore

from sklearn.svm import SVC

config = {
    "algorithm": "bayes",
    "parameters": {
        "C": {
            "type": "float",
            "min": 0.01,
            "max": 100.0,
            "scalingType": "loguniform",
        },
        "gamma": {
            "type": "discrete",
            "values": ["scale", "auto"],
        },
        "kernel": {
            "type": "categorical",
            "values": ["rbf", "poly", "sigmoid"],
        },
    },
    "spec": {
        "maxCombo": 30,
        "objective": "maximize",
        "metric": "cvaccuracy",
    },
}

X, y = loaddigits(returnXy=True)


optimizer = Optimizer(config, projectname="svm-optimization")

for experiment in optimizer.getexperiments():

    C = experiment.getparameter("C")
    gamma = experiment.getparameter("gamma")

    kernel = experiment.getparameter("kernel")

    model = SVC(C=float(C), gamma=gamma, kernel=kernel)

    scores = crossvalscore(model, X, y, cv=5, scoring="accuracy")
    meanaccuracy = scores.mean()


    experiment.logmetric("cvaccuracy", meanaccuracy)
    experiment.logmetric("cvstd", scores.std())
    experiment.end()

Custom Panels dan Visualisasi

import cometml
import matplotlib.pyplot as plt
import numpy as np
from sklearn.metrics import precisionrecallcurve, roccurve

experiment = cometml.Experiment(projectname="custom-panels-demo")

ytrue = np.random.randint(0, 2, 200)

yscores = np.random.random(200)

precision, recall,  = precisionrecallcurve(ytrue, yscores)

fpr, tpr,  = roccurve(ytrue, yscores)


fig, axes = plt.subplots(1, 2, figsize=(14, 5))

axes[0].plot(recall, precision)
axes[0].setxlabel("Recall")
axes[0].setylabel("Precision")

axes[0].settitle("Precision-Recall Curve")
axes[0].grid(True)

axes[1].plot(fpr, tpr)
axes[1].plot([0, 1], [0, 1], "k--")
axes[1].setxlabel("False Positive Rate")

axes[1].setylabel("True Positive Rate")
axes[1].settitle("ROC Curve")

axes[1].grid(True)

plt.tightlayout()
experiment.logfigure(figurename="evaluationcurves", figure=plt)

plt.close()

data = [[x, np.sin(x), np.cos(x)] for x in np.linspace(0, 10, 100)]
experiment.logtable("trigfunctions.csv", tabulardata=data, headers=["x", "sin", "cos"])

experiment.loghtml("
Model Summary
This model achieved 95% accuracy on the test set.
")


experiment.end()

import cometml

offlineexperiment = cometml.OfflineExperiment(
    projectname="offline-project",

    offlinedirectory="./cometoffline",

)

offlineexperiment.logparameter("model", "randomforest")
offlineexperiment.logmetric("accuracy", 0.92)

offlineexperiment.end()

comet upload ./cometoffline/*.zip

Experiment Comparison secara Programatik

from cometml import API

api = API()

experiments = api.get("my-workspace/my-project")

results = []
for exp in experiments:
    metrics = exp.getmetricssummary()

    params = exp.getparameterssummary()


    accuracy = next(
        (m["valueCurrent"] for m in metrics if m["name"] == "accuracy"), None
    )
    results.append({
        "experimentkey": exp.key,
        "name": exp.name,
        "accuracy": accuracy,
    })

results.sort(key=lambda x: float(x["accuracy"] or 0), reverse=True)

print("Top 5 Experiments:")
for i, r in enumerate(results[:5], 1):
    print(f"  {i}. {r['name']}: accuracy={r['accuracy']}")

bestexp = api.getexperimentbykey(results[0]["experimentkey"])

print(f"\nBest experiment: {bestexp.name}")
print(f"URL: {bestexp.url}")

import cometml
from datetime import datetime

experiment = cometml.Experiment(

    projectname="text-classification",
    workspace="my-team",
)

experiment.setname(f"bert-base-lr0001-bs32")


experiment.addtags([
    "bert",
    "text-classification",
    "production-candidate",
])

experiment.logother("datasetversion", "v2.3")
experiment.logother("datasplitseed", 42)

experiment.logother("gputype", "A100")

import cometml

experiment = cometml.Experiment(

    projectname="reproducibility-demo",
    logcode=True,

    logenvdetails=True,

    logenvgpu=True,

    logenvcpu=True,

    loggitmetadata=True,

    loggitpatch=True,

)

experiment.logdependency("torch", "2.1.0")
experiment.logdependency("transformers", "4.35.0")

import cometml

with cometml.Experiment(projectname="context-manager-demo") as experiment:
    experiment.logparameter("learningrate", 0.001)
    experiment.logmetric("accuracy", 0.95)

# .github/workflows/train.yml
name: Model Training
on:
  push:
    branches: [main]

jobs:
  train:
    runs-on: ubuntu-latest
    steps:
uses: actions/checkout@v4
name: Setup Python
        uses: actions/setup-python@v5

        with:
          python-version: "3.11"
name: Install dependencies
        run: pip install -r requirements.txt

name: Train model
        env:

          COMETAPIKEY: ${{ secrets.COMETAPIKEY }}

          COMETPROJECTNAME: "production-training"

        run: python train.py

import cometml

try:
    experiment = cometml.ExistingExperiment(

        previousexperiment="experiment-key-to-resume"
    )
    print("Resuming existing experiment")
except Exception:
    experiment = cometml.Experiment(projectname="robust-training")
    print("Starting new experiment")

try:
    for epoch in range(100):
        loss = 1.0 / (epoch + 1)
        experiment.logmetric("loss", loss, step=epoch)


        if epoch % 10 == 0:
            experiment.logmodel(f"checkpointepoch{epoch}", "./checkpoints/")
except Exception as e:
    experiment.logother("error", str(e))

    raise
finally:
    experiment.end()

• Gunakan cometml.Experiment untuk setiap eksperimen baru dan pastikan memanggil experiment.end()
• Manfaatkan auto-logging untuk framework yang didukung
• Organisir eksperimen dengan project, tags, dan naming convention yang konsisten
• Gunakan Artifacts untuk versioning dataset dan model
• Manfaatkan Model Registry untuk mengelola model di production
• Integrasikan dengan CI/CD pipeline untuk automated training
• Gunakan Optimizer untuk hyperparameter tuning yang efisien

Dengan menerapkan praktik-praktik ini, Anda dapat membangun workflow ML yang terstruktur, reproducible, dan siap untuk skala production.