Tutorial CatBoost: Gradient Boosting untuk Data Tabular dengan Penanganan Fitur Kategorikal Otomatis
CatBoost (Categorical Boosting) adalah library gradient boosting open-source yang dikembangkan oleh Yandex. Library ini dirancang khusus untuk menangani fitur kategorikal secara native tanpa perlu encoding manual, sehingga menjadi salah satu pilihan terbaik untuk pemodelan data tabular. Dalam tutorial ini, kita akan mempelajari cara menggunakan CatBoost mulai dari instalasi, training model dasar, hingga teknik lanjutan seperti hyperparameter tuning, feature importance, dan deployment model.
Mengapa CatBoost?
Sebelum masuk ke implementasi, berikut alasan mengapa CatBoost layak dipertimbangkan dibanding library gradient boosting lainnya:
Instalasi
Instalasi CatBoost sangat mudah menggunakan pip:
pip install catboost
Untuk instalasi dengan dukungan GPU (memerlukan CUDA):
pip install catboost-gpu
Pastikan juga library pendukung sudah terinstal:
pip install pandas numpy scikit-learn matplotlib seaborn
Verifikasi instalasi:
import catboost
print(f"CatBoost version: {catboost.version}")
Penggunaan Dasar: Klasifikasi
Mari mulai dengan contoh klasifikasi sederhana menggunakan dataset Titanic.
Persiapan Data
import pandas as pd
import numpy as np
from sklearn.modelselection import traintestsplit
from catboost import CatBoostClassifier, Pool
Load dataset
url = "https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv"
df = pd.readcsv(url)
Pilih fitur yang relevan
features = ['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked']
target = 'Survived'
df = df[features + [target]].dropna()
Identifikasi fitur kategorikal
catfeatures = ['Sex', 'Embarked']
Split data
X = df[features]
y = df[target]
Xtrain, Xtest, ytrain, ytest = traintestsplit(
X, y, testsize=0.2, randomstate=42, stratify=y
)
print(f"Training set: {Xtrain.shape}")
print(f"Test set: {Xtest.shape}")
Training Model Klasifikasi
# Buat CatBoost Pool (format data khusus CatBoost)
trainpool = Pool(
data=Xtrain,
label=ytrain,
catfeatures=catfeatures
)
testpool = Pool(
data=Xtest,
label=ytest,
catfeatures=catfeatures
)
Inisialisasi dan training model
model = CatBoostClassifier(
iterations=500,
learningrate=0.05,
depth=6,
lossfunction='Logloss',
evalmetric='Accuracy',
randomseed=42,
verbose=100 # Print progress setiap 100 iterasi
)
model.fit(
trainpool,
evalset=testpool,
earlystoppingrounds=50
)
Evaluasi
accuracy = model.score(Xtest, ytest)
print(f"\nTest Accuracy: {accuracy:.4f}")
Prediksi dan Probabilitas
# Prediksi kelas
predictions = model.predict(Xtest)
Prediksi probabilitas
probabilities = model.predictproba(Xtest)
Tampilkan beberapa hasil
results = pd.DataFrame({
'Actual': ytest.values,
'Predicted': predictions.flatten(),
'ProbSurvived': probabilities[:, 1]
})
print(results.head(10))
Penggunaan Dasar: Regresi
CatBoost juga sangat baik untuk task regresi. Berikut contoh memprediksi harga rumah.
from catboost import CatBoostRegressor
from sklearn.datasets import fetchcaliforniahousing
from sklearn.metrics import meansquarederror, r2score
Load dataset
housing = fetchcaliforniahousing(asframe=True)
X = housing.data
y = housing.target
Xtrain, Xtest, ytrain, ytest = traintestsplit(
X, y, testsize=0.2, randomstate=42
)
Training model regresi
regmodel = CatBoostRegressor(
iterations=1000,
learningrate=0.05,
depth=8,
lossfunction='RMSE',
evalmetric='R2',
randomseed=42,
verbose=200
)
regmodel.fit(
Xtrain, ytrain,
evalset=(Xtest, ytest),
earlystoppingrounds=50
)
Evaluasi
predictions = regmodel.predict(Xtest)
rmse = meansquarederror(ytest, predictions, squared=False)
r2 = r2score(ytest, predictions)
print(f"RMSE: {rmse:.4f}")
print(f"R2 Score: {r2:.4f}")
Penanganan Fitur Kategorikal
Salah satu keunggulan utama CatBoost adalah penanganan fitur kategorikal. Berikut penjelasan mendalam tentang cara kerjanya.
Metode 1: Menggunakan Parameter catfeatures
import pandas as pd
from catboost import CatBoostClassifier, Pool
Dataset dengan fitur kategorikal
data = pd.DataFrame({
'city': ['Jakarta', 'Surabaya', 'Bandung', 'Jakarta', 'Surabaya',
'Bandung', 'Jakarta', 'Surabaya', 'Bandung', 'Jakarta'],
'education': ['S1', 'S2', 'S1', 'S3', 'S1', 'S2', 'S1', 'S3', 'S2', 'S1'],
'experience': [5, 10, 3, 15, 7, 8, 2, 12, 6, 4],
'salaryhigh': [0, 1, 0, 1, 1, 1, 0, 1, 0, 0]
})
X = data[['city', 'education', 'experience']]
y = data['salaryhigh']
Tentukan indeks fitur kategorikal
catfeaturesindices = [0, 1] # city dan education
Atau gunakan nama kolom
catfeaturesnames = ['city', 'education']
model = CatBoostClassifier(iterations=100, verbose=0)
model.fit(X, y, catfeatures=catfeaturesnames)
Metode 2: Menggunakan Pool dengan Tipe Data
# CatBoost otomatis mendeteksi kolom string sebagai kategorikal
pool = Pool(
data=X,
label=y,
catfeatures=catfeaturesnames,
featurenames=list(X.columns)
)
model = CatBoostClassifier(iterations=100, verbose=0)
model.fit(pool)
Metode 3: Fitur Teks (Text Features)
CatBoost juga mendukung fitur teks secara native:
datawithtext = pd.DataFrame({
'review': [
'produk sangat bagus dan berkualitas',
'mengecewakan tidak sesuai deskripsi',
'pengiriman cepat barang oke',
'buruk sekali tidak akan beli lagi',
'recommended seller terpercaya'
],
'category': ['elektronik', 'fashion', 'elektronik', 'fashion', 'elektronik'],
'rating': [5, 1, 4, 1, 5]
})
X
text = datawithtext[['review', 'category']]
ytext = datawithtext['rating']
textpool = Pool(
data=Xtext,
label=ytext,
catfeatures=['category'],
textfeatures=['review']
)
textmodel = CatBoostClassifier(iterations=100, verbose=0)
textmodel.fit(textpool)
Hyperparameter Tuning
Grid Search
from catboost import CatBoostClassifier
model = CatBoostClassifier(
lossfunction='Logloss',
randomseed=42,
verbose=0
)
grid = {
'iterations': [500, 1000],
'learningrate': [0.01, 0.05, 0.1],
'depth': [4, 6, 8],
'l2leafreg': [1, 3, 5]
}
gridsearchresult = model.gridsearch(
grid,
X=Xtrain,
y=ytrain,
cv=5,
plot=False,
verbose=0
)
print("Best parameters:", gridsearchresult['params'])
Randomized Search
randomizedsearchresult = model.randomizedsearch(
{
'iterations': [500, 1000, 1500],
'learningrate': [0.01, 0.03, 0.05, 0.1],
'depth': [4, 6, 8, 10],
'l2leafreg': [1, 3, 5, 7, 9],
'bordercount': [32, 64, 128, 254],
'baggingtemperature': [0, 0.5, 1]
},
X=Xtrain,
y=ytrain,
cv=5,
niter=20,
plot=False,
verbose=0
)
print("Best parameters:", randomizedsearchresult['params'])
Integrasi dengan Optuna
import optuna
from catboost import CatBoostClassifier
from sklearn.modelselection import crossvalscore
def objective(trial):
params = {
'iterations': trial.suggestint('iterations', 300, 1500),
'learningrate': trial.suggestfloat('learningrate', 0.01, 0.3, log=True),
'depth': trial.suggestint('depth', 3, 10),
'l2leafreg': trial.suggestfloat('l2leafreg', 1e-3, 10, log=True),
'bordercount': trial.suggestint('bordercount', 32, 255),
'baggingtemperature': trial.suggestfloat('baggingtemperature', 0, 1),
'randomstrength': trial.suggestfloat('randomstrength', 0, 10),
'mindatainleaf': trial.suggestint('mindatainleaf', 1, 50),
'lossfunction': 'Logloss',
'evalmetric': 'Accuracy',
'randomseed': 42,
'verbose': 0
}
model = CatBoostClassifier(**params)
scores = crossvalscore(model, Xtrain, ytrain, cv=5, scoring='accuracy')
return scores.mean()
study = optuna.createstudy(direction='maximize')
study.optimize(objective, ntrials=50, showprogressbar=True)
print(f"Best accuracy: {study.bestvalue:.4f}")
print(f"Best params: {study.bestparams}")
Feature Importance dan Model Interpretasi
Feature Importance
import matplotlib.pyplot as plt
Training model dengan dataset Titanic
model = CatBoostClassifier(iterations=500, verbose=0)
model.fit(trainpool, evalset=testpool)
Default feature importance (PredictionValuesChange)
featureimportance = model.getfeatureimportance(prettified=True)
print("Feature Importance:")
print(featureimportance)
Visualisasi
plt.figure(figsize=(10, 6))
plt.barh(
featureimportance['Feature Id'],
featureimportance['Importances']
)
plt.xlabel('Importance')
plt.title('CatBoost Feature Importance')
plt.tightlayout()
plt.savefig('featureimportance.png', dpi=150)
plt.show()
SHAP Values
# SHAP values untuk interpretasi per-prediksi
shapvalues = model.getfeatureimportance(
data=testpool,
type='ShapValues'
)
print(f"SHAP values shape: {shapvalues.shape}")
Kolom terakhir adalah expected value (base value)
Visualisasi SHAP untuk satu sampel
import shap
explainer = shap.TreeExplainer(model)
shapvals = explainer.shapvalues(Xtest)
Summary plot
shap.summaryplot(shapvals, Xtest, featurenames=features)
Feature Interaction
# Feature interaction strength
interaction = model.getfeatureimportance(
data=testpool,
type='Interaction'
)
print("Top feature interactions:")
for i, row in enumerate(interaction[:5]):
f1 = features[int(row[0])]
f2 = features[int(row[1])]
strength = row[2]
print(f" {f1} x {f2}: {strength:.4f}")
Cross-Validation
from catboost import cv
params = {
'iterations': 1000,
'learningrate': 0.05,
'depth': 6,
'lossfunction': 'Logloss',
'evalmetric': 'Accuracy',
'randomseed': 42,
'verbose': 200
}
cvresults = cv(
pool=trainpool,
params=params,
foldcount=5,
shuffle=True,
stratified=True,
earlystoppingrounds=50,
plot=False
)
print(f"\nBest iteration: {cvresults['test-Accuracy-mean'].idxmax()}")
print(f"Best CV Accuracy: {cvresults['test-Accuracy-mean'].max():.4f}")
print(f"Std: {cvresults['test-Accuracy-std'].min():.4f}")
Training dengan GPU
# Training menggunakan GPU
gpumodel = CatBoostClassifier(
iterations=1000,
learningrate=0.05,
depth=8,
lossfunction='Logloss',
tasktype='GPU', # Aktifkan GPU
devices='0', # GPU device ID
gpurampart=0.5, # Gunakan 50% GPU RAM
randomseed=42,
verbose=100
)
gpumodel.fit(trainpool, evalset=testpool)
Multiclass Classification
from sklearn.datasets import loadiris
Load dataset multiclass
iris = loadiris(asframe=True)
Xiris = iris.data
yiris = iris.target
Xtr, Xte, ytr, yte = traintestsplit(
Xiris, yiris, testsize=0.2, randomstate=42
)
Model multiclass
multimodel = CatBoostClassifier(
iterations=300,
lossfunction='MultiClass',
evalmetric='MultiClass',
randomseed=42,
verbose=0
)
multimodel.fit(Xtr, ytr, evalset=(Xte, yte))
Evaluasi
from sklearn.metrics import classificationreport
predictions = multimodel.predict(Xte).flatten().astype(int)
print(classificationreport(yte, predictions, targetnames=iris.targetnames))
Ranking (Learning to Rank)
CatBoost mendukung task ranking untuk mesin pencari atau sistem rekomendasi:
from catboost import CatBoostRanker, Pool
Simulasi data ranking
np.random.seed(42)
nsamples = 100
datarank = pd.DataFrame({
'queryid': np.repeat(range(10), 10),
'feature1': np.random.randn(nsamples),
'feature2': np.random.randn(nsamples),
'feature3': np.random.randn(nsamples),
'relevance': np.random.randint(0, 5, nsamples)
})
Xrank = datarank[['feature1', 'feature2', 'feature3']]
yrank = datarank['relevance']
groups = datarank['queryid']
rankpool = Pool(
data=Xrank,
label=yrank,
groupid=groups
)
ranker = CatBoostRanker(
iterations=300,
lossfunction='YetiRank',
randomseed=42,
verbose=0
)
ranker.fit(rankpool)
scores = ranker.predict(Xrank)
print("Ranking scores (first query):", scores[:10])
Save dan Load Model
# Simpan model dalam berbagai format
model.savemodel('model.cbm') # Format CatBoost native
model.savemodel('model.json', format='json') # Format JSON
model.savemodel('model.onnx', format='onnx') # Format ONNX
Load model
loadedmodel = CatBoostClassifier()
loadedmodel.loadmodel('model.cbm')
Verifikasi
originalpred = model.predict(Xtest[:5])
loadedpred = loadedmodel.predict(Xtest[:5])
assert np.arrayequal(originalpred, loadedpred)
print("Model loaded successfully!")
CatBoost dengan Scikit-learn Pipeline
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.compose import ColumnTransformer
from catboost import CatBoostClassifier
Definisikan pipeline
numericfeatures = ['Age', 'SibSp', 'Parch', 'Fare']
categoricalfeatures = ['Sex', 'Embarked']
preprocessor = ColumnTransformer(
transformers=[
('num', StandardScaler(), numericfeatures),
],
remainder='passthrough'
)
pipeline = Pipeline([
('preprocessor', preprocessor),
('classifier', CatBoostClassifier(
iterations=500,
verbose=0,
catfeatures=[4, 5] # Index setelah transformasi
))
])
pipeline.fit(Xtrain, ytrain)
score = pipeline.score(Xtest, ytest)
print(f"Pipeline accuracy: {score:.4f}")
Monitoring Training dengan Visualisasi
from catboost import CatBoostClassifier
Training dengan logging metrik
model = CatBoostClassifier(
iterations=500,
learningrate=0.05,
depth=6,
lossfunction='Logloss',
evalmetric='AUC',
custommetric=['Accuracy', 'Precision', 'Recall', 'F1'],
randomseed=42,
verbose=0,
traindir='catboostinfo' # Direktori untuk log training
)
model.fit(
trainpool,
evalset=testpool,
plot=False
)
Akses metrik training
evalsresult = model.getevalsresult()
print("Available metrics:", list(evalsresult['validation'].keys()))
Plot learning curves secara manual
import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
Loss curve
axes[0].plot(evalsresult['learn']['Logloss'], label='Train')
axes[0].plot(evalsresult['validation']['Logloss'], label='Validation')
axes[0].setxlabel('Iteration')
axes[0].setylabel('Logloss')
axes[0].settitle('Loss Curve')
axes[0].legend()
AUC curve
axes[1].plot(evalsresult['validation']['AUC'], label='Validation AUC')
axes[1].setxlabel('Iteration')
axes[1].setylabel('AUC')
axes[1].settitle('AUC Curve')
axes[1].legend()
plt.tightlayout()
plt.savefig('trainingcurves.png', dpi=150)
plt.show()
Penanganan Missing Values
CatBoost menangani missing values secara native:
# CatBoost menangani NaN secara otomatis
datawithnan = Xtrain.copy()
Sengaja tambahkan missing values
mask = np.random.random(datawithnan.shape) < 0.1
datawithnan = datawithnan.mask(
pd.DataFrame(mask, columns=datawithnan.columns, index=datawithnan.index)
)
print(f"Missing values:\n{datawithnan.isnull().sum()}")
CatBoost otomatis menangani NaN
modelnan = CatBoostClassifier(iterations=300, verbose=0)
modelnan.fit(
datawithnan, ytrain,
catfeatures=catfeatures
)
accuracy = modelnan.score(Xtest, ytest)
print(f"Accuracy with missing values: {accuracy:.4f}")
Model Deployment dengan ONNX
# Export ke ONNX untuk deployment
model.savemodel(
'catboostmodel.onnx',
format='onnx',
exportparameters={
'onnxdomain': 'ai.catboost',
'onnxmodelversion': 1,
}
)
Inferensi menggunakan ONNX Runtime
import onnxruntime as ort
session = ort.InferenceSession('catboostmodel.onnx')
Persiapan input
inputname = session.getinputs()[0].name
sample = Xtest.iloc[:5].values.astype(np.float32)
Prediksi
onnxpredictions = session.run(None, {inputname: sample})
print("ONNX predictions:", onnxpredictions[0])
Best Practices
1. Pemilihan Hyperparameter
# Rekomendasi hyperparameter berdasarkan ukuran dataset
def getrecommendedparams(nsamples, nfeatures):
if nsamples < 1000:
return {
'iterations': 300,
'learningrate': 0.1,
'depth': 4,
'l2leafreg': 5,
'mindatainleaf': 1
}
elif nsamples < 10000:
return {
'iterations': 1000,
'learningrate': 0.05,
'depth': 6,
'l2leafreg': 3,
'mindatainleaf': 5
}
else:
return {
'iterations': 2000,
'learningrate': 0.03,
'depth': 8,
'l2leafreg': 1,
'mindatainleaf': 10
}
params = getrecommendedparams(len(Xtrain), Xtrain.shape[1])
print("Recommended params:", params)
2. Menghindari Overfitting
# Konfigurasi anti-overfitting
model = CatBoostClassifier(
iterations=2000,
learningrate=0.03,
depth=6,
l2leafreg=5, # Regularisasi L2
randomstrength=1, # Randomness untuk split
baggingtemperature=1, # Bayesian bootstrap temperature
odtype='Iter', # Overfitting detector
odwait=50, # Patience untuk early stopping
verbose=0
)
3. Handling Imbalanced Dataset
# Untuk dataset tidak seimbang
from sklearn.utils.classweight import computeclassweight
classweights = computeclassweight(
'balanced', classes=np.unique(ytrain), y=ytrain
)
classweightdict = dict(zip(np.unique(ytrain), classweights))
model = CatBoostClassifier(
iterations=500,
classweights=classweightdict,
autoclassweights='Balanced', # Atau gunakan ini
evalmetric='F1',
verbose=0
)
4. Snapshot Training (Resume Training)
# Simpan snapshot untuk resume training
model = CatBoostClassifier(
iterations=1000,
learningrate=0.05,
savesnapshot=True,
snapshotfile='trainingsnapshot',
snapshotinterval=100,
verbose=100
)
Training pertama (misal terputus di iterasi 400)
model.fit(trainpool, evalset=testpool)
Resume training dari snapshot terakhir
modelresumed = CatBoostClassifier(
iterations=2000, # Tambah iterasi
learningrate=0.05,
savesnapshot=True,
snapshotfile='trainingsnapshot',
verbose=100
)
modelresumed.fit(trainpool, evalset=testpool)
Perbandingan CatBoost vs XGBoost vs LightGBM
Berikut perbandingan singkat ketiga library gradient boosting populer:
| Aspek | CatBoost | XGBoost | LightGBM |
|-------|----------|---------|----------|
| Fitur Kategorikal | Native support | Perlu encoding | Partial support |
| Missing Values | Native | Native | Native |
| Default Performance | Sangat baik | Baik | Baik |
| GPU Training | Ya | Ya | Ya |
| Kecepatan Training | Sedang | Sedang | Cepat |
| Overfitting Resistance | Tinggi | Sedang | Sedang |
| ONNX Export | Ya | Ya | Ya |
CatBoost cenderung memberikan performa terbaik out-of-the-box tanpa banyak tuning, terutama pada dataset dengan fitur kategorikal. XGBoost dan LightGBM mungkin memerlukan feature engineering tambahan untuk mencapai performa serupa.
Kesimpulan
CatBoost adalah library gradient boosting yang sangat powerful dengan beberapa keunggulan utama: penanganan fitur kategorikal native, ketahanan terhadap overfitting, dan performa default yang sudah optimal. Library ini cocok digunakan untuk berbagai task machine learning pada data tabular, mulai dari klasifikasi, regresi, hingga ranking.
Poin penting yang perlu diingat:
- Selalu tentukan
catfeaturesuntuk memaksimalkan kemampuan CatBoost - Gunakan
Poolobject untuk efisiensi memori dan kecepatan - Manfaatkan built-in cross-validation dan grid search
- Export model ke ONNX untuk deployment yang efisien
- Gunakan early stopping untuk mencegah overfitting
Dengan pemahaman yang baik tentang CatBoost, Anda dapat membangun model machine learning yang akurat dan robust untuk berbagai kebutuhan bisnis dan riset.