CatBoost Tutorial: Gradient Boosting for Tabular Data with Automatic Categorical Feature Handling
CatBoost (Categorical Boosting) is an open-source gradient boosting library developed by Yandex. It is specifically designed to handle categorical features natively without manual encoding, making it one of the best choices for tabular data modeling. In this tutorial, we will learn how to use CatBoost from installation, basic model training, to advanced techniques such as hyperparameter tuning, feature importance analysis, and model deployment.
Why CatBoost?
Before diving into the implementation, here are the reasons why CatBoost deserves consideration over other gradient boosting libraries:
Installation
Installing CatBoost is straightforward using pip:
pip install catboost
For installation with GPU support (requires CUDA):
pip install catboost-gpu
Also ensure supporting libraries are installed:
pip install pandas numpy scikit-learn matplotlib seaborn
Verify the installation:
import catboost
print(f"CatBoost version: {catboost.version}")
Basic Usage: Classification
Let's start with a simple classification example using the Titanic dataset.
Data Preparation
import pandas as pd
import numpy as np
from sklearn.modelselection import traintestsplit
from catboost import CatBoostClassifier, Pool
Load dataset
url = "https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv"
df = pd.readcsv(url)
Select relevant features
features = ['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked']
target = 'Survived'
df = df[features + [target]].dropna()
Identify categorical features
catfeatures = ['Sex', 'Embarked']
Split data
X = df[features]
y = df[target]
Xtrain, Xtest, ytrain, ytest = traintestsplit(
X, y, testsize=0.2, randomstate=42, stratify=y
)
print(f"Training set: {Xtrain.shape}")
print(f"Test set: {Xtest.shape}")
Training a Classification Model
# Create CatBoost Pool (CatBoost's special data format)
trainpool = Pool(
data=Xtrain,
label=ytrain,
catfeatures=catfeatures
)
testpool = Pool(
data=Xtest,
label=ytest,
catfeatures=catfeatures
)
Initialize and train the model
model = CatBoostClassifier(
iterations=500,
learningrate=0.05,
depth=6,
lossfunction='Logloss',
evalmetric='Accuracy',
randomseed=42,
verbose=100 # Print progress every 100 iterations
)
model.fit(
trainpool,
evalset=testpool,
earlystoppingrounds=50
)
Evaluate
accuracy = model.score(Xtest, ytest)
print(f"\nTest Accuracy: {accuracy:.4f}")
Predictions and Probabilities
# Class predictions
predictions = model.predict(Xtest)
Probability predictions
probabilities = model.predictproba(Xtest)
Display some results
results = pd.DataFrame({
'Actual': ytest.values,
'Predicted': predictions.flatten(),
'ProbSurvived': probabilities[:, 1]
})
print(results.head(10))
Basic Usage: Regression
CatBoost is also excellent for regression tasks. Here is an example predicting house prices.
from catboost import CatBoostRegressor
from sklearn.datasets import fetchcaliforniahousing
from sklearn.metrics import meansquarederror, r2score
Load dataset
housing = fetchcaliforniahousing(asframe=True)
X = housing.data
y = housing.target
Xtrain, Xtest, ytrain, ytest = traintestsplit(
X, y, testsize=0.2, randomstate=42
)
Train regression model
regmodel = CatBoostRegressor(
iterations=1000,
learningrate=0.05,
depth=8,
lossfunction='RMSE',
evalmetric='R2',
randomseed=42,
verbose=200
)
regmodel.fit(
Xtrain, ytrain,
evalset=(Xtest, ytest),
earlystoppingrounds=50
)
Evaluate
predictions = regmodel.predict(Xtest)
rmse = meansquarederror(ytest, predictions, squared=False)
r2 = r2score(ytest, predictions)
print(f"RMSE: {rmse:.4f}")
print(f"R2 Score: {r2:.4f}")
Handling Categorical Features
One of CatBoost's primary advantages is its handling of categorical features. Here is an in-depth explanation of how it works.
Method 1: Using the catfeatures Parameter
import pandas as pd
from catboost import CatBoostClassifier, Pool
Dataset with categorical features
data = pd.DataFrame({
'city': ['Jakarta', 'Surabaya', 'Bandung', 'Jakarta', 'Surabaya',
'Bandung', 'Jakarta', 'Surabaya', 'Bandung', 'Jakarta'],
'education': ['S1', 'S2', 'S1', 'S3', 'S1', 'S2', 'S1', 'S3', 'S2', 'S1'],
'experience': [5, 10, 3, 15, 7, 8, 2, 12, 6, 4],
'salaryhigh': [0, 1, 0, 1, 1, 1, 0, 1, 0, 0]
})
X = data[['city', 'education', 'experience']]
y = data['salaryhigh']
Specify categorical feature indices
catfeaturesindices = [0, 1] # city and education
Or use column names
catfeaturesnames = ['city', 'education']
model = CatBoostClassifier(iterations=100, verbose=0)
model.fit(X, y, catfeatures=catfeaturesnames)
Method 2: Using Pool with Data Types
# CatBoost automatically detects string columns as categorical
pool = Pool(
data=X,
label=y,
catfeatures=catfeaturesnames,
featurenames=list(X.columns)
)
model = CatBoostClassifier(iterations=100, verbose=0)
model.fit(pool)
Method 3: Text Features
CatBoost also supports text features natively:
datawithtext = pd.DataFrame({
'review': [
'great product excellent quality',
'disappointing does not match description',
'fast shipping item is okay',
'terrible will never buy again',
'recommended trusted seller'
],
'category': ['electronics', 'fashion', 'electronics', 'fashion', 'electronics'],
'rating': [5, 1, 4, 1, 5]
})
X
text = datawithtext[['review', 'category']]
ytext = datawithtext['rating']
textpool = Pool(
data=Xtext,
label=ytext,
catfeatures=['category'],
textfeatures=['review']
)
textmodel = CatBoostClassifier(iterations=100, verbose=0)
textmodel.fit(textpool)
Hyperparameter Tuning
Grid Search
from catboost import CatBoostClassifier
model = CatBoostClassifier(
lossfunction='Logloss',
randomseed=42,
verbose=0
)
grid = {
'iterations': [500, 1000],
'learningrate': [0.01, 0.05, 0.1],
'depth': [4, 6, 8],
'l2leafreg': [1, 3, 5]
}
gridsearchresult = model.gridsearch(
grid,
X=Xtrain,
y=ytrain,
cv=5,
plot=False,
verbose=0
)
print("Best parameters:", gridsearchresult['params'])
Randomized Search
randomizedsearchresult = model.randomizedsearch(
{
'iterations': [500, 1000, 1500],
'learningrate': [0.01, 0.03, 0.05, 0.1],
'depth': [4, 6, 8, 10],
'l2leafreg': [1, 3, 5, 7, 9],
'bordercount': [32, 64, 128, 254],
'baggingtemperature': [0, 0.5, 1]
},
X=Xtrain,
y=ytrain,
cv=5,
niter=20,
plot=False,
verbose=0
)
print("Best parameters:", randomizedsearchresult['params'])
Integration with Optuna
import optuna
from catboost import CatBoostClassifier
from sklearn.modelselection import crossvalscore
def objective(trial):
params = {
'iterations': trial.suggestint('iterations', 300, 1500),
'learningrate': trial.suggestfloat('learningrate', 0.01, 0.3, log=True),
'depth': trial.suggestint('depth', 3, 10),
'l2leafreg': trial.suggestfloat('l2leafreg', 1e-3, 10, log=True),
'bordercount': trial.suggestint('bordercount', 32, 255),
'baggingtemperature': trial.suggestfloat('baggingtemperature', 0, 1),
'randomstrength': trial.suggestfloat('randomstrength', 0, 10),
'mindatainleaf': trial.suggestint('mindatainleaf', 1, 50),
'lossfunction': 'Logloss',
'evalmetric': 'Accuracy',
'randomseed': 42,
'verbose': 0
}
model = CatBoostClassifier(**params)
scores = crossvalscore(model, Xtrain, ytrain, cv=5, scoring='accuracy')
return scores.mean()
study = optuna.createstudy(direction='maximize')
study.optimize(objective, ntrials=50, showprogressbar=True)
print(f"Best accuracy: {study.bestvalue:.4f}")
print(f"Best params: {study.bestparams}")
Feature Importance and Model Interpretation
Feature Importance
import matplotlib.pyplot as plt
Train model with Titanic dataset
model = CatBoostClassifier(iterations=500, verbose=0)
model.fit(trainpool, evalset=testpool)
Default feature importance (PredictionValuesChange)
featureimportance = model.getfeatureimportance(prettified=True)
print("Feature Importance:")
print(featureimportance)
Visualization
plt.figure(figsize=(10, 6))
plt.barh(
featureimportance['Feature Id'],
featureimportance['Importances']
)
plt.xlabel('Importance')
plt.title('CatBoost Feature Importance')
plt.tightlayout()
plt.savefig('featureimportance.png', dpi=150)
plt.show()
SHAP Values
# SHAP values for per-prediction interpretation
shapvalues = model.getfeatureimportance(
data=testpool,
type='ShapValues'
)
print(f"SHAP values shape: {shapvalues.shape}")
The last column is the expected value (base value)
Visualize SHAP for a single sample
import shap
explainer = shap.TreeExplainer(model)
shapvals = explainer.shapvalues(Xtest)
Summary plot
shap.summaryplot(shapvals, Xtest, featurenames=features)
Feature Interaction
# Feature interaction strength
interaction = model.getfeatureimportance(
data=testpool,
type='Interaction'
)
print("Top feature interactions:")
for i, row in enumerate(interaction[:5]):
f1 = features[int(row[0])]
f2 = features[int(row[1])]
strength = row[2]
print(f" {f1} x {f2}: {strength:.4f}")
Cross-Validation
from catboost import cv
params = {
'iterations': 1000,
'learningrate': 0.05,
'depth': 6,
'lossfunction': 'Logloss',
'evalmetric': 'Accuracy',
'randomseed': 42,
'verbose': 200
}
cvresults = cv(
pool=trainpool,
params=params,
foldcount=5,
shuffle=True,
stratified=True,
earlystoppingrounds=50,
plot=False
)
print(f"\nBest iteration: {cvresults['test-Accuracy-mean'].idxmax()}")
print(f"Best CV Accuracy: {cvresults['test-Accuracy-mean'].max():.4f}")
print(f"Std: {cvresults['test-Accuracy-std'].min():.4f}")
GPU Training
# Train using GPU
gpumodel = CatBoostClassifier(
iterations=1000,
learningrate=0.05,
depth=8,
lossfunction='Logloss',
tasktype='GPU', # Enable GPU
devices='0', # GPU device ID
gpurampart=0.5, # Use 50% GPU RAM
randomseed=42,
verbose=100
)
gpumodel.fit(trainpool, evalset=testpool)
Multiclass Classification
from sklearn.datasets import loadiris
Load multiclass dataset
iris = loadiris(asframe=True)
Xiris = iris.data
yiris = iris.target
Xtr, Xte, ytr, yte = traintestsplit(
Xiris, yiris, testsize=0.2, randomstate=42
)
Multiclass model
multimodel = CatBoostClassifier(
iterations=300,
lossfunction='MultiClass',
evalmetric='MultiClass',
randomseed=42,
verbose=0
)
multimodel.fit(Xtr, ytr, evalset=(Xte, yte))
Evaluate
from sklearn.metrics import classificationreport
predictions = multimodel.predict(Xte).flatten().astype(int)
print(classificationreport(yte, predictions, targetnames=iris.targetnames))
Ranking (Learning to Rank)
CatBoost supports ranking tasks for search engines or recommendation systems:
from catboost import CatBoostRanker, Pool
Simulate ranking data
np.random.seed(42)
nsamples = 100
datarank = pd.DataFrame({
'queryid': np.repeat(range(10), 10),
'feature1': np.random.randn(nsamples),
'feature2': np.random.randn(nsamples),
'feature3': np.random.randn(nsamples),
'relevance': np.random.randint(0, 5, nsamples)
})
Xrank = datarank[['feature1', 'feature2', 'feature3']]
yrank = datarank['relevance']
groups = datarank['queryid']
rankpool = Pool(
data=Xrank,
label=yrank,
groupid=groups
)
ranker = CatBoostRanker(
iterations=300,
lossfunction='YetiRank',
randomseed=42,
verbose=0
)
ranker.fit(rankpool)
scores = ranker.predict(Xrank)
print("Ranking scores (first query):", scores[:10])
Save and Load Models
# Save model in various formats
model.savemodel('model.cbm') # CatBoost native format
model.savemodel('model.json', format='json') # JSON format
model.savemodel('model.onnx', format='onnx') # ONNX format
Load model
loadedmodel = CatBoostClassifier()
loadedmodel.loadmodel('model.cbm')
Verify
originalpred = model.predict(Xtest[:5])
loadedpred = loadedmodel.predict(Xtest[:5])
assert np.arrayequal(originalpred, loadedpred)
print("Model loaded successfully!")
CatBoost with Scikit-learn Pipeline
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.compose import ColumnTransformer
from catboost import CatBoostClassifier
Define pipeline
numericfeatures = ['Age', 'SibSp', 'Parch', 'Fare']
categoricalfeatures = ['Sex', 'Embarked']
preprocessor = ColumnTransformer(
transformers=[
('num', StandardScaler(), numericfeatures),
],
remainder='passthrough'
)
pipeline = Pipeline([
('preprocessor', preprocessor),
('classifier', CatBoostClassifier(
iterations=500,
verbose=0,
catfeatures=[4, 5] # Indices after transformation
))
])
pipeline.fit(Xtrain, ytrain)
score = pipeline.score(Xtest, ytest)
print(f"Pipeline accuracy: {score:.4f}")
Training Monitoring with Visualization
from catboost import CatBoostClassifier
Training with metric logging
model = CatBoostClassifier(
iterations=500,
learningrate=0.05,
depth=6,
lossfunction='Logloss',
evalmetric='AUC',
custommetric=['Accuracy', 'Precision', 'Recall', 'F1'],
randomseed=42,
verbose=0,
traindir='catboostinfo' # Directory for training logs
)
model.fit(
trainpool,
evalset=testpool,
plot=False
)
Access training metrics
evalsresult = model.getevalsresult()
print("Available metrics:", list(evalsresult['validation'].keys()))
Plot learning curves manually
import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
Loss curve
axes[0].plot(evalsresult['learn']['Logloss'], label='Train')
axes[0].plot(evalsresult['validation']['Logloss'], label='Validation')
axes[0].setxlabel('Iteration')
axes[0].setylabel('Logloss')
axes[0].settitle('Loss Curve')
axes[0].legend()
AUC curve
axes[1].plot(evalsresult['validation']['AUC'], label='Validation AUC')
axes[1].setxlabel('Iteration')
axes[1].setylabel('AUC')
axes[1].settitle('AUC Curve')
axes[1].legend()
plt.tightlayout()
plt.savefig('trainingcurves.png', dpi=150)
plt.show()
Handling Missing Values
CatBoost handles missing values natively:
# CatBoost handles NaN automatically
datawithnan = Xtrain.copy()
Deliberately add missing values
mask = np.random.random(datawithnan.shape) < 0.1
datawithnan = datawithnan.mask(
pd.DataFrame(mask, columns=datawithnan.columns, index=datawithnan.index)
)
print(f"Missing values:\n{datawithnan.isnull().sum()}")
CatBoost automatically handles NaN
modelnan = CatBoostClassifier(iterations=300, verbose=0)
modelnan.fit(
datawithnan, ytrain,
catfeatures=catfeatures
)
accuracy = modelnan.score(Xtest, ytest)
print(f"Accuracy with missing values: {accuracy:.4f}")
Model Deployment with ONNX
# Export to ONNX for deployment
model.savemodel(
'catboostmodel.onnx',
format='onnx',
exportparameters={
'onnxdomain': 'ai.catboost',
'onnxmodelversion': 1,
}
)
Inference using ONNX Runtime
import onnxruntime as ort
session = ort.InferenceSession('catboostmodel.onnx')
Prepare input
inputname = session.getinputs()[0].name
sample = Xtest.iloc[:5].values.astype(np.float32)
Predict
onnxpredictions = session.run(None, {inputname: sample})
print("ONNX predictions:", onnxpredictions[0])
Best Practices
1. Hyperparameter Selection
# Recommended hyperparameters based on dataset size
def getrecommendedparams(nsamples, nfeatures):
if nsamples < 1000:
return {
'iterations': 300,
'learningrate': 0.1,
'depth': 4,
'l2leafreg': 5,
'mindatainleaf': 1
}
elif nsamples < 10000:
return {
'iterations': 1000,
'learningrate': 0.05,
'depth': 6,
'l2leafreg': 3,
'mindatainleaf': 5
}
else:
return {
'iterations': 2000,
'learningrate': 0.03,
'depth': 8,
'l2leafreg': 1,
'mindatainleaf': 10
}
params = getrecommendedparams(len(Xtrain), Xtrain.shape[1])
print("Recommended params:", params)
2. Preventing Overfitting
# Anti-overfitting configuration
model = CatBoostClassifier(
iterations=2000,
learningrate=0.03,
depth=6,
l2leafreg=5, # L2 regularization
randomstrength=1, # Randomness for splits
baggingtemperature=1, # Bayesian bootstrap temperature
odtype='Iter', # Overfitting detector
odwait=50, # Patience for early stopping
verbose=0
)
3. Handling Imbalanced Datasets
# For imbalanced datasets
from sklearn.utils.classweight import computeclassweight
classweights = computeclassweight(
'balanced', classes=np.unique(ytrain), y=ytrain
)
classweightdict = dict(zip(np.unique(ytrain), classweights))
model = CatBoostClassifier(
iterations=500,
classweights=classweightdict,
autoclassweights='Balanced', # Or use this instead
evalmetric='F1',
verbose=0
)
4. Snapshot Training (Resume Training)
# Save snapshots to resume training
model = CatBoostClassifier(
iterations=1000,
learningrate=0.05,
savesnapshot=True,
snapshotfile='trainingsnapshot',
snapshotinterval=100,
verbose=100
)
First training run (e.g., interrupted at iteration 400)
model.fit(trainpool, evalset=testpool)
Resume training from the last snapshot
modelresumed = CatBoostClassifier(
iterations=2000, # Increase iterations
learningrate=0.05,
savesnapshot=True,
snapshotfile='trainingsnapshot',
verbose=100
)
modelresumed.fit(trainpool, evalset=testpool)
CatBoost vs XGBoost vs LightGBM Comparison
Here is a brief comparison of the three popular gradient boosting libraries:
| Aspect | CatBoost | XGBoost | LightGBM |
|--------|----------|---------|----------|
| Categorical Features | Native support | Requires encoding | Partial support |
| Missing Values | Native | Native | Native |
| Default Performance | Excellent | Good | Good |
| GPU Training | Yes | Yes | Yes |
| Training Speed | Moderate | Moderate | Fast |
| Overfitting Resistance | High | Moderate | Moderate |
| ONNX Export | Yes | Yes | Yes |
CatBoost tends to deliver the best out-of-the-box performance without extensive tuning, especially on datasets with categorical features. XGBoost and LightGBM may require additional feature engineering to achieve similar performance.
Conclusion
CatBoost is a powerful gradient boosting library with several key advantages: native categorical feature handling, overfitting resistance, and already-optimized default performance. It is suitable for various machine learning tasks on tabular data, from classification and regression to ranking.
Key takeaways:
- Always specify
catfeaturesto maximize CatBoost's capabilities - Use
Poolobjects for memory efficiency and speed - Leverage built-in cross-validation and grid search
- Export models to ONNX for efficient deployment
- Use early stopping to prevent overfitting
With a solid understanding of CatBoost, you can build accurate and robust machine learning models for various business and research needs.