CatBoost Tutorial: Gradient Boosting for Tabular Data with Automatic Categorical Feature Handling

# Tutorial CatBoost: Gradient Boosting untuk Data Tabular dengan Penanganan Fitur Kategorikal Otomatis CatBoost (Categorical Boosting) adalah library gradient boosting open-source yang dikembangkan o...

By Ruby Abdullah · · tutorial
CatBoostGradient BoostingMachine LearningTabular DataPython

CatBoost Tutorial: Gradient Boosting for Tabular Data with Automatic Categorical Feature Handling

CatBoost (Categorical Boosting) is an open-source gradient boosting library developed by Yandex. It is specifically designed to handle categorical features natively without manual encoding, making it one of the best choices for tabular data modeling. In this tutorial, we will learn how to use CatBoost from installation, basic model training, to advanced techniques such as hyperparameter tuning, feature importance analysis, and model deployment.

Why CatBoost?

Before diving into the implementation, here are the reasons why CatBoost deserves consideration over other gradient boosting libraries:

  • Native Categorical Feature Handling: CatBoost uses ordered target encoding that avoids target leakage, without requiring manual one-hot encoding or label encoding.
  • Ordered Boosting: A unique algorithm that reduces prediction shift (bias that occurs in traditional gradient boosting).
  • High Performance: Competitive with XGBoost and LightGBM, often superior on datasets with many categorical features.
  • GPU Training: Built-in GPU support to accelerate the learning process.
  • Built-in Visualization: Native tools for training monitoring and model analysis.
  • Robustness: More resistant to overfitting with already-optimized default hyperparameters.
  • Installation

    Installing CatBoost is straightforward using pip:

    pip install catboost
    

    For installation with GPU support (requires CUDA):

    pip install catboost-gpu
    

    Also ensure supporting libraries are installed:

    pip install pandas numpy scikit-learn matplotlib seaborn
    

    Verify the installation:

    import catboost
    

    print(f"CatBoost version: {catboost.version}")

    Basic Usage: Classification

    Let's start with a simple classification example using the Titanic dataset.

    Data Preparation

    import pandas as pd
    

    import numpy as np

    from sklearn.modelselection import traintestsplit

    from catboost import CatBoostClassifier, Pool

    Load dataset

    url = "https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv"

    df = pd.readcsv(url)

    Select relevant features

    features = ['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked']

    target = 'Survived'

    df = df[features + [target]].dropna()

    Identify categorical features

    catfeatures = ['Sex', 'Embarked']

    Split data

    X = df[features]

    y = df[target]

    Xtrain, Xtest, ytrain, ytest = traintestsplit(

    X, y, testsize=0.2, randomstate=42, stratify=y

    )

    print(f"Training set: {Xtrain.shape}")

    print(f"Test set: {Xtest.shape}")

    Training a Classification Model

    # Create CatBoost Pool (CatBoost's special data format)
    

    trainpool = Pool(

    data=Xtrain,

    label=ytrain,

    catfeatures=catfeatures

    )

    testpool = Pool(

    data=Xtest,

    label=ytest,

    catfeatures=catfeatures

    )

    Initialize and train the model

    model = CatBoostClassifier(

    iterations=500,

    learningrate=0.05,

    depth=6,

    lossfunction='Logloss',

    evalmetric='Accuracy',

    randomseed=42,

    verbose=100 # Print progress every 100 iterations

    )

    model.fit(

    trainpool,

    evalset=testpool,

    earlystoppingrounds=50

    )

    Evaluate

    accuracy = model.score(Xtest, ytest)

    print(f"\nTest Accuracy: {accuracy:.4f}")

    Predictions and Probabilities

    # Class predictions
    

    predictions = model.predict(Xtest)

    Probability predictions

    probabilities = model.predictproba(Xtest)

    Display some results

    results = pd.DataFrame({

    'Actual': ytest.values,

    'Predicted': predictions.flatten(),

    'ProbSurvived': probabilities[:, 1]

    })

    print(results.head(10))

    Basic Usage: Regression

    CatBoost is also excellent for regression tasks. Here is an example predicting house prices.

    from catboost import CatBoostRegressor
    

    from sklearn.datasets import fetchcaliforniahousing

    from sklearn.metrics import meansquarederror, r2score

    Load dataset

    housing = fetchcaliforniahousing(asframe=True)

    X = housing.data

    y = housing.target

    Xtrain, Xtest, ytrain, ytest = traintestsplit(

    X, y, testsize=0.2, randomstate=42

    )

    Train regression model

    regmodel = CatBoostRegressor(

    iterations=1000,

    learningrate=0.05,

    depth=8,

    lossfunction='RMSE',

    evalmetric='R2',

    randomseed=42,

    verbose=200

    )

    regmodel.fit(

    Xtrain, ytrain,

    evalset=(Xtest, ytest),

    earlystoppingrounds=50

    )

    Evaluate

    predictions = regmodel.predict(Xtest)

    rmse = meansquarederror(ytest, predictions, squared=False)

    r2 = r2score(ytest, predictions)

    print(f"RMSE: {rmse:.4f}")

    print(f"R2 Score: {r2:.4f}")

    Handling Categorical Features

    One of CatBoost's primary advantages is its handling of categorical features. Here is an in-depth explanation of how it works.

    Method 1: Using the catfeatures Parameter

    import pandas as pd
    

    from catboost import CatBoostClassifier, Pool

    Dataset with categorical features

    data = pd.DataFrame({

    'city': ['Jakarta', 'Surabaya', 'Bandung', 'Jakarta', 'Surabaya',

    'Bandung', 'Jakarta', 'Surabaya', 'Bandung', 'Jakarta'],

    'education': ['S1', 'S2', 'S1', 'S3', 'S1', 'S2', 'S1', 'S3', 'S2', 'S1'],

    'experience': [5, 10, 3, 15, 7, 8, 2, 12, 6, 4],

    'salaryhigh': [0, 1, 0, 1, 1, 1, 0, 1, 0, 0]

    })

    X = data[['city', 'education', 'experience']]

    y = data['salaryhigh']

    Specify categorical feature indices

    catfeaturesindices = [0, 1] # city and education

    Or use column names

    catfeaturesnames = ['city', 'education']

    model = CatBoostClassifier(iterations=100, verbose=0)

    model.fit(X, y, catfeatures=catfeaturesnames)

    Method 2: Using Pool with Data Types

    # CatBoost automatically detects string columns as categorical
    

    pool = Pool(

    data=X,

    label=y,

    catfeatures=catfeaturesnames,

    featurenames=list(X.columns)

    )

    model = CatBoostClassifier(iterations=100, verbose=0)

    model.fit(pool)

    Method 3: Text Features

    CatBoost also supports text features natively:

    datawithtext = pd.DataFrame({
    

    'review': [

    'great product excellent quality',

    'disappointing does not match description',

    'fast shipping item is okay',

    'terrible will never buy again',

    'recommended trusted seller'

    ],

    'category': ['electronics', 'fashion', 'electronics', 'fashion', 'electronics'],

    'rating': [5, 1, 4, 1, 5]

    })

    Xtext = datawithtext[['review', 'category']]

    ytext = datawithtext['rating']

    textpool = Pool(

    data=Xtext,

    label=ytext,

    catfeatures=['category'],

    textfeatures=['review']

    )

    textmodel = CatBoostClassifier(iterations=100, verbose=0)

    textmodel.fit(textpool)

    Hyperparameter Tuning

    from catboost import CatBoostClassifier
    
    

    model = CatBoostClassifier(

    lossfunction='Logloss',

    randomseed=42,

    verbose=0

    )

    grid = {

    'iterations': [500, 1000],

    'learningrate': [0.01, 0.05, 0.1],

    'depth': [4, 6, 8],

    'l2leafreg': [1, 3, 5]

    }

    gridsearchresult = model.gridsearch(

    grid,

    X=Xtrain,

    y=ytrain,

    cv=5,

    plot=False,

    verbose=0

    )

    print("Best parameters:", gridsearchresult['params'])

    randomizedsearchresult = model.randomizedsearch(
    

    {

    'iterations': [500, 1000, 1500],

    'learningrate': [0.01, 0.03, 0.05, 0.1],

    'depth': [4, 6, 8, 10],

    'l2leafreg': [1, 3, 5, 7, 9],

    'bordercount': [32, 64, 128, 254],

    'baggingtemperature': [0, 0.5, 1]

    },

    X=Xtrain,

    y=ytrain,

    cv=5,

    niter=20,

    plot=False,

    verbose=0

    )

    print("Best parameters:", randomizedsearchresult['params'])

    Integration with Optuna

    import optuna
    

    from catboost import CatBoostClassifier

    from sklearn.modelselection import crossvalscore

    def objective(trial):

    params = {

    'iterations': trial.suggestint('iterations', 300, 1500),

    'learningrate': trial.suggestfloat('learningrate', 0.01, 0.3, log=True),

    'depth': trial.suggestint('depth', 3, 10),

    'l2leafreg': trial.suggestfloat('l2leafreg', 1e-3, 10, log=True),

    'bordercount': trial.suggestint('bordercount', 32, 255),

    'baggingtemperature': trial.suggestfloat('baggingtemperature', 0, 1),

    'randomstrength': trial.suggestfloat('randomstrength', 0, 10),

    'mindatainleaf': trial.suggestint('mindatainleaf', 1, 50),

    'lossfunction': 'Logloss',

    'evalmetric': 'Accuracy',

    'randomseed': 42,

    'verbose': 0

    }

    model = CatBoostClassifier(**params)

    scores = crossvalscore(model, Xtrain, ytrain, cv=5, scoring='accuracy')

    return scores.mean()

    study = optuna.createstudy(direction='maximize')

    study.optimize(objective, ntrials=50, showprogressbar=True)

    print(f"Best accuracy: {study.bestvalue:.4f}")

    print(f"Best params: {study.bestparams}")

    Feature Importance and Model Interpretation

    Feature Importance

    import matplotlib.pyplot as plt
    
    

    Train model with Titanic dataset

    model = CatBoostClassifier(iterations=500, verbose=0)

    model.fit(trainpool, evalset=testpool)

    Default feature importance (PredictionValuesChange)

    featureimportance = model.getfeatureimportance(prettified=True)

    print("Feature Importance:")

    print(featureimportance)

    Visualization

    plt.figure(figsize=(10, 6))

    plt.barh(

    featureimportance['Feature Id'],

    featureimportance['Importances']

    )

    plt.xlabel('Importance')

    plt.title('CatBoost Feature Importance')

    plt.tightlayout()

    plt.savefig('featureimportance.png', dpi=150)

    plt.show()

    SHAP Values

    # SHAP values for per-prediction interpretation
    

    shapvalues = model.getfeatureimportance(

    data=testpool,

    type='ShapValues'

    )

    print(f"SHAP values shape: {shapvalues.shape}")

    The last column is the expected value (base value)

    Visualize SHAP for a single sample

    import shap

    explainer = shap.TreeExplainer(model)

    shapvals = explainer.shapvalues(Xtest)

    Summary plot

    shap.summaryplot(shapvals, Xtest, featurenames=features)

    Feature Interaction

    # Feature interaction strength
    

    interaction = model.getfeatureimportance(

    data=testpool,

    type='Interaction'

    )

    print("Top feature interactions:")

    for i, row in enumerate(interaction[:5]):

    f1 = features[int(row[0])]

    f2 = features[int(row[1])]

    strength = row[2]

    print(f" {f1} x {f2}: {strength:.4f}")

    Cross-Validation

    from catboost import cv
    
    

    params = {

    'iterations': 1000,

    'learningrate': 0.05,

    'depth': 6,

    'lossfunction': 'Logloss',

    'evalmetric': 'Accuracy',

    'randomseed': 42,

    'verbose': 200

    }

    cvresults = cv(

    pool=trainpool,

    params=params,

    foldcount=5,

    shuffle=True,

    stratified=True,

    earlystoppingrounds=50,

    plot=False

    )

    print(f"\nBest iteration: {cvresults['test-Accuracy-mean'].idxmax()}")

    print(f"Best CV Accuracy: {cvresults['test-Accuracy-mean'].max():.4f}")

    print(f"Std: {cvresults['test-Accuracy-std'].min():.4f}")

    GPU Training

    # Train using GPU
    

    gpumodel = CatBoostClassifier(

    iterations=1000,

    learningrate=0.05,

    depth=8,

    lossfunction='Logloss',

    tasktype='GPU', # Enable GPU

    devices='0', # GPU device ID

    gpurampart=0.5, # Use 50% GPU RAM

    randomseed=42,

    verbose=100

    )

    gpumodel.fit(trainpool, evalset=testpool)

    Multiclass Classification

    from sklearn.datasets import loadiris
    
    

    Load multiclass dataset

    iris = loadiris(asframe=True)

    Xiris = iris.data

    yiris = iris.target

    Xtr, Xte, ytr, yte = traintestsplit(

    Xiris, yiris, testsize=0.2, randomstate=42

    )

    Multiclass model

    multimodel = CatBoostClassifier(

    iterations=300,

    lossfunction='MultiClass',

    evalmetric='MultiClass',

    randomseed=42,

    verbose=0

    )

    multimodel.fit(Xtr, ytr, evalset=(Xte, yte))

    Evaluate

    from sklearn.metrics import classificationreport

    predictions = multimodel.predict(Xte).flatten().astype(int)

    print(classificationreport(yte, predictions, targetnames=iris.targetnames))

    Ranking (Learning to Rank)

    CatBoost supports ranking tasks for search engines or recommendation systems:

    from catboost import CatBoostRanker, Pool
    
    

    Simulate ranking data

    np.random.seed(42)

    nsamples = 100

    datarank = pd.DataFrame({

    'queryid': np.repeat(range(10), 10),

    'feature1': np.random.randn(nsamples),

    'feature2': np.random.randn(nsamples),

    'feature3': np.random.randn(nsamples),

    'relevance': np.random.randint(0, 5, nsamples)

    })

    Xrank = datarank[['feature1', 'feature2', 'feature3']]

    yrank = datarank['relevance']

    groups = datarank['queryid']

    rankpool = Pool(

    data=Xrank,

    label=yrank,

    groupid=groups

    )

    ranker = CatBoostRanker(

    iterations=300,

    lossfunction='YetiRank',

    randomseed=42,

    verbose=0

    )

    ranker.fit(rankpool)

    scores = ranker.predict(Xrank)

    print("Ranking scores (first query):", scores[:10])

    Save and Load Models

    # Save model in various formats
    

    model.savemodel('model.cbm') # CatBoost native format

    model.savemodel('model.json', format='json') # JSON format

    model.savemodel('model.onnx', format='onnx') # ONNX format

    Load model

    loadedmodel = CatBoostClassifier()

    loadedmodel.loadmodel('model.cbm')

    Verify

    originalpred = model.predict(Xtest[:5])

    loadedpred = loadedmodel.predict(Xtest[:5])

    assert np.arrayequal(originalpred, loadedpred)

    print("Model loaded successfully!")

    CatBoost with Scikit-learn Pipeline

    from sklearn.pipeline import Pipeline
    

    from sklearn.preprocessing import StandardScaler

    from sklearn.compose import ColumnTransformer

    from catboost import CatBoostClassifier

    Define pipeline

    numericfeatures = ['Age', 'SibSp', 'Parch', 'Fare']

    categoricalfeatures = ['Sex', 'Embarked']

    preprocessor = ColumnTransformer(

    transformers=[

    ('num', StandardScaler(), numericfeatures),

    ],

    remainder='passthrough'

    )

    pipeline = Pipeline([

    ('preprocessor', preprocessor),

    ('classifier', CatBoostClassifier(

    iterations=500,

    verbose=0,

    catfeatures=[4, 5] # Indices after transformation

    ))

    ])

    pipeline.fit(Xtrain, ytrain)

    score = pipeline.score(Xtest, ytest)

    print(f"Pipeline accuracy: {score:.4f}")

    Training Monitoring with Visualization

    from catboost import CatBoostClassifier
    
    

    Training with metric logging

    model = CatBoostClassifier(

    iterations=500,

    learningrate=0.05,

    depth=6,

    lossfunction='Logloss',

    evalmetric='AUC',

    custommetric=['Accuracy', 'Precision', 'Recall', 'F1'],

    randomseed=42,

    verbose=0,

    traindir='catboostinfo' # Directory for training logs

    )

    model.fit(

    trainpool,

    evalset=testpool,

    plot=False

    )

    Access training metrics

    evalsresult = model.getevalsresult()

    print("Available metrics:", list(evalsresult['validation'].keys()))

    Plot learning curves manually

    import matplotlib.pyplot as plt

    fig, axes = plt.subplots(1, 2, figsize=(14, 5))

    Loss curve

    axes[0].plot(evalsresult['learn']['Logloss'], label='Train')

    axes[0].plot(evalsresult['validation']['Logloss'], label='Validation')

    axes[0].setxlabel('Iteration')

    axes[0].setylabel('Logloss')

    axes[0].settitle('Loss Curve')

    axes[0].legend()

    AUC curve

    axes[1].plot(evalsresult['validation']['AUC'], label='Validation AUC')

    axes[1].setxlabel('Iteration')

    axes[1].setylabel('AUC')

    axes[1].settitle('AUC Curve')

    axes[1].legend()

    plt.tightlayout()

    plt.savefig('trainingcurves.png', dpi=150)

    plt.show()

    Handling Missing Values

    CatBoost handles missing values natively:

    # CatBoost handles NaN automatically
    

    datawithnan = Xtrain.copy()

    Deliberately add missing values

    mask = np.random.random(datawithnan.shape) < 0.1

    datawithnan = datawithnan.mask(

    pd.DataFrame(mask, columns=datawithnan.columns, index=datawithnan.index)

    )

    print(f"Missing values:\n{datawithnan.isnull().sum()}")

    CatBoost automatically handles NaN

    modelnan = CatBoostClassifier(iterations=300, verbose=0)

    modelnan.fit(

    datawithnan, ytrain,

    catfeatures=catfeatures

    )

    accuracy = modelnan.score(Xtest, ytest)

    print(f"Accuracy with missing values: {accuracy:.4f}")

    Model Deployment with ONNX

    # Export to ONNX for deployment
    

    model.savemodel(

    'catboostmodel.onnx',

    format='onnx',

    exportparameters={

    'onnxdomain': 'ai.catboost',

    'onnxmodelversion': 1,

    }

    )

    Inference using ONNX Runtime

    import onnxruntime as ort

    session = ort.InferenceSession('catboostmodel.onnx')

    Prepare input

    inputname = session.getinputs()[0].name

    sample = Xtest.iloc[:5].values.astype(np.float32)

    Predict

    onnxpredictions = session.run(None, {inputname: sample})

    print("ONNX predictions:", onnxpredictions[0])

    Best Practices

    1. Hyperparameter Selection

    # Recommended hyperparameters based on dataset size
    
    

    def getrecommendedparams(nsamples, nfeatures):

    if nsamples < 1000:

    return {

    'iterations': 300,

    'learningrate': 0.1,

    'depth': 4,

    'l2leafreg': 5,

    'mindatainleaf': 1

    }

    elif nsamples < 10000:

    return {

    'iterations': 1000,

    'learningrate': 0.05,

    'depth': 6,

    'l2leafreg': 3,

    'mindatainleaf': 5

    }

    else:

    return {

    'iterations': 2000,

    'learningrate': 0.03,

    'depth': 8,

    'l2leafreg': 1,

    'mindatainleaf': 10

    }

    params = getrecommendedparams(len(Xtrain), Xtrain.shape[1])

    print("Recommended params:", params)

    2. Preventing Overfitting

    # Anti-overfitting configuration
    

    model = CatBoostClassifier(

    iterations=2000,

    learningrate=0.03,

    depth=6,

    l2leafreg=5, # L2 regularization

    randomstrength=1, # Randomness for splits

    baggingtemperature=1, # Bayesian bootstrap temperature

    odtype='Iter', # Overfitting detector

    odwait=50, # Patience for early stopping

    verbose=0

    )

    3. Handling Imbalanced Datasets

    # For imbalanced datasets
    

    from sklearn.utils.classweight import computeclassweight

    classweights = computeclassweight(

    'balanced', classes=np.unique(ytrain), y=ytrain

    )

    classweightdict = dict(zip(np.unique(ytrain), classweights))

    model = CatBoostClassifier(

    iterations=500,

    classweights=classweightdict,

    autoclassweights='Balanced', # Or use this instead

    evalmetric='F1',

    verbose=0

    )

    4. Snapshot Training (Resume Training)

    # Save snapshots to resume training
    

    model = CatBoostClassifier(

    iterations=1000,

    learningrate=0.05,

    savesnapshot=True,

    snapshotfile='trainingsnapshot',

    snapshotinterval=100,

    verbose=100

    )

    First training run (e.g., interrupted at iteration 400)

    model.fit(trainpool, evalset=testpool)

    Resume training from the last snapshot

    modelresumed = CatBoostClassifier(

    iterations=2000, # Increase iterations

    learningrate=0.05,

    savesnapshot=True,

    snapshotfile='trainingsnapshot',

    verbose=100

    )

    modelresumed.fit(trainpool, evalset=testpool)

    CatBoost vs XGBoost vs LightGBM Comparison

    Here is a brief comparison of the three popular gradient boosting libraries:

    | Aspect | CatBoost | XGBoost | LightGBM |

    |--------|----------|---------|----------|

    | Categorical Features | Native support | Requires encoding | Partial support |

    | Missing Values | Native | Native | Native |

    | Default Performance | Excellent | Good | Good |

    | GPU Training | Yes | Yes | Yes |

    | Training Speed | Moderate | Moderate | Fast |

    | Overfitting Resistance | High | Moderate | Moderate |

    | ONNX Export | Yes | Yes | Yes |

    CatBoost tends to deliver the best out-of-the-box performance without extensive tuning, especially on datasets with categorical features. XGBoost and LightGBM may require additional feature engineering to achieve similar performance.

    Conclusion

    CatBoost is a powerful gradient boosting library with several key advantages: native categorical feature handling, overfitting resistance, and already-optimized default performance. It is suitable for various machine learning tasks on tabular data, from classification and regression to ranking.

    Key takeaways:

    • Always specify catfeatures to maximize CatBoost's capabilities
    • Use Pool objects for memory efficiency and speed
    • Leverage built-in cross-validation and grid search
    • Export models to ONNX for efficient deployment
    • Use early stopping to prevent overfitting

    With a solid understanding of CatBoost, you can build accurate and robust machine learning models for various business and research needs.

    Related Articles

    XGBoost & LightGBM Tutorial: Gradient Boosting Masterclass

    XGBoost & LightGBM - Masterclass Gradient Boosting Daftar Isi Pendahuluan Prasyarat Memahami Gradient Boosting [Pelatiha...

    PyCaret Tutorial: Low-Code Machine Learning for Rapid Prototyping

    Tutorial PyCaret: Machine Learning Low-Code untuk Prototyping Cepat PyCaret adalah library open-source Python yang menye...

    Cleanlab Tutorial: Improving Data Quality for Machine Learning

    Tutorial Cleanlab: Meningkatkan Kualitas Data untuk Machine Learning Kualitas data adalah fondasi dari setiap proyek mac...

    Complete Comet ML Tutorial: MLOps Platform for Experiment Tracking and Model Management

    Tutorial Lengkap Comet ML: Platform MLOps untuk Experiment Tracking dan Model Management Dalam dunia machine learning mo...