XGBoost & LightGBM - Masterclass Gradient Boosting

Daftar Isi

Pendahuluan

Gradient boosting adalah salah satu teknik machine learning paling kuat untuk data terstruktur/tabular. XGBoost (eXtreme Gradient Boosting) dan LightGBM (Light Gradient Boosting Machine) adalah dua implementasi terdepan yang secara konsisten memenangkan kompetisi Kaggle dan menggerakkan sistem ML produksi di berbagai industri. Masterclass ini mencakup kedua library dari dasar hingga deployment produksi.

XGBoost, dikembangkan oleh Tianqi Chen, memperkenalkan regularisasi ke gradient boosting dan mempopulerkan teknik ini. LightGBM, dikembangkan oleh Microsoft, membawa inovasi seperti pemisahan berbasis histogram dan pertumbuhan pohon berbasis daun (leaf-wise) untuk pelatihan yang lebih cepat. Memahami keduanya memungkinkan Anda memilih alat yang tepat untuk setiap masalah.

Prasyarat

• Python 3.8 atau lebih tinggi
• Pemahaman dasar tentang konsep machine learning (klasifikasi, regresi, overfitting)
• Keakraban dengan konvensi API scikit-learn

pip install xgboost lightgbm
pip install scikit-learn pandas numpy
pip install shap matplotlib seaborn
pip install optuna  # Untuk optimasi hyperparameter

Memahami Gradient Boosting

Gradient boosting membangun ansambel dari pelajar lemah (biasanya pohon keputusan) secara berurutan, di mana setiap pohon baru mengoreksi kesalahan dari ansambel sebelumnya:

import numpy as np

Ilustrasi konseptual gradient boosting
Langkah 1: Mulai dengan prediksi awal (misal, rata-rata target)
Langkah 2: Hitung residual (aktual - prediksi)
Langkah 3: Pasang pohon ke residual
Langkah 4: Perbarui prediksi: prediksibaru = prediksilama + learningrate  prediksipohon

Langkah 5: Ulangi langkah 2-4

Perbedaan utama antara XGBoost dan LightGBM:
XGBoost: pertumbuhan pohon level-wise (breadth-first)
LightGBM: pertumbuhan pohon leaf-wise (best-first) - lebih cepat tapi risiko overfitting

XGBoost memisahkan node level per level
LightGBM memisahkan daun dengan pengurangan loss tertinggi

import xgboost as xgb
import numpy as np
from sklearn.datasets import makeclassification

from sklearn.modelselection import traintestsplit
from sklearn.metrics import accuracyscore, classificationreport, rocaucscore

Buat data sampel
X, y = makeclassification(

    nsamples=10000, nfeatures=20, ninformative=15,
    nredundant=3, randomstate=42
)
Xtrain, Xtest, ytrain, ytest = traintestsplit(X, y, testsize=0.2, randomstate=42)

Menggunakan API scikit-learn
clf = xgb.XGBClassifier(
    nestimators=500,

    maxdepth=6,
    learningrate=0.1,

    subsample=0.8,
    colsamplebytree=0.8,
    minchildweight=5,
    gamma=0.1,
    regalpha=0.1,        # Regularisasi L1

    reglambda=1.0,       # Regularisasi L2
    objective="binary:logistic",
    evalmetric="logloss",

    treemethod="hist",   # Metode berbasis histogram (lebih cepat)
    device="cuda",        # Gunakan GPU jika tersedia
    randomstate=42,

    njobs=-1
)

clf.fit(
    Xtrain, ytrain,
    evalset=[(Xtest, ytest)],

    verbose=50
)

ypred = clf.predict(Xtest)

yproba = clf.predictproba(Xtest)[:, 1]

print(f"Akurasi: {accuracyscore(ytest, ypred):.4f}")

print(f"AUC-ROC: {rocaucscore(ytest, yproba):.4f}")

print(classificationreport(ytest, ypred))

from sklearn.datasets import makeregression
from sklearn.metrics import meansquarederror, r2score

X, y = makeregression(nsamples=10000, nfeatures=20, noise=10, randomstate=42)
Xtrain, Xtest, ytrain, ytest = traintestsplit(X, y, testsize=0.2, randomstate=42)

reg = xgb.XGBRegressor(
    nestimators=500,

    maxdepth=6,
    learningrate=0.05,

    subsample=0.8,
    colsamplebytree=0.8,
    minchildweight=5,
    regalpha=0.1,

    reglambda=1.0,
    objective="reg:squarederror",
    treemethod="hist",

    randomstate=42
)

reg.fit(
    Xtrain, ytrain,
    evalset=[(Xtest, ytest)],

    verbose=100
)

ypred = reg.predict(Xtest)

print(f"RMSE: {np.sqrt(meansquarederror(ytest, ypred)):.4f}")

print(f"R2:   {r2score(ytest, ypred):.4f}")

# API native memberikan kontrol lebih dan seringkali lebih cepat
dtrain = xgb.DMatrix(Xtrain, label=ytrain, featurenames=[f"f{i}" for i in range(20)])

dtest = xgb.DMatrix(Xtest, label=ytest, featurenames=[f"f{i}" for i in range(20)])

params = {
    "maxdepth": 6,

    "eta": 0.1,
    "objective": "binary:logistic",
    "evalmetric": ["logloss", "auc"],
    "subsample": 0.8,
    "colsamplebytree": 0.8,

    "minchildweight": 5,

    "gamma": 0.1,
    "alpha": 0.1,
    "lambda": 1.0,
    "treemethod": "hist",
    "seed": 42
}

evals = [(dtrain, "train"), (dtest, "eval")]
model = xgb.train(
    params, dtrain,
    numboostround=500,
    evals=evals,
    earlystoppingrounds=50,
    verboseeval=50

)

print(f"Iterasi terbaik: {model.bestiteration}")
print(f"Skor terbaik: {model.bestscore:.4f}")

import lightgbm as lgb
from sklearn.metrics import accuracyscore, rocaucscore, classificationreport


Menggunakan API scikit-learn
clf = lgb.LGBMClassifier(
    nestimators=500,
    maxdepth=-1,           # Tanpa batas (dikontrol oleh numleaves)
    numleaves=31,          # Parameter kunci untuk LightGBM

    learningrate=0.1,
    subsample=0.8,
    colsamplebytree=0.8,

    minchildsamples=20,

    minchildweight=1e-3,

    regalpha=0.1,
    reglambda=1.0,

    objective="binary",
    metric="binarylogloss",
    boostingtype="gbdt",   # Opsi: gbdt, dart, rf

    njobs=-1,
    randomstate=42,

    verbose=-1
)

clf.fit(
    Xtrain, ytrain,

    evalset=[(Xtest, ytest)],
    callbacks=[
        lgb.earlystopping(50),

        lgb.logevaluation(50)
    ]
)

ypred = clf.predict(Xtest)
yproba = clf.predictproba(Xtest)[:, 1]


print(f"Akurasi: {accuracyscore(ytest, ypred):.4f}")
print(f"AUC-ROC: {rocaucscore(ytest, yproba):.4f}")
print(classificationreport(ytest, ypred))

# API native untuk kontrol lebih
traindata = lgb.Dataset(Xtrain, label=ytrain)
testdata = lgb.Dataset(Xtest, label=ytest, reference=traindata)

params = {
    "objective": "binary",
    "metric": ["binarylogloss", "auc"],

    "boostingtype": "gbdt",
    "numleaves": 31,

    "learningrate": 0.1,
    "featurefraction": 0.8,

    "baggingfraction": 0.8,
    "baggingfreq": 5,

    "minchildsamples": 20,

    "lambdal1": 0.1,
    "lambdal2": 1.0,

    "verbose": -1,
    "seed": 42,
    "numthreads": -1
}

callbacks = [
    lgb.earlystopping(50),

    lgb.logevaluation(50)
]

model = lgb.train(
    params, traindata,

    numboostround=500,

    validsets=[traindata, testdata],
    validnames=["train", "eval"],

    callbacks=callbacks
)

print(f"Iterasi terbaik: {model.bestiteration}")
print(f"Skor terbaik: {model.bestscore}")

import pandas as pd
import lightgbm as lgb

LightGBM menangani fitur kategorikal secara native
df = pd.DataFrame({
    "warna": pd.Categorical(["merah", "biru", "hijau", "merah", "biru"]  2000),
    "ukuran": pd.Categorical(["S", "M", "L", "XL", "S"]  2000),

    "berat": np.random.randn(10000),
    "harga": np.random.uniform(10, 100, 10000),
    "target": np.random.randint(0, 2, 10000)
})

X = df.drop("target", axis=1)
y = df["target"]
Xtrain, Xtest, ytrain, ytest = traintestsplit(X, y, testsize=0.2, randomstate=42)


traindata = lgb.Dataset(Xtrain, label=ytrain, categoricalfeature=["warna", "ukuran"])

testdata = lgb.Dataset(Xtest, label=ytest, reference=traindata)


params = {
    "objective": "binary",
    "metric": "binarylogloss",
    "numleaves": 31,

    "learningrate": 0.1,
    "verbose": -1
}

model = lgb.train(
    params, traindata,

    numboostround=200,

    validsets=[testdata],

    callbacks=[lgb.earlystopping(20), lgb.logevaluation(50)]

)

import time
import numpy as np
from sklearn.datasets import makeclassification
from sklearn.modelselection import traintestsplit

from sklearn.metrics import rocaucscore

import xgboost as xgb
import lightgbm as lgb

def bandingkanmodel(nsamples=100000, nfeatures=50):
    X, y = makeclassification(

        nsamples=nsamples, nfeatures=nfeatures,

        ninformative=30, randomstate=42

    )
    Xtrain, Xtest, ytrain, ytest = traintestsplit(X, y, testsize=0.2, randomstate=42)


    # XGBoost
    mulai = time.time()
    modelxgb = xgb.XGBClassifier(
        nestimators=500, maxdepth=6, learningrate=0.1,

        treemethod="hist", randomstate=42, verbosity=0,

        earlystoppingrounds=50, evalmetric="logloss"
    )
    modelxgb.fit(Xtrain, ytrain, evalset=[(Xtest, ytest)], verbose=False)
    waktuxgb = time.time() - mulai

    aucxgb = rocaucscore(ytest, modelxgb.predictproba(Xtest)[:, 1])

    # LightGBM
    mulai = time.time()
    modellgb = lgb.LGBMClassifier(

        nestimators=500, numleaves=31, learningrate=0.1,
        randomstate=42, verbose=-1

    )
    modellgb.fit(
        Xtrain, ytrain, evalset=[(Xtest, ytest)],

        callbacks=[lgb.earlystopping(50), lgb.logevaluation(0)]

    )
    waktulgb = time.time() - mulai
    auclgb = rocaucscore(ytest, modellgb.predictproba(Xtest)[:, 1])


    print(f"{'Metrik':<25} {'XGBoost':<15} {'LightGBM':<15}")
    print("-"  55)
    print(f"{'Waktu Pelatihan':<25} {waktuxgb:.2f}d{'':<10} {waktulgb:.2f}d")

    print(f"{'AUC-ROC':<25} {aucxgb:.4f}{'':<10} {auclgb:.4f}")

    print(f"{'Jumlah Pohon':<25} {modelxgb.bestiteration}{'':<10} {modellgb.bestiteration}")
    print(f"{'Rasio Kecepatan':<25} 1.0x{'':<11} {waktuxgb/waktulgb:.1f}x lebih cepat")

bandingkanmodel(100000, 50)

| Risiko Overfitting | Lebih rendah | Lebih tinggi (mitigasi dengan numleaves) |

| Dukungan GPU | Ya | Ya |

| Nilai Hilang | Penanganan bawaan | Penanganan bawaan |

Optimasi Hyperparameter

Menggunakan Optuna untuk pencarian hyperparameter yang sistematis:

import optuna
from sklearn.modelselection import crossvalscore

import xgboost as xgb
import lightgbm as lgb

def objectivexgboost(trial):
    params = {
        "nestimators": trial.suggestint("nestimators", 100, 1000),

        "maxdepth": trial.suggestint("maxdepth", 3, 10),
        "learningrate": trial.suggestfloat("learningrate", 0.01, 0.3, log=True),

        "subsample": trial.suggestfloat("subsample", 0.5, 1.0),
        "colsamplebytree": trial.suggestfloat("colsamplebytree", 0.5, 1.0),

        "minchildweight": trial.suggestint("minchildweight", 1, 20),
        "gamma": trial.suggestfloat("gamma", 0.0, 5.0),

        "regalpha": trial.suggestfloat("regalpha", 1e-8, 10.0, log=True),
        "reglambda": trial.suggestfloat("reglambda", 1e-8, 10.0, log=True),

        "treemethod": "hist",
        "randomstate": 42,

        "verbosity": 0
    }

    model = xgb.XGBClassifier(params)

    scores = crossvalscore(model, Xtrain, ytrain, cv=5, scoring="rocauc", njobs=-1)

    return scores.mean()

def objectivelightgbm(trial):
    params = {
        "nestimators": trial.suggestint("nestimators", 100, 1000),

        "numleaves": trial.suggestint("numleaves", 15, 127),
        "maxdepth": trial.suggestint("maxdepth", 3, 12),

        "learningrate": trial.suggestfloat("learningrate", 0.01, 0.3, log=True),
        "subsample": trial.suggestfloat("subsample", 0.5, 1.0),

        "colsamplebytree": trial.suggestfloat("colsamplebytree", 0.5, 1.0),
        "minchildsamples": trial.suggestint("minchildsamples", 5, 100),

        "regalpha": trial.suggestfloat("regalpha", 1e-8, 10.0, log=True),
        "reglambda": trial.suggestfloat("reglambda", 1e-8, 10.0, log=True),

        "randomstate": 42,
        "verbose": -1
    }

    model = lgb.LGBMClassifier(params)
    scores = crossvalscore(model, Xtrain, ytrain, cv=5, scoring="rocauc", njobs=-1)
    return scores.mean()

Jalankan optimasi
studixgb = optuna.createstudy(direction="maximize", studyname="xgboost")

studixgb.optimize(objectivexgboost, ntrials=100, showprogressbar=True)

studilgb = optuna.createstudy(direction="maximize", studyname="lightgbm")

studilgb.optimize(objectivelightgbm, ntrials=100, showprogressbar=True)

print(f"AUC terbaik XGBoost: {studixgb.bestvalue:.4f}")
print(f"Parameter terbaik XGBoost: {studixgb.bestparams}")
print(f"AUC terbaik LightGBM: {studilgb.bestvalue:.4f}")
print(f"Parameter terbaik LightGBM: {studilgb.bestparams}")

Explainability dengan SHAP

SHAP (SHapley Additive exPlanations) memberikan penjelasan yang dapat diinterpretasi untuk prediksi model:

import shap
import xgboost as xgb
import matplotlib.pyplot as plt

Latih model
model = xgb.XGBClassifier(nestimators=200, maxdepth=6, learningrate=0.1, randomstate=42)
model.fit(Xtrain, ytrain)

Buat explainer SHAP
explainer = shap.TreeExplainer(model)
shapvalues = explainer.shapvalues(Xtest)


Plot ringkasan - kepentingan fitur global dengan arah
namafitur = [f"fitur{i}" for i in range(Xtest.shape[1])]
shap.summaryplot(shapvalues, Xtest, featurenames=namafitur)

plt.tightlayout()
plt.savefig("shapringkasan.png", dpi=150, bboxinches="tight")
plt.close()

Plot batang - rata-rata absolut nilai SHAP
shap.summaryplot(shapvalues, Xtest, featurenames=namafitur, plottype="bar")
plt.tightlayout()

plt.savefig("shapbatang.png", dpi=150, bboxinches="tight")

plt.close()

Plot waterfall untuk prediksi tunggal
shap.waterfallplot(shap.Explanation(
    values=shapvalues[0],

    basevalues=explainer.expectedvalue,

    data=Xtest[0],
    featurenames=namafitur
))
plt.tightlayout()

plt.savefig("shapwaterfall.png", dpi=150, bboxinches="tight")

plt.close()

Plot dependensi - efek interaksi
shap.dependenceplot(
    "fitur0", shapvalues, Xtest,

    featurenames=namafitur,

    interactionindex="fitur1"

)
plt.tightlayout()
plt.savefig("shapdependensi.png", dpi=150, bboxinches="tight")
plt.close()

Analisis Feature Importance

import xgboost as xgb
import lightgbm as lgb
import pandas as pd

namafitur = [f"fitur{i}" for i in range(Xtrain.shape[1])]


Kepentingan fitur XGBoost (beberapa metode)
modelxgb = xgb.XGBClassifier(nestimators=200, randomstate=42, verbosity=0)
modelxgb.fit(Xtrain, ytrain)


Tipe kepentingan bawaan
tipekepentingan = ["weight", "gain", "cover", "totalgain", "totalcover"]
dfkepentingan = pd.DataFrame({"fitur": namafitur})

for tipe in tipekepentingan:

    skor = modelxgb.getbooster().getscore(importancetype=tipe)

    dfkepentingan[tipe] = [skor.get(f, 0) for f in modelxgb.getbooster().featurenames]


print("Kepentingan Fitur XGBoost (berdasarkan gain):")
print(dfkepentingan.sortvalues("gain", ascending=False).head(10))


Kepentingan fitur LightGBM
modellgb = lgb.LGBMClassifier(nestimators=200, randomstate=42, verbose=-1)
modellgb.fit(Xtrain, ytrain)


kepentinganlgb = pd.DataFrame({
    "fitur": namafitur,

    "kepentingansplit": modellgb.featureimportances,

    "kepentingangain": modellgb.booster.featureimportance(importancetype="gain")
}).sortvalues("kepentingangain", ascending=False)

print("\nKepentingan Fitur LightGBM (berdasarkan gain):")
print(kepentinganlgb.head(10))


Permutation importance (model-agnostik)
from sklearn.inspection import permutationimportance

permimp = permutationimportance(
    modelxgb, Xtest, ytest,

    nrepeats=10, randomstate=42, njobs=-1, scoring="rocauc"

)

dfperm = pd.DataFrame({
    "fitur": namafitur,

    "ratakepentingan": permimp.importancesmean,
    "stdkepentingan": permimp.importancesstd

}).sortvalues("ratakepentingan", ascending=False)


print("\nPermutation Importance:")
print(dfperm.head(10))

Early Stopping dan Cross-Validation

import xgboost as xgb
import lightgbm as lgb
from sklearn.modelselection import StratifiedKFold

import numpy as np

Cross-validation bawaan XGBoost
dtrain = xgb.DMatrix(Xtrain, label=ytrain)


params = {
    "maxdepth": 6,
    "eta": 0.1,
    "objective": "binary:logistic",
    "evalmetric": "auc",

    "treemethod": "hist",
    "seed": 42
}

cvresults = xgb.cv(

    params, dtrain,
    numboostround=1000,

    nfold=5,
    stratified=True,
    earlystoppingrounds=50,

    verboseeval=100,
    seed=42
)

print(f"Numboostround terbaik: {len(cvresults)}")

print(f"AUC terbaik: {cvresults['test-auc-mean'].iloc[-1]:.4f} +/- {cvresults['test-auc-std'].iloc[-1]:.4f}")


Cross-validation bawaan LightGBM
traindata = lgb.Dataset(Xtrain, label=ytrain)

paramslgb = {

    "objective": "binary",
    "metric": "auc",
    "numleaves": 31,
    "learningrate": 0.1,

    "verbose": -1,
    "seed": 42
}

cvresultslgb = lgb.cv(

    paramslgb, traindata,

    numboostround=1000,

    nfold=5,
    stratified=True,
    callbacks=[lgb.earlystopping(50), lgb.logevaluation(100)],

    seed=42,
    returncvbooster=True
)

print(f"Iterasi terbaik: {cvresultslgb['cvbooster'].bestiteration}")

print(f"AUC terbaik: {max(cvresultslgb['valid auc-mean']):.4f}")


Cross-validation manual dengan logika kustom
skf = StratifiedKFold(nsplits=5, shuffle=True, randomstate=42)

skorlipatan = []
modellipatan = []


for lipatan, (idxtrain, idxval) in enumerate(skf.split(Xtrain, ytrain)):

    Xlipatantrain, Xlipatanval = Xtrain[idxtrain], Xtrain[idxval]

    ylipatantrain, ylipatanval = ytrain[idxtrain], ytrain[idxval]


    model = xgb.XGBClassifier(
        nestimators=1000, maxdepth=6, learningrate=0.1,
        earlystoppingrounds=50, evalmetric="auc",

        treemethod="hist", randomstate=42, verbosity=0

    )

    model.fit(
        Xlipatantrain, ylipatantrain,

        evalset=[(Xlipatanval, ylipatanval)],
        verbose=False
    )

    valpred = model.predictproba(Xlipatanval)[:, 1]
    auc = rocaucscore(ylipatanval, valpred)

    skorlipatan.append(auc)
    modellipatan.append(model)

    print(f"Lipatan {lipatan + 1}: AUC = {auc:.4f} (iterasi terbaik: {model.bestiteration})")

print(f"\nRata-rata AUC: {np.mean(skorlipatan):.4f} +/- {np.std(skorlipatan):.4f}")

Deployment Model

Menyimpan dan Memuat Model

import xgboost as xgb
import lightgbm as lgb
import joblib

XGBoost - beberapa format penyimpanan
modelxgb.savemodel("modelxgb.json")              # Format JSON (direkomendasikan)

modelxgb.savemodel("modelxgb.ubj")               # Universal Binary JSON
modelxgb.getbooster().savemodel("modelxgb.bin")  # Format biner

Muat model XGBoost
xgbdimuat = xgb.XGBClassifier()

xgbdimuat.loadmodel("modelxgb.json")

LightGBM
modellgb.booster.savemodel("modellgb.txt")       # Format teks
joblib.dump(modellgb, "modellgb.joblib")            # Joblib (mempertahankan wrapper sklearn)

Muat model LightGBM
lgbdimuat = lgb.Booster(modelfile="modellgb.txt")

lgbsklearndimuat = joblib.load("modellgb.joblib")

Deployment dengan FastAPI

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import xgboost as xgb
import numpy as np
from typing import List

app = FastAPI(title="API Prediksi Gradient Boosting")

Muat model saat startup
modelxgb = xgb.XGBClassifier()

modelxgb.loadmodel("modelxgb.json")

class PermintaanPrediksi(BaseModel):
    fitur: List[List[float]]

class ResponPrediksi(BaseModel):
    prediksi: List[int]
    probabilitas: List[List[float]]

@app.post("/prediksi", responsemodel=ResponPrediksi)

async def prediksi(permintaan: PermintaanPrediksi):
    try:
        X = np.array(permintaan.fitur)
        prediksi = modelxgb.predict(X).tolist()
        probabilitas = modelxgb.predictproba(X).tolist()

        return ResponPrediksi(
            prediksi=prediksi,
            probabilitas=probabilitas
        )
    except Exception as e:
        raise HTTPException(statuscode=400, detail=str(e))


@app.get("/model/info")
async def infomodel():
    booster = modelxgb.getbooster()
    return {
        "jumlahpohon": booster.numboostedrounds(),

        "jumlahfitur": booster.numfeatures(),

        "namafitur": booster.featurenames

    }

Jalankan: uvicorn deployment:app --host 0.0.0.0 --port 8000

from skl2onnx import convertsklearn
from skl2onnx.common.datatypes import FloatTensorType

import onnxruntime as ort
import numpy as np

Konversi model XGBoost ke ONNX
tipeawal = [("floatinput", FloatTensorType([None, Xtrain.shape[1]]))]
modelonnx = convertsklearn(modelxgb, initialtypes=tipeawal)


with open("model.onnx", "wb") as f:
    f.write(modelonnx.SerializeToString())

Inferensi dengan ONNX Runtime
session = ort.InferenceSession("model.onnx")
namainput = session.getinputs()[0].name

Prediksi
Xsampel = Xtest[:5].astype(np.float32)
hasil = session.run(None, {namainput: Xsampel})
print(f"Prediksi: {hasil[0]}")
print(f"Probabilitas: {hasil[1]}")

Praktik Terbaik

Kesimpulan

XGBoost dan LightGBM tetap menjadi standar emas untuk machine learning data terstruktur. XGBoost menawarkan performa yang kuat dan teruji dengan opsi regularisasi yang solid, sementara LightGBM menyediakan pelatihan lebih cepat dan dukungan fitur kategorikal native. Untuk sebagian besar aplikasi praktis, kedua library akan memberikan hasil yang sangat baik. Kunci keberhasilan terletak pada rekayasa fitur yang tepat, optimasi hyperparameter yang sistematis dengan alat seperti Optuna, cross-validation yang ketat, dan interpretabilitas berbasis SHAP untuk membangun kepercayaan pada prediksi model. Pilih XGBoost saat Anda membutuhkan ketahanan maksimal, dan LightGBM saat kecepatan pelatihan dan efisiensi memori menjadi prioritas.