Cleanlab Tutorial: Improving Data Quality for Machine Learning

Data quality is the foundation of every successful machine learning project. No matter how sophisticated your model is, if the training data contains mislabeled examples or excessive noise, model performance will degrade significantly. Cleanlab is a powerful open-source solution that automatically detects and fixes data quality issues.

In this tutorial, we will learn how to use Cleanlab to find label errors, identify outliers, and build cleaner datasets so your machine learning models can achieve optimal performance.

What Is Cleanlab?

Cleanlab is an open-source Python library that implements the Confident Learning algorithm to detect issues in datasets. The library can find:

• Label errors: Data with incorrect labels in the training dataset
• Outliers: Unusual or anomalous data points
• Near-duplicates: Nearly identical data that could cause data leakage
• Class overlap: Areas where classes overlap with each other

Cleanlab works with any machine learning model — scikit-learn, PyTorch, TensorFlow, XGBoost, and more. You simply provide the predicted probabilities from your model, and Cleanlab will analyze the results.

Installation

Basic Installation

pip install cleanlab

Installation with Additional Dependencies

pip install "cleanlab[datalab]"

Development Installation

pip install "cleanlab[all]"

Verify Installation

import cleanlab
print(cleanlab.version)

Environment Setup

Ensure you have the following dependencies:

pip install scikit-learn pandas numpy matplotlib seaborn

Basic Usage: Finding Label Errors

Example 1: Simple Classification Dataset

Let's start with a simple example using the Iris dataset:

import numpy as np
import pandas as pd
from sklearn.datasets import loadiris

from sklearn.modelselection import crossvalpredict
from sklearn.linearmodel import LogisticRegression

from cleanlab.filter import findlabelissues


iris = loadiris()
X = iris.data
y = iris.target

np.random.seed(42)
noisyindices = np.random.choice(len(y), size=15, replace=False)

ynoisy = y.copy()
for idx in noisyindices:

    possiblelabels = [l for l in range(3) if l != y[idx]]
    ynoisy[idx] = np.random.choice(possiblelabels)

print(f"Number of intentionally corrupted labels: {len(noisyindices)}")

print(f"Corrupted indices: {sorted(noisyindices)}")

Getting Predicted Probabilities

Cleanlab requires out-of-sample predicted probabilities. The best approach is using cross-validation:

model = LogisticRegression(maxiter=500, randomstate=42)

predprobs = crossvalpredict(

    model, X, ynoisy,
    cv=5,
    method="predictproba"

)

print(f"predprobs shape: {predprobs.shape}")

print(f"Example probabilities for first sample: {predprobs[0]}")

Finding Label Issues

labelissues = findlabelissues(
    labels=ynoisy,
    predprobs=predprobs,
    returnindicesrankedby="selfconfidence"
)

print(f"\nNumber of label issues found: {len(labelissues)}")

print(f"Detected indices: {sorted(labelissues)}")

detected = set(labelissues)

actual = set(noisyindices)
truepositives = detected & actual

precision = len(truepositives) / len(detected) if detected else 0
recall = len(truepositives) / len(actual) if actual else 0


print(f"\nPrecision: {precision:.2f}")
print(f"Recall: {recall:.2f}")

from cleanlab.classification import CleanLearning

cl = CleanLearning(
    clf=LogisticRegression(maxiter=500, randomstate=42),

    seed=42
)

cl.fit(X, ynoisy)

labelissuesdf = cl.getlabelissues()
print(labelissuesdf.head(10))

print(f"\nAvailable columns: {labelissuesdf.columns.tolist()}")
print(f"Number of label issues: {labelissuesdf['islabelissue'].sum()}")

Comparing Performance

from sklearn.modelselection import crossvalscore

baselinemodel = LogisticRegression(maxiter=500, randomstate=42)
baselinescores = crossvalscore(baselinemodel, X, ynoisy, cv=5)

print(f"Accuracy with noisy data: {baselinescores.mean():.4f} (+/- {baselinescores.std():.4f})")


cleanmask = ~labelissuesdf["islabelissue"].values
Xclean = X[cleanmask]
yclean = ynoisy[cleanmask]


cleanmodel = LogisticRegression(maxiter=500, randomstate=42)
cleanscores = crossvalscore(cleanmodel, Xclean, yclean, cv=5)
print(f"Accuracy with clean data: {cleanscores.mean():.4f} (+/- {cleanscores.std():.4f})")

Using Datalab for Comprehensive Analysis

from cleanlab import Datalab

lab = Datalab(
    data=pd.DataFrame(X, columns=iris.featurenames),

    labelname="label",
    task="classification"
)

lab.findissues(predprobs=predprobs, features=X)


lab.report()

issuesummary = lab.getissuesummary()
print(issuesummary)


labelissues = lab.getissues("label")

print("\nTop 10 data points with lowest label issue score:")
print(labelissues.nsmallest(10, "labelscore"))


outlierissues = lab.getissues("outlier")

print("\nTop 10 outliers:")
print(outlierissues.nsmallest(10, "outlierscore"))

duplicateissues = lab.getissues("nearduplicate")
neardupes = duplicateissues[duplicateissues["isnearduplicateissue"]]
print(f"\nNumber of near-duplicates: {len(neardupes)}")


if len(neardupes) > 0:
    for idx in neardupes.index[:5]:

        neardupset = duplicateissues.loc[idx, "nearduplicatesets"]
        score = duplicateissues.loc[idx, "nearduplicatescore"]

        print(f"  Index {idx} (score: {score:.4f}): near-duplicate with {neardupset}")

import pandas as pd
from sklearn.featureextraction.text import TfidfVectorizer
from sklearn.naivebayes import MultinomialNB

from sklearn.modelselection import crossvalpredict
from cleanlab import Datalab

texts = [
    "Stock prices surged sharply today",
    "The football team won the championship final",
    "Special fried rice recipe for the family",
    "Indonesia's economy grew by 5 percent",
    "Basketball player set a new scoring record",
    "How to make a delicious chocolate cake",
    "Central bank raised interest rates",
    "Tennis tournament starts next week",
    "Tips for cooking tender beef rendang",
    "Stock market experienced a correction",
    "The badminton match was very exciting",
    "Homemade pizza tutorial for beginners",
    "Gold investment is increasingly popular",
    "New football league season schedule",
    "Fresh fruit smoothie recipe",
    "Stock prices increased dramatically",
    "Swimming athlete broke world record",
    "How to cook traditional chicken soup",
    "Rupiah strengthened against the dollar",
    "Esports competition gaining popularity",
]

labels = [0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1]
labels[3] = 1   # Intentionally wrong: business -> sports
labels[11] = 0  # Intentionally wrong: food -> business
labels[16] = 2  # Intentionally wrong: sports -> food

labelnames = {0: "business", 1: "sports", 2: "food"}

df = pd.DataFrame({"text": texts, "label": labels})

vectorizer = TfidfVectorizer(maxfeatures=100)
Xtfidf = vectorizer.fittransform(df["text"]).toarray()

model = MultinomialNB()
predprobs = crossvalpredict(model, Xtfidf, df["label"], cv=3, method="predictproba")


lab = Datalab(data=df, labelname="label", task="classification")
lab.findissues(predprobs=predprobs, features=Xtfidf)
lab.report()

issues = lab.getissues("label")

problematic = issues[issues["islabelissue"]].sortvalues("labelscore")


for idx in problematic.index:
    text = df.loc[idx, "text"]
    givenlabel = labelnames[df.loc[idx, "label"]]

    predicted = labelnames[issues.loc[idx, "predictedlabel"]]

    score = issues.loc[idx, "labelscore"]
    print(f"  [{idx}] '{text}'")
    print(f"       Label: {givenlabel} -> Predicted: {predicted} (score: {score:.4f})")

import torch
import torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset
from sklearn.modelselection import StratifiedKFold
from cleanlab.filter import findlabelissues
import numpy as np

class SimpleClassifier(nn.Module):
    def init(self, inputdim, numclasses):
        super().init()
        self.network = nn.Sequential(
            nn.Linear(inputdim, 128),

            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(64, numclasses)
        )

    def forward(self, x):
        return self.network(x)

def getcrossvalpredprobs(X, y, nsplits=5, epochs=50):

    """Get out-of-sample predicted probabilities using cross-validation."""
    skf = StratifiedKFold(nsplits=nsplits, shuffle=True, randomstate=42)
    predprobs = np.zeros((len(y), len(np.unique(y))))


    for fold, (trainidx, validx) in enumerate(skf.split(X, y)):

        Xtrain = torch.FloatTensor(X[trainidx])

        ytrain = torch.LongTensor(y[trainidx])

        Xval = torch.FloatTensor(X[validx])


        traindataset = TensorDataset(Xtrain, ytrain)
        trainloader = DataLoader(traindataset, batchsize=32, shuffle=True)


        model = SimpleClassifier(X.shape[1], len(np.unique(y)))
        optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
        criterion = nn.CrossEntropyLoss()

        model.train()
        for epoch in range(epochs):
            for batchX, batchy in trainloader:
                optimizer.zerograd()

                outputs = model(batchX)
                loss = criterion(outputs, batchy)

                loss.backward()
                optimizer.step()

        model.eval()
        with torch.nograd():
            valoutputs = model(Xval)
            valprobs = torch.softmax(valoutputs, dim=1).numpy()
            predprobs[validx] = valprobs


        print(f"  Fold {fold+1}/{nsplits} complete")

    return predprobs


from sklearn.datasets import makeclassification

X, y = makeclassification(

    nsamples=1000, nfeatures=20, ninformative=15,
    nclasses=5, nclustersperclass=1, randomstate=42

)

noiserate = 0.1
nnoisy = int(len(y)  noiserate)

noisyidx = np.random.choice(len(y), size=nnoisy, replace=False)
ynoisy = y.copy()

for idx in noisyidx:
    possible = [l for l in range(5) if l != y[idx]]
    ynoisy[idx] = np.random.choice(possible)


print("Running cross-validation to get predicted probabilities...")
predprobs = getcrossvalpredprobs(X, ynoisy, nsplits=5, epochs=30)

issues = findlabelissues(
    labels=ynoisy,

    predprobs=predprobs,

    returnindicesrankedby="selfconfidence"

)

print(f"\nFound {len(issues)} label issues")
print(f"Actual noisy labels: {len(noisyidx)}")

overlap = len(set(issues[:nnoisy]) & set(noisyidx))
print(f"True positives (in top-{nnoisy}): {overlap}")

print(f"Recall: {overlap/nnoisy:.2%}")

import xgboost as xgb
from sklearn.modelselection import crossvalpredict

from cleanlab.classification import CleanLearning
from sklearn.datasets import loadbreastcancer


data = loadbreastcancer()

X, y = data.data, data.target

np.random.seed(42)
noiseidx = np.random.choice(len(y), size=30, replace=False)
ynoisy = y.copy()

ynoisy[noiseidx] = 1 - ynoisy[noiseidx]


xgbmodel = xgb.XGBClassifier(
    nestimators=100,

    maxdepth=3,
    learningrate=0.1,

    randomstate=42,
    evalmetric="logloss"

)

cl = CleanLearning(clf=xgbmodel, seed=42)
cl.fit(X, ynoisy)


labelissues = cl.getlabelissues()
detected = labelissues[labelissues["islabelissue"]].index.tolist()

print(f"Detected {len(detected)} label issues")
print(f"Actually noisy: {len(noiseidx)}")


tp = len(set(detected) & set(noiseidx))
precision = tp / len(detected) if detected else 0
recall = tp / len(noiseidx)

print(f"Precision: {precision:.2%}")
print(f"Recall: {recall:.2%}")

from cleanlab.count import computeconfidentjoint, estimatelatent

confidentjoint = computeconfidentjoint(

    labels=ynoisy,
    predprobs=predprobs
)

print("Confident Joint Matrix:")
print(confidentjoint)


py, noisematrix, invnoisematrix = estimatelatent(

    confidentjoint=confidentjoint,

    labels=ynoisy
)

print("\nNoise Matrix (P(noisy label | true label)):")
print(np.round(noisematrix, 3))

print("\nInverse Noise Matrix (P(true label | noisy label)):")
print(np.round(invnoisematrix, 3))

from cleanlab.rank import getlabelqualityscores

qualityscores = getlabelqualityscores(
    labels=ynoisy,

    predprobs=predprobs,

    method="selfconfidence"
)

print(f"Mean quality score: {qualityscores.mean():.4f}")

print(f"Min quality score: {qualityscores.min():.4f}")
print(f"Max quality score: {qualityscores.max():.4f}")


import matplotlib.pyplot as plt

fig, axes = plt.subplots(1, 2, figsize=(14, 5))

axes[0].hist(qualityscores, bins=50, edgecolor="black", alpha=0.7)
axes[0].setxlabel("Label Quality Score")

axes[0].setylabel("Frequency")
axes[0].settitle("Distribution of Label Quality Scores")

axes[0].axvline(x=0.5, color="red", linestyle="--", label="Threshold 0.5")
axes[0].legend()

cleanscores = qualityscores[~np.isin(np.arange(len(ynoisy)), noisyidx)]

noisyscores = qualityscores[noisyidx]

axes[1].hist(cleanscores, bins=30, alpha=0.6, label="Clean labels", color="green")

axes[1].hist(noisyscores, bins=30, alpha=0.6, label="Noisy labels", color="red")
axes[1].setxlabel("Label Quality Score")

axes[1].setylabel("Frequency")
axes[1].settitle("Quality Score: Clean vs Noisy Labels")

axes[1].legend()

plt.tightlayout()
plt.savefig("labelqualitydistribution.png", dpi=150)
plt.show()

from cleanlab.multilabelclassification.filter import findlabelissues as findmultilabelissues
from sklearn.multiclass import OneVsRestClassifier
from sklearn.linearmodel import LogisticRegression
from sklearn.preprocessing import MultiLabelBinarizer

np.random.seed(42)
nsamples = 500

nfeatures = 20
nclasses = 4


X = np.random.randn(nsamples, nfeatures)


truelabels = []
for i in range(nsamples):

    nlabels = np.random.randint(1, 3)
    labels = sorted(np.random.choice(nclasses, size=nlabels, replace=False).tolist())
    truelabels.append(labels)


mlb = MultiLabelBinarizer()
ybinary = mlb.fittransform(truelabels)

noiseindices = np.random.choice(nsamples, size=50, replace=False)
ynoisybinary = ybinary.copy()

for idx in noiseindices:
    flipcol = np.random.randint(nclasses)
    ynoisybinary[idx, flipcol] = 1 - ynoisybinary[idx, flipcol]

model = OneVsRestClassifier(LogisticRegression(maxiter=500, randomstate=42))
predprobsmultilabel = crossvalpredict(model, X, ynoisybinary, cv=5, method="predictproba")


issues = findmultilabelissues(

    labels=list(mlb.inversetransform(ynoisybinary)),
    predprobs=[predprobsmultilabel[:, i] for i in range(nclasses)],
)

print(f"Found {issues.sum()} label issues in multi-label setting")

# CORRECT: Cross-validation
predprobs = crossvalpredict(model, X, y, cv=5, method="predictproba")

WRONG: In-sample predictions
model.fit(X, y)
predprobs = model.predictproba(X)  # Overfit, results unreliable

from sklearn.ensemble import GradientBoostingClassifier

model = GradientBoostingClassifier(
    nestimators=200,

    maxdepth=5,
    learningrate=0.1,

    randomstate=42
)
predprobs = crossvalpredict(model, X, y, cv=5, method="predictproba")

from cleanlab.filter import findlabelissues

issues = findlabelissues(
    labels=y, predprobs=predprobs,
    returnindicesrankedby="selfconfidence"
)

print("Top 20 most suspicious data points:")
for rank, idx in enumerate(issues[:20], 1):
    print(f"  #{rank}: Index {idx}")
    print(f"    Current label: {y[idx]}")
    print(f"    Predicted probs: {predprobs[idx]}")

    print(f"    Suggested label: {np.argmax(predprobs[idx])}")
    print()

def iterativecleaning(X, y, model, maxiterations=3):
    """Perform iterative data cleaning."""
    currentX = X.copy()

    currenty = y.copy()
    totalremoved = 0


    for iteration in range(maxiterations):
        predprobs = crossvalpredict(

            model, currentX, currenty, cv=5, method="predictproba"
        )

        issues = findlabelissues(
            labels=currenty, predprobs=predprobs,

            returnindicesrankedby="selfconfidence"

        )

        if len(issues) == 0:
            print(f"Iteration {iteration+1}: No more label issues found")
            break

        print(f"Iteration {iteration+1}: Found {len(issues)} label issues")
        totalremoved += len(issues)

        mask = np.ones(len(currenty), dtype=bool)

        mask[issues] = False
        currentX = currentX[mask]

        currenty = currenty[mask]


    print(f"\nTotal data removed: {totalremoved}")
    print(f"Final dataset size: {len(currenty)}")

    return currentX, currenty

# Most conservative method (low false positives)
issuesprune = findlabelissues(
    labels=y, predprobs=predprobs,
    filterby="prunebyclass"

)

Most aggressive method (high recall)
issuesnoise = findlabelissues(
    labels=y, predprobs=predprobs,
    filterby="prunebynoiserate"
)

Balanced method (default)
issuesboth = findlabelissues(

    labels=y, predprobs=predprobs,

    filterby="both"
)

Confident learning method
issuescl = findlabelissues(

    labels=y, predprobs=predprobs,

    filterby="confidentlearning"

)

print(f"prunebyclass: {len(issuesprune)} issues")
print(f"prunebynoiserate: {len(issuesnoise)} issues")
print(f"both: {len(issuesboth)} issues")

print(f"confidentlearning: {len(issuescl)} issues")

import numpy as np
import pandas as pd
from sklearn.modelselection import crossvalpredict, crossvalscore
from sklearn.ensemble import GradientBoostingClassifier
from cleanlab import Datalab
from cleanlab.classification import CleanLearning

def dataqualitypipeline(X, y, featurenames=None):

    """Complete pipeline for data quality analysis and improvement."""

    if featurenames is None:
        featurenames = [f"feature{i}" for i in range(X.shape[1])]

    df = pd.DataFrame(X, columns=featurenames)

    df["label"] = y

    print("="  60)
    print("STEP 1: Baseline Performance")
    print("="  60)


    model = GradientBoostingClassifier(nestimators=100, randomstate=42)

    baselinescores = crossvalscore(model, X, y, cv=5, scoring="accuracy")
    print(f"Baseline Accuracy: {baselinescores.mean():.4f} (+/- {baselinescores.std():.4f})")

    print("\n" + "="  60)
    print("STEP 2: Data Quality Analysis with Datalab")
    print("="  60)


    predprobs = crossvalpredict(model, X, y, cv=5, method="predictproba")
    lab = Datalab(data=df, labelname="label", task="classification")

    lab.findissues(predprobs=predprobs, features=X)
    lab.report()

    print("\n" + "="  60)
    print("STEP 3: Clean Data and Re-train")
    print("="  60)


    cl = CleanLearning(
        clf=GradientBoostingClassifier(nestimators=100, randomstate=42),
        seed=42
    )
    cl.fit(X, y)

    issuesdf = cl.getlabelissues()

    nissues = issuesdf["islabelissue"].sum()

    print(f"Label issues found: {nissues} ({nissues/len(y)100:.1f}%)")

    cleanmask = ~issuesdf["islabelissue"].values

    Xclean = X[cleanmask]

    yclean = y[cleanmask]


    cleanscores = crossvalscore(model, Xclean, yclean, cv=5, scoring="accuracy")
    print(f"Clean Accuracy: {cleanscores.mean():.4f} (+/- {cleanscores.std():.4f})")
    print(f"Improvement: {(cleanscores.mean() - baselinescores.mean())100:.2f}%")


    return {
        "baselineaccuracy": baselinescores.mean(),
        "cleanaccuracy": cleanscores.mean(),
        "nissues": nissues,
        "issuesdf": issuesdf,
        "Xclean": Xclean,
        "yclean": yclean
    }

from sklearn.datasets import loaddigits


digits = loaddigits()
X, y = digits.data, digits.target

np.random.seed(42)
nnoisy = int(0.05  len(y))
noisyidx = np.random.choice(len(y), size=nnoisy, replace=False)

ynoisy = y.copy()
for idx in noisyidx:

    ynoisy[idx] = np.random.choice([l for l in range(10) if l != y[idx]])

results = dataqualitypipeline(X, ynoisy, featurenames=[f"pixel_{i}" for i in range(64)])

Conclusion

Cleanlab is an invaluable tool in the modern data scientist's toolkit. By implementing Confident Learning, Cleanlab enables you to:

Keys to successfully using Cleanlab:

• Always use out-of-sample predictions (cross-validation)
• Use a sufficiently good model to generate predicted probabilities
• Manually verify findings, especially for ambiguous cases
• Perform cleaning iteratively for optimal results

With cleaner data, your machine learning models will be more accurate, more robust, and more reliable in production. Remember: garbage in, garbage out — and Cleanlab helps ensure that what goes into your model is not garbage.