Complete Optuna Tutorial: Automatic Hyperparameter Optimization

Table of Contents

Introduction

Hyperparameter tuning is one of the most time-consuming tasks in machine learning. Optuna is an automatic hyperparameter optimization framework that uses efficient search algorithms to find optimal configurations significantly faster than grid or random search.

In this tutorial, you'll learn how to use Optuna for everything from basic optimization to distributed multi-objective tuning across ML frameworks.

Prerequisites

• Python 3.8+
• Basic understanding of machine learning
• Familiarity with scikit-learn or PyTorch

Installation and Setup

pip install optuna optuna-dashboard plotly scikit-learn xgboost lightgbm torch

Optional: for distributed optimization
pip install optuna[rdb] psycopg2-binary

Core Concepts: Study and Trial

Optuna revolves around two main concepts:

• Study: An optimization session consisting of multiple trials
• Trial: A single evaluation of the objective function with specific hyperparameters

import optuna

Define objective function
def objective(trial):
    # Suggest hyperparameters
    x = trial.suggestfloat("x", -10, 10)

    y = trial.suggestfloat("y", -10, 10)

    # Return value to minimize
    return (x - 2)  2 + (y + 3)  2

Create study and optimize
study = optuna.createstudy(direction="minimize")

study.optimize(objective, ntrials=100)

print(f"Best value: {study.bestvalue:.4f}")

print(f"Best params: {study.bestparams}")

Parameter Suggestion Methods

def objective(trial):
    # Integer parameter
    nlayers = trial.suggestint("nlayers", 1, 5)


    # Float parameter
    learningrate = trial.suggestfloat("lr", 1e-5, 1e-1, log=True)


    # Categorical parameter
    optimizer = trial.suggestcategorical("optimizer", ["adam", "sgd", "rmsprop"])

    # Discrete float
    dropout = trial.suggestfloat("dropout", 0.1, 0.5, step=0.1)


    # Conditional parameters
    if optimizer == "sgd":
        momentum = trial.suggestfloat("momentum", 0.0, 0.99)

    return trainmodel(nlayers, learningrate, optimizer, dropout)

import optuna
from sklearn.ensemble import RandomForestClassifier
from sklearn.modelselection import crossvalscore
from sklearn.datasets import loadbreastcancer

X, y = loadbreastcancer(returnXy=True)

def objective(trial):
    params = {
        "nestimators": trial.suggestint("nestimators", 50, 500),

        "maxdepth": trial.suggestint("maxdepth", 3, 20),
        "minsamplessplit": trial.suggestint("minsamplessplit", 2, 20),

        "minsamplesleaf": trial.suggestint("minsamplesleaf", 1, 10),
        "maxfeatures": trial.suggestcategorical("maxfeatures", ["sqrt", "log2", None]),

        "criterion": trial.suggestcategorical("criterion", ["gini", "entropy"]),
    }

    clf = RandomForestClassifier(params, randomstate=42, njobs=-1)

    scores = crossvalscore(clf, X, y, cv=5, scoring="accuracy")

    return scores.mean()

study = optuna.createstudy(direction="maximize")

study.optimize(objective, ntrials=200, showprogressbar=True)

print(f"Best accuracy: {study.bestvalue:.4f}")