Python: How to retrive the best model from Optuna LightGBM study?

Question

I would like to get the best model to use later in the notebook to predict using a different test batch.

reproducible example (taken from Optuna Github) :

import lightgbm as lgb
import numpy as np
import sklearn.datasets
import sklearn.metrics
from sklearn.model_selection import train_test_split

import optuna


# FYI: Objective functions can take additional arguments
# (https://optuna.readthedocs.io/en/stable/faq.html#objective-func-additional-args).
def objective(trial):
    data, target = sklearn.datasets.load_breast_cancer(return_X_y=True)
    train_x, valid_x, train_y, valid_y = train_test_split(data, target, test_size=0.25)
    dtrain = lgb.Dataset(train_x, label=train_y)
    dvalid = lgb.Dataset(valid_x, label=valid_y)

    param = {
        "objective": "binary",
        "metric": "auc",
        "verbosity": -1,
        "boosting_type": "gbdt",
        "lambda_l1": trial.suggest_loguniform("lambda_l1", 1e-8, 10.0),
        "lambda_l2": trial.suggest_loguniform("lambda_l2", 1e-8, 10.0),
        "num_leaves": trial.suggest_int("num_leaves", 2, 256),
        "feature_fraction": trial.suggest_uniform("feature_fraction", 0.4, 1.0),
        "bagging_fraction": trial.suggest_uniform("bagging_fraction", 0.4, 1.0),
        "bagging_freq": trial.suggest_int("bagging_freq", 1, 7),
        "min_child_samples": trial.suggest_int("min_child_samples", 5, 100),
    }

    # Add a callback for pruning.
    pruning_callback = optuna.integration.LightGBMPruningCallback(trial, "auc")
    gbm = lgb.train(
        param, dtrain, valid_sets=[dvalid], verbose_eval=False, callbacks=[pruning_callback]
    )

    preds = gbm.predict(valid_x)
    pred_labels = np.rint(preds)
    accuracy = sklearn.metrics.accuracy_score(valid_y, pred_labels)
    return accuracy

my understanding is that the study below will tune for accuracy. I would like to somehow retrieve the best model from the study (not just the parameters) without saving it as a pickle, I just want to use the model somewhere else in my notebook.


if __name__ == "__main__":
    study = optuna.create_study(
        pruner=optuna.pruners.MedianPruner(n_warmup_steps=10), direction="maximize"
    )
    study.optimize(objective, n_trials=100)

    print("Best trial:")
    trial = study.best_trial

    print("  Params: ")
    for key, value in trial.params.items():
        print("    {}: {}".format(key, value))

desired output would be

best_model = ~model from above~
new_target_pred = best_model.predict(new_data_test)
metrics.accuracy_score(new_target_test, new__target_pred)

avinashpancham · Answer 1 · 2020-08-12T09:41:38.040

Short addition to @Toshihiko Yanase's answer, because the condition study.best_trial==trial was never True for me. This was even the case when both (Frozen)Trial objects had the same content, so it is likely a bug in Optuna. Changing the condition to study.best_trial.number==trial.number solves the problem for me.

Also if you prefer to not use globals in Python, you can use the study and trial user attributes

def objective(trial):
    gmb = ...
    trial.set_user_attr(key="best_booster", value=gbm)

def callback(study, trial):
    if study.best_trial.number == trial.number:
        study.set_user_attr(key="best_booster", value=trial.user_attrs["best_booster"])


if __name__ == "__main__":
    study = optuna.create_study(
        pruner=optuna.pruners.MedianPruner(n_warmup_steps=10), direction="maximize"
    )
    study.optimize(objective, n_trials=100, callbacks=[callback])
    best_model=study.user_attrs["best_booster"]

I can validate that your solution also works with xgboost, I validate it with comparing loss metrics after obtaining de best_model via best_model=study.user_attrs["best_booster"]. Amazing thanks to both of you guys. — Mehmet Burak Sayıcı, Dec 25 '20 at 15:20

score 3 · Accepted Answer · answered Jun 03 '20 at 03:11

I think you can use the callback argument of Study.optimize to save the best model. In the following code example, the callback checks if a given trial is corresponding to the best trial and saves the model as a global variable best_booster.

best_booster = None
gbm = None

def objective(trial):
    global gbm
    # ...

def callback(study, trial):
    global best_booster
    if study.best_trial == trial:
        best_booster = gbm

if __name__ == "__main__":
    study = optuna.create_study(
        pruner=optuna.pruners.MedianPruner(n_warmup_steps=10), direction="maximize"
    )
    study.optimize(objective, n_trials=100, callbacks=[callback])

If you define your objective function as a class, you can remove the global variables. I created a notebook as a code example. Please take a look at it: https://colab.research.google.com/drive/1ssjXp74bJ8bCAbvXFOC4EIycBto_ONp_?usp=sharing

I would like to somehow retrieve the best model from the study (not just the parameters) without saving it as a pickle

FYI, if you can pickle the boosters, I think you can make the code simple by following this FAQ.

score 2 · Answer 3 · answered Mar 03 '21 at 16:37

I know this is has been already answered, there is a straightforward way of doing this with the optuna-lightgbm integration lightgbmtuner released in late 2020.

In short you could perform what you want to do, i.e. save the best booster as follows

import optuna.integration.lightgbm as lgb

dtrain = lgb.Dataset(X,Y,categorical_feature = 'auto')

params = {
"objective": "binary",
"metric": "auc",
"verbosity": -1,
"boosting_type": "gbdt",
}

tuner = lgb.LightGBMTuner(
     params, dtrain, verbose_eval=100, early_stopping_rounds=1000, 
  model_dir= 'directory_to_save_boosters'
)

tuner.run()

Please note that the main thing here is to specify a model_dir directory to save the models in each of the iterations.

There is usually no need for a pruning callback as the optimization is done using a combination of Bayesian methods and expert heuristics and the search is usually over in around 60-64 iterations.

Then you can get the best model from the model directory you specified above using the single line

tuner.get_best_booster()

Python: How to retrive the best model from Optuna LightGBM study?

3 Answers3