1

I am using TPE sampler from optuna to optimize hyperparameters for Deep Learning vision models. I was wondering if optuna adapt search depending of the number of trials.

If I train for 1000 trials and stop at 500, I can see that many parameters were not tried by the algorithm. If I reduce n_trials, does TPE explore faster (and less precisely) ? In other terms, is interupting optuna at 500 with n_step=1000 the same as using n_trials = 500 and waiting until the end.

I only have basic understanding of how TPE works.

Thanks.

Phi
  • 21
  • 3
  • 1
    Hello Phi, let me clarify your question. Which sense do you use the word `iteration`? The training loop count that is corresponding to the number of mini-batches, or the number of Optuna trials that is corresponding to the number of hyperparameter sets? – Toshihiko Yanase Aug 06 '20 at 12:06
  • 1
    I was thinking of the n_trials parameters in the optimize function. – Phi Aug 06 '20 at 15:19
  • Thanks for your reply and update of the description. I understand. – Toshihiko Yanase Aug 28 '20 at 00:05

1 Answers1

2

If I reduce n_trials, does TPE explore faster (and less precisely)?

No.

... is interupting optuna at 500 with n_step=1000 the same as using n_trials = 500 and waiting until the end.

Yes.

This is because TPE (from Optuna) is not aware of the number of remaining (or total) trials.

hvy
  • 141
  • 5
  • @user11409592 would you be able/willing to cite source(s) for your answer please? – brethvoice Oct 20 '20 at 12:31
  • 1
    This is the source code of the implementation https://github.com/optuna/optuna/blob/master/optuna/samplers/_tpe/sampler.py. It does not rely on the number of trials specified by the user. One can also see from the outer trial loop that there are other stop conditions besides `n_trials`, namely running time that's specified using `timeout` https://github.com/optuna/optuna/blob/master/optuna/_optimize.py – hvy Oct 21 '20 at 23:48