3

The background for the question is optimizing hyper params of neural network training by running study.optimize() with default pruning enabled and learning rate as parameter to optimize (this question can be generalized to other hyperparams).

high learning rates can have good results initially lowering the loss substantially every intermediate step at first but results in poor final result.

low learning rates can have poor initial results but better end value.

with pruning enabled wouldn't low learning rate be pruned because of poor intermediate results? which means the final best param would be incorrectly selected?

how can this be fixed? would using another pruner help? thanks!

yonatan medan
  • 33
  • 5

1 Answers1

2

I think protecting the trials which show low initial scores but high final scores (like late bloomers) still an open problem. As you mentioned, the existing pruners (e.g., MedianPruner) tend to prune such late bloomers.

One possible solution may be learning curve prediction, which is a task to estimates the final score based on the intermediate scores. If a predictor successfully estimates the final score, we can create a pruner which protects the late bloomers. I'm looking forward to the research progress of the learning curve prediction.

Toshihiko Yanase
  • 111
  • 3