[1J3-OS-10] OS-10

[1J3-OS-10-02] The Characteristics Required in Hyperparameter Optimization of Deep Learning Algorithms

〇Shuhei Watanabe¹, Masahiro Nomura², Masaki Onishi¹ (1. The National Institute of Advanced Industrial Science and Technology, 2. CyberAgent Inc.)

Keywords:Hyperparameter Optimization, Black-box Optimization, Deep Learning

Since the performance of deep learning algorithms depends seriously on the selection of hyperparameter values, hyperparameter optimization is essential in real-world applications.
Many researchers have studied hyperparameter optimization algorithms extensively.
However, there has been not enough discussion on the characteristics of the relationship, which we call objective function, between the searching space and its performance in the context of hyperparameter optimization.
Therefore, we elucidate the properties that the objective functions in this paper.
More specifically, we evaluated hyperparameter settings of deep learning algorithms with 5 hyperparameters.
Then, a regression model for each objective function was constructed to analyze the objective function.
Finally, we reached the conclusion that hyperparameter methods should be able to deal with ill-conditioned first and then consider non-separable and less important hyperparameters and convexity is a strong assumption on hyperparameter optimization.