[1D1-GS-2] Machine learning: algorithm

[1D1-GS-2-02] Scheduling of Damping in Natural Gradient Method

〇Hiroki Naganuma^1,2, Gaku Fujimori³, Mari Takeuchi⁴, Jumpei Nagase⁵ (1. University of Montreal, 2. Mila, 3. Tokyo University of Science, 4. University College London, 5. Shibaura Institute of Technology)

[[Online]]

Keywords:Deep Learning, Second-Order Optimization, Damping

In recent years, second-order optimization with a fast convergence rate has been used in deep learning owing to fast approximation methods for natural gradient methods. Second-order optimization requires the inverse computation of the information matrix, which generally degenerates in the deep learning problem. Therefore, as a heuristic, a damping method adds a unit matrix multiplied by a constant. This study proposed a method for scheduling damping motivated by the Levenberg-Marquardt method for determining damping and investigated its effectiveness.