[3K3-J-2] Machine learning: analysis and validations of models

[3K3-J-2-01] Statistical Mechanical Formulation of Learning Dynamics of Two-Layered Neural Networks with Batch Normalization

〇Shiro Takagi¹, Yuki Yoshida¹, Masato Okada¹ (1. Graduate School of Frontier Sciences, The University of Tokyo)

Keywords:Neural Network, Batch Normalization, Statistical mechanics

Batch Normalization is known as a method to shorten training time, stabilize training and improve the performance of neural networks. Despite its wide use, the impact of Batch Normalization on the learning dynamics of neural networks is yet to be clarified. Though some recent studies tried to tackle this problem, few of them derived the exact learning dynamics of neural networks with Batch Normalization. Because deriving the learning dynamics is helpful for understanding what Batch Normalization is doing during training, we derived an exact learning dynamics of two-layered neural networks with Batch Normalization by drawing on the previous work about a statistical mechanical method of neural network analysis. Specifically, for neural networks with Batch Normalization, we derived differential equations of order parameters, which represent a macroscopic behavior of neural networks.