[1B5-GS-2] Machine learning

[1B5-GS-2-02] Explicit modeling of the implicit regularization effect of SGD

〇Shota Nakamura¹, Rio Yokota¹ (1. Tokyo Institute of Technology)

Keywords:Deep learning, Optimization, Distributed Parallel Learning

Distributed parallel learning is needed due to the growth of deep learning models and datasets. Data parallelization is the easiest distributed learning method to implement, where each GPU has redundant models and batches are distributed. However, as the number of GPUs increases, the batch size increases proportionally and the generalization performance deteriorates due to the loss of the implicit regularization effect of SGD. In this study, we aim to alleviate this large-batch problem by regularizing by the gradient norm.