[2D5-GS-2] Machine learning: Statistical learning

[2D5-GS-2-03] Accounting for directional outliers in robust gradient descent

〇Kosuke Nakatani¹, Matthew J. Holland¹ (1. Osaka university)

Keywords:Machine Learning, gradient descent

In the field of machine learning, numerous methods have been proposed for the robustness of stochastic gradient descent, with the majority focusing on gradient norms and neglecting the abnormality in the update direction. We introduce a method that incorporates the abnormality of directions into the norm suppression technique, aiming to prioritize the suppression of outliers that impede the smooth progress of learning. We compare this attempt with conventional methods and evaluate it in terms of convergence stability and generalization ability.