[3Rin2] Interactive Session 1

[3Rin2-03] Binarized Variational Information Bottleneck

〇Makoto Kawano¹, Yu Oya², Satoshi Yagi², Jin Nakazawa¹ (1. Keio University, 2. NTT Corporation)

Keywords:Deep Learning, Binarization, Variational Information Bottleneck

Deep neural networks are utilized in various applications in real worlds, thanks to their capabilities. One of the fashions of it is their deployment on edge devices. With edge devices, deep neural networks can be used in the context of IoT. However, the specification of those edge devices is often poor so that deep neural networks cannot be deployed. Binary neural networks, whose weights and activations are binarized, is one of the solutions. There is a well-known issue, the drastic drop in accuracy compared to its full precision networks. We consider that this is because the binary neural networks can only represent a subset of discrete functions so that they become sensitive to the input perturbation: the lack of robustness for inputs. In this paper, we propose a regularization approach that helps to alleviate the over-fitting problem by introducing variational information bottleneck. We show ablation studies on CIFAR-10 that reduce loss value the though accuracy is maintained on AlexNet-like networks with different binary activation functions.