[1Q3-J-2] Machine learning: structural modeling

[1Q3-J-2-02] Construction of pooling layer by skip connection and analysis based on expressive power of these models

〇Jumpei Nagase¹, Tetsuya Ishiwata² (1. Graduate School of Engineering and Science, Shibaura Institute of Technology, 2. College of Systems Engineering and Science, Shibaura Institute of Technology)

Keywords:deep neural networks, skip connection, model design, deep learning, expressive power

In this research, we consider structures such as pooling layer and skip connection from the viewpoint of expressive power in order to organize design of neural networks models.
We showed that widely used these structures can be understood as a composition of affine functions and concatenated activation functions.
Moreover, we show the followings:
(i) the pooling layer explicitly decreases expressive power,
(ii) there is no deference in expressive power between addition and concatenation as skip connection for fully connected neural networks, and
(iii) the single activation block has superior expressive power compared to the multiple activation block.
These results propose one guideline for design of neural networks models.