[2Q1-GS-10] Vision, speech: Fundamental theory and application

[2Q1-GS-10-05] Differentiable Search for Deep Neural Network with Attention

〇Kohei Nakai¹, Takashi Matsubara¹, Kuniaki Uehara¹ (1. Kobe University)

Keywords:Neural Architecture Search, Attention mechanism, Image recognition

Manually designing neural network architecture requires a considerable amount of expertise and time. Neural architecture search (NAS) aims to automatically identify neural architectures, and recent NAS methods enabled us to identify architectures that achieve state-of-the-art performance for image classification in a few days. For image classification, we usually use convolutional neural networks (CNNs), whose main operations are convolution and pooling. Recent studies on neural architectures indicate that attentions can improve the performances of CNNs with a comparable number of parameters by discarding information of no interest, while existing NAS methods have put little focus on it. In this study, we propose Att-DARTS, which is based on DARTS and searches attentions as well as convolution and pooling operations simultaneously. In our experiments on CIFAR-10 and CIFAR-100 datasets, we demonstrate that Att-DARTS can find architectures that achieve lower classification error rates and require fewer parameters compared to those found by DARTS.