[3J4-OS-3b] AutoML（自動機械学習）(2/2)

[3J4-OS-3b-05] Neural Architecture Search for Transformers on Vision and Language Tasks

〇Masanori Suganuma¹ (1. Graduate School of Information Sciences, Tohoku University)

Keywords:AutoML, Neural Architecture Search (NAS), Vision and Language

Since Transformer was first proposed, it has shown remarkable performance in a wide range of fields such as image recognition, natural language processing, and their fusion tasks. In general, the network structure of deep neural networks has a significant impact on its performance, and Transformer is no exception. However, the structure of Transformer has not been explored sufficiently due to the high training cost, and thus its potential has not been fully exploited. In this paper, we first design a search space that can represent various Transformer architectures. We then propose a search method that can efficiently search the architectures in the search space. We evaluate our method on several vision and language tasks and show experimentally that the Transformers found by the search outperform the vanilla Transformers. Moreover, we provide what architecture components are important for the Transformer's performance by analyzing the architectures obtained by the search.