Presentation information

Organized Session

Organized Session » OS-3

[3J4-OS-3b] AutoML(自動機械学習)(2/2)

Thu. Jun 16, 2022 3:30 PM - 5:10 PM Room J (Room J)

オーガナイザ:大西 正輝(産業技術総合研究所)[現地]、日野 英逸(統計数理研究所/理化学研究所)

4:50 PM - 5:10 PM

[3J4-OS-3b-05] Neural Architecture Search for Transformers on Vision and Language Tasks

〇Masanori Suganuma1 (1. Graduate School of Information Sciences, Tohoku University)

Keywords:AutoML, Neural Architecture Search (NAS), Vision and Language

Since Transformer was first proposed, it has shown remarkable performance in a wide range of fields such as image recognition, natural language processing, and their fusion tasks. In general, the network structure of deep neural networks has a significant impact on its performance, and Transformer is no exception. However, the structure of Transformer has not been explored sufficiently due to the high training cost, and thus its potential has not been fully exploited. In this paper, we first design a search space that can represent various Transformer architectures. We then propose a search method that can efficiently search the architectures in the search space. We evaluate our method on several vision and language tasks and show experimentally that the Transformers found by the search outperform the vanilla Transformers. Moreover, we provide what architecture components are important for the Transformer's performance by analyzing the architectures obtained by the search.

Authentication for paper PDF access

A password is required to view paper PDFs. If you are a registered participant, please log on the site from Participant Log In.
You could view the PDF with entering the PDF viewing password bellow.