Presentation information

International Session

International Session » ES-2 Machine learning

[3S3-IS-2e] Machine learning

Thu. Jun 16, 2022 1:30 PM - 2:50 PM Room S (Online S)

Chair: Akinori Abe (Chiba University)

2:30 PM - 2:50 PM

[3S3-IS-2e-04] Robustifying Vision Transformer Without Retraining From Scratch Using Attention Based Test-Time Adaptation

〇Takeshi Kojima1, Yusuke Iwasawa1, Yutaka Matsuo1 (1. Graduate School of Engineering, The University of Tokyo)


Keywords:Vision Transformer, Test-Time Adaptation, Self-Attention

Vision Transformer (ViT) is becoming more and more popular in the field of image processing. This study aims to improve the robustness against the unknown perturbations without retraining the ViT model from scratch. Since our approach does not alter the training phase, it does not need to repeat computationally heavy pre-training of ViT. Specifically, we use test-time adaptation for this purpose, which corrects its prediction during test-time by itself. We first show the existing test-time adaptation method (Tent), which was only validated for CNN model, is also applicable by proper parameter tuning and gradient clipping. However, we observed Tent sometimes catastrophically fails, especially under severe perturbations. To stabilize the optimization, we propose a new loss function called Attent, which minimizes the distributional differences of the attention entropy between the source and target. Experiments of image classification task on CIFAR-10-C, CIFAR-100-C, and ImageNet-C show that both Tent and Attent are effective on a wide variety of corruptions. The results also show that by combining Attent and Tent, the classification accuracy on corrupted data is further improved.

Authentication for paper PDF access

A password is required to view paper PDFs. If you are a registered participant, please log on the site from Participant Log In.
You could view the PDF with entering the PDF viewing password bellow.