[3S3-IS-2e] Machine learning

[3S3-IS-2e-04] Robustifying Vision Transformer Without Retraining From Scratch Using Attention Based Test-Time Adaptation

〇Takeshi Kojima¹, Yusuke Iwasawa¹, Yutaka Matsuo¹ (1. Graduate School of Engineering, The University of Tokyo)

Regular

Keywords:Vision Transformer, Test-Time Adaptation, Self-Attention

Vision Transformer (ViT) is becoming more and more popular in the field of image processing. This study aims to improve the robustness against the unknown perturbations without retraining the ViT model from scratch. Since our approach does not alter the training phase, it does not need to repeat computationally heavy pre-training of ViT. Specifically, we use test-time adaptation for this purpose, which corrects its prediction during test-time by itself. We first show the existing test-time adaptation method (Tent), which was only validated for CNN model, is also applicable by proper parameter tuning and gradient clipping. However, we observed Tent sometimes catastrophically fails, especially under severe perturbations. To stabilize the optimization, we propose a new loss function called Attent, which minimizes the distributional differences of the attention entropy between the source and target. Experiments of image classification task on CIFAR-10-C, CIFAR-100-C, and ImageNet-C show that both Tent and Attent are effective on a wide variety of corruptions. The results also show that by combining Attent and Tent, the classification accuracy on corrupted data is further improved.