[2O1-GS-7] Vision, speech media processing: generation

[2O1-GS-7-03] Visual Explanation Generation Based on Lambda Attention Branch Networks

〇Tsumugi Iida¹, Kanta Kaneda¹, Tsubasa Hirakawa², Takayoshi Yamashita², Hironobu Fujiyoshi², Komei Sugiura¹ (1. Keio University, 2. Chubu University)

Keywords:Visual Explanation Generation, Attention Branch, Lambda Networks, transformer

Explanation generation for transformers enhances accountability for their predictions.
However, there have been few studies on generating visual explanations for the transformers that use multidimensional context, such as LambdaNetworks.
In this paper, we propose the Lambda Attention Branch Networks, which attend to important regions in detail and generate easily interpretable visual explanations.
We also propose the Patch Insertion-Deletion score, an extension of the Insertion-Deletion score, as an effective evaluation metric for images with sparse important regions.
Experimental results on two public datasets indicate that the proposed method successfully generates visual explanations.