[1D3-GS-7] Language media processing:

[1D3-GS-7-05] Alternative Adapter Model: A Visual Explanation Generation for Vision-Language Foundation Models"

〇Shinnosuke Hirano¹, Tsumugi Iida¹, Komei Sugiura¹ (1. Keio University)

Keywords:Attention Lattice Adapter, Visual Explanation, Side Adapter Network, Attention Branch, CLIP

In the modern era where deep learning is applied across a wide range of fields, the explainability of models is of paramount importance.
However, existing methods are not optimized for vision-language foundation models, leading to lower explanation quality for such models.
Therefore, this study proposes the Alternative Adapter Model, an explanation generation model tailored to vision-language foundation models. By introducing a Side Branch Network connected to the vision-language foundation model, the proposed method extracts features suitable for explanation generation. Furthermore, by implementing the Alternative Epoch Architecture, which dynamically changes the outputs of modules and the layers to be frozen, we address the issue of overly narrow focus areas.
To evaluate the proposed method, experiments were conducted using the CUB-200-2011 dataset. The results demonstrate that the proposed method surpasses existing methods in mean IoU, Insertion Score, Deletion Score, and Insertion-Deletion Score, which are standard metrics for visual explanation generation tasks.