[2D4-GS-2] Machine learning: Image recognition

[2D4-GS-2-04] A Study on Image Classification by Regional Embedding with Large Vision-Language Model

〇Kosuke Sakurai¹, Tatsuya Ishii¹, Ryotaro Shimizu¹, Linxin Song¹, Masayuki Goto¹ (1. Waseda University)

Keywords:Regional embedding, Data augmentation, Domain adaptation, Vision-language model, Image classification

In recent years, Contrastive Language-Image Pre-training (CLIP), a large vision-language model, has been widely used as a highly accurate image classification model and utilized in various models. Latent Augmentation using Domain descriptionS (LADS) is an image classification model that improves accuracy for specific unseen domain by augmenting the image embeddings, which is a single point in the image-language embedding space trained by CLIP, in unseen domain. However, in improving the model’s generalization performance, a simple augmentation method such as LADS cannot take into account the diversity of data arising from various domains (e.g., differences of background or number) that are not included in the training data. Therefore, we present Latent Augmentation using Regional Embedding (LARE), a robust image classification model that applies regional embedding in latent space trained by large vision-language model and augments data to various domains by sampling from those regions. We show that LARE outperforms previous models on two benchmarks under multiple conditions, such as unseen domains, small amounts of data, and unbalanced data.