[4A3-GS-6] Language media processing

[4A3-GS-6-04] Image Captioners Tell More Than Images Given to Them

Honori Udo¹, 〇Takafumi Koshinaka¹ (1. Yokohama City University)

[[Online]]

Keywords:CNN, Transformer, BERT

Image captioning, a.k.a. image-to-text, which generates descriptive text from given images, has been rapidly developing through the era of deep learning. To what extent is the information of the original image preserved in the descriptive text generated by an image captioner? To answer that question, we perform an experiment to classify images only from the descriptive text without looking at the images at all, and compare it with a standard CNN-based image classifier. We evaluate several image captioning models on a disaster image classification task, CrisisNLP, and show that descriptive text classifiers can sometimes achieve higher accuracy than the CNN-based classifier. Furthermore, we show that fusing the CNN-based classifier and the descriptive text classifier can provide further accuracy improvement.