[4G2-OS-24c] 日常生活知識とAI

[4G2-OS-24c-04] Automatic Evaluation and Analysis of Image Captioning Models Based on Scene Graphs

〇Reo Tanaka¹, Yuiga Wada¹, Komei Sugiura¹ (1. Keio University)

Keywords:Image captioning, Automatic evaluation metric, Scene graph, JaSPICE

Image captioning studies rely heavily on automatic evaluation metrics such as BLEU and METEOR, which are based on n-grams. However, these metrics have shown poor correlation with human evaluations, leading to the proposal of alternative metrics such as JaSPICE. JaSPICE has only been validated for a general image captioning task without an error analysis. In this paper, we analyze JaSPICE for a fetching instruction generation task and identify its errors for an image captioning task. We conducted experiments on STAIR Captions and PFN-PIC datasets and JaSPICE outperformed the baseline metrics on the correlation coefficient with human evaluation.