[2D1-GS-9] Natural language processing, information retrieval: Support technology

[2D1-GS-9-05] News Image Caption Generation

〇Zhishen Yang¹, Naoaki Okazaki¹ (1. Tokyo Institute of Technology)

Keywords:vision and language, image captioning, multimodality

Vision and language as a vibrant multimodal machine learning research field aim to create models that serve comprehension of information across vision and language modalities. In this work, we utilized the multimodal Transformer model with joint text-vision representation to approach one of the vision and language tasks: news image caption generation. The multimodal Transformer model leverages context from the article with consideration of the scene in the associated image to generate caption. The experimental result demonstrated the multimodal Transformer significantly improved the quality of generated news image caption.