[2L1-J-9] Natural language processing, information retrieval: fusion with image

[2L1-J-9-03] Towards Building a Logical Inference System for Text and Visual Information

Riko Suzuki¹, 〇Masashi Yoshikawa², Hitomi Yanaka³, Koji Mineshima¹, Daisuke Bekki¹ (1. Ochanomizu University, 2. Nara Institute of Science and Technology, 3. RIKEN Center for Advanced Intelligence Project)

Keywords:Meaning representations for images, Multimodal inference, Theorem Proving, Semantic Parsing, Computer Vision

A large amount of research about multimodal inference across natural language and vision has been recently developed to obtain visually grounded word and sentence representations. In this paper, we use logic-based representations as unified meaning representations for texts and images and present an unsupervised inference system that can effectively prove entailment relations between them. We show that by combining semantic parsing and theorem proving, the system can handle semantically complex queries for image retrieval.