[4I1-GS-7b] 画像音声メディア処理：マルチモーダル処理

[4I1-GS-7b-01] Building a Video-and-Language Dataset with Human Actions for Multimodal Inference

〇Mai Yokozeki¹, Natsuki Murakami¹, Riko Suzuki¹, Hitomi Yanaka^2,1, Koji Mineshima³, Daisuke Bekki¹ (1. Ochanomizu Universitiy, 2. RIKEN, 3. Keio University)

Keywords:Multimodal Inference, Visual Textual Entailment, Video Dataset

This paper introduces a new video-and-language dataset with human actions for multimodal inference.The dataset consists of 200 videos, 5554 action labels, and 1942 action triplets of the form <subject, action, object>. Action labels contain various expressions such as aspectual and intentional phrases that are characteristic of videos but do not appear in existing video and image datasets. The dataset is expected to be applied to the evaluation of the multimodal inference system between the video and semantically complicated sentences such as negation and quantity.