[1G4-OS-26a] OS-26

[1G4-OS-26a-04] The 2nd International Knowledge Graph Reasoning Challenge: Application of LLM to Predicting Behavior from Multimodal Data about Daily Life

○Takanori Ugai^1,2,Shusaku Egami², Takahiro Kawamura^3,2, Kouji Kozaki^4,2, Takeshi Morita^5,2, Kyoumoto Matsushita^2,1, Tomohiro Ogawa⁵, Kango Yoshioka⁵, Tsukasa Hirano⁵, Kengo Ozaki⁵, Ken Fukuda² (1. Fujitsu Limited, 2. National Institute of Advanced Industrial Science and Technology, 3. National Agriculture and Food Research Organization, 4. Osaka Electro-Communication University, 5. Aoyama Gakuin University)

[1G5-OS-26b] 日常生活知識とAI 17:40 〜 18:00 にて発表

Keywords:Knowledge Graph, MultiModalLLM, Video recognition

We held the final presentation of the 2nd International Knowledge Graph Inference Challenge on February 8, 2024 as a workshop in conjunction with the International Conference on Semantic Computing. The main task of the Challenge was to obtain statistics about actions, objects, and locations from videos and knowledge graphs generated using a 3D simulator of parts of daily life.
The unique feature of this challenge is that it provides data with a missing part of the knowledge graph, and it is necessary to compensate for the missing information by extracting information from the video and predicting using machine learning on the knowledge graph.
In this presentation, we provide an overview of the dataset and tasks of this inference challenge and introduce the four submissions. Since several of the submissions used multimodal LLMs, we will compare them and also discuss the challenges and expectations for current multimodal LLMs in this task.