[2G6-ES-3] Agents: Conversation and game

[2G6-ES-3-03] QOL Estimation based on Multimodal Learning through Interaction with a Communication Agent

〇Satoshi Nakagawa¹, Shogo Yonekura¹, Hoshinori Kanazawa¹, Satoshi Nishikawa¹, Yasuo Kuniyoshi¹ (1. The University of Tokyo)

Keywords:human agent interaction, multimodal learning, quality of life, elderly welfare

When a monitoring system or a communication robot for the elderly welfare interacts with a human, it is important to estimate the user's state and to generate a behavior based on it. In the field of welfare for the elderly, quality of life (QOL) is a useful indicator, not only for human physical suffering, but also for treating mental and social activities in a comprehensive manner. In this study, we propose a QOL estimation approach that integrates facial expressions, head movements, and eyes movements in the process of interaction with a communication agent. To this end, we implemented a communication agent and constructed a database based on the information collected through communication experiments with humans. In addition, we implemented a multimodal learning estimator that incorporates C3D, a three-dimensional convolution and performed learning with head fluctuation and gaze feature extraction. Our results show that multimodal learning that integrates all of facial expressions, head fluctuations, and eyes movements was realized with less error than single modal learning using each feature separately. From our experimental results, we concluded that the proposed system can be used sufficiently as a QOL estimator.