[4E2-OS-19a] OS-19 (1)

[4E2-OS-19a-04] Automatic Generation of Responses and Facial Expressions based on Multimodal Information

〇Ryosuke Ueno¹, Jie Zeng¹, Yukiko Nakano¹ (1. Seikei university)

Keywords:multimodal, emotional response, dialogue

It is important for Conversational Agents to display emotional and empathetic responses using verbal and nonverbal behaviors. Towards multimodal generation in Conversational Agents, this study proposes a multimodal deep learning model for predicting a category of verbal acknowledgement and facial expressions at the same time. First, unimodal encoders for audio, language, and face movement were trained, and the output of the encoders were fused to train multimodal decoder which predicts verbal acknowledgement and facial expressions. The model performs much better than a baseline model.