[2K5-ES-2] Machine learning: Multimedia

[2K5-ES-2-02] Detection of Facial Expression Changes in Conversational Interactions based on Deep Learning

Lieu-Hen Chen¹, 〇HOW-YU TSEN¹, ChengYu Sun², Hiroki Shibata², Yasufumi Takama² (1. National Chi-Nan University, Taiwan, 2. Tokyo Metropolitan University, Japan)

Keywords:HMI, Emotion transition, Facial recognition

Facial detection and recognition are traditional research topics in the field of Computer Vision. Recent facial recognition work mostly focus on precision improvement of facial detection and facial alignment. In this paper we present a research work, which detects facial expression changes in conversational interactions based on deep learning. Our system first extract the timing of user’s facial expression changes in a video or continuous image sequence, by using a robust CNN network to segment the region of eyes, eyebrows, and lips. Then, by comparing the difference of the segmentation result, user’s emotional transitions are analyzed. Finally, keywords or phrases that stimulates the user are extracted and associated with the related emotional transition. The experiment result shows this research has great potential whether for psychology or human machine interface applications.