[4D3-OS-4b] Affective Computing(2/3)

[4D3-OS-4b-01] End-to-End Deep Learning for pNN50 Estimation Using a Spatiotemporal Representation

〇Sayyedjavad Ziaratnia¹, Peeraya Sripian¹, Kazuo Ohzeki², Midori Sugaya¹ (1. Shibaura Institute of Technology, 2. Tokyo International institute of Technology)

Keywords:Deep Learning, pNN50, rPPG

Various industries widely use emotion estimation to evaluate their consumer satisfaction towards their product. Generally, emotion can be estimated based on observable expressions such as facial expression, or unobservable expressions such as biological signals. Although used by many research, the Facial Expression Recognition (FER) has a lack of precision for expressions that are very similar to each other or a situation where the shown expression differs from the real subject’s emotion. On the other hand, biological signal indexes such as pNN50 can act as a supportive mechanism to improve emotion estimation from observable expressions such as FER method. pNN50 is a reliable index to estimate stress-relax, and it originates from unconscious emotions that cannot be manipulated. In this work, we propose a method for pNN50 estimation from facial video using a Deep Learning model. Transfer learning technique and a pre-trained Image recognition Convolutional Neural Network (CNN) model are employed to estimate pNN50 based on a spatiotemporal map created from a series of frames in a facial video. The model trained on low, middle, and high pNN50 values, shows an accuracy of about 80%. Therefore, it indicates the potential of our proposed method, and we can expand it to categorize the more detailed level of pNN50 values.