[2I5-OS-9a] Affective Computing(1/2)

[2I5-OS-9a-03] Crossmodal Representation of Emotions Based on Hyperbolic Embedding Trained by Emotion Recognition and Latent Space Unification Tasks

〇Seiichi Harata¹, Takuto Sakuma¹, Shohei Kato^1,2 (1. Dept. of Engineering, Graduate School of Engineering, Nagoya Institute of Technology, 2. Frontier Research Institute for Information Science, Nagoya Institute of Technology)

Keywords:Affective Computing, Crossmodal, Emotion Recognition, Representation Learning, Emotional Space

This study aims to obtain a mathematical representation of emotions (an emotional space) common to the modalities. We compare methods for embedding emotions into non-Euclidean spaces using multimodal DNNs.
The proposed model fuses the emotional spaces for each modality embedded in the latent space based on the Klein model, a hyperbolic space model. Then, we train the model by multitasking the emotion recognition task and the latent space unification task.
In the experiment using audio-visual data, we compare the representation on hyperbolic space with the Euclidean or Hemi-hyperspherical representation considered in previous studies.
We evaluate the robustness of emotion recognition when the modality is missing and confirm that the proposed method obtains shared representations of emotions across modalities in a low-dimensional hyperbolic space.
We also compare the emotion recognition tendency of the proposed model with human raters to examine the representational power of the proposed emotional space.