[3M5-GS-10] AI application

[3M5-GS-10-01] An Assessment System of Online Structured Job Interviews Supported by Multi-Modal Deep Learning

〇Shengzhou Yi¹, Toshiaki Yamasaki², Toshihiko Yamasaki¹ (1. The University of Tokyo, 2. Talent and Assessment Inc.)

Keywords:Multimodal Recognition, Multimodal Neural Network, Class Imbalance, Job Interview

A structured interview is a method that asks predetermined behavioral questions to interviewees to eliminate subjectivity of interviewers. This method is employed in survey research, job interviews, and so on. However, there are still some problems because interviews are usually time- and money-consuming. In this study, we present a multi-modal neural network aiming at online structured job interviews. Text and audio features of the interviews are extracted by the proposed model. Furthermore, class-imbalanced learning methods and margin ranking loss are used to improve the model performance. The interview videos are assigned with the labels of seven assessment criteria to clarify whether the candidates get high or low scores. For the experimental results, the combination of multi-modal features, class-imbalanced learning, and margin ranking loss makes the proposed model achieve an average accuracy of 76.59% in the two-class classification. Such AI-supported job interview systems would give more chances to candidates because they will have online interviews in addition to curriculum vitae screening.