[1D4-GS-10] AI application: Behavioral model

[1D4-GS-10-05] Enhancing Online Structured Job Interviews: A Comprehensive Personality Assessment Using Multimodal Neural Networks

〇Shengzhou Yi¹, Toshiaki Yamasaki², Toshihiko Yamasaki¹ (1. The University of Tokyo, 2. Talent and Assessment Inc.)

Keywords:Multimodal Recognition, Job Interview, Transformer, Prompt Learning

In our earlier study, we introduced a multimodal neural network designed to assess online interviews and appraise candidates' performance. However, the previous study focused solely on a subset of evaluation criteria named question items that assess distinct sections within the interview process. In this study, our evaluation criteria are extended to observation items that encompass the entire interview process rather than targeting specific sections. Because some samples lack audio modality, we use prompt learning to discern between the samples with completed modalities and those without audio modality. Furthermore, we apply the re-sampling method and margin ranking loss to improve the model robustness on imbalanced distribution. For the experimental results, the prompt learning and class-imbalanced learning methods improved the prediction accuracy, and the proposed model finally achieves an average accuracy of 67.41% in binary classification for the extended eight criteria, providing a more holistic assessment of candidate performance.