[1N4-IS-1a] Knowledge engineering (1/3)

[1N4-IS-1a-04] Semantic and Topic fused Multimodal Transformer

〇Shuhei Tateishi¹, Sohei Okui¹, Hirofumi Yashima¹, Makoto Nakatsuji¹ (1. NTT Resonant, Inc.)

Keywords:Multimodal, Transformer, Semantic

The human behaviors observed in multi modalities are naturally related with each other and many AI/ML methods try to catch such implicit relationships to enhance the prediction accuracies on users' behaviors.
Ordinal methods, however, do not try to use the explicit relationships (e.g. taxonomy of emotions) across the modalities.
In this paper, we suggest a new hypothesis for measuring the relevances across the modalities to improve accuracies.