[2G5-OS-18a] 脳波から音声言語情報を抽出・識別・利用する(1/2)

[2G5-OS-18a-01] Verification of general-purpose language models and deep learning models for estimating brain activities evoked by verbal stimuli

〇Rikako Sumida¹, Hiroto Yamaguchi^2,3, Tomoya Nakai³, Shinji Nishimoto^2,3, Ichiro Kobayashi¹ (1. Ochanomizu University, 2. Osaka University, 3. National Institute of Information and Communications Technology)

Keywords:fMRI, Deep Learning , language processing

For the estimation of brain states in spoken conversation stimuli, we conducted an experiment using three types of deep learning models (Bi-LSTM/Bi-GRU/Bi-RNN) to estimate brain activity data using speech spectrogram as speech features, and compared the estimation performance of each model. There was no significant difference in the performance of any of those models, and we confirmed that the brain regions close to the ears, which are considered to be responsible for phonological and grammatical processing, responded better.
In addition, we predicted brain activity using linguistic features transcribed from auditory stimuli into text. We used RoBERTa/BERT/word2vec as a general-purpose language model to convert them into embedded vectors. In this experiment, we could confirm responses in a wide range of language areas in the brain, not limited to the peripheral regions of the ear.