[4P3-J-10] Vision, speech: image recognition learning

[4P3-J-10-04] Reconstructing Dynamic Visual Stimuli from Human Brain Activity using Deep Neural Networks

〇Yudai Nagano¹, Ichiro Kobayashi², Shinji Nishimoto³, Hideki Nakayama¹ (1. Graduate School of Information Science and Technology, The University of Tokyo, 2. Advanced Sciences, Graduate School of Humanities and Sciences, Ochanomizu University, 3. Center for Information and Neural Networks, National Institute of Information and Communications Technology)

Keywords:decoding, brain activity, fMRI

Decoding is one of the important fields in Neuroscience, which is considered to be useful for analysis of brain function, clarification of disease and development of Brain Machine Interfaces. The purpose of this study is to decode visual stimuli from human brain activity. To reconstruct the visual stimuli, we used Neural Networks and GAN-based Neural Networks. We compared recent GAN-based models to confirm which one works the best. We also examined the difference in reconstruction quality when brain area was changed. To improve the quality of reconstruction, we combine multiple consecutive frames of human brain activity. Finally, we calculated the effect of multi-frame by quantative evaluation. The results show the effectiveness of decoding with multi-frame inputs.