[4Pin1] インタラクティブ(2)

[4Pin1-37] A computational system estimating perception evoked by arbitrary visual inputs on the basis of modeling the perceptual representation in the brain

〇Satoshi Nishida^1,2, Shinji Nishimoto^1,2 (1. National Institute of Information and Communications Technology, 2. Osaka University)

Keywords:Brain, Neural decoding, fMRI, Deep learning, Natural language processing

Although brain decoding techniques using functional magnetic resonance imaging (fMRI) have many potential real-world applications, the measuring cost of fMRI makes it difficult to realize many of such applications. Here, we propose a new decoding framework for estimating perceptual experiences evoked by visual materials with no additional fMRI measurement after model construction. Our framework consists of brain-activity prediction and perceptual decoding models constructed from individual brain data. Once the training of these models using movie-evoked fMRI data has been done, the framework combines these models and estimates each person’s perceptual experiences regarding novel scenes without any additional fMRI measurements. Our results showed that our framework well estimated perceptual experiences that were evoked by novel scenes, and the estimated contents varied across the models for individual persons. Thus, our framework may provide a new computational system for estimating personal perceptual experiences, evoked by arbitrary visual inputs, via the perceptual representation in the brain.