[4Xin1] Poster session 2

[4Xin1-44] Effective Neural Representation for Arithmetic Tasks Induced by Multi-modal Deep Learning Models

Kamma Noda¹, 〇Takafumi Soda^1,2, Yuichi Yamashita¹ (1.National Center of Neurology and Psychiatry, 2.Tokyo Medical and Dental University)

Keywords:deep learning, synesthesia, multimodal learning, number sense, computational neuroscience

The associations of multimodal information are essential for human cognitive functions. Recently, multimodal learning has received a lot of attention in the field of machine learning. Investigating the impact of multimodal learning on its representation could facilitate our understanding of multimodal associative learning in humans. Here, a multimodal deep learning model was used as a computational model of multimodal associations in the brain. Representations of numerical information, such as handwritten numbers and images of geometric figures, were learnt and compared using single-modal and multimodal learning. Multimodal learning models acquired better latent representations. Furthermore, the models trained using multimodal method performed better on the downstream arithmetic task. These results showed that changes in latent representations acquired during multimodal associative learning were directly related to cognitive function. This supports the potential of multimodal learning research to provide new insights into the understanding of higher cognitive functions in humans, including mathematical abilities.