[2G1-OS-21c] 世界モデルと知能

[2G1-OS-21c-05] Acquiring Peripersonal Space Representation Shared Between Vision and Touch by Transformer Autoencoder

〇Wataru Noguchi¹, Hiroyuki Iizuka¹, Masahito Yamamoto¹ (1. Hokkaido University)

Keywords:Spatial Recognition, Multimodal Integration, Deep Learning

Peripersonal space, where individuals interact with the environment within their reach, has multimodal representations in the brain. It is assumed that the multimodal representation of peripersonal space is acquired through interaction with the environment. In this study, we propose a neural network model that acquires a representation of peripersonal space shared between vision and touch through the experience of vision, touch, and proprioception. Our proposed model reconstructs visual and tactile observations corresponding to proprioceptive inputs after integrating the observations through Transformer based on self-attention mechanism. By learning on camera vision and arm touch of a simulated robot and proprioceptive inputs of camera and arm poses, a spatial representation like a map between the spatial coordinates of peripersonal space and visual and tactile observations was constructed in the model. In particular, the spatial map was shared between vision and touch by sharing part of the visual and tactile decoding module.