[2A3] [General Session] 11. Robot / Real World

[2A3-01] Bidirectional Translation between Robot Actions and Linguistic Descriptions by Aligned Representations

〇Tatsuro Yamada^1,2, Hiroyuki Matsunaga¹, Tetsuya Ogata¹ (1. Waseda University, 2. Research Fellow of Japan Society for the Promotion of Science)

Keywords:recurrent neural network, symbol grounding problem, representation learning, sequence-to-sequence learning

We propose a novel deep learning framework to bidirectionally translate between robot actions and their linguistic descriptions. The model consists of two recurrent autoencoders, one of which is employed to make vector representations of robot actions and the other is for descriptions. The learning algorithm produces representations shared between actions and their descriptions by creating an additional loss function in which the representation of an action and that of its description become close to each other in the latent vector space. Across the shared representation, the trained model can produce a linguistic description given a robot action. The model is also able to generate an appropriate action by receiving a linguistic instruction, conditioned on the current visual input. Visualization of the shared representations shows that the robot actions are represented in a semantically compositional way in the vector space by being learned jointly with their descriptions.