[2G4-OS-21d] 世界モデルと知能

[2G4-OS-21d-01] Switching Head-Tail Funnel UNITER: Multimodal Instruction Comprehension for Object Manipulation Tasks

〇Ryosuke Korekata¹, Motonari Kambara¹, Yu Yoshida¹, Shintaro Ishikawa¹, Yosuke Kawasaki¹, Masaki Takahashi¹, Komei Sugiura¹ (1. Keio University)

Keywords:Multimodal Language Understanding, Vision & Language, fetch-and-carry, Object Manipulation, Domestic Service Robot

This paper describes a domestic service robot (DSR) that fetches everyday objects and carries them to specified destinations according to free-form natural language instructions. We propose Switching Head-Tail Funnel UNITER, which solves the task by predicting the target object and the destination individually using a single model. We conduct physical experiments in which a DSR delivers standardized everyday objects in a standardized domestic environment as requested by instructions with referring expressions. The experimental results show that our method outperforms the baseline method in terms of language comprehension accuracy and the object grasping and placing actions are achieved with success rates of more than 90%.