[3T5-OS-6b] OS-6

[3T5-OS-6b-04] Fetch-and-Carry Tasks by Domestic Service Robots Based on Multimodal Retrieval Models with Switching Mechanism Using Large Language Models

〇Ryosuke Korekata¹, Kanta Kaneda¹, Shunya Nagashima¹, Yuto Imai¹, Komei Sugiura¹ (1. Keio University)

Keywords:Domestic Service Robot, Object Retrieval, Large Language Model, Multimodal Foundation Model, Fetch-and-Carry

In this study, we aim to develop a domestic service robot (DSR) that carries an everyday object to a piece of furniture by retrieving images of target objects and receptacles from collected images of an environment, based on an open-vocabulary instruction. We propose a multimodal model that retrieves both target objects and receptacles individually using a single model based on the switching mechanism via large language models. The experimental results show that our method outperformed baseline methods on the newly-built datasets in terms of the standard metrics. Furthermore, our method achieved task success rates of more than 80% in the physical experiments.