[1Win4] Poster session 1

[1Win4-50] Open-Vocabulary Mobile Manipulation Based on Double Relaxed Contrastive Loss

〇Daichi Yashima¹, Ryosuke Korekata¹, Komei Sugiura¹ (1.Keio University)

Keywords:Domestic Service Robot, Open-Vocabulary Mobile Manipulation, Physical-World Search Engine

In this study, we develop a DSR that transports everyday objects to specified pieces of furniture based on open-vocabulary instructions. Our approach focuses on retrieving images of target objects and receptacles from pre-collected images of indoor environments. We propose a multimodal model, which learns diverse and robust representations from among positive, unlabeled positive, and negative samples. The experimental results demonstrate that our model outperformed existing baseline models across standard image retrieval metrics. Moreover, we performed physical experiments using a DSR to evaluate the performance of our approach in a zero-shot transfer setting. The experiments involved the DSR to carry objects to specific receptacles based on open-vocabulary instructions, achieving an overall success rate of 75%.