[4E1-GS-8] Robot and real worlds:

[4E1-GS-8-04] Online robot program expanding based on field knowledge of life support robots

〇Yoshiki Obinata¹, Naoto Tsukamoto¹, Kento Kawaharazuka¹, Naoaki Kanazawa¹, Kei Okada¹, Masayuki Inaba¹ (1. The University of Tokyo)

Keywords:Life support robot, Foundation model

This paper proposes a natural language programming system for deterministic and transparent daily life support robot actions. The system allows the user to instruct the robot's actions in natural language and the robot to confirm and give feedback on its action program in natural language. The system also enables users to explicitly request the robot's field judgment by embedding template expressions in the predicates. The robot expands the template arguments based on field knowledge using large-scale language models (LLM) and large-scale vision-language models (VLM). The paper demonstrates the system's usefulness in a real-world scenario where the robot goes to a sandwich shop to buy food for the user.