[2G1-OS-21c] 世界モデルと知能

[2G1-OS-21c-02] Multi-modal NewtonianVAE: High-precision reaching method for autonomous suturing

〇Mai Terashima¹, Pedro Miguel Uriguen Eljuri¹, Yuanyuan Jia¹, Hironobu Shibata¹, Masaki Ito¹, Tadahiro Taniguchi¹ (1. Ritsumeikan University)

Keywords:World model, Multi-modal information

This study focuses on NewtonianVAE, a world model that can learn a proportionally controllable latent space. To achieve precise control in a physical world, it is necessary to construct a latent space of NewtonianVAE representing a physical world from multi-modal observations. However, learning from multi-modal observations using NewtonianVAE has not been studied.
To address this issue, we discuss methods for learning multi-modal observations using NewtonianVAE.
In this paper, we propose Multi-modal NewtonianVAE (MNVAE), which uses Mixture-of-Products-of-Experts (MoPoE) to integrate multi-modal observations.
MNVAE learns a latent space representing a physical environment and it has the potential for precise control in a physical world.