[2M6-OS-19d] 世界モデルと知能(4/4)

[2M6-OS-19d-01] 2D Rendering-based Pseudo 3D Models for Casual VR Experiments

〇Naruya Kondo¹, So Kuroki², Ryosuke Hyakuta¹, Yutaka Matsuo², Shixiang Shane Gu³, Yoichi Ochiai¹ (1. University of Tsukuba, 2. The University of Tokyo, 3. Google Brain)

Keywords:World Model, Virtual Reality

An aspirational goal for virtual reality (VR) is to bring in a rich diversity of real world objects losslessly. Existing VR applications often convert objects into explicit 3D models with meshes, which allow fast interactive rendering but also severely limit its quality and the types of supported objects, fundamentally upper-bounding the “realism” of VR. Inspired by the classic “billboards” technique in gaming, we develop Deep Billboards that model 3D objects implicitly using neural networks, where only 2D image is rendered at a time based on the user’s viewing direction. Our system, connecting a commercial VR headset with a server running neural rendering, allows real-time high-resolution simulation of detailed rigid objects, hairy objects, actuated dynamic objects and more in an interactive VR world, drastically narrowing the existing real-to-simulation (real2sim) gap. Additionally, we augment Deep Billboards with physical interaction capability, adapting classic billboards from screen-based games to immersive VR.