[2G5-OS-21e] 世界モデルと知能

[2G5-OS-21e-02] Scaling Laws of Model Size for World Models

〇Makoto Sato^1,6, Ryosuke Unno^2,6, Masahiro Negishi^2,6, Koudai Tabata^2,6, Taiju Watanabe^3,6, Junnosuke Kamohara^4,6, Taiga Kume^5,6, Ryo Okada^2,6, Yusuke Iwasawa², Yutaka Matsuo² (1. Nara Institute of Science and Technology, 2. The University of Tokyo, 3. Waseda University, 4. Tohoku University, 5. Keio University, 6. Matsuo Institute)

Keywords:World Models, Large Language Models, Scaling Laws

With the development of deep learning, significant performance improvements have been achieved in computer vision and natural language processing. In these advancements, scaling laws that demonstrate exponential changes in model performance with respect to model size, dataset size, and computational resources used for training have played a significant role. These scaling laws have been reported to hold for various tasks, including image classification, image generation, and natural language processing. However, it has not yet been verified whether these scaling laws are effective for tasks that involve long-horizon predictions. In this study, we investigate the validity of scaling laws for world models from the perspective of model size. We conduct experiments that scale the model sizes of two world models in a video prediction task conditioned on actions using the CARLA dataset, and verify that the loss function decreases exponentially and the scaling law holds when including large-scale autoencoder.