[3R5-OS-31] OS-31

[3R5-OS-31-02] Combining IMPALA and Demonstrations to Solve Problems with Hard Exploration and Large State-Action Spaces

Application examples in Yu-Gi-Oh! MASTER DUEL

〇Sora Satake¹, Soichiro Hattori¹, Naoya Kihara¹ (1. Konami Digital Entertainment Co., Ltd.)

Keywords:Digital Game, AI, Reinforcement Learning, Game AI

When applying deep reinforcement learning to current digital games, the difficulty of exploration and the vastness of the state-action space often become challenges. If a lot of play logs can be utilized, these difficulties can be mitigated through imitation learning. However, in cases where sufficient logs are difficult to collect, such as during the development of a game or events with different regulations, imitation learning may not be feasible. In this study, we propose a method to efficiently perform deep reinforcement learning using the IMPALA architecture by guiding exploration with small-scale demonstrations that developers can manually create. We show that it is possible to quickly learn hard exploration problems by devising the correction calculation in V-trace. Additionally, in current competitive digital games, we trained the AI using the proposed method. As a result, we trained an AI that is as strong as existing rule-based AI.