[2Q1-OS-27a] 強化学習の新展開

[2Q1-OS-27a-02] Offline Model-Based Imitation Learning with Entropy Regularization of Model and Policy

〇Eiji Uchibe¹ (1. Advanced Telecommunications Research Institute International)

Keywords:Offline imitation learning, Model-Based Forward and Inverse Reinforcement Learning, Entropy Regularization

Model-Based Entropy-Regularized Imitation Learning (MB-ERIL) is an online model-based generative adversarial imitation learning method that introduces entropy regularization of policy and state transition model. Online-MB-ERIL learns the policy and model from expert data, learner's data, and generated data. Costly interactions with an actual environment are needed to obtain the first two datasets, while the policy and model quickly generate the last one. This report discusses an offline learning setting without using the second data obtained from the interaction between the policy and the actual environment. Next, we propose Offline-MB-ERIL, which introduces the idea of Positive and Unlabeled data learning. Given sub-optimal data, Offline-MB-ERIL can recover policy and model efficiently using them as unlabeled data. Through a vision-based arm-reaching task, we show that Offline-MB-ERIL can better use suboptimal data than Online-MB-ERIL.