JSAI2023

Presentation information

General Session

General Session » GS-2 Machine learning

[1B4-GS-2] Machine learning

Tue. Jun 6, 2023 3:00 PM - 4:40 PM Room B (Civic hall B)

座長:井田 安俊(NTT) [現地]

4:20 PM - 4:40 PM

[1B4-GS-2-05] Deep reinforcement learning with planning based on replay of similar experiences

〇Shunpei Koshikawa1, Jun Kume1, Koki Higuchi1, Tatsuji Takahashi2, Hiroyuki Ohta3 (1. Graduate School of Tokyo Denki University, 2. Tokyo Denki University, 3. National Defense Medical College)

Keywords:Machine learning, Reinforcement learning, Experience replay, Behavior planning, Deep learning

The hippocampus is known to be the brain region that replays past experiences. In the context of deep reinforcement learning, experience replay has traditionally been used primarily to improve the sample efficiency of data used to train artificial neural networks and to maintain independence among samples. However, recent advances in neuroscience research have revealed that hippocampal replays occur prior to the onset of locomotion and involve planning that selects the optimal locomotion path from among previously experienced paths, starting from the current location. Inspired by this phenomena, we proposed a mechanism in the Deep Q-Network (DQN) framework to reflect in the current action selection previously experienced paths. This mechanism is described as follows: first, search for trajectories that start from states similar to the current state in the replay buffer that holds previously observed information. Second, reflect the n-step rewards in the past action selections by adding them to the action value of the current state. Our simulation experiments with CliffWalking confirmed that the proposed method allows the agent to maximize returns earlier and to reach the terminal state with fewer steps than normal DQN.

Authentication for paper PDF access

A password is required to view paper PDFs. If you are a registered participant, please log on the site from Participant Log In.
You could view the PDF with entering the PDF viewing password bellow.

Password