AUTHOR=Massi Elisa , Barthélemy Jeanne , Mailly Juliane , Dromnelle Rémi , Canitrot Julien , Poniatowski Esther , Girard Benoît , Khamassi Mehdi TITLE=Model-Based and Model-Free Replay Mechanisms for Reinforcement Learning in Neurorobotics JOURNAL=Frontiers in Neurorobotics VOLUME=16 YEAR=2022 URL=https://www.frontiersin.org/journals/neurorobotics/articles/10.3389/fnbot.2022.864380 DOI=10.3389/fnbot.2022.864380 ISSN=1662-5218 ABSTRACT=

Experience replay is widely used in AI to bootstrap reinforcement learning (RL) by enabling an agent to remember and reuse past experiences. Classical techniques include shuffled-, reversed-ordered- and prioritized-memory buffers, which have different properties and advantages depending on the nature of the data and problem. Interestingly, recent computational neuroscience work has shown that these techniques are relevant to model hippocampal reactivations recorded during rodent navigation. Nevertheless, the brain mechanisms for orchestrating hippocampal replay are still unclear. In this paper, we present recent neurorobotics research aiming to endow a navigating robot with a neuro-inspired RL architecture (including different learning strategies, such as model-based (MB) and model-free (MF), and different replay techniques). We illustrate through a series of numerical simulations how the specificities of robotic experimentation (e.g., autonomous state decomposition by the robot, noisy perception, state transition uncertainty, non-stationarity) can shed new lights on which replay techniques turn out to be more efficient in different situations. Finally, we close the loop by raising new hypotheses for neuroscience from such robotic models of hippocampal replay.