Prefrontal Meta-Control Incorporating Mental Simulation Enhances the Adaptivity of Reinforcement Learning Agents in Dynamic Environments

Jihun Kim ¹ Jee Hang Lee ^1,2*

• ¹ Sangmyung University, Seoul, Republic of Korea
• ² Institute for Advanced Intelligence Study, Daejeon, Republic of Korea

Recent advances in computational neuroscience highlight the significance of prefrontal cortical meta-control mechanisms in facilitating flexible and adaptive human behaviour. In addition, hippocampal function, particularly mental simulation capacity, proves essential in this adaptive process. Rooted from these neuroscientific insights, we present Meta-Dyna, a novel neuroscienceinspired reinforcement learning architecture that demonstrates rapid adaptation to environmental dynamics whilst managing variable goal states and state-transition uncertainties. This architectural framework implements prefrontal meta-control mechanisms integrated with hippocampal replay function, which in turn optimised task performance with limited experiences. We evaluated this approach through comprehensive experimental simulations across three distinct paradigms: the two-stage Markov decision task, which frequently serves in human learning and decisionmaking research; stochastic GridWorldLoCA, an established benchmark suite for model-based reinforcement learning; and a stochastic Atari Pong variant incorporating multiple goals under uncertainty. Experimental results demonstrate Meta-Dyna's superior performance compared with baseline reinforcement learning algorithms across multiple metrics: average reward, choice optimality, and a number of trials for success. These findings advance our understanding of computational reinforcement learning whilst contributing to the development of brain-inspired learning agents capable of flexible, goal-directed behaviour within dynamic environments.