AUTHOR=Nguyen Quang Dang , Prokopenko Mikhail TITLE=Structure-Preserving Imitation Learning With Delayed Reward: An Evaluation Within the RoboCup Soccer 2D Simulation Environment JOURNAL=Frontiers in Robotics and AI VOLUME=7 YEAR=2020 URL=https://www.frontiersin.org/journals/robotics-and-ai/articles/10.3389/frobt.2020.00123 DOI=10.3389/frobt.2020.00123 ISSN=2296-9144 ABSTRACT=

We describe and evaluate a neural network-based architecture aimed to imitate and improve the performance of a fully autonomous soccer team in RoboCup Soccer 2D Simulation environment. The approach utilizes deep Q-network architecture for action determination and a deep neural network for parameter learning. The proposed solution is shown to be feasible for replacing a selected behavioral module in a well-established RoboCup base team, Gliders2d, in which behavioral modules have been evolved with human experts in the loop. Furthermore, we introduce an additional performance-correlated signal (a delayed reward signal), enabling a search for local maxima during a training phase. The extension is compared against a known benchmark. Finally, we investigate the extent to which preserving the structure of expert-designed behaviors affects the performance of a neural network-based solution.