Skip to main content

ORIGINAL RESEARCH article

Front. Artif. Intell.
Sec. Machine Learning and Artificial Intelligence
Volume 7 - 2024 | doi: 10.3389/frai.2024.1397915
This article is part of the Research Topic Deep Neural Network Architectures and Reservoir Computing View all articles

Multi-scale Dynamics by Adjusting Leaking Rate to Enhance Performance of Deep Echo State Networks

Provisionally accepted
Shuichi Inoue Shuichi Inoue 1Sou Nobukawa Sou Nobukawa 1*Haruhiko Nishimura Haruhiko Nishimura 2Eiji Watanabe Eiji Watanabe 3Teijiro Isokawa Teijiro Isokawa 4
  • 1 Chiba Institute of Technology, Narashino, Chiba, Japan
  • 2 Yamato University, Suita, Miyagi, Japan
  • 3 National Institute for Basic Biology, Okazaki, Aichi, Japan
  • 4 University of Hyogo, Kobe, Hyōgo, Japan

The final, formatted version of the article will be published soon.

    The deep echo state network (Deep-ESN) architecture, which comprises a multi-layered reservoir layer, exhibits superior performance compared to conventional echo state networks (ESNs) owing to the divergent layer-specific time-scale responses in the Deep-ESN. Although researchers have attempted to use experimental trial-and-error grid searches and Bayesian optimization methods to adjust the hyperparameters, suitable guidelines for setting hyperparameters to adjust the time scale of the dynamics in each layer from the perspective of dynamical characteristics have not been established. In this context, we hypothesized that evaluating the dependence of the multi-time-scale dynamical response on the leaking rate as a typical hyperparameter of the time scale in each neuron would help to achieve a guideline for optimizing the hyperparameters of the Deep-ESN. First, we set several leaking rates for each layer of the Deep-ESN and performed multi-scale entropy (MSCE) analysis to analyze the impact of the leaking rate on the dynamics in each layer. Second, we performed layer-by-layer cross-correlation analysis between adjacent layers to elucidate the structural mechanisms to enhance the performance. As a result, an optimum task-specific leaking rate value for producing layer-specific multi-time-scale responses and a queue structure with layer-to-layer signal transmission delays for retaining past applied input enhance the Deep-ESN prediction performance. These findings can help to establish ideal design guidelines for setting the hyperparameters of Deep-ESNs.

    Keywords: Multi-scale dynamics, machine learning, reservoir computing, Echo state network, Deep echo state network

    Received: 08 Mar 2024; Accepted: 18 Jun 2024.

    Copyright: © 2024 Inoue, Nobukawa, Nishimura, Watanabe and Isokawa. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

    * Correspondence: Sou Nobukawa, Chiba Institute of Technology, Narashino, 275-0016, Chiba, Japan

    Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.