General Distributed Neural Control and Sensory Adaptation for Self-Organized Locomotion and Fast Adaptation to Damage of Walking Robots

Walking animals such as invertebrates can effectively perform self-organized and robust locomotion. They can also quickly adapt their gait to deal with injury or damage. Such a complex achievement is mainly performed via coordination between the legs, commonly known as interlimb coordination. Several components underlying the interlimb coordination process (like distributed neural control circuits, local sensory feedback, and body-environment interactions during movement) have been recently identified and applied to the control systems of walking robots. However, while the sensory pathways of biological systems are plastic and can be continuously readjusted (referred to as sensory adaptation), those implemented on robots are typically static. They first need to be manually adjusted or optimized offline to obtain stable locomotion. In this study, we introduce a fast learning mechanism for online sensory adaptation. It can continuously adjust the strength of sensory pathways, thereby introducing flexible plasticity into the connections between sensory feedback and neural control circuits. We combine the sensory adaptation mechanism with distributed neural control circuits to acquire the adaptive and robust interlimb coordination of walking robots. This novel approach is also general and flexible. It can automatically adapt to different walking robots and allow them to perform stable self-organized locomotion as well as quickly deal with damage within a few walking steps. The adaptation of plasticity after damage or injury is considered here as lesion-induced plasticity. We validated our adaptive interlimb coordination approach with continuous online sensory adaptation on simulated 4-, 6-, 8-, and 20-legged robots. This study not only proposes an adaptive neural control system for artificial walking systems but also offers a possibility of invertebrate nervous systems with flexible plasticity for locomotion and adaptation to injury.

1. Introduction

Walking animals show robust and adaptive locomotion. They can form their gaits in a self-organized manner as well as quickly adapt to environmental and body changes, including damage (Wolf and Büschges, 1997; Büschges and Manira, 1998; Grabowska et al., 2012). This complex ability is achieved through adaptive interlimb coordination mechanisms. Biological investigation reveals that the adaptive coordination in walking animals is largely attained by distributed neural control mechanisms with central pattern generators (CPGs), local or proprioceptive feedback, and body dynamics (Pearson and Iles, 1970, 1973; Bässler and Wegner, 1983; Dean, 1989; Berkowitz and Laurent, 1996).

While these components have been identified, their details have not been fully applied to artificial walking systems. For example, while animal experiments show that synaptic connections in sensory motor pathways are plastic (i.e., sensory adaptation) (Whelan and Pearson, 1997; Wolf and Büschges, 1997; Wark et al., 2007) to allow for stable locomotion and adaptation, this plasticity with continuous synaptic changes has been largely ignored in robotic implementation. Typically, the connections between sensory feedback and neural circuits for locomotion control of walking robots are static. To obtain stable locomotion, these connections are usually adjusted manually, or empirically chosen, for specific walking robots (Owaki et al., 2012; Barikhan et al., 2014). In some cases, machine learning techniques are employed first to optimize the connections through simulation before implementing them on real robots (Hwangbo et al., 2019). Accordingly, unexpected situations such as leg damage might lead to unstable locomotion if the sensory connection strength cannot be automatically or continuously adjusted to deliver proper information for adaptation. Furthermore, transferring the control system with the tuned or optimized connections from one walking robot to another might not work effectively.

From this perspective, in this study, we introduce a fast learning mechanism for continuous online adaptation or flexible plasticity in sensory pathways (i.e., synaptic connection strength plasticity of sensory feedback) in order to (i) generate stable self-organized locomotion, (ii) deal with damage (known as lesion-induced plasticity), and (iii) be able to automatically adapt to different walking robots. Specifically, the learning mechanism will continuously adjust the connection strength between proprioceptive feedback (i.e., foot contact feedback) and distributed neural CPG-based control circuits (Figure 1). This approach combines bio-inspired key ingredients including: (1) distributed neural CPG-based control circuits without inter-circuit connections for flexible and independent individual leg control, (2) a learning mechanism for proprioceptive sensory adaptation, and (3) body-environment interaction, to acquire adaptive and flexible interlimb coordination for walking robots. This novel approach has more advantages compared to others (Ijspeert et al., 2007; Manoonpong et al., 2008, 2013; Inagaki et al., 2010; Asif, 2012; Ambe et al., 2013) in the following aspects:

• It does not require predefined interlimb coordination (i.e., hardwired neural connections between the CPG circuits), predefined or preoptimized connection strength in sensory pathways, or even the robot's kinematic model.

• It can be directly applied to different walking robots (i.e., generalization and transferability) allowing them to quickly perform stable, self-organized locomotion.

• It can quickly deal with damage within a few walking steps.

FIGURE 1

Figure 1. (Left) The robot structure for our experiments, consisting of multiple invertebrate-inspired body segments where each has either one or two pairs of identical legs. Legs are inspired by insect limbs and have three joints: Thoraco-coxal (TC-), coxa-trochanteral (CTr-), and femur-tibia (FTi-) joints. The TC-joint enables forward (+) and backward (−) movement. The CTr- joint enables elevation (+) and depression (−) of the leg. The FTi-joint enables extension (+) and flexion (−) of one leg. The connection between body segments is performed by a universal passive connector with two joints(U-joint1 and U-joint2) and limited rotation. (Right) A complete adaptive neural control system for each leg, consisting of (i) CPG-based control, (ii) forward model, and (iii) dual rate learning. This is a distributed and decoupled control structure where each leg is driven by one control system and there is no coupling between the control systems. This system uses foot contact or load sensing feedback to alter the CPG phase of each leg; thereby forming interlimb coordination. The CPG post-processing unit (PCPG) converts the CPG outputs into proper motor commands, which are at the same time copied (efference copy) to the forward model for foot contact sensory prediction. The dual rate learning mechanism affects the CPG inputs (i.e., sensory feedback) by adjusting the synaptic plasticity (i.e., sensory feedback connection, see dashed line) based on an error signal between the prediction and actual foot contact feedback (see also Figure 5). (Bottom) Self-organized locomotion and adaptation of a walking robot under the adaptive neural control system with sensory synaptic plasticity.

We validated our proposed adaptive interlimb coordination approach on simulated 4-, 6-, 8-, and 20-legged robots. We believe that the study pursued here will also sharpen our understanding of how continuous online sensory adaptation with flexible plasticity can be realized and combined with control mechanisms for self-organized locomotion and fast adaptation to damage in walking systems which could not be realized solely by conventional bio-inspired control methods (Espenschied et al., 1996; Ijspeert et al., 2007; Manoonpong et al., 2008, 2013; Inagaki et al., 2010; Asif, 2012; Ambe et al., 2013; Bjelonic et al., 2016) or machine learning techniques (Bongard et al., 2006; Cully et al., 2015; Hwangbo et al., 2019), or their combination (like CPG-based control with reinforcement learning, Ishige et al., 2019) (see section 5 for more details).

2. Materials and Methods

In this section, we describe the development of our neural control system for self-organized locomotion and fast adaptation to damage of walking robots. The system exploits neural dynamics and plasticity, proprioceptive feedback (i.e., load sensing feedback), and robot body dynamics to adaptively coordinate robot limbs, called adaptive interlimb coordination. The proposed control system involves three main components: (1) neural CPG-based control with load sensing feedback for rhythmic movement generation, (2) a forward model for sensory prediction, and (3) a dual rate learning mechanism for continuous online sensory adaptation or synaptic connection strength plasticity of sensory feedback (Figure 1). Each of which is described in detail below.

In this setup, each leg is driven by one control system (i.e., one neural CPG-based control circuit, one forward model, and one dual rate learning mechanism). As a consequence, controlling 4-, 6-, 8-, and 20-legged robots will require 4, 6, 8, and 20 neural control systems. For flexibility, we do not define any connection or coupling between the neural control systems. Instead, the coordination among them is mainly achieved by the interaction between the robot and environment, resulting in self-organized locomotion.

Similar configurations have been used before, showing that it is possible to use sensory feedback to adjust the phase between the legs of a walking robot (Owaki et al., 2012; Barikhan et al., 2014). The feedback-based phase adjustment eventually forms a walking pattern, allowing the robot to walk, while the neural CPGs continue to produce the oscillatory signals driving the leg movements during each step. However, this does not guarantee that the robot will always keep a stable gait during walking, since a change in the sensory feedback can appear as a consequence of structural changes in the robot (e.g., leg damage and different configurations). Thus, sufficient sensory feedback information should be maintained to adapt to a new stable pattern. The mechanism for automatic adjustment of sensory feedback strength (or synaptic connection strength plasticity) on the control loop adds a degree of flexibility to the feedback-based phase adjustment method, allowing for adaptation to the changes. It is also important to note that this control does not cover the trajectory of a step. In this case, the individual leg trajectories (i.e., intralimb coordination) have been predefined to fit the normal movement of these limbs based on that of insect legs. However, to adapt the proposed mechanism to other configurations, it is necessary to create an appropriate translation from CPG outputs to motor commands. For example, the robot legs here include three joints, allowing the movement of their tips with respect to the defined trajectory for self-organized locomotion (see section 3).

2.1. Neural CPG-Based Control With Load Sensing Feedback

The neural CPG-based control is based on two coupled recurrent neurons which generate two periodic signals (Figure 2). The signals are converted into motor commands for driving the leg joints (TC-, CTr-, and FTi-joints) through the CPG postprocessing unit. To automatically adjust the phase of the CPG for interlimb coordination, we use load sensing feedback, projecting to the CPG as its inputs. This technique was proposed by Owaki et al. (2012) and Barikhan et al. (2014). This feedback produces the appropriate phase shifts between the legs to generate a robot walking gait while keeping an understandable and simple control mechanism.

FIGURE 2

Figure 2. (Left) The CPG model of the neural CPG-based control, consisting of two recurrent neurons (N1, N2) with mutual connections. The sensory feedback (S_1,2) is projected to the CPG as its neural inputs. (Right) The CPG output signals. The signals differ in phase by π/2 and are further shaped by a CPG postprocessing unit to obtain smooth signals for motor control. In this study, the weights of all CPGs are set to w₁₁ = 1.4, w₂₂ = 1.4, w₂₁ = −0.78, and w₁₂ = 0.78. With these parameters, the CPG generates periodic signals with a frequency of 1.4 Hz.

The neurons of the CPG control network are modeled as discrete-time non-spiking neurons. They are updated using a frequency of approximately 27 Hz. The activity of each neuron develops according to:

$\begin{array}{ll}{a}_1(t+1)={\displaystyle \sum _{j=1}^2}{w}_{1j}{o}_j(t)+\alpha S_1(t)& (1)\end{array}$

$\begin{array}{ll}{a}_2(t+1)={\displaystyle \sum _{j=1}^2}{w}_{2j}{o}_j(t)+\alpha S_2(t)& (2)\end{array}$

$\begin{array}{ll}{o}_i(t+1)=tanh(a_i(t+1))i=1,2& (3)\end{array}$

where w_1j,2j are the synaptic connection weights between the neurons, S_1,2 are the CPG inputs (i.e., load sensing feedback), and o_i are the CPG outputs. α is a sensory feedback connection (synaptic plasticity), automatically adjusted by dual rate learning (described in detail in the following section. See Equation 11). The CPG inputs are defined as:

$\begin{array}{ll}{S}_1(t)=-F(t)cos(a_1(t))& (4)\end{array}$

$\begin{array}{ll}{S}_2(t)=-F(t)sin(a_2(t))& (5)\end{array}$

where F is the negative continuous load sensing feedback at the leg. The sine and cosine functions of the neural activities a_1,2 are used to derive a proper correlation between the neural activity and feedback. The functions are related to the phases of the CPG outputs¹ o_1,2 which differ by π/2 (see Figure 2). The strength of the sensory feedback connection can be adapted to regulate the amount of sensory feedback to the CPG-based control. Through this connection, the foot contact sensory feedback can reduce the leg speed when highly loaded at the end of the stance phase, while increasing the speed of the leg trajectory when it is unloaded at the end of the swing phase. This will allow the robot to adaptively adjust its leg movement to form a stable gait with good body weight distribution.

When implementing the neural control system on different robots, the proper value of α needs to be used for stable locomotion and this value might have to be changed in the face of unexpected situations, such as leg damage. The difficulty in determining an optimal value for all cases motivated us to develop an automatic process for continuously and dynamically adjusting the value without previous knowledge of the robot's morphology. The effect of the load sensing feedback on the CPG outputs is not the same for all CPGs or legs, but rather corresponds to the correlation of the feedback and the neural activities (Equations 4, 5). The variation of the influence on the CPGs will automatically yield phase differences among them, which will be translated into proper interlimb coordination (i.e., leg coordination). As a result, the robot will perform an adaptive gait. In this case, we do not have a fixed and predefined interlimb coordination, but rather a flexible one, since the gait obtained is derived from load sensing feedback, synaptic plasticity, neural activities, and body-environment interaction.

For intralimb coordination (i.e., joint coordination) in each leg of all tested robots, we project the CPG outputs to the TC-, CTr-, and FTi-joints indirectly through a CPG post processing unit. This post processing unit shapes the CPG outputs into the desired motor commands. In this case, a simple algorithm defining the swing and stance periods of the robots is applied (see Manoonpong et al., 2013). It translates the CPG outputs into ascending and descending slopes, finally controlling the joint movements in basic swing and stance phases. In our robot system here, we use position control. Thus, the motor commands correspond to the target TC-, CTr-, and FTi-joint positions.

2.2. Forward Model for Sensory Prediction

The learning algorithm implemented is inspired by Xiong et al. (2016), and takes advantage of an efference copy of the step movement, allowing comparison between expected or predicted sensor feedback for each leg position and actual sensory feedback. This difference can be used to tune the α value dynamically (Equations 1, 2) to adapt to changes in the sensory feedback.

The forward model implemented (Figure 3) predicts that sensory feedback should be zero while the leg is lifted (swing phase) and a high value when the leg is on the ground (stand phase). In other words, a positive sensor value is expected when the leg touches the ground during the stance phase (downward position of the CTr-joint) and a zero value while the leg is in the air during the swing phase (upward position of the CTr-joint). This simple strategy tries to make the robot follow a stable stroke potentially giving the robot body good propulsion along the whole stance phase of the step. The forward model is given as:

$\begin{array}{ll}{F}_l^{\prime }(t+1)=\gamma ·G_l(t)+(1-\gamma )·F_l^{\prime }(t)& (6)\end{array}$

$\begin{array}{ll}{G}_l(t)=\{\begin{array}{cc}0,& m_0^l(t+1)>m_0^l(t)\\ 1,& m_0^l(t+1)<=m_0^l(t)\end{array}& (7)\end{array}$

where $F_l^{\prime }$ is the expected or predicted sensory value and γ is a factor in a range of [0, 1] defining the shape of the output signal of the forward model. In this study, we empirically adjust and set γ to 0.5. The signal $m_0^l$ is the motor command of the CTr-joint. When this motor value increases (moving the leg upward during swing phase) or decreases (moving the leg downward during stance phase), signal G_l is set to 0 or 1, respectively.

FIGURE 3

Figure 3. The forward model implemented as a linear recurrent neuron. Its output F′ is compared with the actual sensory feedback F. The difference of the comparison leads to an error e which is then used in the dual learning mechanism for sensory adaptation.

A forward model is used for each leg l. The difference between the forward model and its respective sensory input is calculated as:

$\begin{array}{ll}{e}_l(t)=|F_l^{\prime }(t)-F_l(t)|& (8)\end{array}$

Figure 4 shows the signals involved in the forward model. One can observe how the actual sensor value changes providing a better match with the predicted signal generated by the forward model. The change is due to the robot-environment interaction and sensory adaptation, described by the shadowed areas representing the stance phases of the foot contact sensory feedback.

FIGURE 4

Figure 4. Changes in different signals of the control system during robot-environment interaction and sensory adaptation to form a gait. (A) The outputs of the CPG for driving a leg (o₁, o₂). (B) The original efference copy (motor command of the CTr-joint) and the transformed one G (i.e., input of the forward model, Figure 3). (C) The predicted foot contact sensor signal F′ as the output of the forward model (Figure 3). (D) The actual foot contact sensory signal F. (E) The error signal e and sensory feedback connection α. All these signals were recorded from the simulation of the four-legged robot. Areas colored blue show the stance phases while the white areas refer to the swing phases. The red line indicates the point at which the robot starts showing a stable gait. In this case, a stable gait is quickly formed within about 6 s.

2.3. Dual Rate Learning for Sensory Adaptation

In order to reduce the error produced by the mismatch between predicted and actual sensory feedback signals, the strength of the sensory feedback on the CPG (i.e., α of Equations 1, 2) is continuously adjusted online at each leg using the dual rate learning process (Smith et al., 2006). The implemented controller combines fast and slow adaptation processes (i.e., fast and slow learners) arranged in parallel. The fast adaptation performs rapid initial learning. However, it forgets quickly while the slow adaptation contributes to long-term retention, but it adapts slowly. The use of the two parallel adaptations at different time scales leads to the fast and stable convergence of sensory feedback strength.

In this way, a proper feedback strength can be obtained after a few walking steps. Here, each adaptation process receives the same error and adapts the sensory feedback strength accordingly, as shown in the following equations:

$\begin{array}{ll}{D}_f(t+1)=A_f{D}_f(t)+B_{f}e(t)& (9)\end{array}$

$\begin{array}{ll}{D}_s(t+1)=A_s{D}_s(t)+B_{s}e(t)& (10)\end{array}$

$\begin{array}{ll}\alpha (t+1)=D_f(t+1)+D_s(t+1)& (11)\end{array}$

where D_f is the output of the fast adaptation. D_s is the output of the slow adaptation. α is the combination of the two outputs (i.e., sensory feedback strength adaptation). e is the error calculated from the difference between the predicted and actual sensory feedback (see Equation 8). B_s and B_f are the learning rates of the slow and fast adaptation mechanisms, respectively. A_s and A_f are retention factors of the slow and fast adaptations, respectively. The parameters are determined as A_s > A_f and B_s < B_f. The fast adaptation mechanism thus adapts more rapidly as indicated by a higher learning rate but also forgets more rapidly as indicated by a lower retention factor. In this study, we set A_s = 0.992, A_f = 0.57, B_s = 0.0004, B_f = 0.005. These values were used in Xiong et al. (2016). Figure 5 shows the implementation of the dual rate learning process. In this learning process, using only slow adaptation will lead to slow converge while using only fast adaptation will lead to instability (see Figure 6).

FIGURE 5

Figure 5. Dual rate learning for the sensory feedback strength adaptation of CPG-based control. The sensory information from the foot contact sensor at each leg is transmitted to its corresponding CPG-based control. It shapes the CPG outputs to obtain a proper phase between legs. The learning process continuously adapts the strength of the sensory feedback to ensure that proper sensory information is transmitted to its corresponding CPG-based control. The forward model receives an efference copy (i.e., CTr motor command) and translates it into predicted foot contact sensory feedback which is then compared to the actual foot contact sensory feedback. The difference between them is sent to the learning process for sensory feedback strength adaptation.

FIGURE 6

Figure 6. The effect of the learning process when only slow learner, only fast learner, or their combination (dual learner) is used. Using the dual learner leads to fast convergence where a stable gait (showing clear swing and stance phases) is formed at around 7 s. Using only slow learner leads to slow convergence where a walking gait is formed at around 12 s, although this gait is more irregular and unstable. Using only fast learner leads to instability where a stable gait cannot be formed. A video showing robot locomotion with different learners can be seen at: www.manoonpong.com/Frontiers2020/DifferentLearning.mp4 (or see Video S9).

3. Experimental Setup

To validate the performance of our proposed neural control system with sensory adaptation, we simulated different robots with 4, 6, 8, and 20 legs. We used the robot simulation framework LpzRobots (Martius et al., 2010) based on the Open Dynamics Engine (Smith, 2006). Each robot has the same body and leg structures (see Figure 1). Each body segment consists of one or two pairs of legs. Each leg has three joints (see Figure 1) inspired by those of invertebrates. We used a universal passive joint with limited rotation to connect between body segments. This allows for small passive body movements and dynamics for stability.

All simulated 4-, 6-, 8-, and 20-legged robots have similar leg movements with the same amplitude, allowing the observation of different walking patterns or gaits with respect to different numbers of legs. In this study, we focus only on an adaptive process for the interlimb coordination or the coordination between legs while the intralimb coordination or the coordination between joints within the leg are predefined. The leg trajectory resulting from the predefined intralimb coordination is shown in Figure 7A.

FIGURE 7

Figure 7. The simulated robots and leg trajectory of a robot in one step. (A) The trajectory consists of stance and swing phases. The stance phase follows an almost straight trajectory where the TC-joint moves backward and CTr- and FTi-joints are set to certain positions. The swing phase forms an arch with a movement combination of the three joints (i.e., TC-joint moves forward, CTr-joint moves up, and FTi-joint extends). The simple leg movement is implemented for walking on a flat surface only. Walking on rough terrain (which is not the focus of this study), requires additional control mechanisms, such as reflexes (Manoonpong et al., 2013). (B–E) The four-, six-, eight-, and 20-legged robots used in this study.

The tests for different robots were performed on a flat terrain (Figures 7B–E). The leg trajectory, all environment variables (e.g., gravity, friction), and sensory noise had the same settings for all robot experiments. We added a Gaussian-distributed noise with a standard deviation of 10%. In all experiments, we initiated the robots with an irregular gait on a flat terrain, where all legs moved in phases with a frequency of 1.4 Hz. The sensory feedback strength α was also initialized to zero, i.e., no connection between the sensory feedback and CPG-based control. As previously mentioned, in this distributed neural control approach, no CPG-based control module has any connection or direct communication with another. The gaits will emerge from the body-environment interaction, i.e., physical communication (Owaki et al., 2012) through foot contact sensory feedback and the proposed sensory adaptation in this study.

4. Results

In this section, we present the results of the experiments with simulated robots (Figure 7). The performance of the proposed neural control system was evaluated in two scenarios. The first of which accessed the self-organized locomotion of the robots using different morphologies. The second was conducted to show the adaptability of the control approach in dealing with the amputation of different legs. The leg amputation was simply performed by lifting it above the ground or keeping it fixed in a certain position. In order to statistically evaluate the control performance, each robot was tested 200 times in the first scenario and 500 times in the second. During the experiments, we measured the average walking speed once the robots had moved forward.

Figures 8, 9 show that the control approach was able to quickly adapt the sensory feedback strength and generate self-organized locomotion within 5–10 s. The gaits emerging from the 4, 6, 8, and 20-legs were similar to those observed in animals; i.e., a trot gait in the four-legged robot, a bipod gait in the six-legged robot² (Ramdya et al., 2017), and metachronal wave gaits in the 8- and 20-legged robots³ (Bowerman, 1975; Spagna and Peattie, 2012; Kano et al., 2017a; Yasui et al., 2017). The average walking speed of all robots was ≈ 0.2 m/s.

FIGURE 8

Figure 8. Results for the four- and six-legged robots. For each configuration: (A) Scheme of the tested robot. (B,C) Sensory feedback strengths during simulation on the left front leg (l₁) and right front leg (l₂), respectively. A black line indicates the average value to which each strength converges. (D) The change in the robot walking speed during the simulation. (E) The swing and stance phases of each leg of the robot, defined by motor commands. Green areas represent stance phases, while white areas correspond to swing phases. Colored marks are used to visualize the formed gaits, namely a trot gait for four legs and a bipod gait for six legs. (F) Average walking speed from 200 tests. Videos showing examples of self-organized locomotion of the four- and six-legged robots can be viewed at: www.manoonpong.com/Frontiers2020/4legs.mp4 (or see Video S1) and www.manoonpong.com/Frontiers2020/6legs.mp4 (or see Video S2), respectively. Note that all sensory feedback strengths of the four- and six-legged robots are shown in Figures S1, S2.

FIGURE 9

Figure 9. Results for the 8- and 20-legged robots. For each configuration: (A) Scheme of the tested robot. (B,C) Sensory feedback strengths during simulation on the left front leg (l₁) and right front leg (l₂), respectively. A black line indicates the average value to which each strength converges. (D) The change in the robot walking speed during the simulation. (E) The swing and stance phases for each leg of the robot, defined by motor commands. Green areas represent stance phases, while white areas correspond to swing phases. Colored marks are used to visualize the formed gaits of the 8- and 20-legged robots. In this case, both robots show metachronal-like gaits. (F) Average walking speed from 200 tests. Videos showing examples of self-organized locomotion in the 8- and 20-legged robots can be viewed at: www.manoonpong.com/Frontiers2020/8legs.mp4 (or see Video S3) and www.manoonpong.com/Frontiers2020/20legs.mp4 (or see Video S4), respectively. Note that all sensory feedback strengths of the 8- and 20-legged robots are shown in Figures S3, S4.

Figures 10, 11 demonstrate that the control approach was able to quickly adapt the sensory feedback strength to new values within a few seconds of amputation and new gaits emerged to allow the robots to locomote. The average walking speed of all robots was ≈0.125–0.175 m/s. In this case, the statistical study on controller adaptability was conducted using multiple tests with a randomized number of amputations of arbitrary legs on the robots. Since some morphologies did not show any gait when amputating too many legs, the maximum number of amputations was set to half of the total legs for each robot morphology; i.e., two for the four-legged robot, three for the six-legged robot, four for the eight-legged robot, and 10 for the 20-legged robot.

FIGURE 10

Figure 10. Results for the four- and six-legged robots when disabling certain limbs after 15 s of simulation. For each configuration: (A) Scheme of the tested robot, including the limbs amputated (red cross) in this experiment. (B,C) Sensory feedback strengths during simulation on the left front leg (l₁) and right front leg (l₂), respectively. A line indicates the average value to which each strength converges. The convergence values before and after the amputation are drawn in black and red, respectively. (D) The change in the robot walking speed during the simulation. (E) The swing and stance phases of each leg of the robot, defined by motor commands. Green areas represent stance phases, while white areas correspond to swing phases. After amputation occurs, the state of the disabled limbs is represented in gray. Colored marks are used to visualize the formed gaits. (F) Average walking speed from 500 tests. For each simulation, a random number of arbitrary limbs was disabled, investigating the ability of the system to adapt to different morphologies. Videos showing examples of adaptation to leg damage of the four- and six-legged robots can be viewed at: www.manoonpong.com/Frontiers2020/4legsDamage.mp4 (or see Video S5) and www.manoonpong.com/Frontiers2020/6legsDamage.mp4 (or see Video S6), respectively. Note that all sensory feedback strengths of the four- and six-legged robots with leg damage are shown in Figures S5, S6.

FIGURE 11

Figure 11. Results for the 8- and 20-legged robots when disabling certain limbs after 15 s of simulation. For each configuration: (A) Scheme of the tested robot, including the limbs amputated (red cross) in this experiment. (B,C) Sensory feedback strengths during simulation on the left front leg (l₁ in both 8- and 20-legged robots) and a right leg (l₂ in the eight-legged robot and l₄ in the 20-legged robot), respectively. A line indicates the average value to which each strength converges. The convergence values before and after the amputation are drawn in black and red, respectively. (D) The change in the robot walking speed during the simulation. (E) The swing and stance phases of each leg of the robot, defined by motor commands. Green areas represent stance phases, while white areas correspond to swing phases. After amputation occurs, the state of the disabled limbs is represented in gray. Colored marks are used to visualize the formed gaits. (F) Average walking speed from 500 tests. For each simulation, a random number of arbitrary limbs was disabled, investigating the ability of the system to adapt to different morphologies. Videos showing examples of adaptation to leg damage of the 8- and 20-legged robots can be viewed at: www.manoonpong.com/Frontiers2020/8legsDamage.mp4 (or see Video S7) and www.manoonpong.com/Frontiers2020/20legsDamage.mp4 (or see Video S8), respectively. Note that all sensory feedback strengths of the 8- and 20-legged robots with leg damage are shown in Figures S7, S8.

After running multiple simulations with different amputation configurations for each robot, the results show that in most cases the control approach was able to find gaits, allowing the robots to move forward. The variance of the results is higher than in the first experiment due to the large number of configurations formed by the amputations. For some configurations, the robots could manage to move effectively while others made it more difficult for the robots to move forward.

By determining the adaptation of the robots to the amputations, in Figures 10, 11, one can see how the dual rate learning mechanism adapts the feedback strength of the different legs (e.g., left and right front legs) to deal with unexpected new body conditions. It is also possible to observe how the robot gaits change to compensate for the lack of proper support resulting from the leg amputation.

The ability of the robots to find a stable gait depends on the convergence of sensory feedback strength (i.e., synaptic weight). The convergent weights during the experiments are shown in Figure 12. One can observe a range of convergence for each leg of every robot with and without amputations. In simulations where no limbs were amputated, this range is quite narrow, and the robot tends to keep a relatively constant weight. However, the amputation experiments show how the changes in the robot morphology can affect the weights needed to obtain a stable gait. By analyzing the results shown in Figure 12, Figures S5–S8, one can observe that the weights of remaining legs increase after amputation. This is due to the new weight distribution across the remaining legs. Increasing the weights leads to stronger inhibition to the CPGs (see Equations 1, 2, 4, 5), resulting in a longer stance phase. This way, the robot can readjust its gait to obtain a new stable one.

FIGURE 12

Figure 12. Average weights of the foot contact sensory feedback (i.e., sensory synaptic strength). Each li box represents the results for each i leg of the robots. Blue boxes represent the results for 200 runs on robots without any amputations. Gray boxes represent the results for 500 runs where randomized amputations were performed to obtain different configurations. In this case, the robots were amputated from the beginning of the simulation where the robots were still able to form gaits. (A) Results for the four-legged robot. (B) Results for the six-legged robot. (C) Results for the eight-legged robot. (D) Results for the 20-legged robot.

5. Discussion and Conclusion

The previous sections present a self-organized locomotion control system with the ability to quickly adapt to changes in the robot morphology. In this section, we discuss the advantages of this mechanism, comparing the proposed neural control system with other solutions and describing some of the remaining issues for improving this approach.

Previous research (Owaki et al., 2012, 2017; Barikhan et al., 2014; Aoi et al., 2017; Kano et al., 2017b, 2019; Ambe et al., 2018) has shown the possibility of offloading the task of interlimb coordination toward body-environment interactions and adaptation to physical damage. By using sensory feedback to trigger reactions in local (decentralized) CPGs and drive individual leg movements, this approach is able to form walking patterns in robots without direct control over the interlimb coordination process. The approach is supported by experimental results showing that invertebrate locomotion (e.g., insects, millipedes, centipedes) is controlled in a similar way, using decentralized CPGs for individual limbs (Bässler and Büschges, 1998; Kano et al., 2017a; Yasui et al., 2017, 2019). The biological study suggests that walking behavior in invertebrates relies highly on decentralized mechanisms and load sensing feedback.

The use of decentralized CPGs is extremely advantageous for walking robots, as it does not require the costly computation of inverse kinematics or a precise kinematic model. While it is possible to generate locomotion under fixed control parameters (e.g., sensory feedback strength), the parameters need to be empirically adjusted. The addition of learning mechanisms for adjusting control parameters online leads to flexibility and adaptability, and ultimately to more robust and general control. Compared to advanced machine learning techniques [like, the Intelligent Trial and Error algorithm (IT&E, Cully et al., 2015) and a greedy random-mutation hill climber algorithm with self-modeling (GRSM, Bongard et al., 2006)] for locomotion generation and adaptation to damage, our proposed mechanism shows advantages over the techniques in the following aspects:

• It can quickly and continuously adapt to structural changes in the robot (e.g., leg damage and different configurations) online within a few walking steps (i.e., 5–10 s) and without the need of pretrained behavior-performance map (as shown in IT&E⁴) or internal morphological models (as shown in GRSM⁵).

• It does not need a complex trial-and-error or optimization process for locomotion generation and adaptation (as needed in IT&E and GRSM); instead, it simply exploits body-environment interaction with sensory adaptation based on online dual rate learning and synaptic plasticity.

• It is computationally less expensive than IT&E and GRSM.

• It does not need additional exteroceptive feedback (like RGB-D camera as used in IT&E) or global position feedback (as used in GRSM) to measure or evaluate robot performance; instead, it only relies on foot contact or load sensing feedback.

Furthermore, due to the complex processes of the machine learning techniques, they will become much more problematic when applying to a robot with a large number of degrees of freedom, such as the 20-legged robot, which includes 65 moving parts and 60 active joints, tested in this study.

The method proposed here is based on the existing body-environment interaction approach for interlimb coordination to which we have now contributed by introducing online learning. The learning, which incorporates forward models, automatically adapts sensory synaptic plasticity (i.e., the sensory feedback strength parameter) on the CPG network. This new mechanism adds a second timescale adaptation to the system. To fully understand the dynamics of the system, it is necessary to differentiate between the two timescale adaptations: the gait formation process (fast time scale adaptation, 0–5 s in Figures 8, 9) and the tuning of synaptic strength in sensory feedback (slow time scale adaptation, after 5 s in Figures 8, 9). By considering the amputation experiments, one can see two types of sensory plasticity (Pyza, 2013): One is so-called experience-induced plasticity and the other lesion-induced plasticity. The experience-induced plasticity is driven by stimulation (Bozorgmehr et al., 2013). In our case here, it is used to form a gait as can be observed in the first period; 0–15 s in Figures 10, 11. The lesion-induced plasticity occurs after injury (Pfister et al., 2013) and, in our case here, when the robots were amputated during the second period after 15 s, as shown in Figures 10, 11. Such plasticity has been observed in invertebrate sensory systems during development and in the adult stage (Bozorgmehr et al., 2013; Lakes-Harlan, 2013; Pfister et al., 2013; Pflüger and Wolf, 2013).

In our control approach, walking patterns emerge from the reaction of each leg to sensory information. Consequently, the gait formation and sensory plasticity mechanisms need to work simultaneously. On the one hand, the robot cannot form a gait unless the sensory feedback synaptic strength is strong enough. On the other, the adjustment of sensory feedback synaptic strength depends on the formation of a gait to converge. As a result, the learning process of each leg is affected by the movement and dynamics of the robot. The process exploits the actual sensory feedback and the prediction provided by the forward model to adjust the sensory feedback synaptic strength, which is increased when the actual feedback and its expectation do not match. If the robot is not able to form a stable gait, the mismatch or error will keep increasing the synaptic strength. By increasing the strength, the sensory feedback will influence the CPG network and thereby shape the CPG outputs (see Equations 1, 2 and Figure 4). Since the process occurs in parallel in each CPG network of every leg, proper phases between the CPG networks will be finally obtained. In analogy to biological systems, the dual rate learning system can be considered as a serotonergic system or an extrinsic modulator that releases serotonin to modulate or influence sensory synaptic plasticity (i.e., sensory gain) in invertebrate nerve cord (Klein et al., 1982; van Haeften et al., 1993; Majeed et al., 2016; Le Gal et al., 2017; Upreti et al., 2019) or vertebrate spinal cord (Stutzmann et al., 1998; Deemyad et al., 2013; Lottem et al., 2016; Avery and Krichmar, 2017) while the sensory feedback projecting to the CPG network can be determined as an extrinsic modulatory input which alters the network dynamics (Katz, 1998; Morgan et al., 2000; Marder, 2012).

The sensory plasticity or adaptation provides multiple advantages compared to the previous model which has no sensory plasticity (Aoi et al., 2007; Owaki et al., 2012; Ambe et al., 2013; Barikhan et al., 2014). It provides an automatic process to tune or find proper sensory feedback contribution to obtain self-organized locomotion. It also introduces the dynamics of sensory feedback gain which leads to fast adaptation (see Figure S9); implicitly indicating an ability to adapt to changes in robot morphology and weight distribution. Furthermore, simulation results show that our approach allows robots with different morphologies to form walking gaits similar to insects. Recent work on sensory adaptation has been also proposed by Ishige et al. (2019). There, they used a combination of CPG-based control and an episode-based reinforcement learning (RL) method (i.e., policy gradients with parameter-based exploration) which was applied to caterpillar-like soft robots. The RL method was used to optimize mechanosensory feedback (sensory adaptation) to the CPG-based control, which controls actuators in the robot. This method, while impressive in their own right, still requires high computational effort (approx. 100–400 epochs⁶) to optimize the feedback to the CPG-based control for generating effective robot crawling behavior.

Taken together, this study proposes general locomotion control for multiple robot configurations and injury compensation. This control system is based on decentralized CPGs with load sensing feedback, a forward model for sensory prediction, and online learning for continuous sensory adaptation. These three components, interacting with body dynamics, can autonomously form robot gaits and compensate for leg damage without the manual tuning of sensory feedback strength required by the previous studies (mentioned above). When applied to various legged robots, different animal-like gaits can be observed, including a typical trot gait for four legs, a fly-like bipod gait for six legs (Ramdya et al., 2017), a scorpion-like metachronal wave gait for eight legs (Bowerman, 1975; Spagna and Peattie, 2012), and a millipede or centipede-like metachronal wave gait for 20 legs (Kano et al., 2017a; Yasui et al., 2017). While this approach shows effective results, it still has a limitation. So far, posture control has not been integrated into the control system. Therefore, robots with four and six legs can become unstable if their legs are amputated since parts of the body may drop on the ground. Furthermore, the robot can only deal with walking on a flat terrain. Walking on uneven or complex terrains will require additional control mechanisms, like local leg extension, elevation control (Manoonpong et al., 2013), and impedance control with online adaptation (Xiong and Manoonpong, 2018; Sun et al., 2020). Thus, in the future, we will further investigate the integration of posture control, local leg control, and muscle models into the control system to achieve self-organized locomotion with high adaptability for traversing on complex terrains.

Data Availability Statement

All datasets generated for this study are included in the article/Supplementary Material.

Author Contributions

PM provided the general direction of the project, supervised the development of the neural control system, and helped with data analysis. AM-B developed the neural control system, performed the robot experiments, and analyzed the data. The manuscript was written by AM-B and PM. Both authors contributed to the article and approved the submitted version.

Funding

This research was supported in part by the Human Frontier Science Program under grant agreement no. RGP0002/2017 (DLife), Horizon 2020 Framework Programme (FETPROACT-01-2016–FET Proactive: emerging themes and communities) under grant agreement no. 732266 (Plan4Act), and the startup grant on Bio-inspired Robotics of Vidyasirimedhi Institute of Science & Technology (VISTEC).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fncir.2020.00046/full#supplementary-material

Figure S1. Adaptation of the sensory feedback connection of each limb (α_i) of the four-legged robot during 30 s of simulation. A video showing an example of self-organized locomotion of the four-legged robot can be viewed at: www.manoonpong.com/Frontiers2020/4legs.mp4.

Figure S2. Adaptation of the sensory feedback connection of each limb (α_i) of the six-legged robot during 30 s of simulation. A video showing an example of self-organized locomotion of the six-legged robot can be viewed at: www.manoonpong.com/Frontiers2020/6legs.mp4.

Figure S3. Adaptation of the sensory feedback connection of each limb (α_i) of the eight-legged robot during 30 s of the simulation. A video showing an example of self-organized locomotion of the eight-legged robot can be viewed at: www.manoonpong.com/Frontiers2020/8legs.mp4.

Figure S4. Adaptation of the sensory feedback connection of each limb (α_i) of the twenty-legged robot during 30 s of simulation. A video showing an example of self-organized locomotion of the twenty-legged robot can be viewed at: www.manoonpong.com/Frontiers2020/20legs.mp4.

Figure S5. Adaptation of the sensory feedback connection of each limb (α_i) of the four-legged robot during 30 s of simulation, where one limb was disabled after 15 s. A video showing an example of self-organized locomotion of the four-legged robot can be viewed at: www.manoonpong.com/Frontiers2020/4legsDamage.mp4. Note that the sensory feedback connections of the disabled legs were not updated after being disabled (i.e., after 15 s). They remained the last values before being disabled.

Figure S6. Adaptation of the sensory feedback connection of each limb (α_i) of the six-legged robot during 30 s of simulation, where two limbs were disabled after 15 s. A video showing an example of self-organized locomotion of the six-legged robot can be viewed at: www.manoonpong.com/Frontiers2020/6legsDamage.mp4. Note that the sensory feedback connections of the disabled legs were not updated after being disabled (i.e., after 15 s). They remained the last values before being disabled.

Figure S7. Adaptation of the sensory feedback connection of each limb (α_i) of the eight-legged robot during 30 s of simulation, where three limbs were disabled after 15 s. A video showing an example of self-organized locomotion of the eight-legged robot can be viewed at: www.manoonpong.com/Frontiers2020/8legsDamage.mp4. Note that the sensory feedback connections of the disabled legs were not updated after being disabled (i.e., after 15 s). They remained the last values before being disabled.

Figure S8. Adaptation of the sensory feedback connection of each limb (α_i) of the 20-legged robot during 30 s of the simulation, where six limbs were disabled after 15 s. A video showing an example of self-organized locomotion of the 20-legged robot can be viewed at: www.manoonpong.com/Frontiers2020/20legsDamage.mp4. Note that the sensory feedback connections of the disabled legs were not updated after being disabled (i.e., after 15 s). They remained the last values before being disabled.

Figure S9. A comparison of gait formation between adaptive and fixed sensory feedback strength in the six-legged robot. (A,B) Using adaptive sensory feedback strength proposed in this study. (C,D) Using fixed sensory feedback strength. The value of the fixed sensory feedback strength was set to 0.02 in all limbs, which is the average value to which the adaptive sensory feedback strength converges under the intact condition. The red lines in (B,D) indicate the periods that the gaits were formed. The gaits were formed within around 5 s for the adaptive sensory feedback strength (B) and around 7 s for the fixed sensory feedback strength (D). Green areas represent stance phases, while white areas correspond to swing phases. After amputation occurs, the state of the disabled limbs is represented in gray. It can be seen that the robot with the adaptive sensory feedback strength could adapt its remaining legs to stay in the stance phase longer for stable locomotion compared to the one with the fixed sensory feedback strength.

Video S1. The video shows an example of self-organized locomotion of the four-legged robot.

Video S2. The video shows an example of self-organized locomotion of the six-legged robot.

Video S3. The video shows an example of self-organized locomotion of the eight-legged robot.

Video S4. The video shows an example of self-organized locomotion of the 20-legged robot.

Video S5. The video shows an example of adaptation to leg damage of the four-legged robot.

Video S6. The video shows an example of adaptation to leg damage of the six-legged robot.

Video S7. The video shows an example of adaptation to leg damage of the eight-legged robot.

Video S8. The video shows an example of adaptation to leg damage of the 20-legged robot.

Video S9. The video shows robot locomotion with different learners.

Footnotes

1. ^The CPG outputs represent the oscillator phases in the Tegotae-based approach (Owaki et al., 2017).

2. ^This gait is found in Drosophila. It is similar to the vertebrate running trot and faster than a typical tripod gait. In this gait, each front leg moves almost at the same time as the opposite hind leg, while the middle legs move together.

3. ^In scorpions, the gait shows metachronal alternating tetrapods with a 10% lag between legs within a tetrapod. In millipedes/centipedes, the gait exhibits propagating density waves running from the hind legs to the front legs.

4. ^IT&E starts with random controllers and then performs stochastic, population-based, optimization. This process requires 20 million iterations for creating one behavior map (containing over 13,000 high-performing behaviors or gaits), which lasted roughly 2 weeks on one multi-core computer.

5. ^The algorithm is based on an evolutionary process where new morphological models are produced by copying the originals, introducing small random modifications, and reevaluating them. This process uses the mean Euclidean distance between the centroid of each model body part, and where the centroid should be for performance evaluation. The process requires 200 iterations to obtain the best models.

6. ^One epoch consists of 20 independent episodes and one episode lasts 50 s. Thus, the process takes several hours up to a day.

References

Ambe, Y., Aoi, S., Nachstedt, T., Manoonpong, P., Wörgötter, F., and Matsuno, F. (2018). Simple analytical model reveals the functional role of embodied sensorimotor interaction in hexapod gaits. PLoS ONE 13:e192469. doi: 10.1371/journal.pone.0192469

PubMed Abstract | CrossRef Full Text | Google Scholar

Ambe, Y., Nachstedt, T., Manoonpong, P., Worgotter, F., Aoi, S., and Matsuno, F. (2013). “Stability analysis of a hexapod robot driven by distributed nonlinear oscillators with a phase modulation mechanism,” in 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (Tokyo), 5087–5092. doi: 10.1109/IROS.2013.6697092

CrossRef Full Text | Google Scholar

Aoi, S., Manoonpong, P., Ambe, Y., Matsuno, F., and Wörgötter, F. (2017). Adaptive control strategies for interlimb coordination in legged robots: a review. Front. Neurorobot. 11:39. doi: 10.3389/fnbot.2017.00039

PubMed Abstract | CrossRef Full Text | Google Scholar

Aoi, S., Sasaki, H., and Tsuchiya, K. (2007). A multilegged modular robot that meanders: investigation of turning maneuvers using its inherent dynamic characteristics. SIAM J. Appl. Dyn. Syst. 6, 348–377. doi: 10.1137/060664756

CrossRef Full Text | Google Scholar

Asif, U. (2012). Improving the navigability of a hexapod robot using a fault-tolerant adaptive gait. Int. J. Adv. Robot. Syst. 9:34. doi: 10.5772/50604

CrossRef Full Text | Google Scholar

Avery, M. C., and Krichmar, J. L. (2017). Neuromodulatory systems and their interactions: a review of models, theories, and experiments. Front. Neural Circ. 11:108. doi: 10.3389/fncir.2017.00108

PubMed Abstract | CrossRef Full Text | Google Scholar

Barikhan, S. S., Wörgötter, F., and Manoonpong, P. (2014). Multiple Decoupled CPGs With Local Sensory Feedback for Adaptive Locomotion Behaviors of Bio-inspired Walking Robots. Cham: Springer International Publishing, 65–75. doi: 10.1007/978-3-319-08864-8_7

CrossRef Full Text | Google Scholar

Bässler, U., and Büschges, A. (1998). Pattern generation for stick insect walking movements–multisensory control of a locomotor program. Brain Res. Rev. 27, 65–88. doi: 10.1016/S0165-0173(98)00006-X

PubMed Abstract | CrossRef Full Text | Google Scholar

Bässler, U., and Wegner, U. T. A. (1983). Motor output of the denervated thoracic ventral nerve cord in the stick insect carausius morosus. J. Exp. Biol. 105, 127–145.

Google Scholar

Berkowitz, A., and Laurent, G. (1996). Central generation of grooming motor patterns and interlimb coordination in locusts. J. Neurosci. 16, 8079–8091. doi: 10.1523/JNEUROSCI.16-24-08079.1996

PubMed Abstract | CrossRef Full Text | Google Scholar

Bjelonic, M., Kottege, N., and Beckerle, P. (2016). “Proprioceptive control of an over-actuated hexapod robot in unstructured terrain,” in 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (Daejeon), 2042–2049. doi: 10.1109/IROS.2016.7759321

CrossRef Full Text | Google Scholar

Bongard, J., Zykov, V., and Lipson, H. (2006). Resilient machines through continuous self-modeling. 314, 1118–1121. doi: 10.1126/science.1133687

PubMed Abstract | CrossRef Full Text | Google Scholar

Bowerman, R. F. (1975). The control of walking in the scorpion. J. Compar. Physiol. 100, 183–196. doi: 10.1007/BF00614529

CrossRef Full Text | Google Scholar

Bozorgmehr, T., Ardiel, E., McEwan, A., and Rankin, C. (2013). Mechanisms of plasticity in a caenorhabditis elegans mechanosensory circuit. Front. Physiol. 4:88. doi: 10.3389/fphys.2013.00088

PubMed Abstract | CrossRef Full Text | Google Scholar

Büschges, A., and Manira, A. E. (1998). Sensory pathways and their modulation in the control of locomotion. Curr. Opin. Neurobiol. 8, 733–739. doi: 10.1016/S0959-4388(98)80115-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Cully, A., Clune, J., Tarapore, D., and Mouret, J.-B. (2015). Robots that can adapt like animals. Nature 521:503. doi: 10.1038/nature14422

PubMed Abstract | CrossRef Full Text | Google Scholar

Dean, J. (1989). Leg coordination in the stick insect carausius morosus: effects of cutting thoracic connectives. J. Exp. Biol. 145, 103–131.

Google Scholar

Deemyad, T., Metzen, M. G., Pan, Y., and Chacron, M. J. (2013). Serotonin selectively enhances perception and sensory neural responses to stimuli generated by same-sex conspecifics. Proc. Natl. Acad. Sci. U.S.A. 110, 19609–19614. doi: 10.1073/pnas.1314008110

PubMed Abstract | CrossRef Full Text | Google Scholar

Espenschied, K. S., Quinn, R. D., Beer, R. D., and Chiel, H. J. (1996). Biologically based distributed control and local reflexes improve rough terrain locomotion in a hexapod robot. Robot. Auton. Syst. 18, 59–64. doi: 10.1016/0921-8890(96)00003-6

CrossRef Full Text | Google Scholar

Grabowska, M., Hammel, E., Schmidt, J., and Daun, S. (2012). Quadrupedal gaits in hexapod animals–stepping patterns in free-walking adult stick insects. J. Exp. Biol. 215. doi: 10.1242/jeb.073643

PubMed Abstract | CrossRef Full Text

Hwangbo, J., Lee, J., Dosovitskiy, A., Bellicoso, D., Tsounis, V., Koltun, V., et al. (2019). Learning agile and dynamic motor skills for legged robots. Sci. Robot. 4, 1–13. doi: 10.1126/scirobotics.aau5872.

CrossRef Full Text | Google Scholar

Ijspeert, A. J., Crespi, A., Ryczko, D., and Cabelguen, J.-M. (2007). From swimming to walking with a salamander robot driven by a spinal cord model. Science 315, 1416–1420. doi: 10.1126/science.1138353

PubMed Abstract | CrossRef Full Text | Google Scholar

Inagaki, S., Niwa, T., and Suzuki, T. (2010). “Follow-the-contact-point gait control of centipede-like multi-legged robot to navigate and walk on uneven terrain,” in 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems (Taipei), 5341–5346. doi: 10.1109/IROS.2010.5651324

CrossRef Full Text | Google Scholar

Ishige, M., Umedachi, T., Taniguchi, T., and Kawahara, Y. (2019). Exploring behaviors of caterpillar-like soft robots with a central pattern generator-based controller and reinforcement learning. Soft Robot. 6, 579–594. doi: 10.1089/soro.2018.0126

PubMed Abstract | CrossRef Full Text | Google Scholar

Kano, T., Kanauchi, D., Ono, T., Aonuma, H., and Ishiguro, A. (2019). Flexible coordination of flexible limbs: decentralized control scheme for inter- and intra-limb coordination in brittle stars' locomotion. Front. Neurorobot. 13:104. doi: 10.3389/fnbot.2019.00104

PubMed Abstract | CrossRef Full Text | Google Scholar

Kano, T., Sakai, K., Yasui, K., Owaki, D., and Ishiguro, A. (2017a). Decentralized control mechanism underlying interlimb coordination of millipedes. Bioinspir. Biomimet. 12:036007. doi: 10.1088/1748-3190/aa64a5

PubMed Abstract | CrossRef Full Text | Google Scholar

Kano, T., Sato, E., Ono, T., Aonuma, H., Matsuzaka, Y., and Ishiguro, A. (2017b). A brittle star-like robot capable of immediately adapting to unexpected physical damage. R. Soc. Open Sci. 4:171200. doi: 10.1098/rsos.171200

PubMed Abstract | CrossRef Full Text | Google Scholar

Katz, P. S. (1998). Comparison of extrinsic and intrinsic neuromodulation in two central pattern generator circuits in invertebrates. Exp. Physiol. 83, 281–292. doi: 10.1113/expphysiol.1998.sp004113

PubMed Abstract | CrossRef Full Text | Google Scholar

Klein, M., Camardo, J., and Kandel, E. R. (1982). Serotonin modulates a specific potassium current in the sensory neurons that show presynaptic facilitation in aplysia. Proc. Natl. Acad. Sci. U.S.A. 79, 5713–5717. doi: 10.1073/pnas.79.18.5713

PubMed Abstract | CrossRef Full Text | Google Scholar

Lakes-Harlan, R. (2013). Lesion-induced insights in the plasticity of the insect auditory system. Front. Physiol. 4:48. doi: 10.3389/fphys.2013.00048

PubMed Abstract | CrossRef Full Text | Google Scholar

Le Gal, J. P., Dubuc, R., and Smarandache-Wellmann, C. (2017). “Coordination of rhythmic movements,” in Neurobiology of Motor Control: Fundamental Concepts and New Directions, eds S. L. Hooper and A. Büschges (Hoboken, NJ: Wiley Blackwell), 305–340.

Google Scholar

Lottem, E., Lörincz, M. L., and Mainen, Z. F. (2016). Optogenetic activation of dorsal raphe serotonin neurons rapidly inhibits spontaneous but not odor-evoked activity in olfactory cortex. J. Neurosci. 36, 7–18. doi: 10.1523/JNEUROSCI.3008-15.2016

CrossRef Full Text | Google Scholar

Majeed, Z. R., Abdeljaber, E., Soveland, R., Cornwell, K., Bankemper, A., Koch, F., et al. (2016). Modulatory action by the serotonergic system: behavior and neurophysiology in drosophila melanogaster. Neural Plast. 2016:7291438. doi: 10.1155/2016/7291438

PubMed Abstract | CrossRef Full Text | Google Scholar

Manoonpong, P., Parlitz, U., and Wörgötter, F. (2013). Neural control and adaptive neural forward models for insect-like, energy-efficient, and adaptable locomotion of walking machines. Front. Neural Circ. 7:12. doi: 10.3389/fncir.2013.00012

PubMed Abstract | CrossRef Full Text | Google Scholar

Manoonpong, P., Pasemann, F., and Wörgötter, F. (2008). Sensor-driven neural control for omnidirectional locomotion and versatile reactive behaviors of walking machines. Robot. Auton. Syst. 56, 265–288. doi: 10.1016/j.robot.2007.07.004

CrossRef Full Text | Google Scholar

Marder, E. (2012). Neuromodulation of neuronal circuits: back to the future. Neuron 76, 1–11. doi: 10.1016/j.neuron.2012.09.010

PubMed Abstract | CrossRef Full Text | Google Scholar

Martius, G., Hesse, F., Güttler, F., and Der, R. (2010). Lpzrobots: A Free and Powerful Robot Simulator. Available online at: http://robot.informatik.uni-leipzig.de/software/.

Morgan, P. T., Perrins, R., Lloyd, P. E., and Weiss, K. R. (2000). Intrinsic and extrinsic modulation of a single central pattern generating circuit. J. Neurophysiol. 84, 1186–1193. doi: 10.1152/jn.2000.84.3.1186

PubMed Abstract | CrossRef Full Text | Google Scholar

Owaki, D., Goda, M., Miyazawa, S., and Ishiguro, A. (2017). A minimal model describing hexapedal interlimb coordination: the tegotae-based approach. Front. Neurorobot. 11:29. doi: 10.3389/fnbot.2017.00029

PubMed Abstract | CrossRef Full Text | Google Scholar

Owaki, D. T., Nagasawa, K., Tero, A., and Ishiguro, A. (2012). Simple robot suggests physical interlimb communication is essential for quadruped walking. J. R. Soc. Interface 10:20120669. doi: 10.1098/rsif.2012.0669

PubMed Abstract | CrossRef Full Text | Google Scholar

Pearson, K. G., and Iles, J. F. (1970). Discharge patterns of coxal levator and depressor motoneurones of the cockroach, periplaneta americana. J. Exp. Biol. 52, 139–165.

PubMed Abstract | Google Scholar

Pearson, K. G., and Iles, J. F. (1973). Nervous mechanisms underlying intersegmental co-ordination of leg movements during walking in the cockroach. J. Exp. Biol. 58, 725–744.

Google Scholar

Pfister, A., Johnson, A., Ellers, O., and Horch, H. (2013). Quantification of dendritic and axonal growth after injury to the auditory system of the adult cricket gryllus bimaculatus. Front. Physiol. 3:367. doi: 10.3389/fphys.2012.00367

PubMed Abstract | CrossRef Full Text | Google Scholar

Pflüger, H.-J., and Wolf, H. (2013). Developmental and activity-dependent plasticity of filiform hair receptors in the locust. Front. Physiol. 4:70. doi: 10.3389/fphys.2013.00070

PubMed Abstract | CrossRef Full Text | Google Scholar

Pyza, E. (2013). Plasticity in invertebrate sensory systems. Front. Physiol. 4:226. doi: 10.3389/fphys.2013.00226

PubMed Abstract | CrossRef Full Text | Google Scholar

Ramdya, P., Thandiackal, R., Cherney, R., Asselborn, T., Benton, R., Ijspeert, A. J., et al. (2017). Climbing favours the tripod gait over alternative faster insect gaits. Nat. Commun. 8:14494. doi: 10.1038/ncomms14494

PubMed Abstract | CrossRef Full Text | Google Scholar

Smith, M. A., Ghazizadeh, A., and Shadmehr, R. (2006). Interacting adaptive processes with different timescales underlie short-term motor learning. PLoS Biol. 4:e40179. doi: 10.1371/journal.pbio.0040179

PubMed Abstract | CrossRef Full Text | Google Scholar

Smith, R. L. (2006). Open Dynamics Engine Official Site. Available online at: http://www.ode.org/

Spagna, J. C., and Peattie, A. M. (2012). Terrestrial locomotion in arachnids. J. Insect Physiol. 58, 599–606. doi: 10.1016/j.jinsphys.2012.01.019

PubMed Abstract | CrossRef Full Text | Google Scholar

Stutzmann, G. E., McEwen, B. S., and LeDoux, J. E. (1998). Serotonin modulation of sensory inputs to the lateral amygdala: dependency on corticosterone. J. Neurosci. 18, 9529–9538. doi: 10.1523/JNEUROSCI.18-22-09529.1998

PubMed Abstract | CrossRef Full Text | Google Scholar

Sun, T., Xiong, X., Dai, Z., and Manoonpong, P. (2020). Small-sized reconfigurable quadruped robot with multiple sensory feedback for studying adaptive and versatile behaviors. Front. Neurorobot. 14:14. doi: 10.3389/fnbot.2020.00014

PubMed Abstract | CrossRef Full Text | Google Scholar

Upreti, C., Konstantinov, E., Kassabov, S. R., Bailey, C. H., and Kandel, E. R. (2019). Serotonin induces structural plasticity of both extrinsic modulating and intrinsic mediating circuits in vitro in aplysia californica. Cell Rep. 28, 2955–2965. doi: 10.1016/j.celrep.2019.08.016

PubMed Abstract | CrossRef Full Text | Google Scholar

van Haeften, T., Smid, H. M., and Schooneveld, H. (1993). Serotoninergic innervation of the alimentary canal of the Colorado potato beetle, Leptinotarsa decemlineata: structural and functional aspects. Cell Tissue Res. 273, 475–485. doi: 10.1007/BF00333702

CrossRef Full Text | Google Scholar

Wark, B., Lundstrom, B. N., and Fairhall, A. (2007). Sensory adaptation. Curr. Opin. Neurobiol. 17, 423–429. doi: 10.1016/j.conb.2007.07.001

PubMed Abstract | CrossRef Full Text | Google Scholar

Whelan, P. J., and Pearson, K. G. (1997). Plasticity in reflex pathways controlling stepping in the cat. J. Neurophysiol. 78, 1643–1650. doi: 10.1152/jn.1997.78.3.1643

PubMed Abstract | CrossRef Full Text | Google Scholar

Wolf, H., and Büschges, A. (1997). Plasticity of synaptic connections in sensory-motor pathways of the adult locust flight system. J. Neurophysiol. 78, 1276–1284. doi: 10.1152/jn.1997.78.3.1276

PubMed Abstract | CrossRef Full Text | Google Scholar

Xiong, X., and Manoonpong, P. (2018). “Adaptive motor control for human-like spatial-temporal adaptation,” in 2018 IEEE International Conference on Robotics and Biomimetics (ROBIO) (Kuala Lumpur), 2107–2112. doi: 10.1109/ROBIO.2018.8665222

CrossRef Full Text | Google Scholar

Xiong, X., Wörgötter, F., and Manoonpong, P. (2016). Adaptive and energy efficient walking in a hexapod robot under neuromechanical control and sensorimotor learning. IEEE Trans. Cybern. 46, 2521–2534. doi: 10.1109/TCYB.2015.2479237

PubMed Abstract | CrossRef Full Text | Google Scholar

Yasui, K., Kano, T., Standen, E. M., Aonuma, H., Ijspeert, A. J., and Ishiguro, A. (2019). Decoding the essential interplay between central and peripheral control in adaptive locomotion of amphibious centipedes. Sci. Rep. 9:18288. doi: 10.1038/s41598-019-53258-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Yasui, K., Sakai, K., Kano, T., Owaki, D., and Ishiguro, A. (2017). Decentralized control scheme for myriapod robot inspired by adaptive and resilient centipede locomotion. PLoS ONE 12:e171421. doi: 10.1371/journal.pone.0171421

PubMed Abstract | CrossRef Full Text | Google Scholar