Biologically inspired modelling for the control of upper limb movements: from concept studies to future applications

One of the most active and productive research fields in Neuroscience investigates the basic physiological mechanisms of the nervous system when dealing with the control of upper-limb movements. The main obstacle to a deeper knowledge of such a system is due to the difficulty to understand how the control system is able to interact with the anatomy and the physiology in order to build and perfect the desired motor function reflecting the required goal. To solve this point means firstly to discover the mechanisms dealing with the design, construction and maintenance of the neuronal networks. Then it means to understand how the networks succeed in solving problems of increasing complexity, even if they are composed by units whose single behaviour is too simple to explain the acquisition of any competency. To understand how new competencies are learned and adapted to different environments, the observation of the each single units is useless, while the study of the whole network can be much more effective.

Models offer a scientific approach to the comprehension of the nervous system: in particular, motor control mechanisms, which integrate several neuronal structures (i.e. perceptive, associative, cognitive and motor), have often been studied by looking at the behaviour of possible artificial replicae, such as computational models.

It is difficult, if not impossible, to represent such a complexity by using a conventional algorithmic approach where the data flow has to be clearly defined in any single instruction. Instead, soft computing offers a viable solution by moving the rationale of the modelling from the mathematic description of a behaviour to the understanding of the fundamental mechanisms behind it. Moreover, since soft criteria (Perl, 2004 ) accept statistically varying results when lying in a range of acceptable solutions, they simulate biological mechanisms in a way that is inspired to real life. The rationale of the biologically inspired modelling was used in several works of the motor control literature since the pioneering work by Ito (1970) . In motor control theory, a biologically inspired model is an artificial system able to mimic the functional properties of a real biological system with respect to the following four criteria: (1) architecture, (2) learning behaviour (acquisition of motor skills), (3) functioning (in terms of control and execution of movements), (4) adaptability to different environments (force fields, presence of obstacles, etc.). Artificial Neural Networks (ANN) can be considered as the “most biologically inspired” among soft computing approaches, because of their ability to mimic the behaviour of biological neuronal networks with respect to the architecture, the mechanisms of competencies acquisition, and the real-time processing of the information. Furthermore, a deeper insight in the different neural typologies can lead to a differentiation between the classic architectures (i.e. Hopfield, Multi-Layer Perceptron) focusing on activity levels only, and the ones resembling neurophysiologic functioning, including time properties (e.g. Spiking Networks). For simplicity reasons in this work the term ANN will comprehend both the cited structure types.

In this perspective article, the focus will be on a subset of those scientific contributions that proposed biologically inspired models as artificial paradigms for the control of goal-directed upper-limb movements, and tested the feasibility of biologically inspired models in a rehabilitative context. All the works dealing with upper-limb motor control (in vivo, in silico, and for robotic arms) where ANN are not used in a biologically inspired way will be left aside. The review of these latter works, though interesting, is beyond the scope of this contribution. Special attention will be instead devoted to those examples where the application tackles the motor impairment as an effect of neuromuscular disorders. Then the authors will concentrate on future applications of the biologically inspired modelling as envisaged by some proposals ranging from robotics (Miyamoto et al., 1987 ; Glasius et al., 1996 ), to rehabilitation (Lan et al., 1994 ; Goffredo et al., 2008 ).

Modelling the human motor control mechanisms aims at a clear understanding of the functionality of the motor system, from the movement implementation to the learning process. Thus the development of the motor control skills derives from the acquisition of several competencies, including the cognitive ones. It is agreed that accuracy, robustness to changing environment, and adaptability to altered conditions in living organisms result from the intrinsic capacity of a biological system to learn from experience, which, in the motor control framework, is its ability to develop a sensorimotor organization model. This is a direct consequence of the neural plasticity, which, starting from birth, allows changes at the neuronal and network level, thus providing refinement of the model during the entire life cycle. Basically, the scientific modelling of these mechanisms can be divided into two main parts. The first one focuses on the modelling of the kinematics and dynamics of the end-effectors, and directs the attention to an optimal simulation of the motor behaviour. The second one deals with the modelling of higher-level processes and takes into account the relation between the learning and the organization schemes of the movements (i.e. sensorimotor adaptation).

Several control architectures and sophisticated algorithms have been built up and largely adapted to motor control theories in order to aim at a deeper understanding of this process. Some problems are still under investigation, such as the integration of all the different brain areas that contribute to the computational process of motor skills learning (Willingham, 1998 ), and the interaction between the movement and the environment.

In the following, the authors will restrict the field of interest to the modelling of the sensorimotor adaptation as the closest one to the biologically inspired philosophy. In this context, the design of the mechanisms for motor control needs to account for adaptivity, since the neuromuscular strategies depend on the environmental conditions as well as on the mechanical properties of the limbs (mass, geometry, viscosity of the muscles, etc.). This phenomenon was studied experimentally by modifying the environment in several controlled ways, through visual (Abeele and Bock, 2001 ; Miall et al., 2004 ; Caselli et al., 2006 ) or mechanical distortions (Krakauer et al., 1999 ; Scheidt et al., 2000 ), stable (Conditt and Mussa-Ivaldi, 1997 ) and unstable (Osu et al., 2003 ) interactions produced by haptic interfaces. The evidence, provided by experimental data, that the Central Nervous System is able to account for dynamic compensations, led to the hypothesis of the existence of the so-called Internal Models (IM), which implement the neural mechanisms connecting input and output signals of the sensorimotor system in a feedforward fashion (Kawato, 1999 ). These architectures are considered as the keystone upon which the motor control takes shape and dynamically evolves. They intrinsically contain information about the mechanical properties of the human body in relation both to the environment and to the subject’s experience. The cerebellum is considered as the hypothetical site where the IM are gradually built and instantiated through practice and experience (Shadmehr and Mussa-Ivaldi, 1994 ). In order to build an efficient IM, two different functions need to be executed by the cerebellum, and they can respectively be associated with the direct internal models (DIM) and the inverse internal models (IIM) (Wolpert et al., 1998 ). The DIM are able to forecast the sensorial consequences of an action, assessing the limb future state (i.e. position and velocity) from the knowledge of the actual state and the motor commands. On the contrary, the IIM can produce the motor commands, which modify the state in a controlled way. The differentiation concerns the discussion about: (i) the existence of an anticipative control of the movements, and (ii) the role of a central planning actuated before the trigger of the task connected to corrective processes based on the feedback system. It must be taken into account that the main strategies of motor control are the open-loop (feedforward) and the closed-loop (feedback) control: the former is related to all those systems based on the control of an end-effector without having any information about the state during the control (muscle length, end-effector position), but on a predetermined sequence of actuator commands; the latter concerns the theories of control in which the information of the state available at any time step modifies the outgoing of the motor commands in order to correct errors.

The presence of feedforward mechanisms is fundamental since sensorimotor control needs a significant and highly variable amount of time (150–250 ms) to elaborate a motor reaction to a sensory feedback stimulus (Wolpert et al., 1998 ). The way these feedback and feedforward components of motor control interact was discussed in several studies, and among the possible implementations the ones based on ANN reveal good potentialities (Miall and Wolpert, 1996 ; Kawato, 1999 ). This will be object of the following paragraph.

Among the computational models for motor control, those based on a neural approach have often been indicated as biologically inspired. The reason derives from the features of the ANN, which are able to mimic the human neuronal structures in both architecture and behaviour. Indeed, a neural network is composed by a number of units linked through weighted connections, just as the human neuronal structures. Connections and weights linking single units characterize the behaviour of an ANN and affect the contribution given from each unit to the propagation of the action potential through the whole network: this process is tuned through learning. Dealing with this context, in this paragraph the authors will review some representative works concerning biologically inspired neural models for the motor control on the basis of the four criteria proposed in the introduction. Thus, a classification of these contributions will be done focusing on which work can be considered as biologically inspired in terms of:

1. architecture (Glasius et al., 1996 ; Billard and Mataric, 2001 )

2. learning behaviour (Kawato et al., 1988 ; Izawa et al., 2003 )

3. functioning (Fagg et al., 1997 ; Karniel and Inbar 1997 ; Harris and Wolpert, 1998 ; Tanaka et al., 2004 )

4. adaptivity (Stroeve, 1998 ; Meulenbroek et al., 2001 ; Todorov and Jordan, 2002 ; Bernabucci et al., 2007b ).

An interesting example of biologically inspired architecture was proposed by Billard and Mataric (2001) with a model for learning human arm movements by imitation. The model consists of a hierarchy of ANN, and is composed by modules simulating specific brain regions (i.e. temporal cortex, spinal cord, primary motor cortex, premotor area, cerebellum) involved in visuo-motor learning and control. The learning is addressed to the modules of premotor area and cerebellum, the former simulating an abstract representation of mirror neurons (Rizzolatti et al., 1996 ) and the latter allowing to master complex tasks by learning sequences of primitive movements. Both modules are implemented as Dynamical Recurrent Associative Memories (i.e. fully recurrent ANN without hidden units) and a Hebbian strategy is used for learning. The movements executed by an 11 Degrees Of Freedom (DOF) human model driven by this controller are similar to the real ones. Sensory information for movement planning was studied also in Glasius et al. (1996) , where the issues of trajectory formation and obstacle avoidance are solved using a two-layer biologically inspired ANN, trained in an unsupervised way. The layers respectively represent the sensory map, which builds up the activity patterns from sensory information, and the time-evolving motor map, transforming activity patterns into motor commands.

Among the studies using ANN in motor control, those trying to implement IM are probably the ones in which learning behaviour better resembles that of human motor control. Basically three main learning paradigms for ANN can be defined: (1) supervised learning: an “external teacher” provides the net with the input-output pairs. The error is used to adapt the synaptic weights; (2) unsupervised learning: the significant features from the input population are self-recognised; (3) reinforcement learning (Sutton and Barto, 1998 ): it works to maximise a reward or a reinforcement parameter, which influences weights modification (increasing the probability of future rewards) and establishes the end of the learning phase. This paradigm is useful when a task can be decomposed into different sub-tasks, whose sequence can influence the overall reward value. In Kawato et al. (1988) , a computational model for learning and controlling voluntary movements based on Hierarchical ANN was proposed. This scheme used ANN to implement both a DIM and an IIM: the first provides a feedback signal for the training of the second. The IIM is trained by using the feedback error learning algorithm, to provide the proper motor commands when fed by the trajectory of the desired movement in the joints space. A similar approach is that proposed in Izawa et al. (2003) , where an ANN representing the IIM of the upper limb is used as a state predictor in order to stabilize a Reinforcement Learning algorithm for an Actor/Critic ANN controller based on Temporal Difference Learning (Sutton and Barto, 1998 ).

Another fundamental purpose in the modelling of motor control systems for the upper limb is the imitation of human movement (thus intending the biologically inspired functioning). Due to the high redundancy of arm movements, most models imitate sub-sets of motor tasks, generally planar and goal-directed. The works by Fagg et al. (1997) , Karniel and Inbar (1997) , Harris and Wolpert (1998) , Tanaka et al. (2004) , Bernabucci et al. (2007a) , try to mimic the bell-shaped velocity profile of the human movement. Fagg (1997) presented a heuristic computational model based on ANN that learns motor programs such as infant’s reaching strategy, that is bringing an arm to the target and correcting reaching movements. This “trial and error” architecture is able to produce, after some attempts, biologically plausible movements, characterised by bell-shaped tangential velocity profiles. A work illustrating a comprehensive neural-based model of the human arm was presented in Karniel and Inbar (1997) . The proposed model consists of a 2 DOF manipulator driven by three muscle pairs for the direct dynamic modelling, while the control architecture consists of an ANN generating neural outputs and a pulse generator that transforms the neural outputs into representative motor commands. This model is able to produce movements with features consistent with physiology. The working area is restricted to a limited region of the entire workspace and the learning algorithm is not properly biologically inspired. Bernabucci (2007a) proposed a multi-layer Perceptron ANN for the representation of an IIM for ballistic movements of an upper limb in a given environment. Movements generated by this controller reproduce the bell-shaped hand velocity profiles, thus giving results consistent with the movement generation of a human arm.

If the functioning is a well solved problem, on the other hand, the adaptivity of the models to different environment situations is a challenging task. In Stroeve (1998) a multi-layer Perceptron is trained to simulate an IM: a desired task is executed by motor commands which adaptively depend on the environment. The control is performed by generating signals which result very similar to those extracted from real myoelectric data. The adaptability of this model was further assessed by comparing real data with the responses of the model (in terms of arm impedance) to external disturbances: Todorov and Jordan (2002) proposed a fundamental theoretical framework based on stochastic optimal feedback control able to explain adaptivity characteristics of human motor planning as a link between goal achievement and motor details variability. In Bernabucci et al. (2007b) , a controller of an upper human arm model is being exposed to different conditions simulating altered force environment, showing the adaptation ability to environmental modifications such as the insertion of different force fields acting on the end-effector. As a final remark, it is to be outlined that sudden changes in the environmental conditions cannot be managed by IIM and feedforward control schemes, whereas a mixed use of forward and inverse models of the dynamics could solve such a problem.

In a few cases, the computational models for motor control moved from the theoretical framework into the realm of application fields. Most studies aimed indeed at proposing a novel model generally validated by comparing real movements with the artificial ones executed by a virtual arm. Among the few works successfully applied to real contexts, the control scheme proposed in Kawato et al. (1988) , was used to control an industrial manipulator (Miyamoto et al., 1987 ) and the study by Glasius et al. (1996) was tested to drive both a mechanical manipulator and an artificial arm.

A promising perspective for practical applications of computational models is given by the development of neural prostheses for rehabilitation embedding Functional Electrical Stimulation (FES) technology. Some preliminary attempts to use computational models as FES controllers have been proposed. For instance, (Lan et al., 1994 ) implement an open-loop controller for a FES muscle stimulator by using a recurrent feedback-feedforward ANN. The net is fed with kinematic information (i.e. desired trends of trajectory and angular velocity) and gives as output both the muscular stiffness and the motor patterns to tune the FES stimulation parameters. The stimulated movements are executed with limited error on the final position of the end-effector.

Another interesting work is that presented by Sepulveda et al. (1998) where neural networks were used to command a FES system for gait restoration in spinal-chord injured patients.

Some of the authors of this paper presented in Goffredo et al. (2008) the rationale for a novel FES-assisted system devoted to the upper-limb rehabilitation and controlled by a biologically inspired ANN. The system was envisioned as a means to rehabilitate post-stroke patients revealing hemiplegia: the movement of the paretic upper limb is assisted by an “optimal” stimulation provided by the controller previously trained to associate kinematics to muscular activation patterns. The training is done on a set of movements (i.e. the kinematics) executed either by the therapist or by the patient’s sound arm in virtual/augmented reality.

In the authors’ opinion, the merging of biologically inspired modelling and assistive technology could improve the quality of life of people suffering from neuromotor disorders. In the projects funded by the European Commission in the FP7 into different research areas, biologically inspired models are accounted for practical applications (i.e. chemical sensors, rehabilitation, cognitive systems). Among those projects the one called TREMOR (FP7 action ICT-2007.7.2, grant number ICT-2007-224051) proposes a wearable device based on FES technology to assist voluntary movements of the upper limb through tremor minimization (Manto et al., 2009 ). The project, proposing an ANN model to drive the controller of the FES system, will indirectly test how biologically inspired models can renovate applications already proposed. In fact, FES for tremor reduction proposed in Prochazka et al. (1992) and Zhang and Ang (2006) opens the new issue of how to control the FES delivery.

The system proposed in the TREMOR project is composed by several parallel ANN (one for each DOF of elbow and wrist joints) implementing a mapping function between the kinematics of spatial joint configuration (and/or inertial joint values) and the muscular activation patterns devoted to provide the best parameters for stimulation to a FES controller. A multimodal BCI system (intended as a BCI system integrating information gained from EEG signal with that obtained from other signals such as surface EMG and IMUs in a sensor fusion approach) is used to gain information about voluntary movements (EEG signal) and tremor characteristics (that is onset frequency and amplitude from EMG and inertial measurement units). This information might be used to reduce tremor without affecting the execution of voluntary movements. The mapping function is the solution proposed to solve the inverse dynamics problem of the upper arm in an adaptive, subject-specific and real-time way. The training of the ANN is performed by exploiting an advanced biomechanical 3D model (whose anthropometric properties can be tuned according to patient’s data) able to provide kinematics and/or inertial characteristics related to the motor task. The data sets for the learning phase are obtained by associating these values to the muscular activation patterns that generate them. The successful concept study already carried on in the TREMOR project opens new interesting perspectives.

Computational models for motor control, which are generally developed within theoretical frameworks, could be transferred into applications such as those dealing with rehabilitative devices. Even if the literature has not yet offered computational models for motor control that can be considered as biologically inspired as a whole (that is in terms of architecture, learning, functioning and adaptivity), some partial attempts have been proposed in the motor rehabilitation framework.

The TREMOR project has been chosen as a paradigm for the transfer of theoretical studies to applications in the framework of assistive devices and rehabilitation.

This example opens an interesting scenario on the advantages the biologically inspired models can provide to the implementation of technologies for motor planning assistance.

Assessing the feasibility of this approach under real life conditions is thus a necessary yet fundamental step towards the diffusion of biologically inspired models as effective artificial means to control or help control upper limb movements.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Research work carried out with the financial support of the European Union Commission, within FP7 action ICT-2007.7.2 “Accessible and Inclusive ICT”, under grant N. 224051 TREMOR “An ambulatory BCI-driven tremor suppression system based on functional electrical stimulation”.