Alessandro Del Vecchio
Claudio Castellini
Philipp Beckerle- 1Department of Artificial Intelligence in Biomedical Engineering, Faculty of Engineering, Friedrich-Alexander Universität Erlangen-Nürnberg, Erlangen, Germany
- 2Institute of Robotics and Mechatronics, DLR German Aerospace Center, Weßling, Germany
- 3Chair of Autonomous Systems and Mechatronics, Department of Electrical Engineering and Department of Artificial Intelligence in Biomedical Engineering, Faculty of Engineering, Friedrich-Alexander Universität Erlangen-Nürnberg, Erlangen, Germany
- 4Institute for Mechatronic Systems, Mechanical Engineering, Technical University of Darmstadt, Darmstadt, Germany
1. Introduction
The day seems not too far away, in which robots will be an active part of our daily life, just like electric appliances already are. Hence, there is an increasing need for paradigms, tools, and techniques to design proper human-robot interaction in a human-centered fashion (Beckerle et al., 2017). To this end, appropriate Human-Machine Interfaces (HMIs) are required, and there is a growing body of research showing how the Peripheral Nervous System (PNS) might be the ideal channel through which this interaction could proficiently happen.
During daily motor tasks such as grasping, walking, or speaking, the central nervous system (CNS) recruits a number of α-motoneurons in the ventral horn of the spinal cord and modulates the rate at which they discharge action potentials. The α-motoneurons are further modulated by supraspinal, afferent volleys and intrinsic motoneuron properties (Heckman et al., 2005; Enoka, 2008). The motoneuronal axonal action potentials are transformed into forces by a group of muscle fibers (the muscle unit) innervated by one axon. The muscle unit and the motoneuron form the final ensemble of all motor actions, the so-called motor unit. Translating neural commands into muscular forces, (spinal) motor units represent a promising interface to the CNS. However, there are some physiological constraints of motor control that must be taken into account by robotic applications.
In this opinion paper, we claim that better user experience would lead to more intuitive control and tighter human-robot interaction or even human-machine integration and vice-versa (see Figure 1).
Figure 1. Peripheral-Nervous-System–Machine Interfaces (PNS-MIs) in human-robot interaction: PNS data facilitates simultaneous intent recognition and online experience evaluation. This fosters novel sensorimotor interaction paradigms, sheds light on human behavior and reactions, and thereby opens up new directions for human-robot interaction.
Using PNS data for intent detection as well as for online assessment of user experience renders such interfaces technically promising and a tool to understand human behaviors and reactions (Beckerle et al., 2019). To improve on this, we discuss developments in intent detection and user feedback and user feedback emphasizing on anthropomorphic systems, which are directly controlled by humans, e.g., prostheses and teleoperation, and aiming to create novel sensorimotor paradigms.
2. Peripheral-Nervous-System–Machine Interfaces (PNS-MIs)
Interfaces for controlling anthropomorphic robotic systems, e.g., HMIs for self-powered prostheses, cannot function like a joystick or a touch-screen for instance, since the user cannot physically operate such devices. These HMIs must rather resort to interpreting the user's intent based on signals the user is able to produce - usually, relevant biological signals related to the intended muscle activation (Beckerle et al., 2019). Surface electromyography (Merletti et al., 2011) is a primary example, although different kinds of signals are currently being explored, e.g., tactile information (Beckerle et al., 2018, 2019) and also promising for other applications such as anthropomorphic teleoperation (Nostadt et al., 2020) or teaching collaborative robots (Cansev et al., 2021).
In principle, all signals generated by the users can and should be used to interpret their intent, but clearly, the optimal choice of signals and sensors depends on a tradeoff involving several factors. This includes, e.g., how well the sensors can be worn (ergonomy), how expensive the processing is (both economically and computationally), and how invasive their setup is. Furthermore, intent detection does not necessarily coincide with classification of signal patterns; rather, it's the ability to provide the user a seamless control experience based upon such signals, e.g., using regression instead of classification.
2.1. The Pros and Cons of CNS/PNS-MIs
Broadly speaking, signals related to movement and muscle activation can be classified according to whether they are recorded from the CNS or the PNS (Castellini et al., 2014). HMIs relying on CNS signals include brain-machine interfaces using surface electroencephalography as well as electrocorticography with direct implants on the motor cortex and spinal implants (Micera et al., 2010), or by decoding spinal motoneurons from high-density EMG signals (Farina et al., 2017; Del Vecchio and Farina, 2019). PNS-based HMIs, on the contrary, are those using signals recorded from the limbs, either invasively or non-invasively, e.g., implanted electromyography and direct connections to peripheral nerves vs. surface electromyography, force- and magneto-myography (Fang et al., 2015). Although EMG interfaces are placed in the periphery (muscles) the signal carried by the electrical activity generated by muscle fibers is in a one-to-one relation with spinal motoneurons. Moreover, minimally invasive approaches like local tomography of the limbs, which entail no surgery but indeed the injection of energy into the body, exist (Sierra González and Castellini, 2013; Gibas et al., 2019).
Given the extreme density of neural cells found in the CNS, most signals useful for robotic control to be potentially found in it are physically unavailable for direct inspection, unless one resorts to very invasive methods, e.g., the Braingate (Hochberg et al., 2012). Non-invasive methods, on the other hand, strongly suffer from cross-talk: the main problem is to tell signals pertaining to the intent under examination from “all the rest.” Surface electroencephalography, for instance, poses extremely complex problems to interpret and discern the neural firing patterns of interest, since each sensor can only record potentials from a large pool of neurons. Accordingly, damping and distorting effects due to skull bone tissue complicate pattern recognition (Lazarou et al., 2018). As opposed to that, an excellent signal-to-noise ratio can be obtained at the price of getting in contact with the cerebral cortex or the spinal cells (Hochberg et al., 2012).
PNS-based systems, on the other hand, can use better separated and physiologically relevant signals, naturally enforced by the anatomical branching of nerves and neurons as they depart from the brain, brainstem, and spinal cord. If one is interested in detecting the intent to move, exert forces and torques, and/or activate one's own muscles, then detecting such activity from the PNS appears to be a better choice especially if non-invasiveness is desired (Castellini et al., 2014). On top of this, if minimal invasiveness is permitted or desired, PNS approaches are probably even the best choice nowadays. Ultrasound scanning and electromyographic sensors implanted during osseointegration (Ortiz-Catalan et al., 2014) or injected into the muscles (Becerra-Fajardo and Ivorra, 2019) offer high signal-to-noise ratios while entailing rather low risk.
2.2. Improving on PNS-MIs
It has been known to physiologists for the last three decades that the neural activation that is transmitted by the motoneuron is delayed by the muscle tissue over a large range of values, from roughly 50 to more than 200 ms (Partridge, 1965; Baldissera et al., 1998). During fast motor tasks the nervous system compensates this delay by increasing the motoneuron firing frequency and the delay between the recruitment of successive motor units. Therefore, the CNS tunes this delay dynamically. Previous experiments in animal preparations demonstrated that changes in stimulation frequency alters the delay between the myoelectrical signal and the force produced by the muscle tissues in a very large range (Partridge, 1965; Baldissera et al., 1998). Recently, by decoding the activity of a large population of motoneurons during contraction at different speeds, we also found that the human nervous system modulates such delays in a very broad range (50–250 ms for hand and leg muscles Del Vecchio et al., 2018).
In virtually all prosthetic applications, however, this delay is fixed (Farina et al., 2014) yielding devices that do not follow the physiological modulation during natural processes like muscle fatigue (Zhou et al., 1998), adaptation of contraction speed (Del Vecchio et al., 2018), and muscular force output (Del Vecchio et al., 2018). Still, neuroergonomics should indeed translate these basic physiological findings into novel interface designs and, potentially, prosthetic applications for improving human robot-interactions. One potential solution to overcome this limitation is to decode surface EMG in real-time. We have previously shown that it is possible to retrieve the motoneuron discharge timings with delays smaller than 2 ms (Glaser et al., 2013; Barsakcioglu and Farina, 2018; Chen et al., 2020; Ting et al., 2021). Moreover, the potential to identify individual motor unit discharge times allows to label each motor unit to its unique motor space, e.g., encoding flexion/extension or which digit. Therefore, classification of EMG activity can be performed in a highly accurate way by associating each motor unit to its specific spatiotemporal space, as shown in a spinal cord injury case (Ting et al., 2021).
2.3. Considering User Experience Through PNS-MIs
Recent research outlines that PNS-MIs also have potential in directly assessing user experience going beyond established psychometric and physiological methods. An interesting example is the embodiment of robotic systems such as prostheses or teleoperation systems (Beckerle et al., 2019; Nostadt et al., 2020): the embodiment of artificial limbs can be assessed through surveying subjective experience with questionnaires (Longo et al., 2008), objective behavioral measures (e.g., proprioceptive drift), or (neuro)physiology (Christ and Reiner, 2014). This effect was also shown for artificial limbs with myoelectric control (Romano et al., 2015; Sato et al., 2018), but we might ask ourselves whether myoelectric measurements could also be used to analyze neuroergonomics of interaction with such devices. Recent work by Preatoni et al. for instance (Preatoni et al., 2021) indicates that proper sensory feedback makes a leg prosthesis feel lighter.
For patients suffering from stroke, the experience of device embodiment seems to have similar influence on electromyographic activity as for other physiological measures, i.e., electrodermal activity and skin temperature (Llorens et al., 2017). While, Tsuji et al. (2013) even report subjective survey results to be better represented by electromyography than by electrodermal activity, (Llorens et al., 2017) state that interactions between their subjective and neurophysiological results were inconclusive. Besides embodiment, the perception of pleasantness of affective touch can be related to electromyographic as well as to electrodermal measurements (Ree et al., 2019). This is very interesting since providing affective information through touch was shown to increase the embodiment of artificial limbs (Crucianelli et al., 2013, 2018; van Stralen et al., 2014) and, hence, appears worth considering in human-robot interaction (Beckerle et al., 2018).
Although myographic activity was measured at different sites, i.e., hand and face (Tsuji et al., 2013; Llorens et al., 2017; Ree et al., 2019), considering it in the assessment of user experience seems promising. We have ourselves recently put forward the potential connection between control based upon muscle activation and action schemes in the sense developed by Piaget (Piaget, 1950). Here, a proper PNS-MI could foster the creation of novel circular reactions, leading to embodiment as a natural consequence (Bettoni and Castellini, 2021). The factors influencing the effect remain unexplored so far. Understanding and shaping these interactions might be supported by multimodal data from an interface integrating myography with other physiological data, e.g., electrodermal activity or heart rate.
3. Discussion
With this position paper, we advocate peripheral neuroergonomics as an elegant way to improve HRI. Non-invasively interfacing the peripheral nervous system seems to provide very good interpretability and is currently advantageous over CNS-based interfaces, which outline higher invasiveness as well. Moreover, peripheral interfaces can augment or complement other modalities such as eye-tracking and electroencephalography to improve the recognition of user intentd and cognitive status. Generally, we expect considering neuromechanical insights in novel interfaces designs to foster improved HRI characteristics of robotic systems and devices. An accurate closed-loop control of the neuromechanical delays matching the physiological pathways would likely improve sensorimotor interactions. In addition, peripheral neural information can complement psychometric and physiological methods to assess user experience, which indicates that integrating myographic assessment in multimodal PNS-MIs would bring the neuroergonomics of human-robot interaction to a new level of quality.
Author Contributions
PB coordinated its development as well as the integration of individual contributions. All authors conceptualized the structure, contributed content, perspectives, and references as well as discussed and revised the manuscript.
Funding
This work received support from the German Research Foundation (DFG) through the projects Active transfer learning with neural networks through human-robot interactions (TRAIN) (no. BE 5729/16) and Deep-Hand: deep learning + deep sensing for myocontrol of the upper limb (no. CA1389/1-2).
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher's Note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
Baldissera, F., Cavallari, P., and Cerri, G. (1998). Motoneuronal pre-compensation for the low-pass filter characteristics of muscle. A quantitative appraisal in cat muscle units. J. Physiol. 511, 611–627. doi: 10.1111/j.1469-7793.1998.611bh.x
Barsakcioglu, D. Y., and Farina, D. (2018). “A real-time surface EMG decomposition system for non-invasive human-machine interfaces,” in 2018 IEEE Biomedical Circuits and Systems Conference (BioCAS) (Cleveland, OH: IEEE), 1–4. doi: 10.1109/BIOCAS.2018.8584659
Becerra-Fajardo, L., and Ivorra, A. (2019). “First steps towards an implantable electromyography (EMG) sensor powered and controlled by galvanic coupling,” in IFMBE Proceedings, World Congress on Medical Physics and Biomedical Engineering (Prague). doi: 10.1007/978-981-10-9023-3_4
Beckerle, P., Castellini, C., and Lenggenhager, B. (2019). Robotic interfaces for cognitive psychology and embodiment research: a research roadmap. Wiley Interdiscipl. Rev. Cogn. Sci. 10:e1486. doi: 10.1002/wcs.1486
Beckerle, P., Kõiva, R., Kirchner, E. A., Bekrater-Bodmann, R., Dosen, S., Christ, O., et al. (2018). Feel-good robotics: requirements on touch for embodiment in assistive robotics. Front. Neurorobot. 12:84. doi: 10.3389/fnbot.2018.00084
Beckerle, P., Salvietti, G., Unal, R., Prattichizzo, D., Rossi, S., Castellini, C., et al. (2017). A human-robot interaction perspective on assistive and rehabilitation robotics. Front. Neurorobot. 11:24. doi: 10.3389/fnbot.2017.00024
Bettoni, M. C., and Castellini, C. (2021). Interaction in assistive robotics: a radical constructivist design framework. Front. Neurorobot. 15:67. doi: 10.3389/fnbot.2021.675657
Cansev, M. E., Xue, H., Rottmann, N., Bliek, A., Miller, L. E., Rückert, E., et al. (2021). Interactive human-robot skill transfer: a review of learning methods and user experience (accepted). Adv. Intell. Syst. 3:2000247. doi: 10.1002/aisy.202000247
Castellini, C., Artemiadis, P., Wininger, M., Ajoudani, A., Alimusaj, M., Bicchi, A., et al. (2014). Proceedings of the first workshop on Peripheral Machine Interfaces: going beyond traditional surface electromyography. Front. Neurorobot. 8:22. doi: 10.3389/fnbot.2014.00022
Chen, C., Ma, S., Sheng, X., Farina, D., and Zhu, X. (2020). Adaptive real-time identification of motor unit discharges from non-stationary high-density surface electromyographic signals. IEEE Trans. Biomed. Eng. 67, 3501–3509. doi: 10.1109/TBME.2020.2989311
Christ, O., and Reiner, M. (2014). Perspectives and possible applications of the rubber hand and virtual hand illusion in non-invasive rehabilitation: technological improvements and their consequences. Neurosci. Biobehav. Rev. 44, 33–44. doi: 10.1016/j.neubiorev.2014.02.013
Crucianelli, L., Krahé, C., Jenkinson, P. M., and Fotopoulou, A. K. (2018). Interoceptive ingredients of body ownership: affective touch and cardiac awareness in the rubber hand illusion. Cortex 104, 180–192. doi: 10.1016/j.cortex.2017.04.018
Crucianelli, L., Metcalf, N. K., Fotopoulou, A. K., and Jenkinson, P. M. (2013). Bodily pleasure matters: velocity of touch modulates body ownership during the rubber hand illusion. Front. Psychology 4:703. doi: 10.3389/fpsyg.2013.00703
Del Vecchio, A., and Farina, D. (2019). Interfacing the neural output of the spinal cord: robust and reliable longitudinal identification of motor neurons in humans. J. Neural Eng. 17:016003. doi: 10.1088/1741-2552/ab4d05
Del Vecchio, A., Ubeda, A., Sartori, M., Azorin, J. M., Felici, F., and Farina, D. (2018). Central nervous system modulates the neuromechanical delay in a broad range for the control of muscle force. J. Appl. Physiol. 125, 1404–1410. doi: 10.1152/japplphysiol.00135.2018
Fang, Y., Hettiarachchi, N., Zhou, D., and Liu, H. (2015). Multi-modal sensing techniques for interfacing hand prostheses: a review. IEEE Sens. J. 15, 6065–6076. doi: 10.1109/JSEN.2015.2450211
Farina, D., Jiang, N., Rehbaum, H., Holobar, A., Graimann, B., Dietl, H., et al. (2014). The extraction of neural information from the surface EMG for the control of upper-limb prostheses: emerging avenues and challenges. IEEE Trans. Neural Syst. Rehabil. Eng. 22, 797–809. doi: 10.1109/TNSRE.2014.2305111
Farina, D., Vujaklija, I., Sartori, M., Kapelner, T., Negro, F., Jiang, N., et al. (2017). Man/machine interface based on the discharge timings of spinal motor neurons after targeted muscle reinnervation. Nat. Biomed. Eng. 1, 1–12. doi: 10.1038/s41551-016-0025
Gibas, C., Grunewald, A., Wunderlich, H. W., Marx, P., and Bruck, R. (2019). “A wearable EIT system for detection of muscular activity in the extremities,” in 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (Berlin), 2496–2499. doi: 10.1109/EMBC.2019.8856792
Glaser, V., Holobar, A., and Zazula, D. (2013). Real-time motor unit identification from high-density surface EMG. IEEE Trans. Neural Syst. Rehabil. Eng. 21, 949–958. doi: 10.1109/TNSRE.2013.2247631
Heckman, C., Gorassini, M. A., and Bennett, D. J. (2005). Persistent inward currents in motoneuron dendrites: implications for motor output. Muscle Nerve 31, 135–156. doi: 10.1002/mus.20261
Hochberg, L. R., Bacher, D., Jarosiewicz, B., Masse, N. Y., Simeral, J. D., Vogel, J., et al. (2012). Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 7398, 372–375. doi: 10.1038/nature11076
Lazarou, I., Nikolopoulos, S., Petrantonakis, P. C., Kompatsiaris, I., and Tsolaki, M. (2018). EEG-based brain-computer interfaces for communication and rehabilitation of people with motor impairment: a novel approach of the 21st century. Front. Hum. Neurosci. 12:14. doi: 10.3389/fnhum.2018.00014
Llorens, R., Borrego, A., Palomo, P., Cebolla, A., Noé, E., i Badia, S. B., et al. (2017). Body schema plasticity after stroke: subjective and neurophysiological correlates of the rubber hand illusion. Neuropsychologia 96, 61–69. doi: 10.1016/j.neuropsychologia.2017.01.007
Longo, M. R., Schüür, F., Kammers, M. P., Tsakiris, M., and Haggard, P. (2008). What is embodiment? A psychometric approach. Cognition 107, 978–998. doi: 10.1016/j.cognition.2007.12.004
Merletti, R., Botter, A., Cescon, C., Minetto, M., and Vieira, T. (2011). Advances in surface EMG: Recent progress in clinical research applications. Crit. Rev. Biomed. Eng. 38, 347–379. doi: 10.1615/CritRevBiomedEng.v38.i4.20
Micera, S., Carpaneto, J., and Raspopovic, S. (2010). Control of hand prostheses using peripheral information. IEEE Rev. Biomed. Eng. 3, 48–68. doi: 10.1109/RBME.2010.2085429
Nostadt, N., Abbink, D. A., Christ, O., and Beckerle, P. (2020). Embodiment, presence, and their intersections: teleoperation and beyond. ACM Trans. Hum. Robot Interact. 9, 1–19. doi: 10.1145/3389210
Ortiz-Catalan, M., Håkansson, B., and Brånemark, R. (2014). An osseointegrated human-machine gateway for long-term sensory feedback and motor control of artificial limbs. Sci. Transl. Med. 6:257re6. doi: 10.1126/scitranslmed.3008933
Partridge, L. D. (1965). Modifications of neural output signals by muscles: a frequency response study. J. Appl. Physiol. 20, 150–156. doi: 10.1152/jappl.1965.20.1.150
Preatoni, G., Valle, G., Petrini, F. M., and Raspopovic, S. (2021). Lightening the perceived prosthesis weight with neural embodiment promoted by sensory feedback. Curr. Biol. 31, 1065–1071. doi: 10.1016/j.cub.2020.11.069
Ree, A., Mayo, L. M., Leknes, S., and Sailer, U. (2019). Touch targeting C-tactile afferent fibers has a unique physiological pattern: a combined electrodermal and facial electromyography study. Biol. Psychol. 140, 55–63. doi: 10.1016/j.biopsycho.2018.11.006
Romano, D., Caffa, E., Hernandez-Arieta, A., Brugger, P., and Maravita, A. (2015). The robot hand illusion: Inducing proprioceptive drift through visuo-motor congruency. Neuropsychologia 70, 414–420. doi: 10.1016/j.neuropsychologia.2014.10.033
Sato, Y., Kawase, T., Takano, K., Spence, C., and Kansaku, K. (2018). Body ownership and agency altered by an electromyographically controlled robotic arm. R. Soc. Open Sci. 5:172170. doi: 10.1098/rsos.172170
Sierra González, D., and Castellini, C. (2013). A realistic implementation of ultrasound imaging as a human-machine interface for upper-limb amputees. Front. Neurorobot. 7:17. doi: 10.3389/fnbot.2013.00017
Ting, J. E., Del Vecchio, A., Sarma, D., Colachis, S. C., Annetta, N. V., Collinger, J. L., et al. (2021). Sensing and decoding the neural drive to paralyzed muscles during attempted movements of a person with tetraplegia using a sleeve array. medRxiv. doi: 10.1101/2021.02.24.21250962
Tsuji, T., Yamakawa, H., Yamashita, A., Takakusaki, K., Maeda, T., Kato, M., et al. (2013). “Analysis of electromyography and skin conductance response during rubber hand illusion,” in 2013 IEEE Workshop on Advanced Robotics and Its Social Impacts (Tokyo: IEEE), 88–93. doi: 10.1109/ARSO.2013.6705511
van Stralen, H. E., van Zandvoort, M. J., Hoppenbrouwers, S. S., Vissers, L. M., Kappelle, L. J., and Dijkerman, H. C. (2014). Affective touch modulates the rubber hand illusion. Cognition 131, 147–158. doi: 10.1016/j.cognition.2013.11.020
Keywords: peripheral interfaces, myography, user experience, neuroergonomics, human-robot interaction
Citation: Del Vecchio A, Castellini C and Beckerle P (2021) Peripheral Neuroergonomics – An Elegant Way to Improve Human-Robot Interaction? Front. Neurorobot. 15:691508. doi: 10.3389/fnbot.2021.691508
Received: 06 April 2021; Accepted: 28 July 2021;
Published: 20 August 2021.
Edited by:
Ehsan T. Esfahani, University at Buffalo, United StatesReviewed by:
Hemanth Manjunatha, Georgia Institute of Technology, United StatesRanjana K. Mehta, Texas A&M University, United States
Copyright © 2021 Del Vecchio, Castellini and Beckerle. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Philipp Beckerle, philipp.beckerle@fau.de