Rex N. Tien1*
Anand Tekriwal1,2,3,4
Dylan J. Calame2,3,4Jonathan P. Platt5
Sunderland Baker6Lauren C. Seeberger7
Drew S. Kern8Abigail L. Person2
Steven G. Ojemann1
John A. Thompson1,5,8†Daniel R. Kramer1†- 1Department of Neurosurgery, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO, United States
- 2Department of Physiology and Biophysics, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO, United States
- 3Neuroscience Graduate Program, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO, United States
- 4Medical Scientist Training Program, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO, United States
- 5Department of Bioengineering, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO, United States
- 6Colorado College, Colorado Springs, CO, United States
- 7Charleston Area Medical Center, Department of Neurology, Charleston, WV, United States
- 8Department of Neurology, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO, United States
Clinical assessments of movement disorders currently rely on the administration of rating scales, which, while clinimetrically validated and reliable, depend on clinicians’ subjective analyses, resulting in interrater differences. Intraoperative microelectrode recording for deep brain stimulation targeting similarly relies on clinicians’ subjective evaluations of movement-related neural activity. Digital motion tracking can improve the diagnosis, assessment, and treatment of movement disorders by generating objective, standardized measures of patients’ kinematics. Motion tracking with concurrent neural recording also enables motor neuroscience studies to elucidate the neurophysiology underlying movements. Despite these promises, motion tracking has seen limited adoption in clinical settings due to the drawbacks of conventional motion tracking systems and practical limitations associated with clinical settings. However, recent advances in deep learning based computer vision algorithms have made accurate, robust markerless motion tracking viable in any setting where digital video can be captured. Here, we review and discuss the potential clinical applications and technical limitations of deep learning based markerless motion tracking methods with a focus on DeepLabCut (DLC), an open-source software package that has been extensively applied in animal neuroscience research. We first provide a general overview of DLC, discuss its present usage, and describe the advantages that DLC confers over other motion tracking methods for clinical use. We then present our preliminary results from three ongoing studies that demonstrate the use of DLC for 1) movement disorder patient assessment and diagnosis, 2) intraoperative motor mapping for deep brain stimulation targeting and 3) intraoperative neural and kinematic recording for basic human motor neuroscience.
1 Introduction
Movement disorders such as Parkinson’s disease (PD) and essential tremor (ET) are prevalent, debilitating diseases. PD affects 1.6% of the population (Pringsheim et al., 2014) and ET affects 4.6% of people older than 65 (Louis and Ferreira, 2010). These disorders are initially treated with neuromodulatory medications, and in cases where symptoms are not adequately controlled, neurosurgical interventions including deep brain stimulation (DBS) are considered.
The current method of movement disorder severity assessment is the administration of a rating scale by a trained clinician. The most common scale for PD (Mitchell et al., 2000) is the Movement Disorder Society sponsored update of the Unified Parkinson’s Disease Rating Scale (MDS-UPDRS) (Fahn, 1987; Goetz et al., 2007). Despite the MDS-UPDRS being a validated metric for assessment of parkinsonism, interrater reliability is moderate (Heldman et al., 2011; Luiz et al., 2021) or low in the case of tremor ratings (Martinez-Martin et al., 1994; Richards et al., 1994; Goetz et al., 1995). This is remarkable given the extensive training necessary to become a movement disorders expert capable of administering these scales. Similar subjectively rated scales have been developed for ET, such as The Essential Tremor Rating Assessment Scale (TETRAS) (Elble et al., 2008).
DBS surgery also relies on subjective clinician judgments when aided by awake intraoperative microelectrode recording (MER). MER requires clinicians to judge the presence or absence of correspondence between neural activity and patient movement (Hutchison, 2009; Abosch et al., 2013). Tuning of DBS parameters requires additional subjective assessments of motor improvements.
While DBS is effective, its mechanism of action is not fully understood (Lozano et al., 2019). Though much has been learned about PD neuropathophysiology (Gonzalez-Escamilla et al., 2020), many questions remain. In the case of ET, even less is known (Haubenberger and Hallett, 2018).
Digital motion tracking could help to address these issues (Chen et al., 2016). Motion tracking can objectively quantify subtle movement variations during clinical exams, leading to more consistent assessments (Belić et al., 2019). Furthermore, motion tracking can aid clinicians by generating objective measures during intraoperative MER. Motion tracking combined with neural recording enables investigation of the neurophysiology of brain areas involved with movement disorders, informing future treatments.
In recent years, deep learning based markerless motion tracking software, such as DeepLabCut (DLC) (Mathis et al., 2018), has emerged as a flexible and reliable method to measure kinematics. We propose that markerless methods such as DLC have the potential to aid in the diagnosis, assessment, treatment, and neuroscience of movement disorders. In this review, we discuss the methodology of DLC, its current uses, and its applicability in clinical settings. We then describe three of our ongoing studies that utilize DLC: 1) a clinical trial for assessment and diagnosis of movement disorders, 2) the development of a tool for objective MER-assisted motor mapping during DBS surgery for PD and 3) a basic motor neuroscience study characterizing the relationship between thalamic neuron spiking and reach kinematics in ET. The basic setups and preliminary results of these studies are presented here as proofs-of-concept for the use of DLC in clinical settings; rigorous analysis and quantification of results will be presented in future publications.
2 Review of markerless motion tracking methods
2.1 Technical background
Conventional motion tracking methods such as inertial, magnetic, acoustic, and electromechanical sensors, and video-based tracking of reflective markers or LEDs (Zhou and Hu, 2008), all require that a device or marker be affixed to the body part of interest, and often require expensive equipment.
In the last decade, deep learning (the application of many-layered artificial neural networks) has seen considerable progress, especially in image recognition tasks (Voulodimos et al., 2018) such as pose estimation (Dang et al., 2019). Accordingly, several open-source deep learning based tools specifically designed for motion tracking, such as DLC (Mathis et al., 2018), have emerged. These tools allow for motion to be digitized directly from video, eliminating the need for sensors or markers.
DLC is an open-source, Python-based software package adapted from the feature extraction layers of DeeperCut (Insafutdinov et al., 2016), a pose estimation network trained on ImageNet (Deng et al., 2009). DLC networks can be adapted to track subjects in novel environments using transfer learning (Mathis et al., 2018). The typical workflow for motion tracking with DLC is the following: 1) collect video using any digital camera, 2) manually label the body parts or objects to be tracked in a subset of frames, 3) train the network with this input data, and 4) apply the network to the entire video. For each frame, DLC outputs the estimated pixel locations of the tracked points and numerical measures of confidence (Nath et al., 2019).
2.2 Advantages of deeplabcut and suitability for clinical use
The accuracy of DLC has been validated in human studies; when camera views of subjects are unobstructed, DLC performs comparably to or as well as inertial sensors (Pérez et al., 2021), electromagnetic sensors (Pouw et al., 2020), infrared markers (Moro et al., 2020; Vonstad et al., 2020; Drazan et al., 2021; Moro et al., 2021; Needham et al., 2021) and manual video labeling (Papic et al., 2021). DLC performs as well as or better than other markerless methods (Liu et al., 2020; Vonstad et al., 2020; Cronin, 2021; Needham et al., 2021) such as OpenPose (Cao et al., 2019), which does not allow for network retraining, and LEAP (Pereira et al., 2019), which uses a shallower network.
DLC is particularly adept at tracking a wide variety of subjects in unconventional settings (Mathis and Mathis, 2020). This flexibility is evidenced by the widespread adoption of DLC in animal neuroscience across many different species and scenarios. DLC has also been used to track humans in complex environments. The variety of studies utilizing DLC is enumerated in Table 1.
TABLE 1. Enumeration of markerless motion tracking studies in the literature. DeepLabCut (DLC) studies were identified by searching for articles citing the initial DLC publications (Mathis et al., 2018; Nath et al., 2019) using Google Scholar. The Kinect (Microsoft) and Leap Motion controller (Ultraleap) are consumer markerless motion tracking devices that do not utilize deep learning. For the “Human Tracking, Non-Clinical” and “Non-Human Tracking” categories, only studies which utilized DLC were enumerated; studies using other markerless motion tracking methods were excluded for brevity.
An additional advantage of DLC is its open-source availability and associated development community (Mathis et al., 2020; Anderson et al., 2021). DLC development is ongoing (Nath et al., 2019), and third-party contributions have enabled real-time tracking (Forys et al., 2020; Kane et al., 2020; Nourizonoz et al., 2020; Schweihoff et al., 2021; Sehara et al., 2021) and 3D reconstruction (Gosztolai et al., 2020; Sheshadri et al., 2020; Dunn et al., 2021; Karashchuk et al., 2021; Zhang et al., 2021).
Markerless methods in general are attractive for clinical use because they do not require anything to be attached to the body. This reduces setup time, potential for error, and patient encumbrance, which is especially important for movement disorder patients with limited or excessive movements, and who may need to physically interact with clinicians for evaluation. Additionally, the capability of DLC to track motion in a wide range of settings is desirable as clinical environments, especially operating rooms, are visually complex due to reflective surfaces and variable lighting conditions. As such, many clinical studies have been conducted using markerless methods such as DLC, OpenPose, custom neural networks, Kinect (Microsoft), Leap Motion (Ultraleap), and other non-neural network solutions. These preliminary studies have focused on a range of applications, including quantifying gross kinematics for movement disorders or rehabilitation, capturing facial expression changes, and tracking movements of internal anatomy using radiography and endoscopic cameras. These studies are enumerated in Table 1, and studies particularly relevant to the use of markerless tracking for movement disorders are discussed in the following sections.
Based on these advantages, we have chosen to use DLC to pursue three novel clinical research questions, as described below.
3 Study 1: Clinical trial using deeplabcut for movement disorder diagnosis and assessment
3.1 Study 1: Background and related work
Objective quantification of movement can augment movement disorder diagnosis and assessment. Kinematic metrics obtained with conventional tracking methods have shown high correlations with clinical ratings and have predicted disease states. Such studies have been performed with inertial sensors (Burkhard et al., 1999; Hoff et al., 2001; Salarian et al., 2007; Giuffrida et al., 2009; Patel et al., 2009; Kim et al., 2011; Pulliam et al., 2014; Ramsperger et al., 2016; Delrobaei et al., 2017; Jeon et al., 2017), electromagnetic sensors (Espay et al., 2009; Gao et al., 2018), infrared markers (Das et al., 2011) and smartwatches (Malekmohammadi et al., 2016; López-Blanco et al., 2019). Similar studies have also been performed using markerless methods (Li M. H. et al., 2018; Liu et al., 2019; Sato et al., 2019; Wong et al., 2019; Williams et al., 2020a; Pang et al., 2020; Jaber et al., 2021; Shin et al., 2021). We identified four studies that used DLC for movement disorder diagnosis and/or assessment. Miao et al. (2020) tracked gait using iPad videos to distinguish dystonia patients from controls. Stolk et al. (2020) tracked children with dyskinetic cerebral palsy to predict Dyskinesia Impairment Scale scores. Shin et al. (2020) and Williams et al. (2020b) predicted MDS-UPDRS scores by tracking movements in archival footage and cellphone videos, respectively.
3.2 Study 1: Description of current work
We designed a clinical trial (https://clinicaltrials.gov/ct2/show/record/NCT04074772) to evaluate the use of DLC for automated movement disorder disease state evaluation and classification. We built a custom mobile frame to position three synchronized cameras (Blackfly USB3, Teledyne FLIR) in front of subjects (Figure 1A). Healthy control subjects and movement disorder patients with various diagnoses, including PD and ET, were filmed while performing hand movements used in movement disorder rating scales.
FIGURE 1. Description of three ongoing studies utilizing DeepLabCut (DLC) in clinical settings. Top row: Clinical trial using DLC for movement disorder assessment and diagnosis. Middle row: Development of an intraoperative tool for automated motor mapping during DBS implantation of STN for treatment of PD. Bottom row: Motor neuroscience experiment linking VIM unit activity to reach kinematics in ET patients. (A,D and G) depict the physical setups for each study. (B) Hand tracking during a clinical movement disorder exam. (C) Output of DLC tracking in the vertical dimension of pointer finger (blue) and thumb (orange) tips during a finger tap test. (E) Hand tracking during active (left) and passive (right) movements for intraoperative MER based motor mapping. DLC is able to identify and label the patient’s hand and not the clinician’s during passive testing. (F) Average camera coordinate vertical position in pixels of all fingertips during a “chain pull” movement as tracked by DLC. (H) Monitor and cameras used to administer and track the center-out reaching task. While all targets are displayed simultaneously here, only one target was presented at a time during the task. (I) DLC-tracked position in pixels of pointer finger tip over an entire center-out session, with each color representing reaches to and from a different target (seven targets total for this subject).
Subject-specific DLC networks were trained to track 11 points on the hand (Figure 1B), which were then reconstructed in 3D (Figure 1C). Subjects were also rated by a movement disorders neurologist. We expect that DLC output can be used to predict rating scale scores, and that dimensionality reduction and clustering of kinematic data will distinguish movement disorder patients with different diagnoses from each other and from controls.
By using an unobtrusive video recording system and DLC, we can obtain 3D positional data during clinical exams without restricting patient movements. This method can measure kinematic features of hard-to-detect movement variations in clinically relevant tasks like finger tapping and tremor, which are used to characterize disease state and evaluate subtle changes in treatment responses during DBS programming.
4 Study 2: Use of deeplabcut for functional targeting of deep brain stimulation electrodes
4.1 Study 2: Background and related work
DBS applied to the subthalamic nucleus (STN) is an effective treatment for advanced PD (Bronstein et al., 2011). After initial imaging-based targeting, MER is performed to accurately identify the motor territory of the dorsolateral STN for implantation of the therapeutic electrode (Benazzouz et al., 2002; Sterio et al., 2002; Hutchison, 2009; Gonzalez-Escamilla et al., 2020). Studies in which STN MER was performed simultaneously with electromyography have identified regions of STN with neural activity correlated with tremor and active and/or passive movements (Lenz et al., 1988; Magariños-Ascone et al., 2000; Rodriguez-Oroz et al., 2001; Amtage et al., 2008).
To identify the motor territory of STN with MER, a neurologist administers functional tests while visually and auditorily assessing the MER voltage trace. Clinicians seek to identify functional territories of STN defined by correlations between neural activity and tremor, voluntary and/or passive contralateral movements. This procedure relies on clinicians’ subjective judgments of the presence or absence of movement-related neural activity. An automated system to assess MER and kinematics could augment clinicians’ judgments and could be a useful tool for centers where MER expertise is lacking.
4.2 Study 2: Description of current work
Utilizing DLC intraoperatively, we can objectively identify functional relationships between STN neural activity and kinematics through statistical analysis of MER signals and tracked motion. To this end, we designed and executed a study using DLC to track PD patient movement during MER (Colorado Multiple Institution Review Board, COMIRB #17–1291).
Video was obtained using two cameras (Blackfly USB3) mounted on tripods aimed at the operating table (Figure 1D) and synchronized with the MER system (Neuro Omega, AlphaOmega). DLC was used to track 21 points on the hand, and was able to distinguish the patient’s hand from the neurologist’s (Figure 1E). Kinematics (Figure 1F) were compared to the MER signals offline using dynamic time warping and Monte Carlo methods to identify instances of correspondence between movements and neural activity. These objective assessments of motion-related neural activity largely agreed with clinician judgments.
To our knowledge, this study is the first attempt to develop an automated tool to assist clinicians in functional MER for DBS electrode targeting. While this study is only a first step and operates offline, it explores the feasibility of such a technique with potential for real-time intraoperative use. A key advantage of this technique is that it is unobtrusive and does not interfere with standard clinical procedures.
5 Study 3: Use of deeplabcut for human motor neuroscience
5.1 Study 3: Background and related work
MER during DBS surgery presents a rare opportunity to record from single neurons in wakeful humans (Engel et al., 2005; Cash and Hochberg, 2015; Lee et al., 2019; Tekriwal et al., 2019). This allows for basic neuroscience experiments to elucidate the neurophysiology of the targets of DBS, guiding the development of future treatments (Stein and Bar-Gad, 2013; Gonzalez-Escamilla et al., 2020). Several studies have been conducted with conventional motion tracking methods, but because of the constraints of the operating room, kinematic tracking has been limited to the use of joysticks (Amirnovin et al., 2004; Zavala et al., 2017; Tekriwal et al., 2018, 2022), grip force dynamometers (Patil et al., 2004; Fischer et al., 2020), bend-sensitive resistors (Hanson et al., 2012) and inertial sensors (Levy et al., 2002a,b; MacMillan et al., 2004; Tankus et al., 2017, 2018).
Only a single study has used markerless tracking during MER. London et al. (2021) used the Leap Motion controller (Ultraleap) to allow subjects to control a cursor in real-time with hand movements, allowing the identification of one neural population in the STN that encoded kinematics, and another that responded to unexpected action plan changes.
5.2 Study 3: Description of current work
The ventral intermediate nucleus of the thalamus (VIM) is the primary target of DBS for the treatment of ET (Chopra et al., 2013). While it is known that neural activity in the VIM covaries with tremor and active and passive movements (Lenz et al., 1990, 1994; Zirh et al., 1998; Lenz et al., 2002; Hua and Lenz, 2005; Cajigas et al., 2020), the precise relationship between VIM neurons and kinematics remains unknown. Recent work in mice (Becker and Person, 2019) revealed that the interposed nucleus of the cerebellum is implicated in the precise control of braking at the reach endpoint. The deep motor nuclei of the cerebellum project to the VIM in humans (Roostaei et al., 2014), suggesting that the VIM may also contribute to braking.
To investigate whether VIM neurons encode a brake signal, we designed a study involving a center-out reaching task during awake VIM MER in ET patients (COMIRB #20–2979). We utilized a cart-mounted computer monitor with three cameras (Blackfly USB3) mounted above it that can be positioned over the operating table (Figure 1G). This monitor displayed a central target and eight outer targets (Figure 1H). During the task, the central and outer targets were shown sequentially, cycling through all visible outer targets in a pseudorandom order. Patients were instructed to point at the currently visible target. DLC was used to extract fingertip position (Figure 1I). Regression and Monte Carlo methods were used to determine how VIM unit firing rates related to reach kinematics. These analyses reveal a temporal congruence between VIM neural activity and reach braking, and encoding of fingertip position in VIM units.
To our knowledge this is the first study to examine free reaching movements during human VIM MER, and the first use of DLC in such a setting. The flexibility of DLC enables examination of naturalistic movements that have previously been difficult to study during MER, such as handwriting, tool use and facial expressions.
6 Discussion
Deep learning based markerless motion tracking techniques can improve movement disorder diagnosis, assessment, treatment and neuroscience. DLC is especially appropriate for these tasks because of its ease-of-use, flexibility, low cost, open-source availability and development community, as evidenced by its widespread and rapid adoption in animal and human motion tracking applications. We have demonstrated the utility of DLC in three ongoing studies: a clinical trial using DLC for movement disorder assessment and diagnosis, the development of an intraoperative tool for functional DBS implant targeting in the STN, and a motor neuroscience study relating VIM neural activity to reach kinematics.
Though powerful, DLC does have limitations. The DLC analysis pipeline requires initial manual labeling. Because the output of DLC is only as good as its training data, this presents a potential for variation in tracking performance between different users. As DLC networks are trained anew for each recording scenario, standardization of output may be difficult. DLC network training can take tens of hours even with GPU acceleration. The ability of DLC to generalize across subjects of different size, skin color, gender or age has not been well studied. DLC tracks each feature independently and thus does not incorporate biomechanical or temporal constraints on motion. Finally, DLC can occasionally output spurious results, so human review of final output is advisable, especially if tracking is to be used for clinical decision making.
Specific limitations apply to the three ongoing studies discussed herein. In all three studies, a technician was required to position, calibrate and operate the cameras. Simpler systems will be required for integration into clinical workflows. Custom camera mount frames were built for studies 1 and 3, yielding consistency of camera positioning but potentially sub-optimal capture for each subject. In Study 2, tripods were used, resulting in variable views for each subject. Implementing an online version of Study 2 would require initial DLC network training in the operating room, increasing surgical time. The kinematic data presented in Figure 1 are preliminary, and while observed to be qualitatively accurate, further quantification of accuracy through validation with established datasets will be required.
The potential applications of DLC extend beyond the efforts described here. While all three described studies focused on upper limb movements and thus do not account for abnormal lower limb, head or jaw movements, these symptoms can be tracked and studied with DLC. Motion tracking is also critical for developmental research (van Schaik and Dominici, 2020), and DLC is capable of tracking infants (Pérez et al., 2021). DLC can also be used track gaze (Zdarsky et al., 2021) as well as facial expression (Argyle et al., 2021; Namba et al., 2021), making it a useful tool for the assessment of various neuropathies and for cognitive psychology research. Given the promise and utility of DLC, we predict that it and other markerless motion tracking technologies will see widespread adoption for clinical applications.
Data availability statement
The raw datasets presented in this article are not readily available because video and kinematic data are restricted to protect patient privacy. Anonymized data shown in Figures will be made available upon reasonable request. A list of all studies enumerated in Table 1 will be made available upon request. Requests to access the datasets should be directed to UmV4LlRpZW5AY3VhbnNjaHV0ei5lZHU=.
Ethics statement
The studies involving human participants were reviewed and approved by Colorado Multiple Institution Review Board. Written informed consent was obtained from all subjects prior to study participation.
Author contributions
RT designed studies, built apparatus, collected and analyzed data, created sub-Figures, compiled the Figures and wrote the manuscript. AT designed studies, built apparatus, collected and analyzed data and created sub-Figures. DC designed studies, built apparatus, collected and analyzed data and created sub-Figures. JP built apparatus, collected data and analyzed data. SB analyzed data and created sub-Figures. LS designed studies and performed clinical exams. DK performed clinical exams. AP designed studies. SO performed surgeries. JT designed studies and collected data. DK designed studies and performed surgeries. All authors reviewed and revised the manuscript.
Funding
This work was supported by National Institute of Neurological Disorders and Stroke K12 Neurosurgeon Research Career and Development Grant number K12NS080223 as well as two University of Colorado Movement Disorders Center Pilot Grants.
Acknowledgments
The Optogenetics and Neural Engineering (ONE) Core and the Innovation and Design for Experimentation and Analysis (IDEA) Core at the University of Colorado School of Medicine provided engineering support for this research. The ONE Core is part of the University of Colorado NeuroTechnology Center, funded in part by the University of Colorado School of Medicine and by the National Institute of Neurological Disorders and Stroke of the National Institutes of Health under award number P30NS048154. The IDEA Core is part of the University of Colorado NeuroTechnology Center, funded by the University of Colorado School of Medicine.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
Abosch, A., Timmermann, L., Bartley, S., Rietkerk, H. G., Whiting, D., Connolly, P. J., et al. (2013). An international survey of deep brain stimulation procedural steps. Stereotact. Funct. Neurosurg. 91, 1–11. doi:10.1159/000343207
Amirnovin, R., Williams, Z. M., Cosgrove, G. R., and Eskandar, E. N. (2004). Visually guided movements suppress subthalamic oscillations in Parkinson’s disease patients. J. Neurosci. 24, 11302–11306. doi:10.1523/jneurosci.3242-04.2004
Amtage, F., Henschel, K., Schelter, B., Vesper, J., Timmer, J., Lücking, C. H., et al. (2008). Tremor-correlated neuronal activity in the subthalamic nucleus of parkinsonian patients. Neurosci. Lett. 442, 195–199. doi:10.1016/j.neulet.2008.06.087
Anderson, K. R., Harris, J. A., Ng, L., Prins, P., Memar, S., Ljungquist, B., et al. (2021). Highlights from the era of open source web-based tools. J. Neurosci. 41, 927–936. doi:10.1523/jneurosci.1657-20.2020
Argyle, E. M., Marinescu, A., Wilson, M. L., Lawson, G., and Sharples, S. (2021). Physiological indicators of task demand, fatigue, and cognition in future digital manufacturing environments. Int. J. Human-Computer Stud. 145, 102522. doi:10.1016/j.ijhcs.2020.102522
Bank, P. J., Marinus, J., Meskers, C. G., de Groot, J. H., and van Hilten, J. J. (2017). Optical hand tracking: A novel technique for the assessment of bradykinesia in Parkinson’s disease. Mov. Disord. Clin. Pract. 4, 875–883. doi:10.1002/mdc3.12536
Becker, M. I., and Person, A. L. (2019). Cerebellar control of reach kinematics for endpoint precision. Neuron 103, 335–348. doi:10.1016/j.neuron.2019.05.007
Belić, M., Bobić, V., Badža, M., Solaja, N., Durić-Jovičić, M., and Kostić, V. S. (2019). Artificial intelligence for assisting diagnostics and assessment of Parkinson’s disease—A review. Clin. neurology Neurosurg. 184, 105442. doi:10.1016/j.clineuro.2019.105442
Benazzouz, A., Breit, S., Koudsie, A., Pollak, P., Krack, P., and Benabid, A.-L. (2002). Intraoperative microrecordings of the subthalamic nucleus in Parkinson’s disease. Mov. Disord. 17, S145–S149. doi:10.1002/mds.10156
Berger, M., Agha, N. S., and Gail, A. (2020). Wireless recording from unrestrained monkeys reveals motor goal encoding beyond immediate reach in frontoparietal cortex. Elife 9, e51322. doi:10.7554/elife.51322
Bronstein, J. M., Tagliati, M., Alterman, R. L., Lozano, A. M., Volkmann, J., Stefani, A., et al. (2011). Deep brain stimulation for Parkinson disease: An expert consensus and review of key issues. Arch. Neurol. 68, 165. doi:10.1001/archneurol.2010.260
Burkhard, P. R., Shale, H., Langston, J. W., and Tetrud, J. W. (1999). Quantification of dyskinesia in Parkinson’s disease: Validation of a novel instrumental method. Mov. Disord. 14, 754–763. doi:10.1002/1531-8257(199909)14:5<754::aid-mds1007>3.0.co;2-1
Cachot, A., Bilous, M., Liu, Y.-C., Li, X., Saillard, M., Cenerenti, M., et al. (2021). Tumor-specific cytolytic cd4 t cells mediate immunity against human cancer. Sci. Adv. 7, eabe3348. doi:10.1126/sciadv.abe3348
Cajigas, I., Diaz, A., Prins, N., Tan, S. K., Prasad, A., Luca, C., et al. (2020). An inertial sensor-based system for synchronous upper extremity kinematic reconstruction and neural recordings during awake deep brain stimulation. Neurosurgery 67, nyaa447 652. doi:10.1093/neuros/nyaa447_652
Cao, Z., Hidalgo, G., Simon, T., Wei, S.-E., and Sheikh, Y. (2019). Openpose: Realtime multi-person 2d pose estimation using part affinity fields. IEEE Trans. Pattern Anal. Mach. Intell. 43, 172–186. doi:10.1109/tpami.2019.2929257
Cash, S. S., and Hochberg, L. R. (2015). The emergence of single neurons in clinical neurology. Neuron 86, 79–91. doi:10.1016/j.neuron.2015.03.058
Chen, S., Lach, J., Lo, B., and Yang, G.-Z. (2016). Toward pervasive gait analysis with wearable sensors: A systematic review. IEEE J. Biomed. Health Inf. 20, 1521–1537. doi:10.1109/jbhi.2016.2608720
Chopra, A., Klassen, B. T., and Stead, M. (2013). Current clinical application of deep-brain stimulation for essential tremor. Neuropsychiatr. Dis. Treat. 9, 1859–1865. doi:10.2147/ndt.s32342
Cronin, N. J., Rantalainen, T., Ahtiainen, J. P., Hynynen, E., and Waller, B. (2019). Markerless 2d kinematic analysis of underwater running: A deep learning approach. J. biomechanics 87, 75–82. doi:10.1016/j.jbiomech.2019.02.021
Cronin, N. J. (2021). Using deep neural networks for kinematic analysis: Challenges and opportunities. J. Biomechanics 123, 110460. doi:10.1016/j.jbiomech.2021.110460
Dang, Q., Yin, J., Wang, B., and Zheng, W. (2019). Deep learning based 2d human pose estimation: A survey. Tsinghua Sci. Technol. 24, 663–676. doi:10.26599/tst.2018.9010100
Das, S., Trutoiu, L., Murai, A., Alcindor, D., Oh, M., De la Torre, F., et al. (2011). “Quantitative measurement of motor symptoms in Parkinson’s disease: A study with full-body motion capture data,” in 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society (IEEE), 6789–6792.
De Bari, B., Dixon, J. A., Kay, B. A., and Kondepudi, D. (2019). Oscillatory dynamics of an electrically driven dissipative structure. PloS one 14, e0217305. doi:10.1371/journal.pone.0217305
Delrobaei, M., Baktash, N., Gilmore, G., McIsaac, K., and Jog, M. (2017). Using wearable technology to generate objective Parkinson’s disease dyskinesia severity score: Possibilities for home monitoring. IEEE Trans. Neural Syst. Rehabil. Eng. 25, 1853–1863. doi:10.1109/tnsre.2017.2690578
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition (IEEE), 248–255.
Drazan, J. F., Phillips, W. T., Seethapathi, N., Hullfish, T. J., and Baxter, J. R. (2021). Moving outside the lab: Markerless motion capture accurately quantifies sagittal plane kinematics during the vertical jump. J. Biomech. 125, 110547. doi:10.1016/j.jbiomech.2021.110547
Dror, B., Yanai, E., Frid, A., Peleg, N., Goldenthal, N., Schlesinger, I., et al. (2014). “Automatic assessment of Parkinson’s disease from natural hands movements using 3d depth sensor,” in 2014 IEEE 28th Convention of Electrical & Electronics Engineers in Israel (IEEEI) (IEEE), 1–5.
Dunn, T. W., Marshall, J. D., Severson, K. S., Aldarondo, D. E., Hildebrand, D. G., Chettih, S. N., et al. (2021). Geometric deep learning enables 3d kinematic profiling across species and environments. Nat. Methods 18, 564–573. doi:10.1038/s41592-021-01106-6
Elble, R., Comella, C., Fahn, S., Hallett, M., Jankovic, J., Juncos, J., et al. (2008). “The essential tremor rating assessment scale (tetras),” in Movement disorders (WILEY-LISS DIV JOHN WILEY & SONS INC), 23, S357. 111 RIVER ST, HOBOKEN, NJ 07030 USA.
Engel, A. K., Moll, C. K., Fried, I., and Ojemann, G. A. (2005). Invasive recordings from the human brain: Clinical insights and beyond. Nat. Rev. Neurosci. 6, 35–47. doi:10.1038/nrn1585
Espay, A. J., Beaton, D. E., Morgante, F., Gunraj, C. A., Lang, A. E., and Chen, R. (2009). Impairments of speed and amplitude of movement in Parkinson’s disease: A pilot study. Mov. Disord. 24, 1001–1008. doi:10.1002/mds.22480
Fahn, S. (1987). Unified Parkinson’s disease rating scale. recent developments in Parkinson’s disease volume ii. Macmillan Healthc. Inf. 153.
Fischer, P., Lipski, W. J., Neumann, W.-J., Turner, R. S., Fries, P., Brown, P., et al. (2020). Movement-related coupling of human subthalamic nucleus spikes to cortical gamma. Elife 9, e51956. doi:10.7554/elife.51956
Forys, B. J., Xiao, D., Gupta, P., and Murphy, T. H. (2020). Real-time selective markerless tracking of forepaws of head fixed mice using deep neural networks. Eneuro 7, 0096. doi:10.1523/eneuro.0096-20.2020
Gao, C., Smith, S., Lones, M., Jamieson, S., Alty, J., Cosgrove, J., et al. (2018). Objective assessment of bradykinesia in Parkinson’s disease using evolutionary algorithms: Clinical validation. Transl. Neurodegener. 7, 18–8. doi:10.1186/s40035-018-0124-x
Garcia-Agundez, A., and Eickhoff, C. (2021). Towards objective quantification of hand tremors and bradykinesia using contactless sensors: A systematic review. Front. Aging Neurosci. 694, 716102. doi:10.3389/fnagi.2021.716102
Gautam, B. P., Noda, Y., Gautam, R., Sharma, H. P., Sato, K., and Neupane, S. B. (2020). “Body part localization and pose tracking by using deepercut algorithm for king cobra’s bbl (biting behavior learning),” in 2020 International Conference on Networking and Network Applications (NaNA) (IEEE), 422–429.
Giuffrida, J. P., Riley, D. E., Maddux, B. N., and Heldman, D. A. (2009). Clinically deployable kinesia™ technology for automated tremor assessment. Mov. Disord. 24, 723–730. doi:10.1002/mds.22445
Goetz, C. G., Fahn, S., Martinez-Martin, P., Poewe, W., Sampaio, C., Stebbins, G. T., et al. (2007). Movement disorder society-sponsored revision of the unified Parkinson’s disease rating scale (mds-updrs): Process, format, and clinimetric testing plan. Mov. Disord. 22, 41–47. doi:10.1002/mds.21198
Goetz, C. G., Stebbins, G. T., Chmura, T. A., Fahn, S., Klawans, H. L., and Marsden, C. D. (1995). Teaching tape for the motor section of the unified Parkinson’s disease rating scale. Mov. Disord. 10, 263–266. doi:10.1002/mds.870100305
Gonzalez-Escamilla, G., Muthuraman, M., Ciolac, D., Coenen, V. A., Schnitzler, A., and Groppa, S. (2020). Neuroimaging and electrophysiology meet invasive neurostimulation for causal interrogations and modulations of brain states. Neuroimage 220, 117144. doi:10.1016/j.neuroimage.2020.117144
Gosztolai, A., Günel, S., Abrate, M. P., Morales, D., Ríos, V. L., Rhodin, H., et al. (2020). Liftpose3d, a deep learning-based approach for transforming 2d to 3d pose in laboratory animals. bioRxiv.
Green, R. D., Guan, L., and Burne, J. A. (2000). Video analysis of gait for diagnosing movement disorders. J. Electron. Imaging 9, 16–21. doi:10.1117/1.482723
Hanson, T. L., Fuller, A. M., Lebedev, M. A., Turner, D. A., and Nicolelis, M. A. (2012). Subcortical neuronal ensembles: An analysis of motor task association, tremor, oscillations, and synchrony in human patients. J. Neurosci. 32, 8620–8632. doi:10.1523/jneurosci.0750-12.2012
Haubenberger, D., and Hallett, M. (2018). Essential tremor. N. Engl. J. Med. Overseas. Ed. 378, 1802–1810. doi:10.1056/nejmcp1707928
Heldman, D. A., Giuffrida, J. P., Chen, R., Payne, M., Mazzella, F., Duker, A. P., et al. (2011). The modified bradykinesia rating scale for Parkinson’s disease: Reliability and comparison with kinematic measures. Mov. Disord. 26, 1859–1863. doi:10.1002/mds.23740
Hoff, J., Plas, A., Wagemans, E., and Van Hilten, J. (2001). Accelerometric assessment of levodopa-induced dyskinesias in Parkinson’s disease. Mov. Disord. 16, 58–61. doi:10.1002/1531-8257(200101)16:1<58::aid-mds1018>3.0.co;2-9
Hua, S. E., and Lenz, F. A. (2005). Posture-related oscillations in human cerebellar thalamus in essential tremor are enabled by voluntary motor circuits. J. neurophysiology 93, 117–127. doi:10.1152/jn.00527.2004
Hutchison, W. D. (2009). Microelectrode recording and microstimulation for target mapping. Deep Brain Stimul., 37–48. doi:10.1093/med/9780199543717.003.0005
Insafutdinov, E., Pishchulin, L., Andres, B., Andriluka, M., and Schiele, B. (2016). “Deepercut: A deeper, stronger, and faster multi-person pose estimation model,” in European Conference on Computer Vision (Springer), 34–50.
Jaber, R., Qahwaji, R., Abdullatif, A., Buckley, J., and Abd-Alhameed, R. (2021). “Proposing a three-stage model to quantify bradykinesia on a symptom severity level using deep learning,” in UK Workshop on Computational Intelligence (Springer), 428–438.
Jeon, H., Lee, W., Park, H., Lee, H. J., Kim, S. K., Kim, H. B., et al. (2017). Automatic classification of tremor severity in Parkinson’s disease using a wearable device. Sensors 17, 2067. doi:10.3390/s17092067
Kane, G. A., Lopes, G., Saunders, J. L., Mathis, A., and Mathis, M. W. (2020). Real-time, low-latency closed-loop feedback using markerless posture tracking. Elife 9, e61909. doi:10.7554/elife.61909
Karashchuk, P., Rupp, K. L., Dickinson, E. S., Walling-Bell, S., Sanders, E., Azim, E., et al. (2021). Anipose: A toolkit for robust markerless 3d pose estimation. Cell. Rep. 36, 109730. doi:10.1016/j.celrep.2021.109730
Khan, T., Nyholm, D., Westin, J., and Dougherty, M. (2014). A computer vision framework for finger-tapping evaluation in Parkinson’s disease. Artif. Intell. Med. 60, 27–40. doi:10.1016/j.artmed.2013.11.004
Kim, J.-W., Lee, J.-H., Kwon, Y., Kim, C.-S., Eom, G.-M., Koh, S.-B., et al. (2011). Quantification of bradykinesia during clinical finger taps using a gyrosensor in patients with Parkinson’s disease. Med. Biol. Eng. Comput. 49, 365–371. doi:10.1007/s11517-010-0697-8
Lee, H., Guan, L., and Lee, I. (2008). Video analysis of human gait and posture to determine neurological disorders. EURASIP J. Image Video Process. 2008, 1–12. doi:10.1155/2008/380867
Lee, S., Shah, M., Lauro, P. M., and Asaad, W. F. (2019). “Intraoperative research during deep brain stimulation surgery,” in Deep brain stimulation: Techniques and practices (thieme).
Lemaire, B. S. (2020). No evidence of spontaneous preference for slowly moving objects in visually naïve chicks. Sci. Rep. 10, 6277–6278. doi:10.1038/s41598-020-63428-3
Lenz, F., Jaeger, C., Seike, M., Lin, Y., and Reich, S. (2002). Single-neuron analysis of human thalamus in patients with intention tremor and other clinical signs of cerebellar disease. J. neurophysiology 87, 2084–2094. doi:10.1152/jn.00049.2001
Lenz, F., Kwan, H., Dostrovsky, J., Tasker, R., Murphy, J., and Lenz, Y. (1990). Single unit analysis of the human ventral thalamic nuclear group: Activity correlated with movement. Brain 113, 1795–1821. doi:10.1093/brain/113.6.1795
Lenz, F., Kwan, H., Martin, R., Tasker, R., Dostrovsky, J., and Lenz, Y. (1994). Single unit analysis of the human ventral thalamic nuclear group: Tremor-related activity in functionally identified cells. Brain 117, 531–543. doi:10.1093/brain/117.3.531
Lenz, F., Tasker, R., Kwan, H., Schnider, S., Kwong, R., Murayama, Y., et al. (1988). Single unit analysis of the human ventral thalamic nuclear group: Correlation of thalamic “tremor cells” with the 3-6 hz component of parkinsonian tremor. J. Neurosci. 8, 754–764. doi:10.1523/jneurosci.08-03-00754.1988
Levy, R., Ashby, P., Hutchison, W. D., Lang, A. E., Lozano, A. M., and Dostrovsky, J. O. (2002a). Dependence of subthalamic nucleus oscillations on movement and dopamine in Parkinson’s disease. Brain 125, 1196–1209. doi:10.1093/brain/awf128
Levy, R., Hutchison, W. D., Lozano, A. M., and Dostrovsky, J. O. (2002b). Synchronized neuronal discharge in the basal ganglia of parkinsonian patients is limited to oscillatory activity. J. Neurosci. 22, 2855–2861. doi:10.1523/jneurosci.22-07-02855.2002
Li, M. H., Mestre, T. A., Fox, S. H., and Taati, B. (2018a). Vision-based assessment of parkinsonism and levodopa-induced dyskinesia with pose estimation. J. Neuroeng. Rehabil. 15, 97–13. doi:10.1186/s12984-018-0446-z
Li, T., Chen, J., Hu, C., Ma, Y., Wu, Z., Wan, W., et al. (2018b). Automatic timed up-and-go sub- task segmentation for Parkinson’s disease patients using video-based activity classification. IEEE Trans. Neural Syst. Rehabil. Eng. 26, 2189–2199. doi:10.1109/tnsre.2018.2875738
Li, Y., Richter, F., Lu, J., Funk, E. K., Orosco, R. K., Zhu, J., et al. (2020). Super: A surgical perception framework for endoscopic tissue manipulation with surgical robotics. IEEE Robot. Autom. Lett. 5, 2294–2301. doi:10.1109/lra.2020.2970659
Liu, X., Yu, S.-y., Flierman, N., Loyola, S., Kamermans, M., Hoogland, T. M., et al. (2020). Optiflex: Video-based animal pose estimation using deep learning enhanced by optical flow. BioRxiv.
Liu, Y., Chen, J., Hu, C., Ma, Y., Ge, D., Miao, S., et al. (2019). Vision-based method for automatic quantification of parkinsonian bradykinesia. IEEE Trans. Neural Syst. Rehabil. Eng. 27, 1952–1961. doi:10.1109/tnsre.2019.2939596
London, D., Fazl, A., Katlowitz, K., Soula, M., Pourfar, M., Mogilner, A., et al. (2021). Distinct population code for movement kinematics and changes of ongoing movements in human subthalamic nucleus. Elife 10, e64893. doi:10.7554/elife.64893
Louis, E. D., and Ferreira, J. J. (2010). How common is the most common adult movement disorder? Update on the worldwide prevalence of essential tremor. Mov. Disord. 25, 534–541. doi:10.1002/mds.22838
Lozano, A. M., Lipsman, N., Bergman, H., Brown, P., Chabardes, S., Chang, J. W., et al. (2019). Deep brain stimulation: Current challenges and future directions. Nat. Rev. Neurol. 15, 148–160. doi:10.1038/s41582-018-0128-2
López-Blanco, R., Velasco, M. A., Méndez-Guerrero, A., Romero, J. P., Del Castillo, M. D., Serrano, J. I., et al. (2019). Smartwatch for the analysis of rest tremor in patients with Parkinson’s disease. J. neurological Sci. 401, 37–42. doi:10.1016/j.jns.2019.04.011
Luiz, L., Marques, I., Folador, J., and Andrade, A. (2021). Intra and inter-rater remote assessment of bradykinesia in Parkinson’s disease. Neurología. doi:10.1016/j.nrl.2021.08.005
MacMillan, M. L., Dostrovsky, J. O., Lozano, A. M., and Hutchison, W. D. (2004). Involvement of human thalamic neurons in internally and externally generated movements. J. neurophysiology 91, 1085–1090. doi:10.1152/jn.00835.2003
Magariños-Ascone, C. M., Figueiras-Mendez, R., Riva-Meana, C., and Córdoba-Fernández, A. (2000). Subthalamic neuron activity related to tremor and movement in Parkinson’s disease. Eur. J. Neurosci. 12, 2597–2607. doi:10.1046/j.1460-9568.2000.00127.x
Malekmohammadi, M., Herron, J., Velisar, A., Blumenfeld, Z., Trager, M. H., Chizeck, H. J., et al. (2016). Kinematic adaptive deep brain stimulation for resting tremor in parkinson’s disease. Mov. Disord. 31, 426–428. doi:10.1002/mds.26482
Marques, J. C., Li, M., Schaak, D., Robson, D. N., and Li, J. M. (2020). Internal state dynamics shape brainwide activity and foraging behaviour. Nature 577, 239–243. doi:10.1038/s41586-019-1858-z
Martinez-Martin, P., Gil-Nagel, A., Gracia, L. M., Gomez, J. B., Martinez-Sarries, J., and Bermejo, F. (1994). Unified Parkinson's disease rating scale characteristics and structure. Mov. Disord. 9, 76–83. doi:10.1002/mds.870090112
Mathis, A., Mamidanna, P., Cury, K. M., Abe, T., Murthy, V. N., Mathis, M. W., et al. (2018). Deeplabcut: Markerless pose estimation of user-defined body parts with deep learning. Nat. Neurosci. 21, 1281–1289. doi:10.1038/s41593-018-0209-y
Mathis, A., Schneider, S., Lauer, J., and Mathis, M. W. (2020). A primer on motion capture with deep learning: Principles, pitfalls, and perspectives. Neuron 108, 44–65. doi:10.1016/j.neuron.2020.09.017
Mathis, M. W., and Mathis, A. (2020). Deep learning tools for the measurement of animal behavior in neuroscience. Curr. Opin. Neurobiol. 60, 1–11. doi:10.1016/j.conb.2019.10.008
Miao, H., Ueda, K., Pearson, T. S., and Aravamuthan, B. R. (2020). Automated objective dystonia identification using smartphone-quality gait videos acquired in clinic. medRxiv
Mitchell, S. L., Harper, D. W., Lau, A., and Bhalla, R. (2000). Patterns of outcome measurement in Parkinson’s disease clinical trials. Neuroepidemiology 19, 100–108. doi:10.1159/000026244
Moro, M., Casadio, M., Mrotek, L. A., Ranganathan, R., Scheidt, R., and Odone, F. (2021). On the precision of markerless 3d semantic features: An experimental study on violin playing, 2021 IEEE International Conference on Image Processing (ICIP). IEEE, 2733–2737.
Moro, M., Marchesi, G., Odone, F., and Casadio, M. (2020). Markerless gait analysis in stroke survivors based on computer vision and deep learning: A pilot study. In Proceedings of the 35th Annual ACM Symposium on Applied Computing. 2097–2104.
Namba, S., Matsui, H., and Zloteanu, M. (2021). Distinct temporal features of genuine and deliberate facial expressions of surprise. Sci. Rep. 11, 1–10. doi:10.1038/s41598-021-83077-4
Nath, T., Mathis, A., Chen, A. C., Patel, A., Bethge, M., and Mathis, M. W. (2019). Using deeplabcut for 3d markerless pose estimation across species and behaviors. Nat. Protoc. 14, 2152–2176. doi:10.1038/s41596-019-0176-0
Needham, L., Evans, M., Cosker, D. P., Wade, L., McGuigan, P. M., Bilzon, J. L., et al. (2021). The accuracy of several pose estimation methods for 3d joint centre localisation. Sci. Rep. 11, 1–11. doi:10.1038/s41598-021-00212-x
Nourizonoz, A., Zimmermann, R., Ho, C. L. A., Pellat, S., Ormen, Y., Prévost-Solié, C., et al. (2020). Etholoop: Automated closed-loop neuroethology in naturalistic environments. Nat. Methods 17, 1052–1059. doi:10.1038/s41592-020-0961-2
Pang, Y., Christenson, J., Jiang, F., Lei, T., Rhoades, R., Kern, D., et al. (2020). Automatic detection and quantification of hand movements toward development of an objective assessment of tremor and bradykinesia in Parkinson’s disease. J. Neurosci. methods 333, 108576. doi:10.1016/j.jneumeth.2019.108576
Papic, C., Sanders, R. H., Naemi, R., Elipot, M., and Andersen, J. (2021). Improving data acquisition speed and accuracy in sport using neural networks. J. Sports Sci. 39, 513–522. doi:10.1080/02640414.2020.1832735
Patel, S., Lorincz, K., Hughes, R., Huggins, N., Growdon, J., Standaert, D., et al. (2009). Monitoring motor fluctuations in patients with Parkinson’s disease using wearable sensors. IEEE Trans. Inf. Technol. Biomed. 13, 864–873. doi:10.1109/titb.2009.2033471
Patil, P. G., Carmena, J. M., Nicolelis, M. A., and Turner, D. A. (2004). Ensemble recordings of human subcortical neurons as a source of motor control signals for a brain-machine interface. Neurosurgery 55, 27–38. doi:10.1227/01.neu.0000126872.23715.e5
Pereira, T. D., Aldarondo, D. E., Willmore, L., Kislin, M., Wang, S. S.-H., Murthy, M., et al. (2019). Fast animal pose estimation using deep neural networks. Nat. Methods 16, 117–125. doi:10.1038/s41592-018-0234-5
Peterson, D. A., Littlewort, G. C., Bartlett, M. S., Macerollo, A., Perlmutter, J. S., Jinnah, H., et al. (2016). Objective, computerized video-based rating of blepharospasm severity. Neurology 87, 2146–2153. doi:10.1212/WNL.0000000000003336
Pérez, D. L., Laudańska, Z., Radkowska, A., Babis, K., Kozioł, A., and Tomalski, P. (2021). “Do we need expensive equipment to quantify infants’ movement? A cross-validation study between computer vision methods and sensor data,” in 2021 IEEE International Conference on Development and Learning (ICDL) (IEEE), 1–6.
Pouw, W., Trujillo, J. P., and Dixon, J. A. (2020). The quantification of gesture–speech synchrony: A tutorial and validation of multimodal data acquisition using device-based and video-based motion tracking. Behav. Res. methods 52, 723–740. doi:10.3758/s13428-019-01271-9
Pringsheim, T., Jette, N., Frolkis, A., and Steeves, T. D. (2014). The prevalence of Parkinson’s disease: A systematic review and meta-analysis. Mov. Disord. 29, 1583–1590. doi:10.1002/mds.25945
Procházka, A., Vyšata, O., Vališ, M., Ťupa, O., Schätz, M., and Mařík, V. (2015). Use of the image and depth sensors of the microsoft kinect for the detection of gait disorders. Neural comput. Appl. 26, 1621–1629. doi:10.1007/s00521-015-1827-x
Pulliam, C. L., Burack, M. A., Heldman, D. A., Giuffrida, J. P., and Mera, T. O. (2014). Motion sensor dyskinesia assessment during activities of daily living. J. Parkinson's. Dis. 4, 609–615. doi:10.3233/jpd-140348
Ramsperger, R., Meckler, S., Heger, T., van Uem, J., Hucker, S., Braatz, U., et al. (2016). Continuous leg dyskinesia assessment in Parkinson’s disease–clinical validity and ecological effect. Park. Relat. Disord. 26, 41–46. doi:10.1016/j.parkreldis.2016.02.007
Richards, M., Marder, K., Cote, L., and Mayeux, R. (1994). Interrater reliability of the unified Parkinson’s disease rating scale motor examination. Mov. Disord. 9, 89–91. doi:10.1002/mds.870090114
Rodriguez-Oroz, M. C., Rodriguez, M., Guridi, J., Mewes, K., Chockkman, V., Vitek, J., et al. (2001). The subthalamic nucleus in Parkinson’s disease: Somatotopic organization and physiological characteristics. Brain 124, 1777–1790. doi:10.1093/brain/124.9.1777
Roostaei, T., Nazeri, A., Sahraian, M. A., and Minagar, A. (2014). The human cerebellum: A review of physiologic neuroanatomy. Neurol. Clin. 32, 859–869. doi:10.1016/j.ncl.2014.07.013
Salarian, A., Russmann, H., Wider, C., Burkhard, P. R., Vingerhoets, F. J., and Aminian, K. (2007). Quantification of tremor and bradykinesia in Parkinson’s disease using a novel ambulatory monitoring system. IEEE Trans. Biomed. Eng. 54, 313–322. doi:10.1109/tbme.2006.886670
Sato, K., Nagashima, Y., Mano, T., Iwata, A., and Toda, T. (2019). Quantifying normal and parkinsonian gait features from home movies: Practical application of a deep learning–based 2d pose estimator. PloS one 14, e0223549. doi:10.1371/journal.pone.0223549
Schweihoff, J. F., Loshakov, M., Pavlova, I., Kück, L., Ewell, L. A., and Schwarz, M. K. (2021). Deeplabstream enables closed-loop behavioral experiments using deep learning-based markerless, real-time posture detection. Commun. Biol. 4, 1–11. doi:10.1038/s42003-021-01654-9
Sehara, K., Zimmer-Harwood, P., Larkum, M. E., and Sachdev, R. N. (2021). Real-time closed-loop feedback in behavioral time scales using deeplabcut. Eneuro 8, 0415. doi:10.1523/eneuro.0415-20.2021
Seidel, J., Bockhop, F., Mitkovski, M., Martin, S., Ronnenberg, A., Krueger-Burg, D., et al. (2020). Vascular response to social cognitive performance measured by infrared thermography: A translational study from mouse to man. FASEB BioAdvances 2, 18–32. doi:10.1096/fba.2019-00085
Sheshadri, S., Dann, B., Hueser, T., and Scherberger, H. (2020). 3d reconstruction toolbox for behavior tracked with multiple cameras. J. Open Source Softw. 5, 1849. doi:10.21105/joss.01849
Shin, J. H., Ong, J. N., Kim, R., Park, S.-m., Choi, J., Kim, H.-J., et al. (2020). Objective measurement of limb bradykinesia using a marker-less tracking algorithm with 2d-video in pd patients. Park. Relat. Disord. 81, 129–135. doi:10.1016/j.parkreldis.2020.09.007
Shin, J. H., Yu, R., Ong, J. N., Lee, C. Y., Jeon, S. H., Park, H., et al. (2021). Quantitative gait analysis using a pose-estimation algorithm with a single 2d-video of Parkinson’s disease patients. J. Parkinson’s Dis. 11, 1271–1283. doi:10.3233/jpd-212544
Stein, E., and Bar-Gad, I. (2013). Beta oscillations in the cortico-basal ganglia loop during parkinsonism. Exp. Neurol. 245, 52–59. doi:10.1016/j.expneurol.2012.07.023
Sterio, D., Zonenshayn, M., Mogilner, A. Y., Rezai, A. R., Kiprovski, K., Kelly, P. J., et al. (2002). Neurophysiological refinement of subthalamic nucleus targeting. Neurosurgery 50, 58–69. doi:10.1097/00006123-200201000-00012
Stolk, K., Schwartz, M., van der Krogt, M., van de Ven, S., Bonouvrié, L., Harlaar, J., et al. (2020). Feature selection from markerless movement recordings to assess dystonia in children with cerebral palsy. Gait Posture 81, 354–355. doi:10.1016/j.gaitpost.2020.08.075
Tankus, A., Mirelman, A., Giladi, N., Fried, I., and Hausdorff, J. M. (2018). Pace of movement: The role of single neurons in the subthalamic nucleus. J. Neurosurg. 130, 1835–1840. doi:10.3171/2018.1.jns171859
Tankus, A., Strauss, I., Gurevich, T., Mirelman, A., Giladi, N., Fried, I., et al. (2017). Subthalamic neurons encode both single-and multi-limb movements in Parkinson’s disease patients. Sci. Rep. 7, 1–10. doi:10.1038/srep42467
Tekriwal, A., Afshar, N. M., Santiago-Moreno, J., Kuijper, F. M., Kern, D. S., Halpern, C. H., et al. (2019). Neural circuit and clinical insights from intraoperative recordings during deep brain stimulation surgery. Brain Sci. 9, 173. doi:10.3390/brainsci9070173
Tekriwal, A., Felsen, G., Ojemann, S. G., Abosch, A., and Thompson, J. A. (2022). Motor context modulates substantia nigra pars reticulata spike activity in patients with Parkinson’s disease. J. Neurol. Neurosurg. Psychiatry 93, 386–394. doi:10.1136/jnnp-2021-326962
Tekriwal, A., Felsen, G., and Thompson, J. A. (2018). Modular auditory decision-making behavioral task designed for intraoperative use in humans. J. Neurosci. Methods 304, 162–167. doi:10.1016/j.jneumeth.2018.05.004
Tsunematsu, T., Patel, A. A., Onken, A., and Sakata, S. (2020). State-dependent brainstem ensemble dynamics and their interactions with hippocampus across sleep states. Elife 9, e52244. doi:10.7554/elife.52244
van Schaik, J. E., and Dominici, N. (2020). Motion tracking in developmental research: Methods, considerations, and applications. Prog. Brain Res. 254, 89–111. doi:10.1016/bs.pbr.2020.06.007
Vonstad, E. K., Su, X., Vereijken, B., Bach, K., and Nilsen, J. H. (2020). Comparison of a deep learning-based pose estimation system to marker-based and kinect systems in exergaming for balance training. Sensors 20, 6940. doi:10.3390/s20236940
Voulodimos, A., Doulamis, N., Doulamis, A., and Protopapadakis, E. (2018). Deep learning for computer vision: A brief review. Comput. Intell. Neurosci. 2018, 7068349. doi:10.1155/2018/7068349
Williams, S., Relton, S. D., Fang, H., Alty, J., Qahwaji, R., Graham, C. D., et al. (2020a). Supervised classification of bradykinesia in Parkinson’s disease from smartphone videos. Artif. Intell. Med. 110, 101966. doi:10.1016/j.artmed.2020.101966
Williams, S., Zhao, Z., Hafeez, A., Wong, D. C., Relton, S. D., Fang, H., et al. (2020b). The discerning eye of computer vision: Can it measure Parkinson’s finger tap bradykinesia? J. Neurological Sci. 416, 117003. doi:10.1016/j.jns.2020.117003
Wong, D. C., Relton, S. D., Fang, H., Qhawaji, R., Graham, C. D., Alty, J., et al. (2019). “Supervised classification of bradykinesia for Parkinson’s disease diagnosis from smartphone videos,” in 2019 IEEE 32nd International Symposium on Computer-Based Medical Systems (CBMS) (IEEE), 32–37.
Wu, J. J.-S., Hung, A., Lin, Y.-C., and Chiao, C.-C. (2020). Visual attack on the moving prey by cuttlefish. Front. Physiology 11, 648. doi:10.3389/fphys.2020.00648
Zavala, B., Damera, S., Dong, J. W., Lungu, C., Brown, P., and Zaghloul, K. A. (2017). Human subthalamic nucleus theta and beta oscillations entrain neuronal firing during sensorimotor conflict. Cereb. Cortex 27, 496–508. doi:10.1093/cercor/bhv244
Zdarsky, N., Treue, S., and Esghaei, M. (2021). A deep learning-based approach to video-based eye tracking for human psychophysics. Front. Hum. Neurosci. 15, 685830. doi:10.3389/fnhum.2021.685830
Zhang, L., Dunn, T., Marshall, J., Olveczky, B., and Linderman, S. (2021). “Animal pose estimation from video data with a hierarchical von mises-Fisher-Gaussian model,” in International Conference on Artificial Intelligence and Statistics (PMLR), 2800–2808.
Zhao, Z., Fang, H., Williams, S., Relton, S. D., Alty, J., Casson, A. J., et al. (2020). “Time series clustering to examine presence of decrement in Parkinson’s finger-tapping bradykinesia,” in 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) (IEEE), 780–783.
Zhou, H., and Hu, H. (2008). Human motion tracking for rehabilitation—A survey. Biomed. signal Process. control 3, 1–18. doi:10.1016/j.bspc.2007.09.001
Keywords: movement disorders, markerless motion tracking, deep learning, deeplabcut, deep brain stimulation, intraoperative electrophysiology, Parkinson’s disease, essential tremor
Citation: Tien RN, Tekriwal A, Calame DJ, Platt JP, Baker S, Seeberger LC, Kern DS, Person AL, Ojemann SG, Thompson JA and Kramer DR (2022) Deep learning based markerless motion tracking as a clinical tool for movement disorders: Utility, feasibility and early experience. Front. Sig. Proc. 2:884384. doi: 10.3389/frsip.2022.884384
Received: 26 February 2022; Accepted: 05 September 2022;
Published: 29 September 2022.
Edited by:
Adriano De Oliveira Andrade, Federal University of Uberlândia, BrazilReviewed by:
Kyriaki Kostoglou, Graz University of Technology, AustriaCopyright © 2022 Tien, Tekriwal, Calame, Platt, Baker, Seeberger, Kern, Person, Ojemann, Thompson and Kramer. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Rex N. Tien, cmV4LnRpZW5AY3VhbnNjaHV0ei5lZHU=
†These authors share senior authorship