Sensorimotor Decoding: Characterization and Modeling for Rehabilitation and Assistive Technologies

Advancements in the characterization and modeling of neural activity in the Research Topic sensorimotor decoding. The Research Topic on sensorimotor activity has brought together studies that utilize natural and artificial stimulation, providing new insights into the encoding of sensorimotor activity under normal and altered conditions. Furthermore, various decoding approaches, including sparse algorithms, connectivity analysis, ERD/ERS analysis, power and power ratios, linear decoders and deep learning, have been employed to decode sensorimotor activity and, ultimately, generate motor responses, motor imagery, and BCI control.

Editorial

18 July 2023

Editorial: Sensorimotor decoding: characterization and modeling for rehabilitation and assistive technologies

Miguel Pais-Vieira

Tetiana Aksenova

Vassiliy Tsytsarev

and

Jean Faber

1,385 views

0 citations

Editors

Jean Faber

Neuroscience Division, Department of Neurology and Neurosurgery, Universidade Federal de São Paulo, Escola Paulista de Medicina

Tetiana Aksenova

CEA LETI

Miguel Pais-Vieira

Institute of Biomedicine, Department of Medical Sciences, University of Aveiro

Vassiliy Tsytsarev

School of Medicine, University of Maryland

Impact

Original Research

23 June 2023

Overt speech decoding from cortical activity: a comparison of different linear methods

Gaël Le Godais

, 5 more and

Blaise Yvert

Introduction: Speech BCIs aim at reconstructing speech in real time from ongoing cortical activity. Ideal BCIs would need to reconstruct speech audio signal frame by frame on a millisecond-timescale. Such approaches require fast computation. In this respect, linear decoder are good candidates and have been widely used in motor BCIs. Yet, they have been very seldomly studied for speech reconstruction, and never for reconstruction of articulatory movements from intracranial activity. Here, we compared vanilla linear regression, ridge-regularized linear regressions, and partial least squares regressions for offline decoding of overt speech from cortical activity.

Methods: Two decoding paradigms were investigated: (1) direct decoding of acoustic vocoder features of speech, and (2) indirect decoding of vocoder features through an intermediate articulatory representation chained with a real-time-compatible DNN-based articulatory-to-acoustic synthesizer. Participant's articulatory trajectories were estimated from an electromagnetic-articulography dataset using dynamic time warping. The accuracy of the decoders was evaluated by computing correlations between original and reconstructed features.

Results: We found that similar performance was achieved by all linear methods well above chance levels, albeit without reaching intelligibility. Direct and indirect methods achieved comparable performance, with an advantage for direct decoding.

Discussion: Future work will address the development of an improved neural speech decoder compatible with fast frame-by-frame speech reconstruction from ongoing activity at a millisecond timescale.

3,069 views

4 citations

$Information transfer networks during width discrimination. The upper half of the figure depicts the results from the analysis of Granger causality tests performed independently for each of the blocks analyzed (“between-subjects”). The lower half of the figure depicts the comparison between the first and the second blocks for each subject (“within-subjects”). Analysis of Granger causality was performed only for the C3 electrode and for the electrodes that presented significant correlation with performance. (A–G) The arrows depict the proportion of significant Granger causality tests. Wider and darker arrows indicate an increased fraction of blocks with a significant Granger causality test between two electrodes. Thinner and brighter arrows indicate a decreased fraction of significant Granger tests. The largest number of significant Granger causality tests occurred from the electrode Fp2 to the electrode T4 (90% of the blocks analyzed) and O2 (80% of the blocks) (A), followed by the electrode C3 sending information to electrodes Fp2, P3, and O2 in 80% of the blocks (C), and also electrode O2 sending information to electrode F4 (80% of the blocks) (G). Electrode P4 generally presented the smallest number of significant tests. (H) Summary of the main significant connections for the network of electrodes present in at least 80% of the blocks. The pattern of connectivity most common between electrodes was C3 electrode sending information to electrodes Fp2 (80%), P3 (80%), and O2 (80%); Fp2 electrode sending information to electrodes O2 (80%) and T4 (90%), and lastly, electrode O2 sending information to F4 (80%). (I–O) The arrows depict the changes in the number of significant Granger causality tests between two blocks of the same subject. The wider arrows with increased color saturation represent the fraction of subjects with an increase (red) or decrease (blue) in significant Granger causality between the two blocks. (I–O) Changes in improvement between blocks were characterized by increases or decreases in information in specific pairs of electrodes that led to networks of information transfer that were closer to the network of information transfer for tactile width processing. The main changes observed involved increasing information transfer between electrodes Fp2-O2 (bidirectionally), F4-P3 (bidirectionally), C3-F4, and C3-P3 (bidirectionally). Decreases in information most often involved electrode bidirectional decreases of information transfer between P4 and P3, F4, and T4. (P) Summary of the main significant connections for the information transfer network predicting improvement between different subjects highlighting increases in C3, Fp2, F4, T4, P4 and O2; and decreases in information transfer between P4 and F4, T4, and P3.$

Original Research

12 May 2023

Neurophysiological correlates of tactile width discrimination in humans

Carla Pais-Vieira

, 6 more and

Miguel Pais-Vieira

1,970 views

2 citations

Visualization of 2D embedding of left and right hand data obtained using UMAP. Green dashed line showes SVM decision boundary.

Original Research

16 March 2023

Impact of dataset size and long-term ECoG-based BCI usage on deep learning decoders performance

Maciej Śliwowski

, 3 more and

Tetiana Aksenova

Introduction: In brain-computer interfaces (BCI) research, recording data is time-consuming and expensive, which limits access to big datasets. This may influence the BCI system performance as machine learning methods depend strongly on the training dataset size. Important questions arise: taking into account neuronal signal characteristics (e.g., non-stationarity), can we achieve higher decoding performance with more data to train decoders? What is the perspective for further improvement with time in the case of long-term BCI studies? In this study, we investigated the impact of long-term recordings on motor imagery decoding from two main perspectives: model requirements regarding dataset size and potential for patient adaptation.

Methods: We evaluated the multilinear model and two deep learning (DL) models on a long-term BCI & Tetraplegia (ClinicalTrials.gov identifier: NCT02550522) clinical trial dataset containing 43 sessions of ECoG recordings performed with a tetraplegic patient. In the experiment, a participant executed 3D virtual hand translation using motor imagery patterns. We designed multiple computational experiments in which training datasets were increased or translated to investigate the relationship between models' performance and different factors influencing recordings.

Results: Our results showed that DL decoders showed similar requirements regarding the dataset size compared to the multilinear model while demonstrating higher decoding performance. Moreover, high decoding performance was obtained with relatively small datasets recorded later in the experiment, suggesting motor imagery patterns improvement and patient adaptation during the long-term experiment. Finally, we proposed UMAP embeddings and local intrinsic dimensionality as a way to visualize the data and potentially evaluate data quality.

Discussion: DL-based decoding is a prospective approach in BCI which may be efficiently applied with real-life dataset size. Patient-decoder co-adaptation is an important factor to consider in long-term clinical BCI.

3,520 views

7 citations