Nicolas J. Bourguignon
Salvatore Lo Bue1- 1Department of Life Sciences, Royal Military Academy of Belgium, Brussels, Belgium
- 2GTM-Grup de Recerca en Technologies Mèdia, La Salle-Universitat Ramon Llull, Barcelona, Spain
- 3Center for Research in Cognition and Neuroscience, Université Libre de Bruxelles, Brussels, Belgium
Neuroergonomics focuses on the brain signatures and associated mental states underlying behavior to design human-machine interfaces enhancing performance in the cognitive and physical domains. Brain imaging techniques such as functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) have been considered key methods for achieving this goal. Recent research stresses the value of combining EEG and fNIRS in improving these interface systems' mental state decoding abilities, but little is known about whether these improvements generalize over different paradigms and methodologies, nor about the potentialities for using these systems in the real world. We review 33 studies comparing mental state decoding accuracy between bimodal EEG-fNIRS and unimodal EEG and fNIRS in several subdomains of neuroergonomics. In light of these studies, we also consider the challenges of exploiting wearable versions of these systems in real-world contexts. Overall the studies reviewed suggest that bimodal EEG-fNIRS outperforms unimodal EEG or fNIRS despite major differences in their conceptual and methodological aspects. Much work however remains to be done to reach practical applications of bimodal EEG-fNIRS in naturalistic conditions. We consider these points to identify aspects of bimodal EEG-fNIRS research in which progress is expected or desired.
Introduction
Neuroergonomics is the branch of human factors concerned with the live reading of brain functions underlying real-world human performance in the physical (e.g., walking/running, manipulating tools, driving) and cognitive domains (e.g., calculating, reasoning, communicating). Its main goal is to design human-machine interfaces to enable autonomous daily behaviors in people with movement or communication disorders or to ensure the efficiency and safety of complex, high-risk private or professional activities (e.g., flying, driving, operating machines, etc.). Achieving this goal not only requires understanding how brain activity represents performance-related mental states such as goals (e.g., intending to go left or right), feelings and emotions (e.g., stress or anxiety), or mental/physical effort (e.g., workload, exhaustion, etc.) but also on brain monitoring techniques that can decode these mental states quickly and accurately. Existing techniques with such potential include electroencephalography (EEG), which measures the brain's temporal unfolding of electrophysiological activity with a millisecond accuracy (Casson, 2019), and functional near-infrared spectroscopy (fNIRS), which enables precise localization of brain activity during performance (Ferrari and Quaresima, 2012; Vitorio et al., 2017; Quaresima and Ferrari, 2019; Zhu et al., 2020). A cousin of hemodynamic brain monitoring techniques such as positron emission tomography and fMRI, fNIRS uses the diffusion of infrared light emitted through the skull and refracted at various intensities depending on the concentration of (de)oxygenated hemoglobin (see Pinti et al., 2018a and Pan et al., 2019 for details). The possibility to acquire this signal through sensors placed on the scalp makes fNIRS significantly more portable than PET or fMRI, earning it increasing attention in many sectors of neuroergonomics and human factors design.
As an emergent discipline, neuroergonomics faces two important challenges. One is to miniaturize brain monitoring techniques in a way that allows for natural evolution in the real world without jeopardizing brain signal quality. Indeed, despite their higher flexibility of use compared with other techniques, both EEG and fNIRS remain sensitive to movement artifacts likely to contaminate brain signal. Another concerns the inherent temporal and spatial limitations of fNIRS and EEG, respectively. Like fMRI, fNIRS suffers an approximate 5-second delay between stimulus presentation and associated brain responses (Pinti et al., 2018a). This discrepancy poses a problem for studying the brain bases of performance in naturalistic conditions where events tend to happen quickly and unpredictably. Conversely, EEG's limited spatial resolution does not allow for accurate identification of brain regions at the source of task-relevant brain signal. Finally, fNIRS only provides partial information on the processes taking place within task-relevant brain regions, whereas electrode recordings permit finer-grained analyses of these processes in terms of timing and local or global neuronal interactions (Logothetis et al., 2001; Palva and Palva, 2012).
To overcome these challenges, recent efforts have been made to combine EEG and fNIRS into bimodal systems able the simultaneously record the hemodynamic and electrophysiological correlates of human performance in real time (Safaie et al., 2013; Tomita et al., 2014; Hong and Khan, 2017; von Luhmann and Muller, 2017; Hong et al., 2018). Indeed, besides improving the signal-to-noise ratio by representing brain activity in different formats (Sun et al., 2020), combined EEG and fNIRS signals have the potential to significantly compensate for each other's spatial and temporal limitations, thereby increasing the speed, precision, and richness of mental state decoding in various tasks of interest to human factors (Wallois et al., 2010, 2012; Nguyen et al., 2012, 2017; Tomita et al., 2014; Kaewkamnerdpong, 2016; Balconi et al., 2017; Pinti et al., 2018b; Dehais et al., 2019; Firooz and Setarhdan, 2019). They may ultimately provide motor-disabled individuals with the means to interact with their environment and optimize the detection of mental workload, drowsiness, or brain dysfunction in various high-risk situations.
Given the recency of bimodal EEG-fNIRS technologies, and despite several reports of their higher decoding accuracy compared to unimodal EEG and fNIRS, it is still unclear whether these improvements generalize over most processes typically investigated in human-machine interface research or whether they relate to particular domains of performance and not others (e.g., whether its decoding accuracy is higher for cognitive relative to physical performance). Another pending question is whether these improvements are stable despite variations across studies' methodology. To help elucidate these questions the present article reviews studies comparing the performance of bimodal EEG-fNIRS against unimodal EEG and unimodal fNIRS in performance-based mental state decoding in the context of key paradigms of neuroergonomics: motor imagery and execution for remote-controlled action, navigation for safe and efficient driving or flying, clinical diagnosis for efficient healthcare services, as well as cognitive and affective processing for detecting potentially harmful cognitive or affective states (e.g., stress, mental workload, drowsiness, etc.). We consider the relative value of bimodal vs. unimodal EEG and fNIRS in light of these studies' primary conceptual and methodological differences in order to identify areas in which improvement is expected or desired in the coming years. We also consider the state of progress achieved in building wearable versions of bimodal EEG-fNIRS equipment to be used in naturalistic conditions. These considerations provide the basis for recommendations as to how future neuroergonomics research should be carried out using multimodal brain imaging.
Methods
Given the recency and sparseness of mental state-decoding studies using bimodal EEG-fNIRS, we sought to maximize the chances of finding relevant studies by departing from the standards of systematic literature reviews (e.g., Moher et al., 2009, see Figure 1 for a graphical representation of the study selection process). This review should therefore not be regarded as systematic. The literature search was initiated in 2020 and terminated in 2021. A Google Scholar search was performed via the Publish or Perish search engine (Harzing, 2007, Publish or Perish, available https://harzing.com/resources/publish-or-perish) with the following keywords: “EEG-fNIRS” AND “simultaneous” OR “concurrent” AND “bimodal” OR “multimodal” OR “feature classification”, yielding a total of 455 relevant titles. Additional searches were performed using the same engine in PubMed and Web of Science using the keyword “EEG-fNIRS”, yielding 94 titles for PubMed and 154 titles for Web of Science. Prospective studies were reviewed based on the following criteria: (a) studies performed on human subjects, (b) research contributing to key sectors of neuroergonomics research, (c) description of key methodological points, specifically: (1) the number of subjects tested, (2) sensor montage and type of hardware used (i.e. custom-made vs. commercially distributed sensors), (3) head coverage specifying which parts of the brain were targeted, (4) the sampling parameters for EEG and fNIRS, (5) the EEG and fNIRS features of interest extracted and (6) the feature extraction method used (this information is provided in Table 1), (d) direct, quantitative comparisons of mental state classification accuracy between unimodal EEG, unimodal fNIRS and bimodal EEG-fNIRS.
Figure 1. Flowchart of the literature search.
Table 1. Target methodological details of the studies included in the review.
Complementing these criteria were the following three selection filters: First, when two articles reported results from the same dataset (e.g., Fazli et al., 2012; Lee et al., 2014; Ahn et al., 2016; Nguyen et al., 2017; Shin et al., 2017) only one of these articles was retained to avoid redundancy or selection biases. Second, when the same dataset was reported in peer-reviewed articles and conference proceedings (Fazli et al., 2012; Lee et al., 2014) only the peer-reviewed articles were retained. Finally, studies performed on simulated datasets (e.g., Croce et al., 2017) were excluded. As most of the research using bimodal EEG-fNIRS is still quite recent, the year of publication was not used as an inclusion/exclusion criterion. In total 22 articles were selected for inclusion in the review. In addition to the database search described above, each of the 22 articles found was then screened for its reference section and recent contributions citing it. Eleven additional articles were found that met the above criteria. A total of 33 research articles were therefore included in the review.
To better capture and discuss the conceptual, empirical, and methodological implications of the studies reviewed here, the following information was systematically extracted for each of them (see Table 1 for detail):
1. Paradigm. Studies were grouped into seven categories based on their paradigms. These categories were created in a way that best encapsulates the main domains of performance investigated in the literature. These include: (a) motor imagery (10 studies), (b) motor execution (seven studies), (c) navigation (simulated and/or real-world, 2 studies), (d) cognitive processing (working memory, word generation, mental arithmetic, spatial attention, 8 studies), (e) affective/emotional processing (2 studies) and (f) clinical (5 studies). An additional category ‘other' was created to include one study focused on discriminating mental states related to different tasks belonging to one of the six previous categories as well as one study aimed at discriminating between auditory and visual processing (Putze et al., 2014). It must be noted that some of the studies included in the review involved more than one task that could be assigned to different categories. Furthermore, one study involving clinical subjects differed from the other clinical studies in being focused on mental state decoding during motor imagery in patients and healthy controls (i.e. was not run for diagnostic purposes, cf. Blokland et al., 2014). Accordingly, this study was assigned to the motor imagery category, ensuring that the results of the two groups were given separately.
Naturally, this categorization can to a large extent be regarded as arbitrary. Alternative categorizations might help place greater emphasis on other equally relevant, possibly more specific aspects of performance such as mental workload, engagement, cognitive fatigue, or drowsiness. Similarly, the studies reviewed might be grouped in a way that more explicitly underlines the distinction between “physical” and “cognitive” performance. Several reasons however motivate the categorization proposed here. First, many of these alternative dimensions of performance are embedded in the context of more general activities in a way that makes it difficult to distinguish them. EEG-fNIRS studies on mental fatigue or drowsiness, for instance, have typically been carried out in the context of navigation tasks. Similarly, even the most elementary tasks of physical performance entail some level of cognitive processing (e.g., deciding to move left or right), while classical cognitive tasks such as working memory span or arithmetic are typically translated into physical responses. It is therefore not obvious where the cognitive dimension of performance ends and the physical one begins. Finally, the way the studies are categorized does not for the present purposes significantly affect the primary finding of the present review, as will be shown in the next section. For these reasons, we elected to consider these dimensions in more detail when commenting on the results of our analysis.
2. Study parameters, including the number of subjects, specifics on the fNIRS and EEG sensor setup, EEG and fNIRS sampling parameters (EEG: sampling rate, fNIRS: wavelength and sampling rate), EEG and fNIRS features of interest, as well as a detailed description of data preprocessing (including both feature extraction and feature classification).
3. Mental state classification methods, with a special focus on the machine learning algorithms used for decoding performance-related mental states.
Description of Studies
The penultimate column in Table 2 provides the percentage mental state decoding accuracy between unimodal EEG, unimodal fNIRS, and bimodal EEG-fNIRS for every study reviewed. The reader is also referred to Table 1 for further methodological information on each study, to be discussed in more detail below. Despite substantial differences between the studies' paradigms and methodology results reveal consistently better classification accuracy for bimodal EEG-fNIRS compared to unimodal EEG or fNIRS. In the following we consider in detail the primary methodological differences between studies that are likely to affect bimodal EEG-fNIRS mental state classification accuracy, focusing on their paradigms, sensor layout, and signal pre-processing as well as EEG and fNIRS feature extraction, integration and classification methods for mental state decoding. We then examine the studies' level of ecological validity, which should help consider the main challenges and existing solutions for implementing bimodal EEG-fNIRS experiments outside the laboratory.
Table 2. Articles included in the review.
Paradigms
As shown in Figure 2, bimodal EEG-fNIRS yields consistent improvements in decoding accuracy compared to unimodal EEG and NIRS regardless of paradigm (we note that Figure 2 is provided only for visual observation and does not represent any statistical inference due to the relatively small number of studies included in each category). This section is devoted to commenting on some of the key aspects of the studies carried out within each of these paradigm categories.
Figure 2. Differences in decoding accuracy between unimodal NIRS, unimodal EEG, and bimodal EEG-fNIRS across the seven paradigms considered. Scores were obtained by averaging percent decoding accuracy scores reported for each method by the studies reviewed in each category (see Tables 1, 2 for detail). When more than one possible score was obtained for the same individual method (e.g., when comparing different decoding algorithms), the highest score was systematically chosen. Note that these graphs are displayed for visual presentation and not statistical inference.
Most studies were performed within research on brain-computer interface technologies (BCI) seeking to equip motor-disabled individuals with the means to communicate and interact with their environment (Allison et al., 2007; Ahn and Jun, 2018). Unsurprisingly, thus, many studies used tasks of motor imagery (11 studies) and motor execution (seven studies). Results show that bimodal EEG-fNIRS outperforms unimodal EEG and fNIRS in both cases, but bimodal classification appears more accurate in motor execution (~85%) compared to motor imagery tasks (~80%). This discrepancy could in part be explained by differences in the brain signatures associated with executed vs. imagined movements. Indeed, the hemodynamic signal appears more diffuse and weaker during motor imagery compared to motor execution, with limited involvement of primary motor regions (Deiber et al., 1998; Dechent et al., 2004). EEG studies comparing motor imagery vs. executed movements nevertheless reveal overall similar electrophysiological signatures in the motor regions between the two conditions, though classification in these regions was still highest during executed movements (Neuper et al., 2005). These studies also report significantly different activation profiles in the motor region depending on whether subjects are asked to imagine themselves or another person performing a motor action (Neuper et al., 2005), which highlights the impact of task demands and task instructions in driving classification accuracy. Further progress might therefore be partly contingent on a deeper understanding of the differences between motor imagery and motor execution at both the hemodynamic and electrophysiological levels.
Interestingly, despite the focus on the usefulness of EEG-fNIRS technologies to motor-disabled individuals, only one study actually involved patients with motor disorders (tetraplegia, cf. Blokland et al., 2014). Bimodal EEG-fNIRS in patients yielded higher classification accuracy than unimodal EEG or fNIRS for both imagined movement (EEG-fNIRS: 70%, fNIRS: 65%, EEG: 63%) and attempted movements (EEG-fNIRS: 79%, fNIRS: 70%, EEG: 73%). This finding highlights the potential use of EEG-fNIRS as an implementable neuroprosthetic technology. Other studies involving clinical populations were aimed at discriminating patients with neurological conditions such as Parkinson's disease (Abtahi et al., 2020), ADHD (Güven et al., 2020), Alzheimer's disease (Cicalese et al., 2020), or acute brain injury (Othman et al., 2020) from neurotypical controls, mostly within tasks targeting the neurological condition of interest (motor control in PD, attentional control in ADHD, memory retrieval in AD). These studies mostly showed higher classification accuracy overall for bimodal EEG-fNIRS (~90%) compared to unimodal EEG (~83%) or unimodal fNIRS (~77%), illustrating the utility of EEG-fNIRS for diagnostic purposes. One interesting exception is the study by Abtahi et al. (2020) investigating the accuracy of discrimination between PD patients and neurotypical controls based on unimodal EEG, unimodal fNIRS, bimodal EEG-fNIRS and bimodal EEG-fNIRS information fused with data from motion capture (MoCap) and WearUp flex sensors measuring large and small range hand movements, respectively. This study showed that while bimodal EEG-fNIRS does not perform better than unimodal EEG in discriminating PD from controls (92.27% for fNIRS compared to 92.79% for EEG), data from MoCap and WearUp measurements added to EEG-fNIRS significantly improved discrimination accuracy (93.4%). This suggests that additional information related to particular neurological conditions can further complement information from EEG-fNIRS to improve diagnostic accuracy.
Given the implications of neuroergonomics research in matters of public safety, another relevant sector interest for bimodal EEG-fNIRS technologies is the measurement of cognitive fatigue and engagement during navigation (driving and flying, Ahn et al., 2016; Dehais et al., 2019). The neural correlates of cognitive fatigue have been an important area of neuroergonomics research using EEG (Lal and Craig, 2001; Wascher et al., 2014; Mu et al., 2017), and several studies have now been conducted on cognitive fatigue using fNIRS (Borragán et al., 2018, 2019; Tanveer et al., 2019). A widely cited study in this respect reported higher accuracy of EEG-fNIRS in discriminating driving under well-rested vs. sleep-deprived conditions (Ahn et al., 2016). However, these authors also combined EEG-fNIRS with electrocardiography (ECG), achieving even higher discrimination compared to bimodal EEG-fNIRS (76% against 68.3%). Another study by Dehais et al. (2019) showed that EEG-fNIRS could appropriately decode levels of mental fatigue in pilots during simulated and real flight conditions, with little to no accuracy differences between the two (87.2% in simulation vs. 87.6% in a light aircraft). Altogether these findings not only stress the benefits of using bimodal EEG-fNIRS in parallel with other relevant physiological features to improve classification performance but also open the possibility to equip modern navigation systems with neurophysiological tools apt to precisely detect harmful levels of fatigue in moderate- to high-risk navigation situations.
Another central concept handled in half of the studies included in the “cognitive” category is that of mental workload—defined as the amount of information to be retained in working memory for prospective action (Krueger, 1989; Wickens, 2008; Loeppke et al., 2009; Mizuno et al., 2011; Raslear et al., 2011; Sievertsen et al., 2016). Here again, these studies report that bimodal EEG-fNIRS performs better (~85%) than EEG (~77%) or fNIRS alone (~67%) in decoding mental workload. It must be noted however that cognitive load is a multiplex concept encompassing several distinct but highly interactive components (Cain, 2007; Wickens, 2008). Beyond factors related to stimulus or task information, other intrinsic factors such as stress or emotional state (Al-Shargie et al., 2016) are also susceptible to modulating mental workload in multiple ways, as reflected in both hemodynamic and electrophysiological responses. For instance, Al-Shargie et al. (2016) investigated the neurocognitive correlates of stress in a task designed to tax cognitive processing. Their results revealed superior accuracy in stress identification for the bimodal technique (95.1%) compared to unimodal EEG (91.7%) and unimodal fNIRS (84.1 %). Altogether these studies highlight the practical payoffs of bimodal EEG-fNIRS in revealing the multiple facets of mental workload.
Methodology
Studies varied substantially in terms of their sensor setups, signal recording parameters, and mental state classification methods. To better present the implications of these methodological choices in bimodal EEG-fNIRS research we chose to describe them following a typical EEG-fNIRS recording-to-analysis pipeline represented in Figure 3 (for other examples see Hong and Khan, 2017). At each of the steps along this pipeline, we will describe the main options chosen in the studies reviewed, considering some of the advantages and challenges in the effort to achieve higher mental state decoding accuracy.
Figure 3. A standard acquisition and (pre-)processing pipeline using bimodal EEG-fNIRS systems for performance-related mental state classification.
Sensor Setups, Scalp Coverage, and Signal Denoising
Studies varied substantially in their EEG and fNIRS sensor setup, notably regarding the number of sensors used and scalp coverage. In most cases, EEG and fNIRS channels were laid out on the same scalp locations under the assumption that they should record activity from the same underlying cortical sites. For instance, studies targeting hand-related motor movements mostly placed both EEG and fNIRS channels in the vicinity of the motor cortex (e.g., Leamy and Ward, 2010; Leamy et al., 2011; Blokland et al., 2014). Studies measuring mental workload focused on frontal regions typically associated with attention and working memory (Coffey et al., 2012; Herff et al., 2015), and studies investigating auditory vs. visual discrimination placed their EEG/fNIRS channels in regions corresponding to the primary auditory vs. visual cortex (Putze et al., 2014). Although these targeted placements make sense it should be remembered that EEG and fNIRS differ substantially in their spatial resolution. Indeed, despite a universal appeal to standardized layouts supposed to relate electrode locations to underlying cortical areas (e.g., the international 10–20 system), the notoriously poor spatial precision of traditional electrophysiological methods impede the mapping of EEG responses to functional anatomic patterns revealed by hemodynamic techniques (Laureys et al., 2009). In many cases, the scalp distribution of certain well-known EEG responses can be quite misleading as to their actual cortical sources (Halgren et al., 2002; Osterhout et al., 2004; Lau et al., 2008). Several methodological tools have been developed to overcome this mismatch. These include the recording of EEG signal from a larger number of electrodes, enabling source reconstruction of the signal through multiple dipole modeling (Darvas et al., 2006), or 3D registration of EEG signal onto high-resolution MRI images (Yoo et al., 1997). The latter method has already been explored in research combining EEG and fNIRS signals acquired simultaneously (Aihara et al., 2012; Morioka et al., 2014), with significant improvements in decoding accuracy (see Feature Extraction and Integration for more detail). Crucially, the possibility to incorporate source reconstruction methods into bimodal EEG-fNIRS research is contingent on the specific goals and constraints of individual studies. They may turn out particularly difficult to implement in studies seeking to minimize the number of sensors for wearability or convenience (e.g., in BCI technologies). Progress in developing source reconstruction from low-density EEG coverage (e.g., Guevara et al., 2020) or using anatomical priors from the fNIRS signal (Aihara et al., 2012), see also Feature Extraction and Integration) may provide a partial solution to these constraints and help further increase precision in the correspondence between fNIRS and EEG source localization.
Most studies in the review made use of custom-built EEG-fNIRS sensor setups from separate EEG and fNIRS equipment. However, such setups have been shown to create higher electrical interferences or noise in the EEG signal due to inadequate shielding of NIRS optode circuits and suboptimal return current paths (von Luhmann and Muller, 2017). Additionally, these setups typically involve using separate EEG and fNIRS event files that need to be fused after data acquisition—a procedure likely to alter synchronicity between them and therefore affect precision in event-timing reconstruction (von Luhmann and Muller, 2017). Built-in hybrid EEG-fNIRS systems on the other hand significantly reduce both issues (von Luhmann and Muller, 2017). Several such systems are now commercially available (example systems include Medelopt© by Seenel Imaging [cf. Safaie et al., 2013 and https://seenel-imaging.com/], Brite© by Artinis Medical Systems [cf. https://www.artinis.com/], g.Nautilus© by Gtec [cf. https://www.gtec.at] or NIRScout by NIRx [cf. https://nirx.net/], see also von Lühmann et al., 2017 and Kassab et al., 2018) and offer miniaturized, wireless modular technology allowing for greater mobility and flexibility of use across a wide range of real-life situations.
Feature Extraction and Integration
Bimodal EEG-fNIRS provides substantial amounts of complementary information that can be exploited by advanced signal processing algorithms for mental state classification. A precondition for accurate mental state decoding is the choice and appropriate extraction and integration of EEG and fNIRS features (Hong et al., 2018). Feature extraction implies transforming the raw signal into numerical information (i.e., “features”) that can be processed while preserving the essential information of the original data set. This information can be obtained from the time domain (mean, standard deviation, entropy, etc.), frequency domain (Fourier transform, wavelets, time-frequency distributions, etc.), or synchronicity between two or more spatial channels (coherence, correlation, mutual information, etc.). Naturally, efficient mental state decoding significantly depends on the proper methods of EEG and/or fNIRS feature extractions and classification. This particular aspect is the one in which the studies reviewed here vary the most, revealing the widespread selection of procedures available for achieving this crucial step in mental state decoding using bimodal EEG-fNIRS methodologies. Since no consensus exists as yet about which of these procedures are to be recommended, this point will be devoted to describing them in a way that highlights their primary distinctive characteristics.
EEG features. The most common EEG features selected for classification are the four major EEG frequency bands—i.e., delta (< 3 Hz), theta (3–8 Hz), alpha (and its motor homolog mu rhythm, 8–13Hz), and beta frequency (>13 Hz)—and their various subcategories (e.g., low/high beta (Abtahi et al., 2020), low/high alpha (Cicalese et al., 2020). Considerable research has been aimed at characterizing the physiological/functional significance of each of these frequency bands across a broad variety of tasks (Gevins and Smith, 2006), providing a wealth of prior knowledge to target appropriate task-specific frequencies for classification. Many of the studies reviewed therefore selected their frequency bands of interest based on a priori knowledge of their task relevance. Eight studies of motor execution and motor imagery for example focused on event-related synchronization/desynchronization patterns (ERS/ERD) characterized by short-time local alpha/mu suppression prior to movement onset followed by beta power increases after movement execution (Pfurtscheller et al., 1996; Pfurtscheller and Lopes da Silva, 1999). The functional significance of ERS/ERD-type patterns in motor execution and motor imagery has long been recognized within BCI research (McFarland et al., 2000), but ERD/ERS patterns also constitute reliable indices of high-level cognitive processing (Pfurtscheller and Lopes da Silva, 1999; Friedrich et al., 2013). Most studies investigating navigation (driving or flying) under well-rested vs. sleep-deprived conditions selected their frequency bands of interest based on previous literature or top-down analyses showing that lower frequencies (beta, alpha, and theta) reliably discriminated between well-rested vs. sleep-deprived conditions (Ahn et al., 2016; Nguyen et al., 2017; Dehais et al., 2019). Another study focused on decoding mental stress during task performance selected alpha and beta frequencies based on previous research showing that these frequency bands are reliable indicators of mental stress and cognitive engagement (Al-Shargie et al., 2016).
Other EEG features used, though less extensively than oscillatory frequencies, included wavelet approximation coefficients for motor event identification (Zhu et al., 2017; Sereshkeh et al., 2019), current density for EEG source reconstruction (Ge et al., 2019) or perception- and decision-specific ERP waveforms (Güven et al., 2020; e.g., N100, P200, P300, see Putze et al., 2014). One study (Fu et al., 2020) combined instantaneous amplitude, instantaneous phase, and instantaneous frequency of the EEG signal into a single vector in order to maximize classification performance, though little justification is given as to why integrating these particular features should enhance classification. Two clinically oriented studies selected their relevant EEG features based on their association with the neurological conditions of interest (i.e., interictal EEG discharge in epilepsy (Sirpal et al., 2019) and attention-related P300 in ADHD (Güven et al., 2020), achieving greater decoding accuracy for bimodal EEG-fNIRS compared to unimodal EEG and fNIRS. This illustrates the relevance of also exploiting abnormal EEG signatures in bimodal EEG-fNIRS decoding studies with diagnostic purposes.
fNIRS Features
fNIRS signal reflects variations in the concentration of oxygenated hemoglobin (conventionally indicated HbO) and deoxygenated hemoglobin (HbR, Pan et al., 2019), these values usually being negatively correlated with each other (Cui et al., 2009; Guerrero-Mosquera et al., 2016). HbO and HbR values together enable the calculation of the total hemoglobin mobilized during task performance (HbT). The studies reviewed here used these three values either alone or in combination. A majority of studies used HbO and HbR in combination. Eleven studies only used HbO and three used a combination of HbO, HbR, and HbT. Of the remaining studies, one used the difference between HbO and HbR as a relevant feature of interest (HbD). Since HbO and HbR typically show negative correlations with each other, its authors reasoned that HbD would further increase the amplitude of concentration changes as a function of task. This feature used in combination with HbO and HbR significantly improved classification accuracy in bimodal EEG-fNIRS. Another study computed the Hurst coefficient of HbO (Ge et al., 2017). The Hurst coefficient has been shown as a reliable measure of the internal consistency of time series produced by biological systems (Vorobyov and Cichocki, 2002). The authors however do not explain how this measure would improve the classification accuracy of the fNIRS signal. More generally, and contrary to EEG feature selection, little justification is explicitly provided across studies for choosing one fNIRS feature over others. In some cases (Morioka et al., 2014; Güven et al., 2020), the selection is based on prior evidence for the higher sensitivity of HbO compared to other fNIRS features in detecting cerebral blood flow (Hoshi et al., 2001). Other research using fMRI and NIRS however showed that the BOLD response as measured with fMRI was more correlated with HbR as measured by fNIRS (Huppert et al., 2006). In the case of studies with clinical populations (e.g., ADHD, cf. Morioka et al., 2014), the choice of HbO is in part motivated by prior evidence highlighting the suitability of this particular feature in detecting certain neurological conditions. Alternatively, other indicators of hemodynamic activity can be derived from recorded HbO and HbR such as cerebral oxygen exchange (COE) and cerebral blood volume vector (CBV) (Tanaka et al., 2014; Oka et al., 2015; Borragán et al., 2019). COE can show better sensitivity to detect changes in brain activity than HbO alone and is less affected by changes in CBV (Oka et al., 2015). The amplitude of low-frequency fluctuations (ALFF) is computed by assessing the oscillations within the frequency range 0.01-0.1 Hz and represents a good measure of local spontaneous activity. This measure has shown good performance to identify processes related to resting states or rumination (Lu et al., 2010; Rosenbaum et al., 2020).
EEG-fNIRS Integration
Efficient mental state decoding in bimodal EEG-fNIRS critically requires an appropriate combination of fNIRS and EEG features (Ahn and Jun, 2018; Li et al., 2019, 2020a,b). As explained in the introduction, however, fNIRS has substantially lower temporal resolution compared to EEG, a mismatch likely to severely limit the use of EEG-fNIRS in real-time studies typically requiring relatively high information transfer rates (Ahn and Jun, 2018). This temporal issue may also hinder proper source reconstruction of EEG signals based on the hemodynamic response provided by fNIRS. Several studies have addressed this issue by proposing a hierarchical Bayesian approach incorporating fNIRS and EEG (Aihara et al., 2012). In particular, rather than estimating source localization from electrophysiological current alone through current dipoles or distributed currents methods, both of which are insufficient for resolving the limited spatial resolution of EEG, the Bayesian approach imposes functional imaging data as a hierarchical prior to constrain estimation of the EEG current sources. This additional information has been shown to significantly help current source localization even with a small number of electrodes (Aihara et al., 2012). This approach was tested in the context of bimodal EEG-fNIRS research by Morioka et al. (2014) who compared its mental state decoding accuracy against an approach using EEG alone. Results showed that using fNIRS as a prior significantly increased decoding accuracy (79.1%) compared to using EEG alone (71.4%). Complementing these findings, Morioka et al. compared decoding performance between the Bayesian approach and another method of source reconstruction that did not use a hemodynamic signal as a prior (i.e., minimum L2-norm, cf. Wang et al., 1992). Results confirmed higher decoding accuracy for the hierarchical Bayesian approach (79.1 vs. 75.9%). Another potential avenue of progress lies in the detection of the so-called ‘initial dip' in the fNIRS signal, characterized by a local decrease in the HbO response around 1 second post-event onset and prior to the conventional increase in HbO (Zafar et al., 2016; Hong and Zafar, 2018). Improved techniques for detecting this initial dip have been considered among the methods for enhancing mental state decoding in BCI systems (Hong and Zafar, 2018). The study by Li et al. (2017) explored this issue in a bimodal EEG-fNIRS experiment on motor control in which fNIRS was used for initial dip detection. Improvement in decoding accuracy was significantly higher for bimodal EEG-fNIRS (91.02%) compared with unimodal EEG or fNIRS (< 86%). Finally, very recent studies have explored the possibility to use fluctuations in specific EEG frequency bands (e.g., gamma, alpha, or beta) to predict fluctuations in local fNIRS signal (Li et al., 2020a,b). Future research is expected to test which of these techniques might best improve decoding accuracy through more precise localization of task-related cortical activity.
EEG-fNIRS Classification and Detection Methods
Immediately following EEG and fNIRS feature extraction and integration is the actual processing of this integrated signal for accurate mental state decoding (cf. Figure 1). This decoding capacity largely depends on machine learning algorithms applied to brain imaging data (Lemm et al., 2011; Hong et al., 2018). Machine learning methods, including supervised/unsupervised learning algorithms and reinforcement learning, have now reached unprecedented levels of complexity in task outcomes such as classification, detection, or regression. These levels were achieved through increased computational power, efficient learning algorithms, valuable activation functions, and restricted or back-fed neuron connections (Pinti et al., 2015). The formal principles underlying the decoding of brain imaging data are detailed in Lemm et al. (2011). The following paragraphs provide an overview of the main decoding algorithms used in the studies reviewed here, assessing to the extent possible their relative merits and potential challenges.
The most common mental state classification technique used in the studies reviewed was the Linear Discriminant Analysis (LDA, 14 studies). LDA is a supervised algorithm that consists in breaking datasets down into two or more classes by maximizing the distance between the means of these classes while minimizing the variance within each class. This method's success lies in its low computational costs, making it suitable for use in online BCI systems. Second come Support Vector Machines (SVM, 11 studies), which work by first training a model in assigning data points to one or another category (supervised), ensuring that the distance between them and the class boundary is maximal, then using this trained model for assigning new incoming data to one or the other category. This method's primary strength is to enable nonlinear classification through the use of kernel functions (Hong et al., 2018). Seven studies used neural network algorithms for classification, including multilayered or Deep Neural Networks (DNNs), (Cichy and Kaiser, 2019). A variant of simple neural networks aimed at simulating learning as instantiated in the human brain, DNNs feature several (more than two) interconnected layers of processing units (i.e. neurons) which increase their reciprocal connections as a function of learning. These algorithms' principal specificity is to directly use non-preprocessed data for classification (Schirrmeister et al., 2017; Saadati et al., 2020), feeding it into varying numbers of convolutional layers enabling them to decode the signal at different levels of specificity and abstraction. Their primary advantage compared to LDA or SVM is to bypass a priori feature selection and automatically detect hierarchical patterns of information from the raw signal. This improvement however comes with the potential drawback of yielding wrong learning outputs or requiring longer periods of learning (Schirrmeister et al., 2017, but see Yin et al., 2015). Saadati et al. (2020) directly compared classification accuracies between SVM and a deep neural network (DNN) from bimodal EEG-fNIRS signals recorded in a number of cognitive and motor tasks. Not only did EEG-fNIRS perform better than unimodal EEG or fNIRS, but classification accuracy was overall higher for DNN (~90%) than for SVM (~84%). Another study aimed at discriminating children with ADHD vs. control during an ‘oddball' detection paradigm (Güven et al., 2020) directly compared discrimination accuracies between three supervised algorithms using EEG, fNIRS, and EEG-fNIRS features: SVM, the multilayer perception network and the naïve Bayes classifier. Results showed that accuracy was highest for EEG-fNIRS but did not differ between SVM and the neural network (86.3%). The highest accuracy level was obtained through the use of the naïve Bayes classifier (93.2%).
New methods of mental state decoding from bimodal EEG-fNIRS signals are still being developed. Most recent work in this respect has focused on optimizing procedures for fusing EEG and fNIRS signals, with important improvements in decoding accuracy compared to previous classification algorithms (Khan and Hasan, 2020; Sun et al., 2020). Also, the spatial relationship between EEG and fNIRS signals have been used to feed up deep learning architectures for classification, such as convolutional neural networks (RCNNs) for mental task (precision of 99.6%, cf. Ghonchi et al., 2020) or CNNs for mental workload (accuracy 89%, cf. Saadati et al., 2020). Furthermore, with the rapid development of BCI technology, the feasibility of new hardware setups using few EEG channels and fNIRS source-detector pairs has been tested in three different mental tasks using shrinkage LDA classifier), and also for channel selection through k-nearest neighbor (kNNs) and Tree classifiers (Hasan et al., 2020). These results show classification accuracy high enough to be used in practical BCI applications.
EEG-fNIRS Use in Ecologically Valid Contexts
As mentioned at the outset the enhanced wearability of bimodal EEG and fNIRS instrumentation should in principle facilitate the monitoring of brain activity in real-world situations. It must be noted however that almost all the studies reviewed here were still performed in laboratory settings, highlighting the many remaining challenges of using bimodal EEG-fNIRS equipment in ecologically valid contexts. Further progress in this domain depends in part on the availability of miniaturized hybrid EEG-fNIRS setups and signal (pre-)processing algorithms handling motion-related artifacts in a near-instantaneous fashion. Another challenge comes from the versatility of real-world over laboratory conditions where events of interest can be precisely identified through computer-controlled experimental scenarios (Pinti et al., 2017; von Lühmann et al., 2021). An important short-term objective is to be able to retrace the history of an experiment in the absence of scenario files. While little if any progress has been achieved in the particular case of bimodal EEG-fNIRS several methods have been developed for unimodal fNIRS that could be extended to bimodal EEG-fNIRS research. Pinti et al. (2015) for example made use of video cameras fitted onto subjects and accompanying experimenters. Careful analysis of video recordings allowed authors to recover functional events of interest and to use them for tolerably good HbO and HbR measurements in two separate conditions. Another approach developed by the same authors consists in recovering functional events in the fNIRS signal through a GLM-based data-driven method (Pinti et al., 2017). Used in combination with the video recordings on the same 2015 study dataset, this method allowed authors to recover 50 to 75 percent of functionally relevant events. Crucially, however, these methods remain limited by the temporal precision of fNIRS, which is not fitted to capture the versatility and rapidity of real-world environments the way EEG is (Casson, 2019). In this respect, recent research has concentrated on building wearable EEG setups capable of retracing functional events through eye-fixation data (Casson and Trimble, 2018). Future efforts should be geared at combining these methods.
Recommendations For Future Neuroergonomics Research Using EEG-fNIRS
The main focus of neuroergonomics is to build human-machine interfaces for assisting safe and autonomous real-world performance in populations with motor or communicative disabilities, or healthy individuals engaged in high-risk private or professional activities. The recent emergence of bimodal EEG-fNIRS systems apt to decode performance-related mental states as part of these interfaces has received increased attention. This review article aimed to examine the level of improvement achieved in mental state decoding using bimodal EEG-fNIRS compared to unimodal EEG or fNIRS. It also considered the state of progress and challenges in implementing wearable versions of these interfaces. Its results reveal consistent increases in decoding accuracy for bimodal over unimodal EEG and fNIRS despite a limited number of studies and significant variation in the studies' research questions and methods. They also highlight several important challenges in the real-world use of bimodal EEG-fNIRS systems. The remaining paragraphs synthesize these points in the form of recommendations for future research.
First, increased effort should be placed on achieving higher consistency in terms of sensor setups, feature selection, and extraction methods. In terms of sensor setup hybrid EEG-fNIRS hardware should be favored over custom EEG-fNIRS systems built from separate fNIRS and EEG equipment to guard against potential crosstalk and event-timing offsets between EEG and fNIRS signals. Additionally, sensor layouts should be designed by taking into consideration the discrepancies between EEG and fNIRS regarding temporal vs. spatial granularity. Compared with the relatively unambiguous spatial distribution of fNIRS responses the scalp distribution of EEG signatures can be misleading in terms of signal sources. Depending on the level of sophistication in the experimental paradigm and sensor layout, it should therefore be ensured that the fNIRS and EEG signal captured in the same scalp area are tied to the same underlying processes. Conversely, the greater spatial resolution of fNIRS compared to EEG comes at the cost of larger temporal latencies likely to cause difficulties in the real-time monitoring of performance-related brain-signal. Though several avenues have been explored to minimize the temporal discrepancy between EEG and fNIRS, it remains difficult if not impossible to align both modalities at the same level of temporal resolution. Future research should hopefully help minimize or compensate for these temporal discrepancies.
Second, as decoding accuracy naturally depends on signal quality the relevance of signal preprocessing and artifact rejection methods cannot be overlooked. These methods should take into account both modality-specific and modality-independent artifacts, appropriately distinguish between systemic and nonsystemic sources of noise and account for the constraints associated with signal preprocessing such as stimulus repetition and averaging. The latter point poses a particular challenge in the attempt to decode mental states in real-world contexts devoid of repetition and regularity. It is therefore central to focus on the development of online preprocessing algorithms and systems that can optimize noise rejection on single events.
Third, increased effort should also be put into a more targeted and justified selection and integration of EEG and fNIRS features. In the case of EEG, the most common features are the major brain oscillatory frequencies, while the use of event-related potentials remains relatively marginal. In most cases, the frequency bands of interest were selected based on prior knowledge of their functional involvement in the target tasks. In the case of fNIRS, the main features of interest included local changes in HbO, HbR, and HbT. Compared to EEG features, however, little justification is given for selecting one over the other fNIRS features. Possible reasons for this gap include limited knowledge of the relation between hemodynamic signal and underlying neurocognitive processes and of the functional significance of variations in local oxy- or deoxyhemoglobin concentration. Available evidence indicates a correspondence between HbR and HbT concentration and negative bold responses as measured through fMRI (Maggioni et al., 2015). The exact meaning of negative blood dynamics however has remained elusive (Wade, 2002).
Fourth, whereas an immediate goal of neuroergonomics research is to decode task-related mental states in naturalistic conditions, most studies featured in the review were still performed in highly controlled laboratory settings. This indicates that the multimodal exploitation of EEG-fNIRS in real-life applications is still in its infancy. The most likely conditions to meet in achieving this goal include proper miniaturization of EEG-fNIRS systems, the development of advanced algorithms for signal preprocessing, and more powerful methods of real-time EEG and fNIRS analysis. Another major challenge concerns the precise tracking of a sufficient number of events of interest. Overcoming these challenges should not only pave the way toward a better understanding of the neurocognitive correlates of human performance but should open new possibilities for equipping modern hardware (e.g., vehicles, prosthetic devices, etc.) apt to usefully exploit mental states for assisting or enhancing behaviors. On this particular point, it is worth acknowledging the possibility to complement EEG-fNIRS signal with other behavioral and physiological measures. These include motion capture, heart rate, skin conductance, or eye-tracking technologies. The studies reviewed that exploited these additional measures report further improvements in decoding accuracy.
Finally, although decoding is shown to be better in the bimodal than the unimodal approach, this observation is for many studies based on absolute percentage values rather than formal statistical comparisons. Until such comparisons are systematically made the merits of bimodal EEG-fNIRS over unimodal EEG or fNIRS in mental state decoding should therefore be considered with caution. It must be remarked how few studies reviewed approach their research question from a theoretical vantage. Regardless of the issue investigated, little conceptual work is exploited to predict or account for the patterns reported. This point might seem unfortunate given the considerable potential of bimodal EEG-fNIRS in moving neuroergonomics research forward at both fundamental and practical levels (Pfurtscheller et al., 2012; Wallois et al., 2012; Balconi et al., 2015; Pinti et al., 2018b). The development of advanced tools for investigating the neurocognitive correlates of behavior does not replace careful theoretical analysis of the phenomenon addressed (Krakauer et al., 2017). A strong working definition of the cognitive processes targeted by specific experiments is of fundamental importance in an effort to decode mental states from electrophysiological or hemodynamic signal.
Conclusion
The advent of portable methods of multimodal brain monitoring foreshadows significant progress in understanding and exploiting performance-related mental states in naturalistic situations. Amongst these methods, bimodal EEG-fNIRS technologies potentially represent an important step forward in this direction. Available evidence, though recent and relatively limited in scope and amount, consistently points to significant improvements in the decoding of performance-related mental states, with crucial implications in both the conceptual and practical domains of NE. Many challenges however still stand in the way of fully exploiting these technologies in the service of human factors research and development. Relevant questions to be addressed in future research in our view comprise the following: First, is bimodal EEG-fNIRS systematically more desirable than unimodal EEG or fNIRS across all domains of NE? The superior decoding accuracy of bimodal EEG-fNIRS would intuitively seem to bring a positive answer to this question, but one must also weigh the gains of combining these technologies in human-machine interface systems against the costs incurred by their limitations. It is possible in practice that unimodal technologies remain at present more efficient due to their greater flexibility of use in real-world situations as well as a higher degree of technical and conceptual knowledge of their neuroergonomic potential. Most recent state-of-the-art reports highlight the considerable technological advances afforded by portable EEG in neuroergonomics research (Wascher et al., 2021). In contrast, neuroergonomic applications of fNIRS still exhibit comparably greater conceptual and methodological shortcomings (Zhu et al., 2020). Only when these gaps are overcome will it be possible to appreciate the scope and limits of multimodal brain imaging in human-machine interface applications.
Second, do the improvements afforded by bimodal EEG-fNIRS alleviate the need for theoretically grounded approaches to their exploitation in the neuroergonomic context? As has been noted above few of the studies reviewed ground their research into fine-grained theoretical considerations of human performance and its underlying cognitive processes as reflected in the EEG and fNIRS signal. This theory-free approach might in part be symptomatic of the general enthusiasm for the ever-increasing multiplication and sophistication of novel techniques of neuroscientific investigation, which may in turn overshadow the need for deep conceptual analyses of how behavior emerges from the computational operations of the brain (Krakauer et al., 2017). Such analyses however remain in our view an essential precondition for improvement in the use of multimodal brain imaging in human-machine interface systems, particularly with regard to the choice of appropriate decoding algorithms (Kriegeskorte and Douglas, 2018).
Author Contributions
NB, SL, GB, and CG-M conceived the review and wrote the paper. NB performed the review. All authors contributed to the article and approved the submitted version.
Funding
This work was supported by the Belgian Royal Higher Institute for Defense (Grant HFM20-02 awarded to SL).
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher's Note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary Material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnrgo.2022.934234/full#supplementary-material
References
Abtahi, M., Borgheai, S. B., Jafari, R., Constant, N., Diouf, R., Shahriari, Y., et al. (2020). Merging fNIRS-EEG brain monitoring and body motion capture to distinguish parkinsons disease. IEEE Trans. Neural Syst. Rehabilitation Eng. 28, 1246–1253. doi: 10.1109/TNSRE.2020.2987888
Aghajani, H., Garbey, M., and Omurtag, A. (2017). Measuring mental workload with EEG+fNIRS. Front. Hum. Neurosci. 11, 359. doi: 10.3389/fnhum.2017.00359
Ahn, S., Jun, S. C. (2018). Multi-modal integration of EEG-fNIRS for brain-computer interfaces – current limitations and future directions. Front. Hum. Neurosci. 11. doi: 10.3389/fnhum.2017.00503
Ahn, S., Nguyen, T., Jang, H., Kim, J. G., and Jun, S. C. (2016). Exploring neuro-physiological correlates of drivers' mental fatigue caused by sleep deprivation using simultaneous EEG, ECG, fNIRS data. Front. Hum. Neurosci. 10. doi: 10.3389/fnhum.2016.00219
Aihara, T., Takeda, Y., Takeda, K., Yasuda, W., Sato, T., Otaka, Y., et al. (2012). Cortical current source estimation from electroencephalography in combination with near-infrared spectroscopy as a hierarchical prior. Neuroimage. 59, 4006–4021. doi: 10.1016/j.neuroimage.2011.09.087
Allison, B. Z., Wolpaw, E. W., and Wolpaw, J. R. (2007). Brain–computer interface systems: progress and prospects. Expert Rev. Med. Devices. 4, 463–474. doi: 10.1586/17434440.4.4.463
Al-Quraishi, M. S., Elamvazhuthi, I., Tang, T. B., Al-Qurishi, M., Adil, S. H., and Ebrahim, M. (2021). Bimodal data fusion of simultaneous measurements of EEG and fNIRS during lower limb movements. Brain Sci. 11, 713. doi: 10.3390/brainsci11060713
Al-Shargie, F., Kiguchi, M., Badruddin, N., Dass, S. C., Hani, A. F. M., and Tang, T. B. (2016). Mental stress assessment using simultaneous measurement of EEG and fNIRS. Biomed. Opt. Express 7, 3882–3898. doi: 10.1364/BOE.7.003882
Balconi, M., Cortesi, L., and Crivelli, D. (2017). Motor planning and performance in transitive and intransitive gesture execution and imagination: Does EEG (RP) activity predict hemodynamic (fNIRS) response? Neurosci. Lett. 648, 59–65. doi: 10.1016/j.neulet.2017.03.049
Balconi, M., Grippa, E., and Vanutelli, M. E. (2015). What hemodynamic (fNIRS), electrophysiological (EEG) and autonomic integrated measures can tell us about emotional processing. Brain Cogn. 95, 67–76. doi: 10.1016/j.bandc.2015.02.001
Blokland, Y., Spyrou, L., Thujssen, D., Eijsvogels, T., Colier, W., Floor-Westerdijk, M., et al. (2014). Combined EEG-fNIRS decoding of motor attempt and imagery for brain switch control/ an offline study in patients with tetraplegia. IEEE Trans. Neural Syst. Rehabilitation Eng. 22, 222–229. doi: 10.1109/TNSRE.2013.2292995
Borragán, G., Gilson, M., Guerrero-Mosquera, C., Di Ricci, E., Slama, H., and Peigneux, P. (2018). Transcranial direct current stimulation does not counteract cognitive fatigue, but induces sleepiness and an inter-hemispheric shift in brain oxygenation. Front. Psychol. 9. doi: 10.3389/fpsyg.2018.02351
Borragán, G., Guerrero-Mosquera, C., Guillaume, C., Slama, H., and Peigneux, P. (2019). Decreased prefrontal connectivity parallels cognitive fatigue-related performance decline after sleep deprivation. An optical imaging study. Biol. Psychol. 144, 115–124. doi: 10.1016/j.biopsycho.2019.03.004
Buccino, A. P., Keles, H. O., and Omurtag, A. (2016). Hybrid EEG-fNIRS asynchronous brain- computer interface for multiple motor tasks. PLoS ONE. 11, e0146610. doi: 10.1371/journal.pone.0146610
Cain, B. (2007). A Review of the Mental Workload Literature. NATO technical reports. Available online at: https://apps.dtic.mil/sti/citations/ADA474193
Casson, A. J. (2019). Wearable EEG and beyond. Biomed. Eng. Lett. 9, 53–71. doi: 10.1007/s13534-018-00093-6
Casson, A. J., and Trimble, E. V. (2018). Enabling free movement EEG tasks by eye fixation and gyroscope motion correction: EEG effects of color priming in dress shopping. IEEE Access. 6, 62975–62987. doi: 10.1109/ACCESS.2018.2877158
Chiarelli, A. M., P. Croce, A. Merla, and Zappasodi, F. (2018). Deep learning for hybrid EEG-fNIRS brain-computer interface: application to motor imagery classification. J Neural Eng. 15, 036028. doi: 10.1088/1741-2552/aaaf82
Cicalese, P. A., Li, R., Ahmadi, M. B., Wang, C., Francis, J. T., Selvaraj, S., et al. (2020). An EEG-fNIRS hybridization technique in the four-class classi!cation of alzheimer's disease. J. Neurosci. Methods. 336, 108618. doi: 10.1016/j.jneumeth.2020.108618
Cichy, R. M., and Kaiser, D. (2019). Deep neural networks as scientific models. Trends Cogn. Sci. 23, 305–317. doi: 10.1016/j.tics.2019.01.009
Coffey, E. B. J., Brouwer, A.-M., and van Erp, J. B. F. (2012). Measuring Workload Using a Combination of Electroencephalography and Near Infrared Spectroscopy. San Diego CA: Proceedings of the Human Factors and Ergonomics Society 56th annual meeting. doi: 10.1177/1071181312561367
Croce, P., Zappasodi, F., Merla, A., and Chiarelli, A. M. (2017). Exploiting neurovascular coupling: a Bayesian sequential Monte Carlo approach applied to simulated EEG fNIRS data. J. Neural Eng. 14, 046029. doi: 10.1088/1741-2552/aa7321
Cui, X., Bray, S., and Reiss, A. L. (2009). Functional near infrared spectroscopy (NIRS) signal improvement based on negative correlation between oxygenated and deoxygenated hemoglobin dynamics. Neuroimage 49, 3039–3046. doi: 10.1016/j.neuroimage.2009.11.050
Darvas, F., Ermer, J. J., Mosher, J. C., and Leahy, R. M. (2006). Generic head models for Atlas-based EEG source analysis. Hum. Brain Mapp. 27, 129–143. doi: 10.1002/hbm.20171
Dechent, P., Merboldt, K.-D., and Frahm, J. (2004). Is the human primary motor cortex involved in motor imagery? Cognitive Brain Res. 19, 138–144. doi: 10.1016/j.cogbrainres.2003.11.012
Dehais, F., Duprès, A., Flumeri, G., Verdière, K. J., Borghini, G., Babiloni, F., et al. (2019). Monitoring pilot's cognitive fatigue with engagement features in simulated and actual flight conditions using an hybrid fNIRS-EEG passive BCI. IEEE Trans. Syst. Man Cybern. 544–549. doi: 10.1109/SMC.2018.00102
Deiber, M.-P., Ibañez, V., Honda, M., Sadato, N., Raman, R., and Hallett, M. (1998). Cerebral processes related to visuomotor imagery and generation of simple finger movements studied with positron emission tomography. Neuroimage. 7, 73–85. doi: 10.1006/nimg.1997.0314
Fazli, S., Mehnert, J., Steinbrink, J., Curio, G., Villringer, A., Müller, K.-R., et al. (2012). Enhanced performance by a hybrid NIRS–EEG brain computer interface. Neuroimage. 59, 519–5129. doi: 10.1016/j.neuroimage.2011.07.084
Ferrari, M., and Quaresima, V. (2012). A brief review on the history of human functional near-infrared spectroscopy (fNIRS) development and fields of application. Neuroimage. 63, 921–935. doi: 10.1016/j.neuroimage.2012.03.049
Firooz, S., and Setarhdan, S. K. (2019). IQ estimation by means of EEG-fNIRS recordings during a logical-mathematical intelligence test. Comput. Biol. Med. 110, 218–226. doi: 10.1016/j.compbiomed.2019.05.017
Friedrich, E. V. C., Scherer, R., and Neuper, C. (2013). Stability of event-related (de-) synchronization during brain–computer interface-relevant mental tasks. Clini. Neurophysiol. 124, 61–69. doi: 10.1016/j.clinph.2012.05.020
Fu, Y., Xiong, X., Jiang, C., Xu, B., Li, Y., and Li, H. (2020). Imagined hand clenching force and speed modulate brain activity and are classified by nirs combined with EEG. IEEE Trans. Neural Syst. Rehabilitation Eng. 25, 1641–1652. doi: 10.1109/TNSRE.2016.2627809
Ge, S., Wang, P., Liu, H., Lin, P., Gao, J., Wang, R., et al. (2019). Neural activity and decoding of action observation using combined EEG and fNIRS measurement. Front. Hum. Neurosci. 13, 357. doi: 10.3389/fnhum.2019.00357
Ge, S., Yang, Q., Wang, R., Lin, P., Gao, J., Leng, Y., et al. (2017). A brain-computer interface based on a few-channel EEG-fNIRS bimodal system. IEEE Access. 5, 208–218. doi: 10.1109/ACCESS.2016.2637409
Gevins, A., and Smith, M. E. (2006). “Electroencephalograpy (EEG) in Neuroergonomics,” in Neuroergonomics: The Brain at Work, Parasuraman R, Rizzo M. (Eds.). Oxford: Oxford University Press p. 15–31. doi: 10.1093/acprof:oso/9780195177619.003.0002
Ghonchi, H., Fateh, M., Abolghasemi, V., Ferdowsi, S., and Rezvani, M. (2020). Deep recurrent-convolutional neural network for classification of simultaneous EEG-fNIRS signals. IET Signal Process 14, 142–153. doi: 10.1049/iet-spr.2019.0297
Guerrero-Mosquera, C., Borragàn, G., and Peigneux, P. (2016). Automatic detection of noisy channels in fNIRS signal based on correlation analysis. J. Neurosci. Methods 271, 128–138. doi: 10.1016/j.jneumeth.2016.07.010
Guevara, S., Felipe, A., Giraldo, E., M. M., and Molinas, C. (2020). “Low-density EEG for source activity reconstruction using partial brain models,” in 13th International Joint Conference on Biomedical Engineering Systems and Technologies. Valetta, Malta.
Güven, A., Altinkaynak, M., Dolu, N., Izzetoglu, M., Pektas, F., Ozmen, S., et al. (2020). Combining functional near-infrared spectroscopy and EEG measurements for the diagnosis of attention-deficit hyperactivity disorder. Neural Computing Appl. 32, 8367–8380. doi: 10.1007/s00521-019-04294-7
Halgren, E., Dhond, R. P., Christensen, N., Van Petten, C., Marinkovic, K., Lewine, J. D., et al. (2002). N400-like magnetoencephalography responses modulated by semantic context, word frequency, and lexical class in sentences. Neuroimage 17, 1101–1116. doi: 10.1006/nimg.2002.1268
Hasan, M. A. H., Khan, M. U., and Mishra, D. (2020). A computationally efficient method for hybrid EEG-fNIRS BCI based on the pearson correlation. Biomedical Res. Int. 020, 1838140. doi: 10.1155/2020/1838140
Herff, C., Fortmann, O., Tse, C.-Y., Cheng, X., Putze, F., Heger, D., et al. (2015). “Hybrid fNIRS-EEG based discrimination of 5 levels of memory load,” in 7th Annual International IEEE EMBS Conference on Neural Engineering. France: Montpellier. doi: 10.1109/NER.2015.7146546
Hong, K.-S., and Khan, M. J. (2017). Hybrid brain–computer interface techniques for improved classification accuracy and increased number of commands: a review. Front. Neurorobot. 11, 35. doi: 10.3389/fnbot.2017.00035
Hong, K.-S., and Zafar, A. (2018). Existence of initial dip for BCI: an illusion or reality. Front. Neurorobot. 12, 69. doi: 10.3389/fnbot.2018.00069
Hong, K. S., Khan, M. J., and Hong, M. J. (2018). Feature extraction and classification methods for hybrid fNIRS-EEG brain-computer interfaces. Front. Hum. Neurosci. 12(246). doi: 10.3389/fnhum.2018.00246
Hoshi, Y., Kobayashi, N., and Tamura, M. (2001). Interpretation of near-infrared spectroscopy signals: a study with a newly developed perfused rat brain model. J. Appl. Physiol. 90, 1657–1662. doi: 10.1152/jappl.2001.90.5.1657
Huppert, T. J., Hoge, R. D., Diamond, S. G., Francesconi, M. A., and Boas, D. A. (2006). A temporal comparison of BOLD, ASL, and NIRS hemodynamic responses to motor stimuli in adult humans. Neuroimage. 29, 368–382. doi: 10.1016/j.neuroimage.2005.08.065
Kaewkamnerdpong, B. (2016). “A framework for human learning ability study using simultaneous EEG/fNIRS and portable EEG for learning and teaching development,” in Smart Education and e-Learning. (2016). Smart Innovation, Systems and Technologies 59, Uskov, V., Howlett R, Jain L. (Eds.). Cham: Springer. doi: 10.1007/978-3-319-39690-3_14
Kassab, A., Le Lan, J., Tremblay, J., Vannasing, P., Dehbozorgi, M., Pouliot, P., et al. (2018). Multichannel wearable fNIRS-EEG system for long-term clinical monitoring. Hum. Brain Mapp. 39, 7–23. doi: 10.1002/hbm.23849
Khan, M. U., and Hasan, M. A. H. (2020). Hybrid EEG-fNIRS BCI fusion using multi-resolution singular value decomposition (MSVD). Front. Hum. Neurosci. 14, 599802. doi: 10.3389/fnhum.2020.599802
Koo, B., Lee, H.-G., Nam, Y., Kang, H., Koh, C. S., Shin, H.-C., et al. (2015). A hybrid NIRS-EEG system for self-paced brain computer interface with online motor imagery. J. Neurosci. Method. 244, 26–32. doi: 10.1016/j.jneumeth.2014.04.016
Krakauer, J. W., Ghazanfar, A. A., Gomez-Marin, A., MacIver, M. A., and Poeppel, D. (2017). Neuroscience needs behavior: correcting a reductionist bias. Neuron. 93, 480–490. doi: 10.1016/j.neuron.2016.12.041
Kriegeskorte, N., Douglas, P. K. (2018). Cognitive computational neuroscience. Nat. Neurosci. 21, 1148–1160. doi: 10.1038/s41593-018-0210-5
Krueger, G. P. (1989). Sustained work, fatigue, sleep loss and performance: a review of the issues. Work, Stress. 3, 129–141. doi: 10.1080/02678378908256939
Lal, S. K. L., and Craig, A. (2001). A critical review of the psychophysiology of driver fatigue. Biol. Psychol. 55, 173–194. doi: 10.1016/S0301-0511(00)00085-5
Lau, E. F., Phillips, C., and Poeppel, D. (2008). A cortical network for semantics: (de)constructing the N400. Nat. Rev. Neurosci. 9, 920–933. doi: 10.1038/nrn2532
Laureys, S., Boly, M., and Tononi, G. (2009). “Functional Neuroimaging,” in The Neurology of Consciousness Laureys, S, and Tononi, G. (Eds.). New York, NY: Academic Press p. 31–42. doi: 10.1016/B978-0-12-374168-4.00003-4
Leamy, D. J., Collins, R., and Ward, T. E. (2011). “Combining fNIRS and EEG to improve motor cortex activity classification during an imagined movement- based task,” in Foundations of Augmented Cognition. Directing the Future of Adaptive Systems, Schmorrow D. D., and Fidopiastis, C. M. (Eds.). p. 177–185. doi: 10.1007/978-3-642-21852-1_22
Leamy, D. J., and Ward, T. E. (2010). “A novel co-locational and concurrent fNIRS/EEG measurement system: design and initial results,” in 32nd Annual International Conference of the IEEE EMBS. Buenos Aires. doi: 10.1109/IEMBS.2010.5627377
Lee, M.-H., Fazli, S., Mehnert, J., and Lee, S. W. (2014). “Hybrid brain-computer interface based on EEG and NIRS modalities,” 2014 International Winter Workshop on Brain-Computer Interface. South Korea: Gangwon. doi: 10.1109/iww-BCI.2014.6782577
Lemm, S., Blankertz, B., Dickhaus, T., and Müller, K. R. (2011). Introduction to machine learning for brain imaging. Neuroimage. 56, 387–399. doi: 10.1016/j.neuroimage.2010.11.004
Li, R., Li, S., Roh, J., Wang, C., and Zhang, Y. (2020a). Multimodal neuroimaging using concurrent EEG/fNIRS for poststroke recovery assessment: An exploratory study. Neurorehabil. Neural Repair 34, 1099–1110. doi: 10.1177/1545968320969937
Li, R., Nguyen, T., Potter, T., and Zhang, Y. (2019). Dynamic cortical connectivity alterations associated with Alzheimer's disease: AN EEG and fNIRS integration study. NeuroImage: Clinical 21(101622), doi.org/10.1016/j.nicl.2018.101622 doi: 10.1016/j.nicl.2018.101622
Li, R., Potter, T., Huang, W., and Zhang, Y. (2017). Enhancing performance of a hybrid EEG-fNIRS system using channel selection and early temporal features. Front. Hum. Neurosci. 11, 462. doi: 10.3389/fnhum.2017.00462
Li, R., Zhao, C., Wang, C., Wang, J., and Zhang, Y. (2020b). Enhancing fNRIS analysis using EEG rhythmic signatures/ an EEG-informed fNRIS analysis study. IEEE Trans. Neural Syst. Rehabilitation Eng. 67, 2789–2797. doi: 10.1109/TBME.2020.2971679
Loeppke, R., Taitel, M., Haufle, V., Parry, T., Kessler, R., and Kimberly, J. (2009). Health and productivity as a business strategy: a multiemployer study. J. Occupational Environm. Med. 51, 411–428. doi: 10.1097/JOM.0b013e3181a39180
Logothetis, N. K., Pauls, J., Augath, M., Trinath, T., and Oeltermann, A. (2001). Neurophysiological investigation of the basis of the fMRI signal. Nature. 412, 150–157. doi: 10.1038/35084005
Lu, C.-M., Zhang, Y., Biswal, B. B. B., Zang, Y.-F., Peng, D.-L., and Zhu, C. Z. (2010). Use of fNIRS to assess resting state functional connectivity. J. Neurosci. Methods 186, 242–249. doi: 10.1016/j.jneumeth.2009.11.010
Maggioni, E., Molteni, E., Zucca, C., Reni, G., Cerutti, S., Triulzi, F. M., Arrigoni, F., and Bianchi, S. (2015). Investigation of negative BOLD responses in human brain through NIRS technique. A visual stimulation study. Neuroimage. 108, 410–422. doi: 10.1016/j.neuroimage.2014.12.074
McFarland, D. J., Miner, L. A., Vaughan, T. M., and Wolpaw, J. R. (2000). Mu and beta rhythm topographies during motor imagery and actual movements. Brain Topogr. 12, 177–186. doi: 10.1023/A:1023437823106
Mizuno, K., Tanaka, M., Fukuda, S., Imai-Matsumura, K., and Watanabe, Y. (2011). Relationship between cognitive functions and prevalence of fatigue in elementary and junior high school students. Brain Dev. 33, 470–479. doi: 10.1016/j.braindev.2010.08.012
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., and the PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 6, 7. doi: 10.1371/journal.pmed.1000097
Morioka, H., Kanemura, A., Morimoto, S., Yoshioka, T., Oba, S., Kawanabe, M., et al. (2014). Decoding spatial attention by using cortical currents estimated from electroencephalography with near-infrared spectroscopy prior information. Neuroimage. 90, 128–139. doi: 10.1016/j.neuroimage.2013.12.035
Mu, Z., Hu, J., and Min, J. (2017). Driver fatigue detection system using electroencephalography signals based on combined entropy features. Applied Sci. 7, 150. doi: 10.3390/app7020150
Neuper, C., Scherer, R., Reiner, M., and Pfurtscheller, G. (2005). Imagery of motor actions/ Differential effects of kinesthetic and visual-motor mode of imagery in single-trial EEG. Cognitive Brain Res. 25, 668–677. doi: 10.1016/j.cogbrainres.2005.08.014
Nguyen, D. K., Tremblay, J., Pouliot, P., Vannasing, P., Florea, O., Carmant, L., et al. (2012). Non-invasive continuous EEG-fNIRS recording of temporal lobe seizures. Epilepsy Res. 99, 112–126. doi: 10.1016/j.eplepsyres.2011.10.035
Nguyen, T., Ahn, S., Jang, H., Jun, S. C., and Kim, J. G. (2017). Utilization of a combined EEG/NIRS system to predict driver drowsiness. Sci. Rep. 7(43933). doi: 10.1038/srep43933
Oka, N., Yoshino, K., Yamamoto, K., Takahashi, H., Li, S., Sugimachi, T., et al. (2015). Greater activity in the frontal cortex on left curves: a vector-based fnirs study of left and right curve driving. PLoS ONE. 10, e0127594. doi: 10.1371/journal.pone.0127594
Osterhout, L., McLaughlin, J., Kim, A., Greenwald, R., and Inoue, K. (2004). “Sentences in the Brain: Event-Related Potentials as Real-Time Reflections of Sentence Comprehension and Language Learning,” in The on-line study of sentence comprehension: eyetracking, ERPs and beyond, Carreiras, M., and Clifton, C. (Eds.). London: Psychology Press.
Othman, M., Bhattacharya, M., Møller, K., Kjeldsen, S., Gand, J., Kjaergaard, J., et al. (2020). Resting-state NIRS–EEG in unresponsive patients with acute brain injury: a proof-of-concept study. Neurocrit. Care 34, 31–44. doi: 10.1007/s12028-020-00971-x
Palva, S., and Palva, J. M. (2012). Discovering oscillatory interaction networks with M/EEG: challenges and breakthroughs. Trends Cogn. Sci. 16, 219–230. doi: 10.1016/j.tics.2012.02.004
Pan, Y., Borragan, G., and Peigneux, P. (2019). Applications of functional near-infrared spectroscopy in fatigue, sleep deprivation, social cognition. Brain Topogr. 32, 998–1012. doi: 10.1007/s10548-019-00740-w
Pfurtscheller, G., Bauernfeind, G., Neuper, C., and Lopes da Silva, F. H. (2012). Does conscious intention to perform a motor act depend on slow prefrontal (de)oxyhemoglobin oscillations in the resting brain? Neurosci. Lett. 508, 89–94. doi: 10.1016/j.neulet.2011.12.025
Pfurtscheller, G., and Lopes da Silva, F. H. (1999). Event-related EEG/MEG synchronization and desynchronization: basic principles. Clinical Neurophysiology 110, 1842–1857. doi: 10.1016/S1388-2457(99)00141-8
Pfurtscheller, G., Stancak, A., and Neuper, C. (1996). Post-movement beta synchronization. A correlate of an idling motor area? Electroencephalogr. Clin. Neurophysiol. 98, 281–293. doi: 10.1016/0013-4694(95)00258-8
Pinti, P., Aichelburg, C., Gilbert, S. J., Hamilton, A. F., Hirsch, J., Burgess, P. W., et al. (2018a). A review on continuous wave functional near-infrared spectroscopy and imaging instrumentation and methodology. Japanese Psychological Res. 60, 347–373. doi: 10.1111/jpr.12206
Pinti, P., Aichelburg, C., Lind, F., Power, S., Swingler, E., Merla, A., et al. (2015). Using fiberless, wearable fNRIS to monitor brain activity in real-world cognitive tasks. J. Visualized Experiments. 106, e53336. doi: 10.3791/53336
Pinti, P., Merla, A., Aichelburg, C., Lind, F., Power, S., Swingler, E., et al. (2017). A novel GLM-based method for the Automatic IDentification of functional Events (AIDE) in fNIRS data recorded in naturalistic environments. Neuroimage. 155, 291–304. doi: 10.1016/j.neuroimage.2017.05.001
Pinti, P., Tachtsidis, I., Hamilton, A. F., Hirsch, J., Aichelburg, C., Gilbert, S. J., et al. (2018b). The present and future use of functional near-infrared spectroscopy (fNIRS) for cognitive neuroscience. Ann. N. Y. Acad. Sci. 1464, 5–29. doi: 10.1111/nyas.13948
Putze, F., Hesslinger, S., Tse, C.-Y., Huang, Y., Herff, C., Guan, C., et al. (2014). Hybrid fNIRS-EEG based classification of auditory and visual perception processes. Front. Neurosci. 8, 373. doi: 10.3389/fnins.2014.00373
Quaresima, V., and Ferrari, M. (2019). Functional near-infrared spectroscopy (fNIRS) for assessing cerebral cortex function during human behavior in natural/social situations: a concise review. Organ. Res. Methods 22, 46–68. doi: 10.1177/1094428116658959
Raslear, T. G., Hursh, S., and Van Dongen, H. P. A. (2011). Predicting cognitive impairment and accident risk. Prog. Brain Res. 190, 155–167. doi: 10.1016/B978-0-444-53817-8.00010-4
Rosenbaum, D., Int-Veen, I., Kroczek, A., Hilsendegen, P., Velten-Schurian, K., Bihlmaier, I., et al. (2020). Amplitude of low frequency fluctuations (ALFF) of spontaneous and induced rumination in major depression: An fNRIS study. Scientific Rep. 10, 21520. doi: 10.1038/s41598-020-78317-y
Saadati, M., Nelson, J., and Ayaz, H. (2020). “Multimodal fNRIS-EEG classification using deep learning algorithms for brain-computer interface purposes,” in Advances in Neuroergonomics and Cognitive Engineering. AHFE (2019). Advances in Intelligent Systems and Computing, Ayaz H. (Ed.) Cham: Springer. p. 953. doi: 10.1007/978-3-030-20473-0_21
Safaie, J., Grebe, R., Moghaddam, H. A., and Wallois, F. (2013). Toward a fully integrated wireless wearable EEG-fNIRS bimodal acquisition system. J. Neural Eng. 10, 056001. doi: 10.1088/1741-2560/10/5/056001
Schirrmeister, R. T., Springenberg, J. T., Fiederer, L. D. J., Glasstetter, M., Eggensperger, K., Tangermann, M., et al. (2017). Deep learning with convolutional neural networks for eeg decoding and visualization. Hum. Brain Mapp. 38, 5391–5420. doi: 10.1002/hbm.23730
Sereshkeh, A. R., Yousefi, R., Wong, A. T., Rudzicz, F., and Chau, T. (2019). Development of a ternary hybrid fNIRS-EEG brain–computer interface based on imagined speech. Brain-Computer Interfaces. 6, 128–140. doi: 10.1080/2326263X.2019.1698928
Shin, J., Kim, D. W., Müller, K. R., and Hwang, H. J. (2018a). Improvement of information transfer rates using a hybrid EEG-NIRS brain-computer interface with a short trial length: offline and pseudo-online analyses. Sensors 18, 1827. doi: 10.3390/s18061827
Shin, J., Kwon, J., and Im, C. H. (2018b). A ternary hybrid EEG-NIRS brain-computer interface for the classification of brain activation patterns during mental arithmetic, motor imagery, and idle state. Front Neuroinform. 12, 5. doi: 10.3389/fninf.2018.00005
Shin, J., von Luhmann, A., Blankertz, B., Kim, D. W., Jeong, J., Hwang, H. J., et al. (2017). open access dataset for EEG+NIRS single-trial classification. IEEE Trans. Neural Syst. Rehabilitation Eng. 25, 1735–1745. doi: 10.1109/TNSRE.2016.2628057
Sievertsen, H. H., Gino, F., and Piovesan, M. (2016). Cognitive fatigue influences students' performance on standardized tests. Proc. Natl. Acad. Sci. USA. 113, 2621–2624. doi: 10.1073/pnas.1516947113
Sirpal, P., Kassab, A., Pouliot, P., Nguyen, D. K., and Lesage, F. (2019). fNIRS improves seizure detection in multimodal EEG-fNIRS recordings. J. Biomed. Opt. 24, 051408. doi: 10.1117/1.JBO.24.5.051408
Sun, Z., Huang, Z., Duan, F., and Liu, A. (2020). A novel-multimodal approach for hybrid brain-computer interface. IEEE Access. 8, 89909. doi: 10.1109/ACCESS.2020.2994226
Tanaka, H., Katura, T., and Sato, H. (2014). Task-related oxygenation and cerebral blood volume changes estimated from NIRS signals in motor and cognitive tasks. Neuroimage 94, 107–119. doi: 10.1016/j.neuroimage.2014.02.036
Tanveer, M. A., Khan, M. J., Qureshi, M. J., Naseer, N., and Hong, K. S. (2019). Enhanced drowsiness detection using deep learning: an fNIRS study. IEEE Access. 7, 137920–137929. doi: 10.1109/ACCESS.2019.2942838
Tomita, Y., Vialatte, F.-B., Dreyfus, G., Mitsukura, Y., Bakardjian, H., and Cichocki, A. (2014). bimodal BCI using simultaneously NIRS and EEG. IEEE Trans. Neural Syst. Rehabilitation Eng. 61, 1274–1284. doi: 10.1109/TBME.2014.2300492
Verma, P., Heilinger, A., Reitner, P., Grünwald, J., Guger, C., and Franklin, D. (2019). “Performance investigation of brain computer interfaces that combine EEG and fNRIS for motor imagery tasks,” in IEEE International Conference on Systems, Man and Cybernetics (Bari: IEEE).
Vitorio, R., Stuart, S., Rochester, L., Alcock, L., and Pantall, A. (2017). fNIRS response during walking — Artefact or cortical activity? A systematic review. Neurosci. Biobehav. Rev. 83, 160–172. doi: 10.1016/j.neubiorev.2017.10.002
von Luhmann, A., and Muller, K. R. (2017). Why build an integrated EEG-fNIRS? About the advantages of hybrid bio-acquisition hardware. Annual International Conference, IEEE Engineering in Medicine and Biology Society. 2017, 4475–4478. doi: 10.1109/EMBC.2017.8037850
von Lühmann, A., Wabnitz, H., Sander, T., and Müller, K. R. (2017). M3BA: a mobile, modular, multimodal biosignal acquisition architecture for miniaturized EEG-fNIRS-based hybrid BCI and monitoring. IEEE Trans. Neural Syst. Rehabilitation Eng. 64, 1199–1210. doi: 10.1109/TBME.2016.2594127
von Lühmann, A., Zheng, Y., Ortega-Martinez, A., Kiran, S., Somers, D. C., Cronin-Golomb, A., et al. (2021). Toward neuroscience of the everyday world (NEW) using functional near-infrared spectroscopy. Curr. Opin. Biomed. Eng. 18, 100272. doi: 10.1016/j.cobme.2021.100272
Vorobyov, S., and Cichocki, A. (2002). Blind noise reduction for multisensory signals using ICA and subspace filtering, with application to EEG analysis. Biol. Cybern. 86, 293–303. doi: 10.1007/s00422-001-0298-6
Wade, A. R. (2002). The negative BOLD signal unmasked. Neuron. 36, 993–995. doi: 10.1016/S0896-6273(02)01138-8
Wallois, F., Mahmoudzadeh, M., Patil, A., and Grebe, R. (2012). Usefulness of simultaneous EEG-fNIRS recording in language studies. Brain Lang. 121, 110–123. doi: 10.1016/j.bandl.2011.03.010
Wallois, F., Patil, A., Héberlé C., and Grebe, R. (2010). EEG-fNIRS in epilepsy in children and neonates. Neurophysiologie Clinique/Clinical Neurophysiol. 40, 281–292. doi: 10.1016/j.neucli.2010.08.004
Wang, J.-Z., Williamson, S. J., and Kaufman, L. (1992). Magnetic source images determined by a lead-field analysis: the unique minimum-norm least-squares estimation. IEEE Trans. Neural Syst. Rehabilitation Eng. 39, 665–675. doi: 10.1109/10.142641
Wascher, E., Reiser, J., Rinkenauer, G., Larrá M., Dreger, F. A., Schneider, D., et al. (2021). Neuroergonomics on the go: an evaluation of the potential of mobile eeg for workplace assessment and design. Hum. Factors. 00187208211007707. doi: 10.31219/osf.io/vkf79
Wascher, E., Rosch, B., Sänger, J., Hoffman, S., Schneider, D., Rinkenauer, G., et al. (2014). Frontal theta activity reflects distinct aspects of mental fatigue. Biol. Psychol. 96, 57–65. doi: 10.1016/j.biopsycho.2013.11.010
Wickens, C. D. (2008). Multiple resources and mental workload. Hum. Factors 50, 449–455. doi: 10.1518/001872008X288394
Yin, X., Xu, B., Jiang, C., Fu, Y., Wang, Z., Li, H., et al. (2015). A hybrid BCI based on EEG and fNIRS signals improves the performance of decoding motor imagery of both force and speed of hand clenching. J. Neural Eng. 12(36004). doi: 10.1088/1741-2560/12/3/036004
Yoo, S.-S., Guttmann, C. R. G., Ives, J. R., Panych, L. P., Kikinis, R., Schomer, D. L., et al. (1997). 3D Localization of surface 10-20 EEG electrodes on high resolution anatomical MR images. Electroencephalogr. Clin. Neurophysiol. 102, 335–339. doi: 10.1016/S0013-4694(96)95088-9
Zafar, A., Hong, K.-S., and Khan, M. J. (2016). “Initial dip detection based on both HbO and HbR vector-based phase analysis,” in Proceedings of the SICE Annual Conference (Tsukuba). doi: 10.1109/SICE.2016.7749234
Zhu, G., Li, R., Zhang, T., Lou, D., and Wang, R., Y., Zhang. (2017). “A Simplified Hybrid EEG-fNIRS Brain-Computer Interface for Motor Task Classification,” in 8th International IEEE EMBS Conference on Neural Engineering (Shanghai). doi: 10.1109/NER.2017.8008310
Keywords: multimodal brain imaging, electroencephalography, near-infrared spectroscopy, neuroergonomics, human-machine interfaces
Citation: Bourguignon NJ, Lo Bue S, Guerrero-Mosquera C and Borragán G (2022) Bimodal EEG-fNIRS in Neuroergonomics. Current Evidence and Prospects for Future Research. Front. Neuroergon. 3:934234. doi: 10.3389/fnrgo.2022.934234
Received: 02 May 2022; Accepted: 20 June 2022;
Published: 12 August 2022.
Edited by:
Giovanni Vecchiato, National Research Council of Italy, ItalyReviewed by:
Rihui Li, Stanford University, United StatesDavid Perpetuini, University of Studies G. d'Annunzio Chieti and Pescara, Italy
Copyright © 2022 Bourguignon, Lo Bue, Guerrero-Mosquera and Borragán. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Nicolas J. Bourguignon, nicolas.bourguignon@mil.be