Dual Passive Reactive Brain-Computer Interface: A Novel Approach to Human-Machine Symbiosis

The present study proposes a novel concept of neuroadaptive technology, namely a dual passive-reactive Brain-Computer Interface (BCI), that enables bi-directional interaction between humans and machines. We have implemented such a system in a realistic flight simulator using the NextMind classification algorithms and framework to decode pilots' intention (reactive BCI) and to infer their level of attention (passive BCI). Twelve pilots used the reactive BCI to perform checklists along with an anti-collision radar monitoring task that was supervised by the passive BCI. The latter simulated an automatic avoidance maneuver when it detected that pilots missed an incoming collision. The reactive BCI reached 100% classification accuracy with a mean reaction time of 1.6 s when exclusively performing the checklist task. Accuracy was up to 98.5% with a mean reaction time of 2.5 s when pilots also had to fly the aircraft and monitor the anti-collision radar. The passive BCI achieved a F₁−score of 0.94. This first demonstration shows the potential of a dual BCI to improve human-machine teaming which could be applied to a variety of applications.

1. Introduction

Brain-Computer Interfaces (BCI) offer a direct communication pathway between a user and a machine without requiring any muscular engagement (Clerc et al., 2016). To this end, BCI derives the user's intentions and mental states from neural signals. The decoding of specific neural signals triggers interactions with aspects of the computerized environment (e.g., moving a cursor, keystrokes) or external devices (e.g., prostheses) to which they are associated with. Its non-invasiveness and high temporal resolution along with its ease of setup and relatively low cost have established surface electroencephalography (EEG) as the most widely used brain imaging method for BCI (Lotte et al., 2018). While BCI was initially developed within the confines of standard laboratory conditions, recent advances in mobile neurophysiological sensing devices and artificial intelligence have led to a renewed interest in BCI applied to real-world contexts (Fairclough and Lotte, 2020). As a result, these neurotechnologies are now expanding in the clinical field (assistive technologies, motor rehabilitation, etc.), spreading to the entertainment industry to enhance gaming experience, but also extending to the general public through wellbeing (e.g., meditation and relaxation induction, sleep improvement) and domotics (e.g., home automation) applications (Brouwer, 2021). Following this trend, the range of neuroergonomics applications that can benefit from BCI broadens as sensors and interfaces become ever less intrusive (Dehais et al., 2020a; Gramann et al., 2021). Indeed, BCI has the potential to alleviate mental and physical loads associated with the repetition of straining actions (Carelli et al., 2017; Maksimenko et al., 2018), to improve task performance both in terms of its precision and speed, and to promote new forms of interactions to enhance human-machine teaming (Dehais et al., 2020b). Such assistance is particularly desirable in the context of aircraft operations. Flying a plane is a highly demanding task from a cognitive standpoint since it takes place in a dynamic and uncertain environment. It is well-established in cognitive science literature that attentional resources are limited (Kahneman, 1973). This limited capacity implies that the pool of cognitive resources has to be distributed across competing sensory modalities (Wickens, 2008). The amount of cognitive resources available at any given time is also mediated by other factors such as stress and mental fatigue. Following prolonged periods requiring attentional focus, individuals typically exhibit performance decline as fatigue ensues (Dehais et al., 2018). Pilots have to attend to and assess several sources of visual and auditory information spread around the cockpit, and have to take decisions under time pressure and execute maneuvers in a timely manner (Wickens and Dehais, 2019; Behrend and Dehais, 2020). The combination of the stress-inducing context of flight operations and the need to sustain attentional focus over long periods of time is particularly taxing and tiring for pilots. The accumulation of mental fatigue and stress hinders access to the pool of cognitive resources (Hancock and Szalma, 2008), which in turn can leads to completely overlooking information (e.g.., attentional tunneling). This phenomenon has been observed in experienced pilots and can have dramatic consequences (Wickens and Alexander, 2009; Dehais et al., 2019a; Mumaw et al., 2019). The on-line estimation of pilots' monitoring ability combined with implementing new types of interactions through neuro-adaptive technologies may therefore have critical implications for flight safety.

There are several ways whereby BCI can enhance pilot-cockpit teaming. Firstly, active and reactive BCIs allow users to perform interactions under voluntary control via their brain waves (Hong and Khan, 2017; Lotte and Roy, 2019). Several studies disclosed that such technology can be used by pilot to control the flight-path of airplanes and drones (Fricke et al., 2014; Nourmohammadi et al., 2018; Rodriguez-Bermudez et al., 2019). Active BCI require the users to deliberately produce brain signals to interact with the BCI, as with mental imagery. In contrast, reactive BCI (rBCI) makes advantage of the user's cerebral responses elicited by different stimuli. Each stimuli is associated with a different command. Due to their robustness and quick onset (i.e., below 50 ms), the Visual Evoked Potentials (VEP) elicited through the presentation of modulated visual stimuli are a popular and efficient approach for rBCI (Zhu et al., 2010; Chevallier et al., 2021). They offer very high classification performance (Nakanishi et al., 2018; Nagel and Spüler, 2019). Neural activity recorded in the visual cortex with surface EEG is sensitive to temporal and frequency features of the visual stimuli. Two main types of VEP-based paradigms can be distinguished: Steady-States Visually Evoked Potentials (SSVEP) and code-Visually Evoked Potentials (c-VEP). While SSVEP consists of the periodic modulation of visual features (e.g., contrast, color) at a regular frequency sustained over time, c-VEP waveforms are generated by pseudo-random binary (on/off) sequences. The c-VEP pseudo-random sequences are usually broadband and aperiodic (Shirzhiyan et al., 2019). They are defined so that temporal shifts have minimum cross-correlation and ensure a good separation between classes. VEP-based paradigms present the main advantage of relatively short training time, compared to P300 ERP or Motor Imagery paradigms, as only a low number of short-lasting trials is usually required to achieve accurate calibration (Nagel and Spüler, 2019). In the field of aviation, visual BCI could offer promising perspectives for pilots by allowing them to free their hands when interacting with some actuators (e.g., landing gear, flaps). This could be particularly relevant during high g-force scenarios or critical flight phases (e.g., low altitude situations) that require both hands to control the stick and the thrust.

A second approach to improve pilot-cockpit teaming is to consider the use of passive BCIs (pBCI). This latter type of neuroadaptive technology supports implicit interaction by monitoring mental states (e.g., stress, fatigue) and adapting human-machine interactions to overcome cognitive bottlenecks (Zander and Kothe, 2011; Ewing et al., 2016). Several pBCI studies have been implemented in the field of aviation to infer mental workload (Gateau et al., 2018; Dehais et al., 2019a), failure of attention (Dehais et al., 2019b,c), flying performance (Scholl et al., 2016; Klaproth et al., 2020), and mental fatigue (Dehais et al., 2018). Interestingly enough, some authors managed to close the loop by triggering adaptive automation to prevent mental overload and task disengagement (Prinzel et al., 2000; Aricò et al., 2016). Generally, specific frequency-domain features are computed over the electrophysiological signal to account for different mental states (for a review, see Borghini et al., 2014). For instance, changes in mental demand are related to the variation in the alpha band power and in theta band power over fronto-parietal sites (for a review, see Borghini et al., 2014). Some studies also disclosed that increased beta (Matthews et al., 2017) is a neural marker of higher mental efforts. Alternatively, time-domain analyses over the EEG signal (i.e., event-related potential) can also predict variations of cognitive performance and attentional states (Brouwer et al., 2012; Roy et al., 2016; Dehais et al., 2018). One major drawback of such approach is that the calibration requires the induction of the mental states (e.g., different levels of stress or attention) in a repetitive fashion to train the model. It is difficult to achieve under laboratory conditions and, more importantly, is detrimental to the user experience. One alternative approach would be to take advantage of VEP and to use code-VEP tagging stimuli to implement a pBCI. By placing these flickers within the background of different regions of interest, one can measure the intensity of the brain response and derive the level of attention allocated to these specific areas.

Taken together, all these studies demonstrate the benefit of rBCI and pBCI to improve pilot-cockpit teaming and flight safety. However the rBCI and pBCI technologies to date have been used separately, whereas many of everyday-life tasks involved conjointly some voluntary interactions with a user interface and the monitoring of the state of the machine. Moreover, the same device (e.g., EEG) could be used to collect brain data and feed different algorithms in charge to control an interface and to infer the user mental state. Such an approach would pave the way to design a novel concept of neuroadaptive technology, namely a dual BCI (dBCI). By “dual,” we mean that it combines both “reactive” and “passive” components of the BCI to support direct and implicit interactions for end-users. This approach echoes with the concept of invasive bidirectional BCI for disable people that have been designed to translate motor cortex activity into signals to control an apparatus and to provide feedback by translating artificial sensor to restore sense of touch (Hughes et al., 2020). In our case, the dBCI relies on a non-invasive technology such as surface electrodes placed over the scalp (i) to allow an end-user to directly control a machine through their brain activity and (ii) to allow a machine to communicate feedback to its end-user and thus adapt human-machine interaction to overcome cognitive bottleneck.

The objective of this study was to implement such a dBCI in the cockpit. To meet this goal, we used the NextMind 9-electrode dry EEG system and their Unity framework (www.next-mind.com) that allows to implement asynchronous code-VEP based BCI for up to 10-class problems. We chose the NextMind as their hardware is light, fast to setup, and the classification is plug-and-play but this could have been done with another BCI system. Our main motivation was not to focus on the algorithmic implementation but to demonstrate the effectiveness of our genuine dBCI to decode pilots' intention to interact with the flightdeck (reactive, rBCI) and to infer their level of attention on a monitoring task and to adapt the interaction accordingly (passive, pBCI, see Figure 1).

FIGURE 1

Figure 1. Implementation of our dual passive and reactive Brain Computer Interface in the cockpit. It enables human-machine bi-directional communication: (1) the pilots can directly interact with some flight deck actuators (e.g.., the landing gears—as shown by the red rectangles) using information from the EEG signal, and (2) the flight deck can send visual feedback to the pilot and adapt the interaction when poor cognitive performance is detected.

The designed scenario in the flight simulator requires to tackle both direct (rBCI) and implicit (pBCI) interactions. Indeed, the participants had to perform several checklists operated through rBCI and a radar monitoring task which was handled by the pBCI while completing an engaging traffic pattern exercise. We evaluated the rBCI during two contrasted experimental conditions in terms of task difficulty. In one condition, the participants were interacting with the rBCI alone while the plane was operated by the autopilot. In another condition, they were operating the rBCI while flying the plane and monitoring the radar. We collected objective measures (reaction time and accuracy) and subjective measures (level of mental demand and frustration). It was expected that the multitasking condition would lead to longer reaction times to interact with the rBCI, more frustration and lower accuracy. We evaluated the pBCI component using classical machine learning metrics (F₁ score, sensitivity, and specificity).

2. Materials and Methods

2.1. Participants

Twelve participants (one female, mean age = 29.8 year old and SD = 7.4, mean flight hours = 488.9, and SD = 115.7), all students and staff members from our aeronautical university, took part in the experiment. The study was approved by the Local Ethics Committee (approval number 2020−334). We followed specific COVID procedures implemented by our Health, Safety, and Working Conditions Committee (anti-COVID face masks, sanitization of the flight simulator, disinfection of the eye tracker, and EEG device following a thorough cleaning procedure).

2.2. Dual BCI Implementation

We used the NextMind Unity Software Development Toolkit (SDK)¹ to implement the rBCI and the pBCI. An MSI laptop (processor Intel i7−6700HQ with 16 GB RAM and NVIDIA GeForce GTX 960M graphic card) was used to received the EEG data stream trough Bluetooth Low Energy. This PC executed the decoding algorithms and sent the decoded brain command to the core simulator via socket TCP/IP (using an Ethernet connection) that consisted of a thread of four commands (integer type) to change the states of the flaps, the landing gear, the landing and taxi lights and the autopilot (see Figures 1, 2). Since the NextMind is a closed system, it does not provide direct access to the processing time for the classification. However, we managed to estimate the pace of update for the classification output, before it is displayed (to avoid a bias with the refresh rate of the screen). Our measures indicated that the framework takes in average 10.38 ms (SD = 0.05 ms) to refresh the classification score.

FIGURE 2

Figure 2. (Left) The participants had to perform the checklists with the reactive BCI (in red, upper part of the screen) while their monitoring performance on the radar screen was assessed with the passive BCI (in blue, lower part of the screen). (Right) The dual BCI setup in the flight simulator.

2.2.1. rBCI

The rBCI display consisted of four toggle switches (diameter = 4cm) that are defined as “Neurotag” interactive elements (i.e., a NextMind code-VEP) in Unity. They are depicted in the upper left part of Figure 1, outlined in red. These switches are dedicated to lower/retract the flaps, switch on/switch off the taxi and landing lights, to lower/retract the landing gears and to engage/disengage the autopilot system (see Figure 2, left).

2.2.2. pBCI

For the pBCI, the region of interest was a radar. We have integrated a “Neurotag” (diameter = 13.1 cm) in the background. It is depicted in the lower left part of Figure 1, outlined in blue. The brain response to this background c-VEP was used to assess the level of attention of the pilot to the region. Thus, instead of using the regular rBCI classification procedure, we have taken advantage of a continuous confidence score (∈[0, 1]) defined by the NextMind Engine as a proxy measure for attention to the radar. This score reflects the classification certainty for a rBCI. If this score was too low and a collision was incoming, our system would infer that the pilot is not paying attention at the radar and therefore not able to avoid the collision. It would then automatically activate the anti-collision maneuver and trigger an orange visual alert (see Figure 3). Based on a preliminary experiment with 5 participants, we set a threshold of 0.1 on the confidence score to have good responsiveness to determine whether the pilots were actually monitoring the anti-collision radar or not (see Figure 2, right). The classification processes are different between the rBCI (regular and hard decision of classification) and the pBCI (the certainty of classification by the model as attention probe), though they both use the response to the same type of c-VEP as input.

FIGURE 3

Figure 3. (Left) The pBCI detected that the pilot was correctly monitoring the radar and, in return, displayed a green visual feedback. (Right) The pBCI detected that the pilot did not notice the potential collision. It then displayed an orange warning and automatically simulated the avoidance maneuver.

2.3. Flight Simulator

We used our three-axis hydraulic (pitch, roll, height) motion flight simulator to conduct the experiments. It has eight external panoramic displays that reproduces the outside world based on the Flight-Gear open-source software.² It simulates twin-engine aircraft equipped with two side-sticks, a thrust, two rudders, and an advanced auto flight system (Figure 2). Its user interface is composed of a Primary Flight Display, a Navigation Display, and an Electronic Central Aircraft Monitoring Display and a head-up-display (HUD). The HUD provides basic flight parameters (speed vector, angle of attack, total energy) and thus allows the pilots to control the flight path and the speed of the airplane. The flight simulator refreshes every 20 ms the inputs from the BCI system sent through the TCP/IP connection, in addition to the information coming from the traditional organs of piloting like the autopilot button, the flaps and landing gear levers, etc. The dual-BCI user interface (checklists and anti-collision radar) was displayed on a head down 17″ screen (1, 280 × 1, 024–60 Hz) facing the participants (head distance = 80 cm).

2.4. Scenario

We have designed a scenario in which the participants had to perform two checklists (rBCI) and a radar monitoring task (pBCI). The flying task consists of a first landing at Toulouse Blagnac runway (33R) followed by a go-around, then a traffic pattern exercise leading to a final landing. The participants had to use the rBCI during the:

1. first landing: to lower the gear, to lower the flaps, to switch on the landing and taxi lights;

2. go-around: to retract the gear, to retract the flaps, to switch off the taxi and landing lights and to engage the autopilot;

3. crosswind: to disconnect the autopilot;

4. downwind: to lower the flaps and to switch on the lights;

5. final landing: to lower the landing gear.

The four items of the checklist did not demand immediate responses but their correct activation/deactivation is of critical importance during the execution of the traffic pattern, as specified by the standard operational procedures. As depicted in Figure 1 these actuators (in red) were located out of reached from the left-seated participant-pilot and any interactions with them require torso movements and arm extension. Our objective was to show that a reactive BCI could allow the pilots to keep their hands on the stick and thrust lever while performing the required check-lists.

Meanwhile and while flying the plane, participants should monitor the anti-collision radar (pBCI). The anti-collision task was not performed in the auto-pilot condition (low workload) but only while flying (high workload). The collisions were not linked to the actual flight pattern exercise and only appeared on the radar. In practice, it implies looking frequently at this radar to determine if another plane (represented by a red circle) reaches the center of the radar, indicating a collision within 6 s. In the case of an incoming collision, pilots were asked to focus on the radar. This would mean that they have acknowledged it and would avoid it. When a collision is incoming, an anti-collision system would be triggered by the pBCI either by (i) the detection of sufficient attention to the radar (ii) or by the detection of inattention of the pilot. With this scenario, the expertise of the pilot to avoid collisions is primary and it is bypassed only if the system estimated that the pilot is distracted. It is different to a scenario that would systematically activate the anti-collision maneuvre, which sets aside the pilot assessment of the situation. Thus, the idea of this pBCI is to offer supplementary assistance to the pilot during overwhelming situations but not to automate his or her tasks. It is worth noting that we artificially increased the number of potential collisions compared to what pilots are expected to face during real operations for the purpose of the study. The goal was to have enough events of missed collisions so as to have statistically significant results without substantially extending the experiment duration.

In order to assess the effect of mental workload and multitasking on the use of rBCI, we manipulated two variations of the experimental conditions:

• rBCI alone condition (single task ondition): the participants only had to perform the different checklists without flying (the aircraft was in automated flight mode). They also did not have to do the anti-collision monitoring task;

• r/pBCI and Flying condition (multi-tasks condition): the participants had to perform the checklists and the radar monitoring task while manually flying to perform the five legs of the scenario.

These stand for two realistic conditions of flight: auto-pilot mode and full control. A video that illustrates the experimental scenario can be downloaded in the Supplementary Materials.

2.5. Protocol

The participants underwent 30 min of training for the flight simulator without the dBCI. It included a short tutorial about how the simulator worked (user interface, flight parameters), several landings and a complete traffic pattern exercise. Participants were then equipped with the 9-electrode NextMind EEG headset (Oz, PO7, PO8, PO1, PO3, PO4, P3, P4), placed over the visual cortex with the lowest electrode (Oz) set over the inion, as recommended by the manufacturer best practices.³ Following this, participants went through the BCI calibration phase which output a calibration score (between 1 and 5). During the calibration phase, the participant had to focus on a single c-VEP for 40 s. The calibration was redone if the score was below 4 to ensure satisfactory performances. After this setup, participants completed the two conditions (rBCI alone and r/pBCI + flying). The order of the two conditions was pseudo-randomized to counterbalance between all participants and to control for potential fatigue effects: six participants started with the “rBCI alone” condition and the remaining six with “r/pBCI and flying” condition. Finally, after completing each experimental condition, participants were asked to fill the subjective questionnaire. The total length of the experiment for a subject was about 1 h.

2.6. Measurements

2.6.1. Subjective Measures

The participants reported their subjective levels of mental effort and frustration to perform the checklists using a Likert scale (1 = very low, 10 = very high) in the two experimental conditions. These subjective assessments were done immediately after completing the experiments.

2.6.2. Objective Measures

We used a posteriori the recording from a Tobii Glasses II eye tracking system (Tobii Pro AB, Stockholm, Sweden) to manually evaluate the efficiency of the dBCI system. Tobii Glasses II is a wearable eye tracker with an embedded scene camera of frequency 60 Hz (i.e., a sampling interval of 16.67 ms). The recording consists in the video of the scene camera, with a large angle, with in overlay the gaze point of the subject. Built-in parallax and slippage compensation methods were performed to maintain tracking accuracy throughout the recording. During the calibration procedure a target probe was presented in the cockpit. During recording, if the gap between two retrieved samples was more than 16.67 ms and <75 ms, the sample is considered as missed. Missed samples were interpolated using the median on a rolling window of five samples. If the gap is more than 75 ms, the samples were considered lost and not interpolated. The proportion of missed and lost gaze samples was bellow 20% for all the participants. The continuous eye-tracking sequences were then smoothed using a non-weighted moving median filter with a window size of three samples. The built-in Tobii I-VT Fixation Filter was set with a velocity (expressed in visual degrees per second) threshold of 30°/s over 20 ms window length. Gaze samples above the velocity threshold were classified as saccade samples. Short fixations lasting <50 ms were discarded. Adjacent short fixations were merged when their inter-fixation (saccade) duration was lower than 75 ms or that the visual angle difference between these fixations was lower than 0.5°. A lower threshold of 200 ms was used for the definition of visual fixations. The eye tracker is not part of the dBCI system. The recording was used (i) to compute the accuracy of the rBCI and measure the reaction time to perform the checklist events, and (ii) to quantify the performance of the pBCI as follow:Regarding the reactive BCI:

• A “true positive” was labeled if the participant gazed at one of the checklist item and this latter is then activated/deactivated by the NextMind classification framework;

• A “false positive” was labeled if the state of one checklist item changed without any eye fixation on it;

• The reaction time to interact with checklist item was computed by measuring the time interval from the first fixation on this item until this latter is finally activated/deactivated by the NextMind classification framework;

Regarding the passive BCI:

• A “true positive” was labeled if the participant gazed at the radar and the confident score computed by the NextMind algorithm reached a value of 0.1 within the last 6 s, defined as the time to avoid the collision;

• A “false negative” was labeled if the participant did gaze at the radar but the confident score did not reach a value of 0.1 within the 6 s;

• A “false positive” was labeled if the participant did not gaze at the radar but the confident score reached a value of 0.1 within the last 6 s.

3. Results

In this section, we present the subjective and quantitative results of our dual (reactive and passive) BCI.

3.1. Subjective Results

A paired t-test (p < 0.001) demonstrated that the mean level of frustration to interact with the checklist items was significantly lower in the “rBCI alone” condition (mean = 2.6, SD = 2.0) than in the “r/pBCI and flying” condition (mean = 4, SD = 2.2). Similarly, a paired t-test (p < 0.001) demonstrated that the mean level of mental workload to interact with the checklist items was significantly lower in the “rBCI alone” condition (mean = 2.2, SD = 1.4) than in the “rBCI and flying” condition (mean = 5, SD = 2.0, see Figure 4). Such reported results were expected as the “rBCI and flying” is way more demanding compared to “rBCI alone.” We selected these two conditions as they represent two classical flight situations: auto-pilot with low workload and flying with high workload.

FIGURE 4

Figure 4. Questionnaire responses of the participants to statements regarding frustration and mental workload in the two conditions. The width of the bars and the numbers inside them reflect the number of participants that provide a certain rating on the Likert scale for the associated question (1–8). Lower scores (represented in green) indicate a better user experience whereas higher scores (represented in red) are associated with negative experiences (higher mental workload and frustration).

3.2. rBCI Objective Results

In the “rBCI alone” condition, the classification accuracy reached 100% since all the participants interacted successfully with the checklist items without experiencing any false positives (i.e., activation of an undesired item). Unlike a traditional rBCI with a fixed epoch length, here the epoch length varies. In the “r/pBCI and flying” condition, all the participants managed to fly the different legs of the aircraft while interacting with the different checklist items. The classification accuracy reached 98.5% since only two single false positives occurred out of 132 trials (i.e., 11 checklist items ×12 pilots).

A paired t-test indicated that the mean reaction time to activate the checklist items was significantly lower (p = 0.002) when interacting with the rBCI alone (mean = 1643.7 ms, SD = 390.6 ms) than when interacting with the rBCI while flying and monitoring the radar task (mean = 2493.1 ms, SD = 1055.4 ms). Mean reaction times per participant can be found in Figure 5.

FIGURE 5

Figure 5. Mean reaction time per participant and grand average reaction time to activate the checklist items when interacting with the rBCI alone (in blue) and when interacting with the rBCI along with the flying task and the pBCI (in orange). Bars represent standard deviations.

3.3. pBCI Objective Results

The pBCI system have classified as missed by the pilots, a total of 53 potential collisions out of 203 events. Among these 53 missed collisions, the recordings from the eye tracker, manually examined, showed us that in nine cases, the pilot was actually fixating the radar but the pBCI considered as an attentional lapses (False Negative, FN). For the remaining 44 cases (True Negative, TN) and still based on the manual study of the recordings from the eye tracker, the pBCI had accurately detected that the pilot was distracted and compensated attention errors by automatically triggering the orange alert. These results are summarized in Figure 6. During the post-hoc analysis, we have also determined that in eight cases among the 150 potential collisions classified by the pBCI as acknowledged by the pilots, but the eye tracker disclosed that the participants were not actually paying attention (False Positive, FP). Therefore it makes a total of 150−8 = 142 true positives (TP). As such, the True Positive Rate (TPR, also called recall=$\frac{TP}{TP+FN}$) of the pBCI system was 94.04%. The precision ($\frac{TP}{TP+FP}$) was 94.67% and it gives a F₁-score (harmonic mean of precision and recall) of 0.94. The F₁-score reflects the number of false positives and negatives along the true positives and negatives while traditional classification accuracy only provides information about true positives and negatives. Within our framework, missing a collision could have dramatic consequences, thus it was more informative to consider F₁-score than accuracy. The True Negative Rate (TNR, also called selectivity = $\frac{TN}{TN+FP}$) was 84.61%.

FIGURE 6

Figure 6. Confusion matrix for the pBCI. The ground truth was obtained using post-hoc manual analysis of the recordings from the eye tracker. Acknowledged collision corresponds to a collision actually spotted on time by the pilot.

4. Discussion

This study is the first demonstration of a dBCI system that promotes direct (rBCI) and implicit (pBCI) interaction between the user and the interface. In our task, the reactive and passive aspect did not interact together but were motivated by the task involving interactions with the flight deck and the monitoring of the radar screen. More specifically, the rBCI component allowed the participants to interact with the flight simulator, sending commands through a TCP/IP connection, to change the state of specific flight deck actuators. This approach differs from previous aviation-oriented BCI studies, which apply BCI to directly control the aircraft's trajectory (Fricke et al., 2014; Nourmohammadi et al., 2018; Rodriguez-Bermudez et al., 2019). It is important to note that in all these previous studies, the pilots were required to fully allocate attentional resources toward the BCI interface in order to perform the flying task whereas the flight performance under the BCI condition does not meet the standards of manual flying accuracy. Another important consideration is that operating an aircraft requires to constantly monitor a rich flow of information distributed across the cockpit. Therefore such artificial setups in which pilots only pay attention to a single source of information does not accurately reflect real-flight situations and may not be transferable to these use cases. Moreover, any relevant secondary task (e.g., radar monitoring, communication with air traffic control) or critical stimuli (e.g., alarms) will distract them from flying. The pBCI component was precisely dedicated to assist the pilot when performing a secondary task, namely monitoring an anti-collision radar task. To the authors' best knowledge, this is the first time that a VEP is used to monitor attention. Traditionally, this is performed using frequency and/or time-domain features to assess the level of attention (Brouwer et al., 2012; Dehais et al., 2018).

The rBCI analyses disclosed state of the art results in the control condition (auto-pilot) with an accuracy of 100% and a average reaction time of 1.6 s. It should be compared with other asynchronous, online and with 4 classes BCIs of the literature. Kalunga et al. (2016) have reported a reaction time of 1.1 for a 3 class problem and an accuracy of 87.3% for they asynchronous and online BCI. In Gembler et al. (2020), authors have achieved a reaction time of 2.31 s for a 4 class problem and a mean accuracy of 94.4%, also online and asychronous. The rBCI also led to very high accuracy (98.5%) in the condition where the participants had to operate it while flying the plane and monitoring the radar. The mean reaction was longer but quite acceptable (2.5 s) and allowed the pilots to perform the checklist in a timely manner while still being able to navigate and safely land the plane. This slight decrease in classification accuracy (−1.5%) and increase in reaction time (+0.9 s) can be explained in terms of higher mental demand as previously demonstrated by Vecchiato et al. (2016). Indeed, the participants were engaged in a complex multitasking activity, leading them to divide their visual attentional resources between the flying, the radar, and the rBCI tasks. Our subjective results seemed to confirm this hypothesis as our participants expressed significantly higher mental workload and higher frustration in the multitasking condition. It is worth noting that the variability in reaction times between users was also much higher in the multitasking condition. Indeed, the performance of a BCI is tightly related to the subjective mental workload experienced by the users (Felton et al., 2012), the higher the less the BCI performance would be. While the task difficulty was the same for all subjects, the subjective workloads experienced were different as it depends also on the skills, fatigue, etc. of the subjects. Nevertheless, the implemented rBCI demonstrate sufficient efficiency and responsiveness to be operated, even while flying and with high workload which was the aim.

Beside that, the pBCI findings also seemed to indicate the soundness of VEP to probe the level of attention to a monitoring task. One of the main advantage of this approach is that the same calibration, lasting only 40 s, was used to train the rBCI and the pBCI. To the best of our knowledge, this is the first online demonstration that flickering stimuli could be used for pBCI purpose. The behavioral results showed that our pilots missed a total of 44 out of 203 collisions. The pBCI provided assistance to the pilots by simulating safety maneuvers with an acceptable rate of false negatives. We believe that this hybrid approach provides flexibility since the expertise of the pilot is kept at the center of the design, while providing a safety net in case the pilot's attentional resources are engaged on other aircraft operations. We consider it as an improvement compared to automatically activating the anti-collision safety whenever a collision is coming. In some cases, the pilot could have paid attention to this incoming danger but determined that the anti-collision safety was not necessary. In our scenario, the anti-collision safety was triggered only if the pilot deliberately chose to do it or if he or she was not paying attention. However, it is worthy to note that such adaptive automation could cause some drawbacks such as over-reliance on fallback mechanisms.

Despite these promising results it appeared that in eight cases out of 150, the pBCI did not detect pilot's attentional lapses and failed to trigger the automatic maneuver (false positive). The occurrence of such failure could lead to critical scenarios in real-life situations. One has to keep in mind that we used a an empirical threshold set to 0.1 for all participants to infer the attentional state and trigger the pBCI. This fixed threshold, while proving to be efficient overall may not be optimal for each individual. A possible solution is to define an individualized threshold to obtain an optimal ratio between false positive and true negative. Interestingly enough, the findings disclosed that in nine instances out of 150 events, the pBCI classified that the participants detected the collision whereas the participants were not glancing at the radar as attested by the eye tracker (false negative). However, during the debriefing, the participants declared that they actually spotted the collision via peripheral vision. Such situation is possible since peripheral vision, via the rods, is highly sensitive to motion and flickering objects such as the VEP stimuli that were used in this experiment. For instance, some authors demonstrated the potential of using concurrent presentation of VEPs-based stimuli to assess processes outside of the focus of spatial attention (see for reviews Vialatte et al., 2010; Norcia et al., 2015). Future work should investigate this hypothesis.

Following on these promising results, this work calls for refinement through an actual implementation of the dBCI in the wild, i.e., under operational conditions. For instance and from an operating time point of view, it is blatant that using one's hands is still faster than the rBCI to interact with the cockpit. One solution is to provide hybrid interaction so that the pilots can choose to use either the BCI or physical modalities depending on the flight phase and time pressure. One could imagine that the pilot would interact trough the BCI when the autopilot is engaged, so to increase his comfort and availability, and manually during more dynamic phases of the flights (i.e., maneuvers), when full commitment is required. In the present study the eye-tracking data were used as to provide a ground truth to assess both passive and reactive BCI performances in post-hoc analyses. The gaze data could also have been leveraged as a complementary source of spatial and temporal information to improve BCI speed and accuracy. Indeed, the eye-tracking data could provide contextual information, as to which area of the environment the user focus attention. This contextual information (regions of interest in eye-tracking terminology), would lead to the activation of only a subset of the VEP stimuli based on the localization of user's attention (Lin et al., 2019). This approach would allow to artificially increase the number of classes while using a constant number of distinct VEPs therefore reducing the complexity of the classification problem (Stawicki et al., 2017). This lower number of distinct VEP also implies to acquire less calibration data compared to traditional paradigms in which each class is represented by a distinct VEP. Moreover, by triggering stimuli presentation, the eye-tracking data would provide the precise onset time of the stimuli thus giving valuable information for asynchronous BCI allowing to compute classification only when necessary and optimize computing power and time requirements. Additionally, gaze contingent rBCI would reduce the bottom-up influences of the visual stimuli on the user attentional resources. Indeed, the high contrast nature of stimuli commonly used in VEP-based paradigms may distract attentional resources away from the primary task (Zhao et al., 2018). From an user experience perspective, the reduction in the number of VEP stimuli presented at the same time may improve visual comfort. Eventually, visual fixations on the radar area may be used as a two-step certification process to validate the VEP-based pBCI decision. Similarly for the rBCI, visual fixations within VEP stimuli area could be used to confirm the intention of an user to interact with a command. Overall eye-tracking information concurrent to VEP-based BCI classification outputs may be used to further improve classification performance and reduce the rate of false positives. This latter point is particularly important in the context of translating the proposed dual BCI system to real cockpit day-to-day flight operations as meeting high standards of aviation certification criteria is particularly challenging (10⁻³ allowable failure probability). In the context of aviation, any undesired activation of a command could jeopardize flight safety.

To conclude, we believe that the concept of dBCI opens promising prospects to improve human machine symbiosis for neuroergonomics applications in many domains such as transportation, industrial workplaces, medical care but also for disable people. We have developed a proof of concept that relies on the code-VEP based stimuli and classification tools provided by the NextMind company. However, this approach is limited as it is a closed system. The EEG data streams are not accessible and the processing and classifications algorithms are not provided by the manufacturer. It is thus not possible to assess data quality or any data loss related issue. However, we hope that this study should encourage the development of open-source c-VEP code for the scientific community. Alternatively, other types of stimuli and classification procedures could be explored. For instance, the decoding of user's expectation (Zander et al., 2016) represent an interesting alternative since it does not require to present additional stimuli in the user interface. Combining together active and reactive tasks and the use of hybrid EEG and functional near infrared spectroscopy based BCI could maximize the number of commands (Hong and Khan, 2017) provided by a BCI. Regarding the pBCI, the use of more traditional features (e.g., changes in the well defined band of power for EEG) or more advanced ones like brain connectivity could be considered to target specific degraded mental states (cognitive fatigue, failure of auditory attention) so as to trigger the most appropriate neuro-adaptive solutions (Dehais et al., 2020b). Indeed, several solutions could be designed to dynamically optimize human-machine teaming by (1) adapting the user interface using notifications to alert of impeding hazards, (2) adapting the task and the level of automation to maintain the performance efficiency of the operators, and (3) “neuro-adaptating” the end-users to warn them of their mental state and stimulate them to augment performance (e.g., neurofeedback). Eventually, some further work should be conducted to assess the effect of training with such system and stimuli, as few participants had a previous experience interacting with a BCI. It is generally reported that regular use of neurotechnology mitigates BCI illiteracy, improves classification accuracy and reduces reaction time (Blankertz et al., 2009). Also, future experiments could include a baseline condition without BCI use, to assess the effect on pilots performance of such neurotechnology. We truthfully hope that this study will foster research efforts to improve the concept of dual BCI for safer, seamless, and efficient human-human and human(s)-machine(s) interactions.

Data Availability Statement

The datasets presented in this article are not readily available because, due to hardware limitation only eye tracking and subjective data are available. Requests to access the datasets should be directed to ZnJlZGVyaWMuZGVoYWlzJiN4MDAwNDA7aXNhZS5mcg==.

Ethics Statement

The studies involving human participants were reviewed and approved by Comite d'Ethique de l'Université Fédérale de Toulouse (approval number 2020-334). The patients/participants provided their written informed consent to participate in this study.

Author Contributions

FD contributed to the conception and the design of experimental paradigm. FD, SL, and T-VN recorded the data. FD, T-VN, SL, and LD performed part of the analyses. SV, PL, GF, and JT implemented the experimental set-up in the flight simulator. FD, SL, and LD wrote the manuscript. All authors revised the manuscript, read, and approved the submitted version.

Funding

This research was funded by the Agence Innovation Défense of the Direction Générale de l'Armement (Neurosynchrone project).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

The authors wish to sincerely thank all the pilots who participated in the experiment. The authors would also like to acknowledge the Artificial and Natural Intelligence Toulouse Institute (ANITI) and the Axa Research fund Neuroergonomics chair for flight safety for currently funding FD.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnrgo.2022.824780/full#supplementary-material

Footnotes

1. ^https://www.next-mind.com/developer

2. ^www.flightgear.org

3. ^https://www.next-mind.com/documentation/sensor-manual/

References

Aricó, P., Borghini, G., Di Flumeri, G., Colosimo, A., Bonelli, S., Golfetti, A., et al. (2016). Adaptive automation triggered by EEG-based mental workload index: a passive brain-computer interface application in realistic air traffic control environment. Front. Hum. Neurosci. 10:539. doi: 10.3389/fnhum.2016.00539

PubMed Abstract | CrossRef Full Text | Google Scholar

Behrend, J., and Dehais, F. (2020). How role assignment impacts decision-making in high-risk environments: evidence from eye-tracking in aviation. Saf. Sci. 127:104738. doi: 10.1016/j.ssci.2020.104738

CrossRef Full Text | Google Scholar

Blankertz, B., Sanelli, C., Halder, S., Hammer, E., Kübler, A., Müller, K.-R., et al. (2009). Predicting BCI performance to study BCI illiteracy. BMC Neurosci. 10(Suppl. 1):P84. doi: 10.1186/1471-2202-10-S1-P84

CrossRef Full Text | Google Scholar

Borghini, G., Astolfi, L., Vecchiato, G., Mattia, D., and Babiloni, F. (2014). Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neurosci. Biobehav. Rev. 44, 58–75. doi: 10.1016/j.neubiorev.2012.10.003

PubMed Abstract | CrossRef Full Text | Google Scholar

Brouwer, A.-M (2021). Challenges and opportunities in consumer neuroergonomics. Front. Neuroergon. 2:3. doi: 10.3389/fnrgo.2021.606646

CrossRef Full Text | Google Scholar

Brouwer, A.-M., Hogervorst, M. A., Van Erp, J. B., Heffelaar, T., Zimmerman, P. H., and Oostenveld, R. (2012). Estimating workload using EEG spectral power and ERPs in the n-back task. J. Neural Eng. 9:045008. doi: 10.1088/1741-2560/9/4/045008

PubMed Abstract | CrossRef Full Text | Google Scholar

Carelli, L., Solca, F., Faini, A., Meriggi, P., Sangalli, D., Cipresso, P., et al. (2017). Brain-computer interface for clinical purposes: cognitive assessment and rehabilitation. BioMed Res. Int. 2017:1695290. doi: 10.1155/2017/1695290

PubMed Abstract | CrossRef Full Text | Google Scholar

Chevallier, S., Kalunga, E. K., Barthélemy, Q., and Monacelli, E. (2021). Review of riemannian distances and divergences, applied to SSVEP-based BCI. Neuroinformatics 19, 93–106. doi: 10.1007/s12021-020-09473-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Clerc, M., Bougrain, L., and Lotte, F. (2016). Brain-Computer Interfaces 1. Hoboken, NJ: Wiley-ISTE. doi: 10.1002/9781119144977

CrossRef Full Text | Google Scholar

Dehais, F., Duprés, A., Blum, S., Drougard, N., Scannella, S., Roy, R. N., et al. (2019a). Monitoring pilot's mental workload using ERPs and spectral power with a six-dry-electrode EEG system in real flight conditions. Sensors 19:1324. doi: 10.3390/s19061324

PubMed Abstract | CrossRef Full Text | Google Scholar

Dehais, F., Dupres, A., Di Flumeri, G., Verdiere, K., Borghini, G., Babiloni, F., et al. (2018). “Monitoring pilot's cognitive fatigue with engagement features in simulated and actual flight conditions using an hybrid fNIRS-EEG passive BCI,” in 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (Miyazaki), 544–549. doi: 10.1109/SMC.2018.00102

CrossRef Full Text | Google Scholar

Dehais, F., Karwowski, W., and Ayaz, H. (2020a). Brain at work and in everyday life as the next frontier: grand field challenges for neuroergonomics. Front. Neuroergon. 1:583733. doi: 10.3389/fnrgo.2020.583733

CrossRef Full Text | Google Scholar

Dehais, F., Lafont, A., Roy, R., and Fairclough, S. (2020b). A neuroergonomics approach to mental workload, engagement and human performance. Front. Neurosci. 14:268. doi: 10.3389/fnins.2020.00268

PubMed Abstract | CrossRef Full Text | Google Scholar

Dehais, F., Rida, I., Roy, R. N., Iversen, J., Mullen, T., and Callan, D. (2019b). “A PBCI to predict attentional error before it happens in real flight conditions,” in 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC) (Bari), 4155–4160. doi: 10.1109/SMC.2019.8914010

CrossRef Full Text | Google Scholar

Dehais, F., Roy, R. N., and Scannella, S. (2019c). Inattentional deafness to auditory alarms: inter-individual differences, electrophysiological signature and single trial classification. Behav. Brain Res. 360, 51–59. doi: 10.1016/j.bbr.2018.11.045

PubMed Abstract | CrossRef Full Text | Google Scholar

Ewing, K. C., Fairclough, S. H., and Gilleade, K. (2016). Evaluation of an adaptive game that uses EEG measures validated during the design process as inputs to a biocybernetic loop. Front. Hum. Neurosci. 10:223. doi: 10.3389/fnhum.2016.00223

PubMed Abstract | CrossRef Full Text | Google Scholar

Fairclough, S. H., and Lotte, F. (2020). Grand challenges in neurotechnology and system neuroergonomics. Front. Neuroergon. 1:2. doi: 10.3389/fnrgo.2020.602504

CrossRef Full Text | Google Scholar

Felton, E., Williams, J., Vanderheiden, G., and Radwin, R. (2012). Mental workload during brain-computer interface training. Ergonomics 55, 526–537. doi: 10.1080/00140139.2012.662526

PubMed Abstract | CrossRef Full Text | Google Scholar

Fricke, T., Zander, T., Gramann, K., and Holzapfel, F. (2014). “First pilot-inthe-loop simulator experiments on brain control of horizontal aircraft motion,” in Deutscher Luft-und Raumfahrtkongress (Bonn).

Google Scholar

Gateau, T., Ayaz, H., and Dehais, F. (2018). In silico vs. over the clouds: on-the-fly mental state estimation of aircraft pilots, using a functional near infrared spectroscopy based passive-BCI. Front. Hum. Neurosci. 12:187. doi: 10.3389/fnhum.2018.00187

PubMed Abstract | CrossRef Full Text | Google Scholar

Gembler, F., Benda, M., Rezeika, A., Stawicki, P., and Volosyak, I. (2020). Asynchronous c-VEP communication tools-efficiency comparison of low-target, multi-target and dictionary-assisted BCI spellers. Sci. Rep. 10:17064. doi: 10.1038/s41598-020-74143-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Gramann, K., McKendrick, R., Baldwin, C., Roy, R. N., Jeunet, C., Mehta, R. K., et al. (2021). Grand field challenges for cognitive neuroergonomics in the coming decade. Front. Neuroergon. 2:6. doi: 10.3389/fnrgo.2021.643969

CrossRef Full Text | Google Scholar

Hancock, P. A., and Szalma, J. L. (2008). “Stress and neuroergonomics,” in Neuroergonomics: The Brain at Work, eds R. Parasuraman and M. Rizzo (New York, NY: Oxford University), 195–206.

Google Scholar

Hong, K.-S., and Khan, M. J. (2017). Hybrid brain-computer interface techniques for improved classification accuracy and increased number of commands: a review. Front. Neurorobot. 11:35. doi: 10.3389/fnbot.2017.00035

PubMed Abstract | CrossRef Full Text | Google Scholar

Hughes, C. L., Herrera, A., Gaunt, R., and Collinger, J. (2020). “Bidirectional brain-computer interfaces,” in Handbook of Clinical Neurology, eds N. F. Ramsey and J. R. Milla (Elsevier), 163–181. doi: 10.1016/B978-0-444-63934-9.00013-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Kahneman, D (1973). Attention and Effort, Vol. 88. Englewood Cliffs, NJ: University of Illinois Press.

Google Scholar

Kalunga, E. K., Chevallier, S., Barthélemy, Q., Djouani, K., Monacelli, E., and Hamam, Y. (2016). Online SSVEP-based BCI using Riemannian geometry. Neurocomputing 191, 55–68. doi: 10.1016/j.neucom.2016.01.007

PubMed Abstract | CrossRef Full Text | Google Scholar

Klaproth, O. W., Halbrügge, M., Krol, L. R., Vernaleken, C., Zander, T. O., and Russwinkel, N. (2020). A neuroadaptive cognitive model for dealing with uncertainty in tracing pilots' cognitive state. Top. Cogn. Sci. 12, 1012–1029. doi: 10.1111/tops.12515

PubMed Abstract | CrossRef Full Text | Google Scholar

Lin, X., Chen, Z., Xu, K., and Zhang, S. (2019). “Development of a high-speed mental spelling system combining eye tracking and SSVEP-based BCI with high scalability,” in Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS (Berlin), 6318–6322. doi: 10.1109/EMBC.2019.8857408

PubMed Abstract | CrossRef Full Text | Google Scholar

Lotte, F., Bougrain, L., Cichocki, A., Clerc, M., Congedo, M., Rakotomamonjy, A., et al. (2018). A review of classification algorithms for EEG-based brain-computer interfaces: a 10 year update. J. Neural Eng. 15:031005. doi: 10.1088/1741-2552/aab2f2

PubMed Abstract | CrossRef Full Text | Google Scholar

Lotte, F., and Roy, R. N. (2019). “Chapter 7: Brain-computer interface contributions to neuroergonomics,” in Neuroergonomics, eds H. Ayaz and F. Dehais (Amsterdam: Academic Press), 43–48. doi: 10.1016/B978-0-12-811926-6.00007-5

CrossRef Full Text | Google Scholar

Maksimenko, V. A., Hramov, A. E., Frolov, N. S., Lüttjohann, A., Nedaivozov, V. O., Grubov, V. V., et al. (2018). Increasing human performance by sharing cognitive load using brain-to-brain interface. Front. Neurosci. 12:949. doi: 10.3389/fnins.2018.00949

PubMed Abstract | CrossRef Full Text | Google Scholar

Matthews, G., Reinerman-Jones, L., Abich, J. IV., and Kustubayeva, A. (2017). Metrics for individual differences in EEG response to cognitive workload: optimizing performance prediction. Pers. Individ. Differ. 118, 22–28. doi: 10.1016/j.paid.2017.03.002

CrossRef Full Text | Google Scholar

Mumaw, R., Haworth, L., and Feary, M. (2019). The Role of Alerting System Failures in Loss of Control Accidents Cast SE-210 Output 2 Report 3 of 6. NASA Technical Memorandum - SeaWiFS Technical Report Series. NASA AMES.

Google Scholar

Nagel, S., and Spüler, M. (2019). World's fastest brain-computer interface: combining EEG2CODE with deep learning. PLoS ONE 14:e221909. doi: 10.1371/journal.pone.0221909

PubMed Abstract | CrossRef Full Text | Google Scholar

Nakanishi, M., Wang, Y., Chen, X., Wang, Y. T., Gao, X., and Jung, T. P. (2018). Enhancing detection of SSVEPs for a high-speed brain speller using task-related component analysis. IEEE Trans. Biomed. Eng. 65, 104–112. doi: 10.1109/TBME.2017.2694818

PubMed Abstract | CrossRef Full Text | Google Scholar

Norcia, A. M., Appelbaum, L. G., Ales, J. M., Cottereau, B. R., and Rossion, B. (2015). The steady-state visual evoked potential in vision research: a review. J. Vis. 15, 4–4. doi: 10.1167/15.6.4

PubMed Abstract | CrossRef Full Text | Google Scholar

Nourmohammadi, A., Jafari, M., and Zander, T. O. (2018). A survey on unmanned aerial vehicle remote control using brain-computer interface. IEEE Trans. Hum. Mach. Syst. 48, 337–348. doi: 10.1109/THMS.2018.2830647

CrossRef Full Text | Google Scholar

Prinzel, L. J., Freeman, F. G., Scerbo, M. W., Mikulka, P. J., and Pope, A. T. (2000). A closed-loop system for examining psychophysiological measures for adaptive task allocation. Int. J. Aviat. Psychol. 10, 393–410. doi: 10.1207/S15327108IJAP1004_6

PubMed Abstract | CrossRef Full Text | Google Scholar

Rodriguez-Bermudez, G., Lopez-Belchi, A., and Girault, A. (2019). Testing brain-computer interfaces with airplane pilots under new motor imagery tasks. Int. J. Comput. Intell. Syst. 12, 937–946. doi: 10.2991/ijcis.d.190806.001

CrossRef Full Text | Google Scholar

Roy, R. N., Bonnet, S., Charbonnier, S., and Campagne, A. (2016). Efficient workload classification based on ignored auditory probes: a proof of concept. Front. Hum. Neurosci. 10:519. doi: 10.3389/fnhum.2016.00519

PubMed Abstract | CrossRef Full Text | Google Scholar

Scholl, C. A., Chi, Y. M., Elconin, M., Gray, W. R., Chevillet, M. A., and Pohlmeyer, E. A. (2016). “Classification of pilot-induced oscillations during in-flight piloting exercises using dry EEG sensor recordings,” in 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (Orlando, FL), 4467–4470. doi: 10.1109/EMBC.2016.7591719

PubMed Abstract | CrossRef Full Text | Google Scholar

Shirzhiyan, Z., Keihani, A., Farahi, M., Shamsi, E., GolMohammadi, M., Mahnam, A., et al. (2019). Introducing chaotic codes for the modulation of code modulated visual evoked potentials (c-VEP) in normal adults for visual fatigue reduction. PLoS ONE 14:e213197. doi: 10.1371/journal.pone.0213197

PubMed Abstract | CrossRef Full Text | Google Scholar

Stawicki, P., Gembler, F., Rezeika, A., and Volosyak, I. (2017). A novel hybrid mental spelling application based on eye tracking and SSVEP-based BCI. Brain Sci. 7:35. doi: 10.3390/brainsci7040035

PubMed Abstract | CrossRef Full Text | Google Scholar

Vecchiato, G., Borghini, G., Aricó, P., Graziani, I., Maglione, A. G., Cherubino, P., et al. (2016). Investigation of the effect of EEG-BCI on the simultaneous execution of flight simulation and attentional tasks. Med. Biol. Eng. Comput. 54, 1503–1513. doi: 10.1007/s11517-015-1420-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Vialatte, F.-B., Maurice, M., Dauwels, J., and Cichocki, A. (2010). Steady-state visually evoked potentials: focus on essential paradigms and future perspectives. Prog. Neurobiol. 90, 418–438. doi: 10.1016/j.pneurobio.2009.11.005

PubMed Abstract | CrossRef Full Text | Google Scholar

Wickens, C., and Dehais, F. (2019). Expertise in Aviation. Oxford: Oxford University Press. doi: 10.1093/oxfordhb/9780198795872.013.29

CrossRef Full Text | Google Scholar

Wickens, C. D (2008). Multiple resources and mental workload. Hum. Factors 50, 449–455. doi: 10.1518/001872008X288394

PubMed Abstract | CrossRef Full Text | Google Scholar

Wickens, C. D., and Alexander, A. L. (2009). Attentional tunneling and task management in synthetic vision displays. Int. J. Aviat. Psychol. 19, 182–199. doi: 10.1080/10508410902766549

CrossRef Full Text | Google Scholar

Zander, T. O., and Kothe, C. (2011). Towards passive brain-computer interfaces: applying brain-computer interface technology to human-machine systems in general. J. Neural Eng. 8:025005. doi: 10.1088/1741-2560/8/2/025005

PubMed Abstract | CrossRef Full Text | Google Scholar

Zander, T. O., Krol, L. R., Birbaumer, N. P., and Gramann, K. (2016). Neuroadaptive technology enables implicit cursor control based on medial prefrontal cortex activity. Proc. Natl. Acad. Sci. U.S.A. 113, 14898–14903. doi: 10.1073/pnas.1605155114

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhao, Y., Tang, J., Cao, Y., Jiao, X., Xu, M., Zhou, P., Ming, D., et al. (2018). Effects of distracting task with different mental workload on steady-state visual evoked potential based brain computer interfaces - an offline study. Front. Neurosci. 12:79. doi: 10.3389/fnins.2018.00079

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhu, D., Bieger, J., Garcia Molina, G., and Aarts, R. M. (2010). A survey of stimulation methods used in SSVEP-based BCIs. Comput. Intell. Neurosci. 2010:702357. doi: 10.1155/2010/702357

PubMed Abstract | CrossRef Full Text | Google Scholar